1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
|
This bug is reopened to track some new UTF-8 filename issues caused by GHC
7.4. In this version of GHC, git-annex's hack to support filenames in any
encoding no longer works. Even unicode filenames fail to work when
git-annex is built with 7.4. --[[Joey]]
**As a stopgap workaround**, I have made a branch `unicode-only`. This
makes git-annex work with unicode filenames with ghc 7.4, but *only*
unicode filenames. If you have filenames with some other encoding, you're
out in the cold, and it will probably just crash with a error about wrong
encoding.
## analysis
What's going on exactly? The new ghc, when presented with
a String of raw bytes like "fo\194\161", and asked to do
something like `getSymbolicLinkStatus`, encodes it
as unicode, yielding "fo\303\202\302\241". Which is
not the same as the original filename, assuming it was "fo¡".
The new ghc requires a new data type, `RawFilePath` be used if you
don't want to impose utf-8 filenames on your users.
The available `RawFilePath` support is quite low-level, so all the nice
readFile and writeFile code, etc has to be reimplemented. So do any utility
libraries that do things with FilePaths, if you need them to use
RawFilePaths. Until the haskell ecosystem adapts to `RawFilePath`
(if it does), using it broadly, as git-annex needs to, will be difficult.
## rawfilepath branch
I have a `rawfilepath` branch in git where I am trying to convert it to use
`RawFilePath`. However, since there is no way to cast a `FilePath` to a
`RawFilePath` or back (because the encoding of `RawFilePath` is not
specified), this means changing essentially all of git-annex. Even the
filenames used for keys in `.git/annex/objects` need to use the new data
type. I didn't get very far on this branch.
## newghc-edges branch
I have a `newghc-edges` branch in git, trying a different approach.
A `RawFilePath` contains only bytes, so it can actually be cast to a string,
containing encoded characters. That string can then be 1) output in binary
mode or 2) manipulated in ways that do not add characters larger than 255,
and cast back to a `RawFilePath`. While not type-safe, such casts should at
least help during bootstrapping, and might allow for a quick fix that only
changes to `RawFilePath` at the edges.
The branch contains an almost complete, although probably also buggy
conversion using this method. It is missing wrappers for a
few things like `readFile` and `writeFile` but otherwise seems to
basically work.
Is this a suitable approach for merging into `master`? It's nasty,
being not type safe, having to reimplent/copy+modify random bits of
libraries, etc. The nastiness is contained, though, in a single file,
of only a few hundred lines of code. --[[Joey]]
----
Old, now fixed bug report follows:
There are problems with displaying filenames in UTF8 encoding, as shown here:
$ echo $LANG
en_GB.UTF-8
$ git init
$ git annex init test
[...]
$ touch "Umlaut Ü.txt"
$ git annex add Uml*
add Umlaut Ã.txt ok
(Recording state in git...)
$ find -name U\* | hexdump -C
00000000 2e 2f 55 6d 6c 61 75 74 20 c3 9c 2e 74 78 74 0a |./Umlaut ...txt.|
00000010
$ git annex find | hexdump -C
00000000 55 6d 6c 61 75 74 20 c3 83 c2 9c 2e 74 78 74 0a |Umlaut .....txt.|
00000010
$
It looks like the common latin1-to-UTF8 encoding. Functionality other than otuput seems not to be affected.
> Yes, I believe that git-annex is reading filename data from git
> as a stream of char8s, and not decoding unicode in it into logical
> characters.
> Haskell then I guess, tries to unicode encode it when it's output to
> the console.
> This only seems to matter WRT its output to the console; the data
> does not get mangled internally and so it accesses the right files
> under the hood.
>
> I am too new to haskell to really have a handle on how to handle
> unicode and other encodings issues with it. In general, there are three
> valid approaches: --[[Joey]]
>
> 1. Convert all input data to unicode and be unicode clean end-to-end
> internally. Problimatic here since filenames may not necessarily be
> encoded in utf-8 (an archive could have historical filenames using
> varying encodings), and you don't want which files are accessed to
> depend on locale settings.
> > I tried to do this by making parts of GitRepo call
> > Codec.Binary.UTF8.String.decodeString when reading filenames from
> > git. This seemed to break attempts to operate on the files,
> > weirdly encoded strings were seen in syscalls in strace.
> 1. Keep input and internal data un-decoded, but decode it when
> outputting a filename (assuming the filename is encoded using the
> user's configured encoding), and allow haskell's output encoding to then
> encode it according to the user's locale configuration.
> > This is now implemented. I'm not very happy that I have to watch
> > out for any place that a filename is output and call `filePathToString`
> > on it, but there are really not too many such places in git-annex.
> >
> > Note that this only affects filenames apparently.
> > (Names of files in the annex, and also some places where names
> > of keys are displayed.) Utf-8 in the uuid.map file etc seems
> > to be handled cleanly.
> 1. Avoid encodings entirely. Mostly what I'm doing now; probably
> could find a way to disable encoding of console output. Then the raw
> filename would be displayed, which should work ok. git-annex does
> not really need to pull apart filenames; they are almost entirely
> opaque blobs. I guess that the `--exclude` option is the exception
> to that, but it is currently not unicode safe anyway. (Update: tried
> `--exclude` again, seems it is unicode clean..)
> One other possible
> issue would be that this could cause problems if git-annex were
> translated.
> > On second thought, I switched to this. Any decoding of a filename
> > is going to make someone unhappy; the previous approach broke
> > non-utf8 filenames.
|
