Fix encoding of data written to git-annex branch. Avoid truncating unicode characters to 8 bits.

Allow any encoding to be used, as with filenames (but utf8 is the sane
choice). Affects metadata and repository descriptions, and preferred
content expressions.

The question of what's the right encoding for the git-annex branch is a
vexing one. utf-8 would be a nice choice, but this leaves the possibility
of bad data getting into a git-annex branch somehow, and this resulting in
git-annex crashing with encoding errors, which is a failure mode I want to
avoid.

(Also, preferred content expressions can refer to filenames, and filenames
can have any encoding, so limiting to utf-8 would not be ideal.)

The union merge code already took care to not assume any encoding for a
file. Except it assumes that any \n is a literal newline, and not part of
some encoding of a character that happens to contain a newline. (At least
utf-8 avoids using newline for anything except liternal newlines.)
Adapted the git-annex branch code to use this same approach.

Note that there is a potential interop problem with Windows, since
FileSystemEncoding doesn't work there, and instead things are always
decoded as utf-8. If someone uses non-utf8 encoding for data on the
git-annex branch, this can lead to an encoding error on windows. However,
this commit doesn't actually make that any worse, because the union merge
code would similarly fail with an encoding error on windows in that
situation.

This commit was sponsored by Kyle Meyer.

1 files changed, 6 insertions, 4 deletions

diff --git a/Annex/Branch.hs b/Annex/Branch.hs
index 94c4c029c..7a75d8acf 100644
--- a/Annex/Branch.hs
+++ b/Annex/Branch.hs
@@ -25,9 +25,10 @@ module Annex.Branch (
 	performTransitions,
 ) where
 
-import qualified Data.ByteString.Lazy.Char8 as L
+import qualified Data.ByteString.Lazy as L
 import qualified Data.Set as S
 import qualified Data.Map as M
+import Data.Bits.Utils
 
 import Common.Annex
 import Annex.BranchState
@@ -199,7 +200,7 @@ getHistorical :: RefDate -> FilePath -> Annex String
 getHistorical date = getRef (Git.Ref.dateRef fullname date)
 
 getRef :: Ref -> FilePath -> Annex String
-getRef ref file = withIndex $ L.unpack <$> catFile ref file
+getRef ref file = withIndex $ decodeBS <$> catFile ref file
 
 {- Applies a function to modifiy the content of a file.
  -
@@ -259,7 +260,8 @@ commitIndex' jl branchref message parents = do
   where
 	-- look for "parent ref" lines and return the refs
 	commitparents = map (Git.Ref . snd) . filter isparent .
-		map (toassoc . L.unpack) . L.lines
+		map (toassoc . decodeBS) . L.split newline
+	newline = c2w8 '\n'
 	toassoc = separate (== ' ')
 	isparent (k,_) = k == "parent"
 		
@@ -432,7 +434,7 @@ handleTransitions jl localts refs = do
 			return True
   where
   	getreftransition ref = do
-		ts <- parseTransitionsStrictly "remote" . L.unpack
+		ts <- parseTransitionsStrictly "remote" . decodeBS
 			<$> catFile ref transitionsLog
 		return (ref, ts)