diff options
Diffstat (limited to 'doc')
163 files changed, 3841 insertions, 129 deletions
diff --git a/doc/bugs/Adding_zip_or_7z_or_tar_archive_builds_for_windows.mdwn b/doc/bugs/Adding_zip_or_7z_or_tar_archive_builds_for_windows.mdwn index 6cca0082c..ae1f7c522 100644 --- a/doc/bugs/Adding_zip_or_7z_or_tar_archive_builds_for_windows.mdwn +++ b/doc/bugs/Adding_zip_or_7z_or_tar_archive_builds_for_windows.mdwn @@ -1,3 +1,5 @@ Having a windows build of Git-Annex in an archived format would be very usefull for automation, and deploy. Could it be possible to add this to the buildserver of gitannex? +[[!tag moreinfo]] + diff --git a/doc/bugs/Adding_zip_or_7z_or_tar_archive_builds_for_windows/comment_1_70480ffd417788f18cd75a9b625ecf3b._comment b/doc/bugs/Adding_zip_or_7z_or_tar_archive_builds_for_windows/comment_1_70480ffd417788f18cd75a9b625ecf3b._comment new file mode 100644 index 000000000..3bd7381e1 --- /dev/null +++ b/doc/bugs/Adding_zip_or_7z_or_tar_archive_builds_for_windows/comment_1_70480ffd417788f18cd75a9b625ecf3b._comment @@ -0,0 +1,19 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 1""" + date="2016-11-16T18:41:25Z" + content=""" +It would be helpful to have more details, such as an example of software +distributed for windows that way, or documentation of how such an archive +is used on windows. + +The git-annex Windows installer is a exe file that uses the NullSoft +installation system. As far as I know that's pretty common in the Windows +world. + +I don't see any point in zipping up the single exe. It would be possible to +make a zip containing all the files that instlling the exe installs. But, +the installation process has to integrate git-annex with git, it installs +menu items, etc. A zip file would not be able to handle that integration. +So its use seems limited to me. +"""]] diff --git a/doc/bugs/Adding_zip_or_7z_or_tar_archive_builds_for_windows/comment_2_afa6a131999feda67876859cd85ebcfc._comment b/doc/bugs/Adding_zip_or_7z_or_tar_archive_builds_for_windows/comment_2_afa6a131999feda67876859cd85ebcfc._comment new file mode 100644 index 000000000..b0db54479 --- /dev/null +++ b/doc/bugs/Adding_zip_or_7z_or_tar_archive_builds_for_windows/comment_2_afa6a131999feda67876859cd85ebcfc._comment @@ -0,0 +1,15 @@ +[[!comment format=mdwn + username="luckcolorsgoo@ab4f3c1c44a7dbcbcb9d9a29315b59ad524ceaaa" + nickname="luckcolorsgoo" + avatar="http://cdn.libravatar.org/avatar/ddff84cd2a97252a05cccb4bc5010448" + subject="comment 2" + date="2016-11-16T22:56:46Z" + content=""" +In my case i was going to make a script for automatically downloading and updating an git portbale / git annex instance, by first fetching git portbale and then downloading the gitannex exe. + +So yeah it's more reliable to extract an archive rather than trying to extract the setup without executing it. +That's why i'm asking for this feature. :) + + + +"""]] diff --git a/doc/bugs/Allow_automatic_retry_git_annex_get.mdwn b/doc/bugs/Allow_automatic_retry_git_annex_get.mdwn index 0e85b7acf..c6e406b49 100644 --- a/doc/bugs/Allow_automatic_retry_git_annex_get.mdwn +++ b/doc/bugs/Allow_automatic_retry_git_annex_get.mdwn @@ -59,5 +59,3 @@ SHA256E-s41311329--69c3b054a3fe9676133605730d85b7fcef8696f6782d402a524e41b836253 [[!meta title="Detect stalled transfer and retry or abort it"]] - -> [[done]] --[[Joey]] diff --git a/doc/bugs/Allow_automatic_retry_git_annex_get/comment_4_899b66a20b8e29a23068d249a461c19f._comment b/doc/bugs/Allow_automatic_retry_git_annex_get/comment_4_899b66a20b8e29a23068d249a461c19f._comment new file mode 100644 index 000000000..6d2506923 --- /dev/null +++ b/doc/bugs/Allow_automatic_retry_git_annex_get/comment_4_899b66a20b8e29a23068d249a461c19f._comment @@ -0,0 +1,16 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 4""" + date="2016-12-13T16:05:42Z" + content=""" +Could the original bug reporter please show what your ~/.ssh/config +contains? As far as I can tell, ssh's TCPKeepAlive option, which is +supposed to be enabled by default, unless you have disabled it, should +avoid such problems. + +It may also help to set ServerAliveInterval. + +Unfortunately, my attempt to make git-annex set ServerAliveInterval +when running ssh broke too many systems with old sshed, and I have had to +revert it. +"""]] diff --git a/doc/bugs/Assistant_drops_files_from_remote_repos_that_it_shouldn__39__t/comment_9_c46cdba62da4f5ccfdc42dfc33aec600._comment b/doc/bugs/Assistant_drops_files_from_remote_repos_that_it_shouldn__39__t/comment_9_c46cdba62da4f5ccfdc42dfc33aec600._comment new file mode 100644 index 000000000..a6a2397e7 --- /dev/null +++ b/doc/bugs/Assistant_drops_files_from_remote_repos_that_it_shouldn__39__t/comment_9_c46cdba62da4f5ccfdc42dfc33aec600._comment @@ -0,0 +1,34 @@ +[[!comment format=mdwn + username="boh" + avatar="http://cdn.libravatar.org/avatar/e7fa2d1c5d95e323fe48887f7f827b1f" + subject="comment 9" + date="2016-11-27T12:23:20Z" + content=""" +Seems as if the problem still exists in 6.20161118 (Debian). + +I have three repositories (among others), `jolla`, `sts-3xx`, and `here`. `jolla` and `here` are in group `manual`, `sts-3xx` is `backup`; `here` and `sts-3xx` have assistants running, `jolla` not. `jolla` and `sts-3xx` have slightly older versions of git-annex installed. + +Now, when I copy a file from `here` to `jolla` like this + + git annex copy real_programmers.png -t jolla + +the file is subsequently dropped by the assistant: + +``` +drop real_programmers.png (locking jolla...) [2016-11-27 13:00:02.667376556] chat: ssh [\"-S\",\".git/annex/ssh/jolla\",\"-o\",\"ControlMaster +=auto\",\"-o\",\"ControlPersist=yes\",\"-F\",\".git/annex/ssh.config\",\"-T\",\"jolla\",\"git-annex-shell 'lockcontent' '/~/Music/media/' '--debug' ' +SHA256E-s84499--ff98a733cc0122858fb11433c720e2d038fec190a3d36380d0e7e8dab468f883.png' --uuid 5298e3ce-1106-4d5e-b052-0aee4b27a344\"] +(locking sts-3xx...) [2016-11-27 13:00:03.252473676] chat: ssh [..., \"git-annex-shell 'lockcontent' '/backups/exot/media/' '--debug' 'SHA256E-s84499--ff98a733cc0122858fb11433c720e2d038fec190a3d 36380d0e7e8dab468f883.png' --uuid 1fec6253-171d-4f86-885b-e233be2d65ec\"] +(lockcontent failed) [2016-11-27 13:00:03.486158016] process done ExitFailure 1 +(checking sts-3xx...) [2016-11-27 13:00:03.487047149] call: ssh [..., \"git-annex-shell 'inannex' '/backups/exot/media/' '--debug' 'SHA256E-s84499--ff98a733cc0122858fb11433c720e2d038fec190a3d363 80d0e7e8dab468f883.png' --uuid 1fec6253-171d-4f86-885b-e233be2d65ec\"] +[2016-11-27 13:00:03.76435136] process done ExitSuccess +[2016-11-27 13:00:03.764705754] Dropping from here proof: Just (SafeDropProof (NumCopies 2) [RecentlyVerifiedCopy UUID \"1fec6253-171d-4 f86-885b-e233be2d65ec\",LockedCopy UUID \"5298e3ce-1106-4d5e-b052-0aee4b27a344\"] (Just (ContentRemovalLock (Key {keyName = \"ff98a733cc012 2858fb11433c720e2d038fec190a3d36380d0e7e8dab468f883.png\", keyBackendName = \"SHA256E\", keySize = Just 84499, keyMtime = Nothing, keyChun kSize = Nothing, keyChunkNum = Nothing})))) +[2016-11-27 13:00:04.24333081] process done ExitFailure 1 +ok +[2016-11-27 13:00:04.251232455] dropped real_programmers.png (from here) (copies now 4) : drop wanted after Upload UUID \"5298e3ce-1106- 4d5e-b052-0aee4b27a344\" real_programmers.png Just 84499 +``` + +However, I failed to reproduce the problem by replicating my setup with fresh repositories … + +Please let me know if you need more information, and *so* many thanks for git-annex! +"""]] diff --git a/doc/bugs/Build_with_aws_head_fails.mdwn b/doc/bugs/Build_with_aws_head_fails.mdwn new file mode 100644 index 000000000..a96dce0ad --- /dev/null +++ b/doc/bugs/Build_with_aws_head_fails.mdwn @@ -0,0 +1,49 @@ +### Please describe the problem. +https://github.com/aristidb/aws/issues/206 was recently resolved in https://github.com/aristidb/aws/pull/213. + +A newer version will be tagged imminently according to https://github.com/aristidb/aws/issues/206#issuecomment-260214736. + +With the http-conduit (<2.2.0) constraint removed from git-annex.cabal, and the aws dependency set to use aws head (currently c8806dc), the git-annex build fails. + +### What steps will reproduce the problem? + +Remove the http-conduit (<2.2.0) constraint and attempt to build git-annex with aws head. + +### What version of git-annex are you using? On what operating system? + +macOS 10.11, git-annex 6.20161118. + +### Please provide any additional information below. +Full build log: https://gist.github.com/ilovezfs/15bcd8f1086b3d825beff58140e04eec +[[!format sh """ +[ 90 of 542] Compiling Types.Crypto ( Types/Crypto.hs, dist/dist-sandbox-6b15e8f0/build/git-annex/git-annex-tmp/Types/Crypto.o ) +[ 91 of 542] Compiling Utility.Metered ( Utility/Metered.hs, dist/dist-sandbox-6b15e8f0/build/git-annex/git-annex-tmp/Utility/Metered.o ) +[ 92 of 542] Compiling Messages.JSON ( Messages/JSON.hs, dist/dist-sandbox-6b15e8f0/build/git-annex/git-annex-tmp/Messages/JSON.o ) +[ 93 of 542] Compiling Utility.Url ( Utility/Url.hs, dist/dist-sandbox-6b15e8f0/build/git-annex/git-annex-tmp/Utility/Url.o ) + +Utility/Url.hs:354:34: error: + • The constructor ‘StatusCodeException’ should have 2 arguments, but has been given 3 + • In the pattern: StatusCodeException s _ _ + In an equation for ‘matchStatusCodeException’: + matchStatusCodeException want e@(StatusCodeException s _ _) + | want s = Just e + | otherwise = Nothing + +Utility/Url.hs:354:34: error: + • Couldn't match expected type ‘HttpException’ + with actual type ‘HttpExceptionContent’ + • In the pattern: StatusCodeException s _ _ + In an equation for ‘matchStatusCodeException’: + matchStatusCodeException want e@(StatusCodeException s _ _) + | want s = Just e + | otherwise = Nothing +cabal: Leaving directory '.' +cabal: Error: some packages failed to install: +git-annex-6.20161118 failed during the building phase. The exception was: +ExitFailure 1 +"""]] + +### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) +Yes :) + +> [[done]] via the nice patch! --[[Joey]] diff --git a/doc/bugs/Build_with_aws_head_fails/comment_1_d48bc2b3eb48c2a3a4d8608803913000._comment b/doc/bugs/Build_with_aws_head_fails/comment_1_d48bc2b3eb48c2a3a4d8608803913000._comment new file mode 100644 index 000000000..536c1569d --- /dev/null +++ b/doc/bugs/Build_with_aws_head_fails/comment_1_d48bc2b3eb48c2a3a4d8608803913000._comment @@ -0,0 +1,149 @@ +[[!comment format=mdwn + username="alpernebbi" + avatar="http://cdn.libravatar.org/avatar/daf2abb14f39e28ad75d5f9a03fcd106" + subject="Patch to fix aws head build issue" + date="2016-12-10T13:08:58Z" + content=""" +I think I fixed this. I'm attaching the output of `git format-patch origin/master`. + +In an Arch Linux chroot with their [PKGBUILD](https://git.archlinux.org/svntogit/community.git/tree/trunk/PKGBUILD?h=packages/git-annex) (with a small modification to apply the patch), `haskell-http-client 0.5.3.3-1` and `http-conduit 2.2.3-5` both the build and the tests are successful. +It's also successful in a Debian Sid chroot, where `sudo apt build-dep git-annex` gives me `libghc-http-client-dev 0.4.31.1-3+b2`, `libghc-http-conduit-dev 2.1.11-3+b2`. + +### Patch + +[[!format patch \"\"\" +From 2ce09420aa8f3d916c6562abea4ed8911a186902 Mon Sep 17 00:00:00 2001 +From: Alper Nebi Yasak <alpernebiyasak@gmail.com> +Date: Sat, 10 Dec 2016 15:24:27 +0300 +Subject: [PATCH] Remove http-conduit (<2.2.0) constraint + +Since https://github.com/aristidb/aws/issues/206 is resolved, this +constraint is no longer necessary. However, http-conduit (>=2.2.0) +requires http-client (>=0.5.0) which introduces some breaking changes. +This commit also implements those changes depending on the version. +Fixes: https://git-annex.branchable.com/bugs/Build_with_aws_head_fails/ + +Signed-off-by: Alper Nebi Yasak <alpernebiyasak@gmail.com> +--- + Remote/S3.hs | 8 +++++++- + Remote/WebDAV.hs | 17 +++++++++++++++++ + Utility/Url.hs | 8 ++++++++ + git-annex.cabal | 3 +-- + 4 files changed, 33 insertions(+), 3 deletions(-) + +diff --git a/Remote/S3.hs b/Remote/S3.hs +index 4c1bd57..9563b5a 100644 +--- a/Remote/S3.hs ++++ b/Remote/S3.hs +@@ -49,6 +49,12 @@ import Annex.Content + import Annex.Url (withUrlOptions) + import Utility.Url (checkBoth, managerSettings, closeManager) + ++#if MIN_VERSION_http_client(0,5,0) ++import Network.HTTP.Client (responseTimeoutNone) ++#else ++responseTimeoutNone = Nothing ++#endif ++ + type BucketName = String + + remote :: RemoteType +@@ -430,7 +436,7 @@ withS3HandleMaybe c gc u a = do + where + s3cfg = s3Configuration c + httpcfg = managerSettings +- { managerResponseTimeout = Nothing } ++ { managerResponseTimeout = responseTimeoutNone } + + s3Configuration :: RemoteConfig -> S3.S3Configuration AWS.NormalQuery + s3Configuration c = cfg +diff --git a/Remote/WebDAV.hs b/Remote/WebDAV.hs +index 19dbaa8..14947f1 100644 +--- a/Remote/WebDAV.hs ++++ b/Remote/WebDAV.hs +@@ -5,6 +5,7 @@ + - Licensed under the GNU GPL version 3 or higher. + -} + ++{-# LANGUAGE CPP #-} + {-# LANGUAGE ScopedTypeVariables #-} + + module Remote.WebDAV (remote, davCreds, configUrl) where +@@ -34,6 +35,10 @@ import Utility.Url (URLString, matchStatusCodeException) + import Annex.UUID + import Remote.WebDAV.DavLocation + ++#if MIN_VERSION_http_client(0,5,0) ++import Network.HTTP.Client (HttpExceptionContent(..), responseStatus) ++#endif ++ + remote :: RemoteType + remote = RemoteType { + typename = \"webdav\", +@@ -302,6 +307,17 @@ goDAV (DavHandle ctx user pass _) a = choke $ run $ prettifyExceptions $ do + {- Catch StatusCodeException and trim it to only the statusMessage part, + - eliminating a lot of noise, which can include the whole request that + - failed. The rethrown exception is no longer a StatusCodeException. -} ++#if MIN_VERSION_http_client(0,5,0) ++prettifyExceptions :: DAVT IO a -> DAVT IO a ++prettifyExceptions a = catchJust (matchStatusCodeException (const True)) a go ++ where ++ go (HttpExceptionRequest _ (StatusCodeException response message)) = error $ unwords ++ [ \"DAV failure:\" ++ , show (responseStatus response) ++ , show (message) ++ ] ++ go e = throwM e ++#else + prettifyExceptions :: DAVT IO a -> DAVT IO a + prettifyExceptions a = catchJust (matchStatusCodeException (const True)) a go + where +@@ -311,6 +327,7 @@ prettifyExceptions a = catchJust (matchStatusCodeException (const True)) a go + , show (statusMessage status) + ] + go e = throwM e ++#endif + + prepDAV :: DavUser -> DavPass -> DAVT IO () + prepDAV user pass = do +diff --git a/Utility/Url.hs b/Utility/Url.hs +index 9b68871..d0e1b37 100644 +--- a/Utility/Url.hs ++++ b/Utility/Url.hs +@@ -350,8 +350,16 @@ hUserAgent = \"User-Agent\" + - + - > catchJust (matchStatusCodeException (== notFound404)) + -} ++#if MIN_VERSION_http_client(0,5,0) ++matchStatusCodeException :: (Status -> Bool) -> HttpException -> Maybe HttpException ++matchStatusCodeException want e@(HttpExceptionRequest _ (StatusCodeException r _)) ++ | want (responseStatus r) = Just e ++ | otherwise = Nothing ++matchStatusCodeException _ _ = Nothing ++#else + matchStatusCodeException :: (Status -> Bool) -> HttpException -> Maybe HttpException + matchStatusCodeException want e@(StatusCodeException s _ _) + | want s = Just e + | otherwise = Nothing + matchStatusCodeException _ _ = Nothing ++#endif +diff --git a/git-annex.cabal b/git-annex.cabal +index ec54a14..83d45a1 100644 +--- a/git-annex.cabal ++++ b/git-annex.cabal +@@ -357,8 +357,7 @@ Executable git-annex + resourcet, + http-client, + http-types, +- -- Old version needed due to https://github.com/aristidb/aws/issues/206 +- http-conduit (<2.2.0), ++ http-conduit, + time, + old-locale, + esqueleto, +-- +2.7.4 + +\"\"\"]] + +"""]] diff --git a/doc/bugs/Corrupted_git___40__but_not_annex__41___controlled_files.mdwn b/doc/bugs/Corrupted_git___40__but_not_annex__41___controlled_files.mdwn new file mode 100644 index 000000000..8f1995697 --- /dev/null +++ b/doc/bugs/Corrupted_git___40__but_not_annex__41___controlled_files.mdwn @@ -0,0 +1,102 @@ +### Please describe the problem. + +I have files that match annex.largefiles and therefore should be added to git but not to annex, they seem to be getting corrupted after cloning the repo. + +### What steps will reproduce the problem? + +I couldn't immediately find the exact steps to reproduce the issue but I have multiple git repositories showing this. + +### What version of git-annex are you using? On what operating system? + +The problem has occurred a while ago but I have just noticed it. This is on macOS if it helps. I also tend to use the latest released version of git-annex (installed via Homebrew) + +### Please provide any additional information below. + +[[!format sh """ +# If you can, paste a complete transcript of the problem occurring here. +# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log + +$ cd Documents +$ cat .gitattributes +* annex.largefiles=((not(mimetype=text/*))or(largerthan=100kb)) + +*.png binary +*.jpg binary +*.jpeg binary +*.gif binary +*.ico binary + +*.mp3 binary +*.fla binary + +*.mov binary +*.mp4 binary +*.flv binary +*.swf binary +*.avi binary +*.mkv binary +*.mpg binary +*.mpeg binary + +*.gz binary +*.zip binary +*.7z binary +*.rar binary +*.bz2 binary + +*.ttf binary + +*.pdf binary + +$ ls -la Docs/2016-XXX/XXX/ +total 696 +drwx------@ 4 denis staff 136 Jul 11 15:05 ./ +drwxr-xr-x@ 9 denis staff 306 Dec 12 19:42 ../ +-rwxr-xr-x@ 1 denis staff 265898 Jul 11 13:03 XXX.pdf* +-rwxr-xr-x@ 1 denis staff 89586 Jul 11 13:03 Summary.pdf* +$ file --mime-type Docs/2016-XXX/XXX/XXX.pdf +Docs/2016-XXX/XXX/XXX.pdf: application/pdf +$ git show 60a76858a57a73967131b929af45a99703f67335 +commit 60a76858a57a73967131b929af45a99703f67335 +Author: Denis Dzyubenko <denis@ddenis.info> +Date: Mon Jul 11 15:05:37 2016 +0200 + + XXX + +diff --git a/Docs/2016-XXX/XXX/XXX.pdf b/Docs/2016-XXX/XXX/XXX.pdf +new file mode 100755 +index 00000000..112f68d0 +Binary files /dev/null and b/Docs/2016-XXX/XXX/XXX.pdf differ +diff --git a/Docs/2016-XXX/XXX/Summary.pdf b/Docs/2016-XXX/XXX/Summary.pdf +new file mode 100755 +index 00000000..3828383e +Binary files /dev/null and b/Docs/2016-XXX/XXX/Summary.pdf differ +diff --git a/Docs/2016-XXX/XXX.pdf b/Docs/2016-XXX/XXX.pdf +deleted file mode 120000 +index 6d347a22..00000000 +--- a/Docs/2016-XXX/XXX.pdf ++++ /dev/null +@@ -1 +0,0 @@ +-../../.git/annex/objects/zJ/X1/SHA256E-s190749--ee0c8329c88f9c1656cc75cf37d4df64060a022e73d199164c5e5222ba1739d1.pdf/SHA256E-s190749--ee0c8329c88f9c1656cc +\ No newline at end of file + + + +$ git clone Documents Documents.tmp +Cloning into 'Documents.tmp'... +done. +$ cd ./Documents.tmp/ +$ ls -la Docs/2016-XXX/XXX/ +total 184 +drwxr-xr-x 4 denis staff 136 Dec 19 00:09 ./ +drwxr-xr-x 8 denis staff 272 Dec 19 00:09 ../ +-rwxr-xr-x 1 denis staff 101 Dec 19 00:09 XXX.pdf* +-rwxr-xr-x 1 denis staff 89586 Dec 19 00:09 Summary.pdf* +$ cat Docs/2016-XXX/XXX/XXX.pdf +/annex/objects/SHA256E-s265898--9c750c01dce9689ac3880224d2e95da6287b0cc89759c0c882e7a9a0fe48d664.pdf + +# End of transcript or log. +"""]] + +### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) + diff --git a/doc/bugs/DBG__58___running___96____47__Users__47__joey__47__homebrew__47__opt__47__gpg-agent__47__bin__47__gpg-agent__39___for_testing_failed.mdwn b/doc/bugs/DBG__58___running___96____47__Users__47__joey__47__homebrew__47__opt__47__gpg-agent__47__bin__47__gpg-agent__39___for_testing_failed.mdwn new file mode 100644 index 000000000..a3b85bdf2 --- /dev/null +++ b/doc/bugs/DBG__58___running___96____47__Users__47__joey__47__homebrew__47__opt__47__gpg-agent__47__bin__47__gpg-agent__39___for_testing_failed.mdwn @@ -0,0 +1,30 @@ +### Please describe the problem. +Issue uploading to S3 remote (Dreamhost) + +### What steps will reproduce the problem? +git-annex copy massart/f16_Web1/screencaptures/IMG_5159.MOV --to=cloud +on my repo + +### What version of git-annex are you using? On what operating system? +6.20161031-g0a58e94 +OS-X 10.11.6 + +### Please provide any additional information below. +I am using a different WIFI I haven't used before. Maybe it is blocking something… + +[[!format sh """ +git-annex copy massart/f16_Web1/screencaptures/IMG_5159.MOV --to=cloud +copy massart/f16_Web1/screencaptures/IMG_5159.MOV (checking cloud...) (to cloud...) +17% 0.0 B/s 0sgpg: error running `/Users/joey/homebrew/opt/gpg-agent/bin/gpg-agent': probably not installed +gpg: DBG: running `/Users/joey/homebrew/opt/gpg-agent/bin/gpg-agent' for testing failed: Configuration error +gpg: can't connect to the agent: IPC connect call failed +gpg: problem with the agent: No agent running +35% 1021.8KB/s 30s + user error (gpg ["--quiet","--trust-model","always","--batch","--passphrase-fd","26","--symmetric","--force-mdc","--no-textmode"] exited 2) +failed +git-annex: copy: 1 failed +"""]] + +### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) +Yes. + diff --git a/doc/bugs/DBG__58___running___96____47__Users__47__joey__47__homebrew__47__opt__47__gpg-agent__47__bin__47__gpg-agent__39___for_testing_failed/comment_1_39718e8a35e42421a8aaf3316ae1d76a._comment b/doc/bugs/DBG__58___running___96____47__Users__47__joey__47__homebrew__47__opt__47__gpg-agent__47__bin__47__gpg-agent__39___for_testing_failed/comment_1_39718e8a35e42421a8aaf3316ae1d76a._comment new file mode 100644 index 000000000..02b62f46e --- /dev/null +++ b/doc/bugs/DBG__58___running___96____47__Users__47__joey__47__homebrew__47__opt__47__gpg-agent__47__bin__47__gpg-agent__39___for_testing_failed/comment_1_39718e8a35e42421a8aaf3316ae1d76a._comment @@ -0,0 +1,14 @@ +[[!comment format=mdwn + username="andrew" + avatar="http://cdn.libravatar.org/avatar/acc0ece1eedf07dd9631e7d7d343c435" + subject="RESOLVED" + date="2016-11-17T14:59:15Z" + content=""" +Ooops. I am on OS-X. I use brew for my gnupg installation. It appears I had removed gpg from the path when installing something. I just needed to run to fix: + + brew link gnupg2 + +Thanks, + +Andrew +"""]] diff --git a/doc/bugs/Inconsistent_results_between_git-annex-fsck_and_git-annex-whereis.mdwn b/doc/bugs/Inconsistent_results_between_git-annex-fsck_and_git-annex-whereis.mdwn new file mode 100644 index 000000000..c9037b575 --- /dev/null +++ b/doc/bugs/Inconsistent_results_between_git-annex-fsck_and_git-annex-whereis.mdwn @@ -0,0 +1,53 @@ +### Please describe the problem. + +I'm seeing some inconsistent results between runs of `git annex fsck` and `git annex whereis` that I'm not able to explain. When I run `git annex fsck`, it reports a few keys that only have 1 copy, and advises me to make more copies. If I run `git annex whereis --key <key>`, git annex confirms that it only knows about 1 copy of this key. If I then use `git log --stat -S'<key>'` to find the actual file that it refers to, and run `git annex whereis <file>`, git annex report 9 copies of this file. Checking on remotes shows that these files do exist on the remote, so why does `git annex fsck` and `git annex whereis` mis-report the number of copies when querying for the key - but not for the actual filename? Additionally, `git annex find --lackingcopies 1` doesn't return any results, but should if there are actually files with not enough copies? + + +### What steps will reproduce the problem? + + +### What version of git-annex are you using? On what operating system? + +5.20151208-1build1 on Ubuntu Xenial, one remote running 5.20141024~bpo70+1 on Debian Wheezy + +### Please provide any additional information below. + +[[!format sh """ +# If you can, paste a complete transcript of the problem occurring here. +# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log + +[william@hactar ~/Pictures/Photo Library]$ git annex whereis SHA256E-s1071765--dbaa7f32ee44c28d6a1f0c8095e8dfd8b4ec433b144085d5097425303a510ea9 +git-annex: SHA256E-s1071765--dbaa7f32ee44c28d6a1f0c8095e8dfd8b4ec433b144085d5097425303a510ea9 not found +git-annex: whereis: 1 failed +[william@hactar ~/Pictures/Photo Library]$ git annex whereis --key SHA256E-s1071765--dbaa7f32ee44c28d6a1f0c8095e8dfd8b4ec433b144085d5097425303a510ea9 +whereis SHA256E-s1071765--dbaa7f32ee44c28d6a1f0c8095e8dfd8b4ec433b144085d5097425303a510ea9 (1 copy) + 7691934f-2542-4103-9122-2db4e6cfc887 -- hactar [here] +ok +[william@hactar ~/Pictures/Photo Library]$ git annex fsck --key SHA256E-s1071765--dbaa7f32ee44c28d6a1f0c8095e8dfd8b4ec433b144085d5097425303a510ea9 +fsck SHA256E-s1071765--dbaa7f32ee44c28d6a1f0c8095e8dfd8b4ec433b144085d5097425303a510ea9 + Only 1 of 3 trustworthy copies exist of SHA256E-s1071765--dbaa7f32ee44c28d6a1f0c8095e8dfd8b4ec433b144085d5097425303a510ea9 + Back it up with git-annex copy. +failed +(recording state in git...) +git-annex: fsck: 1 failed +[william@hactar ~/Pictures/Photo Library]$ git log --stat -S'SHA256E-s1071765--dbaa7f32ee44c28d6a1f0c8095e8dfd8b4ec433b144085d5097425303a510ea9' +[william@hactar ~/Pictures/Photo Library]$ git annex whereis 2009/05/05/P1040890.JPG +whereis 2009/05/05/P1040890.JPG (9 copies) + 0e825a69-1927-4f62-b731-6f3e98bba998 -- william@marvin:/media/backup/annex/photos [marvin] + 1b728ab5-1e32-45a6-bc11-2a4bfdc9d6ab -- backup1 + 5c0caa42-b489-467b-a612-9590fa9d5a94 -- backup2 + 7691934f-2542-4103-9122-2db4e6cfc887 -- hactar [here] + 894b2216-72e0-40e1-8765-1386e1e9e4b4 -- backup3 + 96f19fa8-d385-4e8b-b000-61ee15993a70 -- backup3 + a862b121-d794-4af4-bb56-21adfe8962f2 -- S3 + b083f8ae-42fb-41f0-a2a3-4e7c9f93aadb -- [guide] + bf021ce9-465b-4419-86e7-bddfd208fca4 -- git@newzaphod:~/repositories/annex/photos.git [zaphod] +ok + + +# End of transcript or log. +"""]] + +### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) + +I trust Git Annex to keep hundreds of GB of data safe, and it has never failed me - despite my best efforts diff --git a/doc/bugs/Inconsistent_results_between_git-annex-fsck_and_git-annex-whereis/comment_1_bd56607f228f3480f1355e3bdb755410._comment b/doc/bugs/Inconsistent_results_between_git-annex-fsck_and_git-annex-whereis/comment_1_bd56607f228f3480f1355e3bdb755410._comment new file mode 100644 index 000000000..d65f39fd0 --- /dev/null +++ b/doc/bugs/Inconsistent_results_between_git-annex-fsck_and_git-annex-whereis/comment_1_bd56607f228f3480f1355e3bdb755410._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 1""" + date="2016-12-13T16:42:08Z" + content=""" +The obvious reason for this would be if the file no longer points to that +same key. Perhaps the file got modified and the key is the old version of +the file. + +That would explain everything you showed, so currently I don't see any +bug.. +"""]] diff --git a/doc/bugs/Linux_standalone__39__s_metadata_--batch_can__39__t_parse_UTF-8.mdwn b/doc/bugs/Linux_standalone__39__s_metadata_--batch_can__39__t_parse_UTF-8.mdwn new file mode 100644 index 000000000..73c7ae864 --- /dev/null +++ b/doc/bugs/Linux_standalone__39__s_metadata_--batch_can__39__t_parse_UTF-8.mdwn @@ -0,0 +1,88 @@ +### Please describe the problem. +I had also commented this on [[another bug|bugs/git-annex_fromkey_barfs_on_utf-8_input]], but the original issue there is fixed now. +I tested `fromkey`, `calckey --batch`, `lookupkey --batch` (in standalone) after your fix, they work nicely. + +However, `git-annex metadata --batch --json` using the [[linux standalone|install/Linux_standalone]] (autobuild) still fails when it encounters UTF-8 characters (e.g. ü, ç, ä). +Also, `git-annex metadata --json` gives `"file":"��.txt"` for `ü.txt`. + +This happens only in the standalone builds. + +### What steps will reproduce the problem? + +[[!format sh """ +$ .../git-annex.linux/runshell +$ touch u.txt ü.txt +$ git-annex add . + +$ git-annex metadata --batch --json +{"file":"ü.txt"} +git-annex: Batch input parse failure: Error in $: Failed reading: Cannot decode byte '\xb3': Data.Text.Internal.Encoding.decodeUtf8: Invalid UTF-8 stream + +$ git-annex metadata --batch --json +{"file":"u.txt","fields":{"ç":["b"]}} +git-annex: Batch input parse failure: Error in $: Failed reading: Cannot decode byte '\xb3': Data.Text.Internal.Encoding.decodeUtf8: Invalid UTF-8 stream + +$ git-annex metadata --batch --json +{"file":"u.txt","fields":{"b":["ä"]}} +git-annex: Batch input parse failure: Error in $: Failed reading: Cannot decode byte '\xb3': Data.Text.Internal.Encoding.decodeUtf8: Invalid UTF-8 stream + +$ git-annex metadata --json +{"command":"metadata","note":"","success":true,"key":"SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855.txt","file":"u.txt","fields":{}} +{"command":"metadata","note":"","success":true,"key":"SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855.txt","file":"��.txt","fields":{}} +# success, but the second line should have "file":"ü.txt" +"""]] + +It's the same even if I call `.../git-annex.linux/git-annex` directly (without `runshell`) + +### What version of git-annex are you using? On what operating system? +Using the Linux standalone: [git-annex-standalone-amd64.tar.gz](https://downloads.kitenet.net/git-annex/autobuild/amd64/git-annex-standalone-amd64.tar.gz) on Xubuntu 16.04 + +[[!format sh """ +$ git-annex version +git-annex version: 6.20161213-g55a34b493 +build flags: Assistant Webapp Pairing Testsuite S3(multipartupload)(storageclasses) WebDAV Inotify DBus DesktopNotify XMPP ConcurrentOutput TorrentParser MagicMime Feeds Quvi +key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 SHA1E SHA1 MD5E MD5 WORM URL +remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav tahoe glacier ddar hook external +local repository version: 5 +supported repository versions: 3 5 6 +upgrade supported from repository versions: 0 1 2 3 4 5 +operating system: linux x86_64 +"""]] + +### Please provide any additional information below. + +None of the characters I used have `\xb3` in it, but all the errors happen due to it: +[[!format sh """ +$ .../git-annex.linux/runshell +$ echo -n ü | xxd +00000000: c3bc .. +$ echo -n ç | xxd +00000000: c3a7 .. +$ echo -n ä | xxd +00000000: c3a4 .. +"""]] + +In `runshell`, `ls` can't show UTF-8, but `git-annex status` can: +[[!format sh """ +$ .../git-annex.linux/runshell +$ ls +u.txt ??.txt +$ git-annex status +A u.txt +A ü.txt +"""]] + +`man` complains about locale in `runshell` as well: +[[!format sh """ +$ .../git-annex.linux/runshell +$ man +man: can\'t set the locale; make sure $LC_* and $LANG are correct +What manual page do you want? +# I escaped that \', formatting was messy otherwise +$ set | grep LANG +GDM_LANG='en_GB' +LANG='en_GB.UTF-8' +LANGUAGE='en_GB:en' +$ set | grep LC +# nothing +"""]] diff --git a/doc/bugs/Linux_standalone__39__s_metadata_--batch_can__39__t_parse_UTF-8/comment_1_1765400777911cc61eb591b76c84ae89._comment b/doc/bugs/Linux_standalone__39__s_metadata_--batch_can__39__t_parse_UTF-8/comment_1_1765400777911cc61eb591b76c84ae89._comment new file mode 100644 index 000000000..4a15b1987 --- /dev/null +++ b/doc/bugs/Linux_standalone__39__s_metadata_--batch_can__39__t_parse_UTF-8/comment_1_1765400777911cc61eb591b76c84ae89._comment @@ -0,0 +1,45 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 1""" + date="2016-12-19T20:37:56Z" + content=""" +runshell was recently changed to bypass using the system locales, it +includes its own locale data and attempts to generate a locale definition +file for the locale. The code that did that was failing to notice that +en_GB.UTF-8 was a UTF-8 locale (en_GB.utf8 would work though), which +explains why the locale is not set inside runshell +(git-annex.linux/git-annex is a script that uses runshell). I've corrected +that problem, and verified it fixes the problem you reported. + +---- + +However.. The same thing happens when using LANG=C with git-annex +installed by any method and --json --batch. So the deeper problem is that +it's forcing the batch input to be decoded as utf8 via the current locale. +This happens in Command/MetaData.hs parseJSONInput which uses +`BU.fromString`. + +I tried swapping in `encodeBS` for `BU.fromString`. That prevented the +decoding error, but made git-annex complain that the file was not annexed, +due to a Mojibake problem: + +With `encodeBS`, the input `{"file":"ü.txt"}` is encoded as +`"{\"file\":\"\195\188.txt\"}"`. Aeson parses that input to this: + + JSONActionItem {itemCommand = Nothing, itemKey = Nothing, itemFile = Just "\252.txt", itemAdded = Nothing} + +Note that the first two bytes have been +parsed by Aeson as unicode (since JSON is unicode encoded), +yielding character 252 (ü). + +In a unicode locale, this works ok, because the encoding layer is able to +convert that unicode character back to two bytes 195 188 +and finds the file on disk. But in a non-unicode locale, it doesn't know +what to do with the unicode character, and in fact it gets discarded +and so it looks for a file named ".txt". + +So, to make --batch --json input work in non-unicode locales, it would +need, after parsing the json, to re-encode filenames (and perhaps other +data), from utf8 to the filesystem encoding. I have not yet worked out how +to do that. +"""]] diff --git a/doc/bugs/Metadata_values_get_stuck_when_repeatedly_modified_in_the_same_batch_mode_run.mdwn b/doc/bugs/Metadata_values_get_stuck_when_repeatedly_modified_in_the_same_batch_mode_run.mdwn new file mode 100644 index 000000000..26790bea3 --- /dev/null +++ b/doc/bugs/Metadata_values_get_stuck_when_repeatedly_modified_in_the_same_batch_mode_run.mdwn @@ -0,0 +1,85 @@ +### Please describe the problem. + +While running `git-annex metadata --batch --json`, repeatedly assigning a field to the same value in the same run (with different values in between the assignments of the same value) causes a value to get stuck. + +### What steps will reproduce the problem? + + $ touch test.txt + $ git annex add + $ git-annex metadata --batch --json + {"file":"test.txt","fields":{"f":["a"]}} + # prints { ... "f":["a"] ... } + {"file":"test.txt","fields":{"f":["b"]}} + # prints { ... "f":["b"] ... } + {"file":"test.txt","fields":{"f":["c"]}} + # prints { ... "f":["c"] ... } + {"file":"test.txt","fields":{"f":["a"]}} + # prints { ... "f":["a", "c"] ... } + {"file":"test.txt","fields":{"f":["b"]}} + # prints { ... "f":["c"] ... } + +### What version of git-annex are you using? On what operating system? + + git-annex version: 6.20161122+gitg9f179ae-1~ndall+1 + build flags: Assistant Webapp Pairing Testsuite S3(multipartupload)(storageclasses) WebDAV Inotify DBus DesktopNotify XMPP ConcurrentOutput TorrentParser MagicMime Feeds Quvi + key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 SHA1E SHA1 MD5E MD5 WORM URL + remote types: git gcrypt S3 bup directory rsync web bittorrent webdav tahoe glacier ddar hook external + local repository version: 5 + supported repository versions: 3 5 6 + upgrade supported from repository versions: 0 1 2 3 4 5 + operating system: linux x86_64 + +I'm using Xubuntu 16.04, with the `git-annex-standalone` package from NeuroDebian repository. + +### Please provide any additional information below. + +If you keep reassigning the same values, things get very weird. Full inputs/outputs from a sample run: + + {"file":"test.txt","fields":{"f":["a"]}} + {"command":"metadata","note":"f=a\nf-lastchanged=2016-12-05@21-17-39\nlastchanged=2016-12-05@21-17-39\n","success":true,"key":"SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855.txt","file":"test.txt","fields": {"f-lastchanged":["2016-12-05@21-17-39"],"lastchanged":["2016-12-05@21-17-39"],"f":["a"]}} + {"file":"test.txt","fields":{"f":["b"]}} + {"command":"metadata","note":"f=b\nf-lastchanged=2016-12-05@21-17-39\nlastchanged=2016-12-05@21-17-39\n","success":true,"key":"SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855.txt","file":"test.txt","fields":{"f-lastchanged":["2016-12-05@21-17-39"],"lastchanged":["2016-12-05@21-17-39"],"f":["b"]}} + {"file":"test.txt","fields":{"f":["c"]}} + {"command":"metadata","note":"f=c\nf-lastchanged=2016-12-05@21-17-39\nlastchanged=2016-12-05@21-17-39\n","success":true,"key":"SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855.txt","file":"test.txt","fields":{"f-lastchanged":["2016-12-05@21-17-39"],"lastchanged":["2016-12-05@21-17-39"],"f":["c"]}} + {"file":"test.txt","fields":{"f":["a"]}} + {"command":"metadata","note":"f=a\nf=c\nf-lastchanged=2016-12-05@21-17-39\nlastchanged=2016-12-05@21-17-39\n","success":true,"key":"SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855.txt","file":"test.txt","fields":{"f-lastchanged":["2016-12-05@21-17-39"],"lastchanged":["2016-12-05@21-17-39"],"f":["a","c"]}} + {"file":"test.txt","fields":{"f":["b"]}} + {"command":"metadata","note":"f=c\nf-lastchanged=2016-12-05@21-17-39\nlastchanged=2016-12-05@21-17-39\n","success":true,"key":"SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855.txt","file":"test.txt","fields":{"f-lastchanged":["2016-12-05@21-17-39"],"lastchanged":["2016-12-05@21-17-39"],"f":["c"]}} + {"file":"test.txt","fields":{"f":[]}} + {"command":"metadata","note":"lastchanged=2016-12-05@21-17-39\n","success":true,"key":"SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855.txt","file":"test.txt","fields":{"lastchanged":["2016-12-05@21-17-39"]}} + {"file":"test.txt","fields":{"f":["b"]}} + {"command":"metadata","note":"lastchanged=2016-12-05@21-17-39\n","success":true,"key":"SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855.txt","file":"test.txt","fields":{"lastchanged":["2016-12-05@21-17-39"]}} + {"file":"test.txt","fields":{"f":["c"]}} + {"command":"metadata","note":"f=c\nf-lastchanged=2016-12-05@21-17-39\nlastchanged=2016-12-05@21-17-39\n","success":true,"key":"SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855.txt","file":"test.txt","fields":{"f-lastchanged":["2016-12-05@21-17-39"],"lastchanged":["2016-12-05@21-17-39"],"f":["c"]}} + {"file":"test.txt","fields":{"f":["a"]}} + {"command":"metadata","note":"f=a\nf=c\nf-lastchanged=2016-12-05@21-17-39\nlastchanged=2016-12-05@21-17-39\n","success":true,"key":"SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855.txt","file":"test.txt","fields":{"f-lastchanged":["2016-12-05@21-17-39"],"lastchanged":["2016-12-05@21-17-39"],"f":["a","c"]}} + {"file":"test.txt","fields":{"f":["b"]}} + {"command":"metadata","note":"f=c\nf-lastchanged=2016-12-05@21-17-39\nlastchanged=2016-12-05@21-17-39\n","success":true,"key":"SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855.txt","file":"test.txt","fields":{"f-lastchanged":["2016-12-05@21-17-39"],"lastchanged":["2016-12-05@21-17-39"],"f":["c"]}} + {"file":"test.txt","fields":{"f":["c"]}} + {"command":"metadata","note":"f=c\nf-lastchanged=2016-12-05@21-17-39\nlastchanged=2016-12-05@21-17-39\n","success":true,"key":"SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855.txt","file":"test.txt","fields":{"f-lastchanged":["2016-12-05@21-17-39"],"lastchanged":["2016-12-05@21-17-39"],"f":["c"]}} + {"file":"test.txt","fields":{"f":["a"]}} + {"command":"metadata","note":"f=a\nf=c\nf-lastchanged=2016-12-05@21-17-39\nlastchanged=2016-12-05@21-17-39\n","success":true,"key":"SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855.txt","file":"test.txt","fields":{"f-lastchanged":["2016-12-05@21-17-39"],"lastchanged":["2016-12-05@21-17-39"],"f":["a","c"]}} + {"file":"test.txt","fields":{"f":["b"]}} + {"command":"metadata","note":"f=c\nf-lastchanged=2016-12-05@21-17-39\nlastchanged=2016-12-05@21-17-39\n","success":true,"key":"SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855.txt","file":"test.txt","fields":{"f-lastchanged":["2016-12-05@21-17-39"],"lastchanged":["2016-12-05@21-17-39"],"f":["c"]}} + {"file":"test.txt","fields":{"f":["b"]}} + {"command":"metadata","note":"f=b\nf=c\nf-lastchanged=2016-12-05@21-17-39\nlastchanged=2016-12-05@21-17-39\n","success":true,"key":"SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855.txt","file":"test.txt","fields":{"f-lastchanged":["2016-12-05@21-17-39"],"lastchanged":["2016-12-05@21-17-39"],"f":["b","c"]}} + {"file":"test.txt","fields":{"f":["a"]}} + {"command":"metadata","note":"f=a\nf=b\nf=c\nf-lastchanged=2016-12-05@21-17-39\nlastchanged=2016-12-05@21-17-39\n","success":true,"key":"SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855.txt","file":"test.txt","fields":{"f-lastchanged":["2016-12-05@21-17-39"],"lastchanged":["2016-12-05@21-17-39"],"f":["a","b","c"]}} + {"file":"test.txt","fields":{"f":["d"]}} + {"command":"metadata","note":"f=b\nf=c\nf=d\nf-lastchanged=2016-12-05@21-17-39\nlastchanged=2016-12-05@21-17-39\n","success":true,"key":"SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855.txt","file":"test.txt","fields":{"f-lastchanged":["2016-12-05@21-17-39"],"lastchanged":["2016-12-05@21-17-39"],"f":["b","c","d"]}} + {"file":"test.txt","fields":{"f":["a"]}} + {"command":"metadata","note":"f=b\nf=c\nf=d\nf-lastchanged=2016-12-05@21-17-39\nlastchanged=2016-12-05@21-17-39\n","success":true,"key":"SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855.txt","file":"test.txt","fields":{"f-lastchanged":["2016-12-05@21-17-39"],"lastchanged":["2016-12-05@21-17-39"],"f":["b","c","d"]}} + {"file":"test.txt","fields":{"f":["a"]}} + {"command":"metadata","note":"f=b\nf=c\nf=d\nf-lastchanged=2016-12-05@21-17-39\nlastchanged=2016-12-05@21-17-39\n","success":true,"key":"SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855.txt","file":"test.txt","fields":{"f-lastchanged":["2016-12-05@21-17-39"],"lastchanged":["2016-12-05@21-17-39"],"f":["b","c","d"]}} + {"file":"test.txt","fields":{"f":[]}} + {"command":"metadata","note":"f=c\nf-lastchanged=2016-12-05@21-17-39\nlastchanged=2016-12-05@21-17-39\n","success":true,"key":"SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855.txt","file":"test.txt","fields":{"f-lastchanged":["2016-12-05@21-17-39"],"lastchanged":["2016-12-05@21-17-39"],"f":["c"]}} + +Restarting the process solves the issue. + +### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) + +I love the metadata functionality so much that I wrote [[tips/a_gui_for_metadata_operations]] and discovered this bug. +Metadata driven views are awesome (but I don't like the entire folder hierarchy being appended to the filename). +I haven't used the other commands much since I have not yet organized most of my stuff (and their naively copy-pasted backups), but I am glad I discovered git-annex before I began organizing. + +> [[fixed|done]] --[[Joey]] diff --git a/doc/bugs/Metadata_values_get_stuck_when_repeatedly_modified_in_the_same_batch_mode_run/comment_1_627bb742a5042741e9a1c294addd69b2._comment b/doc/bugs/Metadata_values_get_stuck_when_repeatedly_modified_in_the_same_batch_mode_run/comment_1_627bb742a5042741e9a1c294addd69b2._comment new file mode 100644 index 000000000..4f82db153 --- /dev/null +++ b/doc/bugs/Metadata_values_get_stuck_when_repeatedly_modified_in_the_same_batch_mode_run/comment_1_627bb742a5042741e9a1c294addd69b2._comment @@ -0,0 +1,24 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 1""" + date="2016-12-13T14:40:02Z" + content=""" +I thought this would involve the journal, but it seems not; same +behavior occurs if the journal is committed after each metadata change. + +Looking at the new metadata value in the case where a and c both get set, +it is: + + MetaData (fromList [(MetaField "f",fromList [MetaValue (CurrentlySet True) "a",MetaValue (CurrentlySet False) "c"])]) + +That is supposed to unset c, with the CurrentlySet False, but instead c +remains set somehow. + +Aha, the use of `addMetaData'` causes the bug. That reuses the same +timestamp, and indeed the same timestamp is used for all the batch +changes. With the same timestamp for the log line that sets c as the line +that removes it, it's indeterminite which line will be acted on first, and +so the removal can be processed before the addition, leaving c "stuck". + +Fixing.. +"""]] diff --git a/doc/bugs/Nearline_bucket_stopped_working___40__can__39__t_even_HEAD_files__41__/comment_2_c227071f23a96ed9928f128e7f77e503._comment b/doc/bugs/Nearline_bucket_stopped_working___40__can__39__t_even_HEAD_files__41__/comment_2_c227071f23a96ed9928f128e7f77e503._comment new file mode 100644 index 000000000..820e6b040 --- /dev/null +++ b/doc/bugs/Nearline_bucket_stopped_working___40__can__39__t_even_HEAD_files__41__/comment_2_c227071f23a96ed9928f128e7f77e503._comment @@ -0,0 +1,17 @@ +[[!comment format=mdwn + username="justin.lebar@7a36fcafc322d9a381e89f08ab6289033c6dde91" + nickname="justin.lebar" + avatar="http://cdn.libravatar.org/avatar/9fca4b61a1ab555f231851e7543f9a3e" + subject="comment 2" + date="2016-11-20T03:47:23Z" + content=""" +Thanks for your reply, Joey. Sorry for the delay getting back to this -- I didn't realize I hadn't enabled notifications on the thread. + +The GCS docs suggest that 400 errors should be accompanied by an explanation in the reply body. + +> Error responses usually include a JSON document in the response body, which contains information about the error. + +https://cloud.google.com/storage/docs/json_api/v1/status-codes + +Do you think we're not getting an http response body here, or that it's not being printed out? +"""]] diff --git a/doc/bugs/Nearline_bucket_stopped_working___40__can__39__t_even_HEAD_files__41__/comment_3_5ac676877feaa7cdb9e05d6b71b1a4c3._comment b/doc/bugs/Nearline_bucket_stopped_working___40__can__39__t_even_HEAD_files__41__/comment_3_5ac676877feaa7cdb9e05d6b71b1a4c3._comment new file mode 100644 index 000000000..67b215bc7 --- /dev/null +++ b/doc/bugs/Nearline_bucket_stopped_working___40__can__39__t_even_HEAD_files__41__/comment_3_5ac676877feaa7cdb9e05d6b71b1a4c3._comment @@ -0,0 +1,11 @@ +[[!comment format=mdwn + username="justin.lebar@7a36fcafc322d9a381e89f08ab6289033c6dde91" + nickname="justin.lebar" + avatar="http://cdn.libravatar.org/avatar/9fca4b61a1ab555f231851e7543f9a3e" + subject="comment 3" + date="2016-12-04T04:30:38Z" + content=""" +For a while things were working, but now it's not working again, same problem as before. + +Do you think maybe it's a timestamp bug in the signature or something? That could explain this \"mysteriously works then stops working\" behavior. +"""]] diff --git a/doc/bugs/Single_space_in_file_name_causes_git_annex_add_to_fail.mdwn b/doc/bugs/Single_space_in_file_name_causes_git_annex_add_to_fail.mdwn new file mode 100644 index 000000000..096562199 --- /dev/null +++ b/doc/bugs/Single_space_in_file_name_causes_git_annex_add_to_fail.mdwn @@ -0,0 +1,57 @@ +### Please describe the problem. + +If a filename has a single space (and only one space), `git annex add` will fail out with the following message: + + add one two git-annex: unknown response from git cat-file ("HEAD:./one two missing",Ref "HEAD:./one two") + CallStack (from HasCallStack): + error, called at ./Git/CatFile.hs:102:28 in main:Git.CatFile + +### What steps will reproduce the problem? + +Run the following: + + git init . + git annex init + touch "one two" + # this will cause error + git annex add "one two" + touch "one two three" + # this is fine + git annex add "one two three" + +### What version of git-annex are you using? On what operating system? + +Output of `git annex version` + + git-annex version: 6.20161027 + build flags: Assistant Webapp Pairing Testsuite S3(multipartupload)(storageclasses) WebDAV Inotify DBus DesktopNotify XMPP ConcurrentOutput TorrentParser MagicMime Feeds Quvi + key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 SHA1E SHA1 MD5E MD5 WORM URL + remote types: git gcrypt S3 bup directory rsync web bittorrent webdav tahoe glacier ddar hook external + local repository version: 5 + supported repository versions: 3 5 6 + upgrade supported from repository versions: 0 1 2 3 4 5 + operating system: linux x86_64 + +Operating System: Linux (NixOS 16.09.909.238c7e0 (Flounder)) + +### Please provide any additional information below. + +Maybe related to [https://git-annex.branchable.com/forum/unknown_response_from_git_cat-file/](https://git-annex.branchable.com/forum/unknown_response_from_git_cat-file/) or [https://git-annex.branchable.com/bugs/git_annex_import_fails_on_filenames_with_newlines_in_them/](https://git-annex.branchable.com/bugs/git_annex_import_fails_on_filenames_with_newlines_in_them/)? + +EDIT: Somewhat surprisingly, if I build from source using `cabal`, everything works fine. + + .cabal-sandbox/bin/git-annex version + git-annex version: 6.20161113-g1e88c12 + build flags: Assistant Webapp Pairing Testsuite WebDAV Inotify ConcurrentOutput TorrentParser MagicMime Feeds Quvi + key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 SHA1E SHA1 MD5E MD5 WORM URL + remote types: git gcrypt bup directory rsync web bittorrent webdav tahoe glacier ddar hook external + +Not sure whether this means that this bug is actually fixed or whether it's an artifact of how things are built in Nix. + +### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) + +Just starting out with it as a way of archiving some of my media seems to work great apart from this. Thanks a bunch! + +> This bug was already fixed in git-annex 6.20161031. I told the Debian +> maintainer about the bug fix at the time; package has not been updated +> yet. [[done]] on git-annex side. --[[Joey]] diff --git a/doc/bugs/Single_space_in_file_name_causes_git_annex_add_to_fail/comment_1_0cf0856c6408c9c588133023a3a6ba8f._comment b/doc/bugs/Single_space_in_file_name_causes_git_annex_add_to_fail/comment_1_0cf0856c6408c9c588133023a3a6ba8f._comment new file mode 100644 index 000000000..5f8b120e2 --- /dev/null +++ b/doc/bugs/Single_space_in_file_name_causes_git_annex_add_to_fail/comment_1_0cf0856c6408c9c588133023a3a6ba8f._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="gfa@1e86118cd41fbfea50004af221471ad97b55af18" + nickname="gfa" + avatar="http://cdn.libravatar.org/avatar/4678da4da55c67fa668e31ea0a76b201" + subject="same on Debian" + date="2016-11-14T09:13:19Z" + content=""" +I face the same issue on Debian testing + +git-annex 6.20161012-1 +git 1:2.10.2-1 +"""]] diff --git a/doc/bugs/Single_space_in_file_name_causes_git_annex_add_to_fail/comment_2_13c242250d1509d933b8f0bcb7b67302._comment b/doc/bugs/Single_space_in_file_name_causes_git_annex_add_to_fail/comment_2_13c242250d1509d933b8f0bcb7b67302._comment new file mode 100644 index 000000000..bf41d40a5 --- /dev/null +++ b/doc/bugs/Single_space_in_file_name_causes_git_annex_add_to_fail/comment_2_13c242250d1509d933b8f0bcb7b67302._comment @@ -0,0 +1,16 @@ +[[!comment format=mdwn + username="https://launchpad.net/~stephane-gourichon-lpad" + nickname="stephane-gourichon-lpad" + avatar="http://cdn.libravatar.org/avatar/02d4a0af59175f9123720b4481d55a769ba954e20f6dd9b2792217d9fa0c6089" + subject="Known bug, fixed." + date="2016-11-23T18:04:27Z" + content=""" +This is a known bug introduced in 6.20161012 and fixed in 6.20161031. + +Solution is: just update your copy of git-annex. At this time most recent is 6.20161119 . + +For more details, see changelog at https://github.com/joeyh/git-annex/blob/master/CHANGELOG#L53 + + + +"""]] diff --git a/doc/bugs/Webapp_missing_CSS_and_JS_resources___40__401_Unauthorized__41__.mdwn b/doc/bugs/Webapp_missing_CSS_and_JS_resources___40__401_Unauthorized__41__.mdwn index 0c3fc16a1..17fe3ac25 100644 --- a/doc/bugs/Webapp_missing_CSS_and_JS_resources___40__401_Unauthorized__41__.mdwn +++ b/doc/bugs/Webapp_missing_CSS_and_JS_resources___40__401_Unauthorized__41__.mdwn @@ -30,4 +30,5 @@ operating system: darwin x86_64 ### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) - +> [[done]], my fix worked! Don't know entirely why it was needed.. +> --[[Joey]] diff --git a/doc/bugs/Webapp_missing_CSS_and_JS_resources___40__401_Unauthorized__41__/comment_3_54bd11140dbe794182263c1a062ad031._comment b/doc/bugs/Webapp_missing_CSS_and_JS_resources___40__401_Unauthorized__41__/comment_3_54bd11140dbe794182263c1a062ad031._comment new file mode 100644 index 000000000..cb6e9b4d2 --- /dev/null +++ b/doc/bugs/Webapp_missing_CSS_and_JS_resources___40__401_Unauthorized__41__/comment_3_54bd11140dbe794182263c1a062ad031._comment @@ -0,0 +1,21 @@ +[[!comment format=mdwn + username="christopher@5845ecd3cef9edadd4dc084df00e1fa60ce311eb" + nickname="christopher" + avatar="http://cdn.libravatar.org/avatar/4b722efb21f38d9944730c93727bc602" + subject="comment 3" + date="2016-11-15T12:15:37Z" + content=""" +Hi Joey, + +I installed git-annex using the homebrew recipe from https://github.com/Homebrew/homebrew-core/blob/master/Formula/git-annex.rb, OS X 10.11.6 (15G31) + +These are the dependencies reported by homebrew: + +Build: ghc ✔, cabal-install ✔, pkg-config ✔ +Required: gsasl ✔, libidn ✔, libmagic ✔, gnutls ✔, quvi ✔ + +I've re-installed using \"brew install git-annex --HEAD\" to pull in your latest commit and I can confirm that everything works as expected and the /static/ resources load correctly. + +Thanks, +Chris +"""]] diff --git a/doc/bugs/When_stopping___96__git_annex_get__96___files_left_broken.mdwn b/doc/bugs/When_stopping___96__git_annex_get__96___files_left_broken.mdwn new file mode 100644 index 000000000..a8df72354 --- /dev/null +++ b/doc/bugs/When_stopping___96__git_annex_get__96___files_left_broken.mdwn @@ -0,0 +1,40 @@ +### Please describe the problem. + +``` +> git annex get Narnia/ +get Narnia/Course of a Generation/01 Sail Around the World.mp3 (from Seagate...) +SHA256E-s8395599--2fea961006a279f0765c45755b35a06f0a4fc6bfbab6118182ebc693d7b47a91.mp3 + 8,395,599 100% 29.65MB/s 0:00:00 (xfr#1, to-chk=0/1) +(checksum...) ^C⏎ +``` + +``` +> mpv ~/Music/sorted/Narnia/Course\ of\ a\ Generation/ +Playing: /home/philip/Music/sorted/Narnia/Course of a Generation/ +[file] This is a directory - adding to playlist. + +Playing: /home/philip/Music/sorted/Narnia/Course of a Generation/01 Sail Around the World.mp3 +Failed to recognize file format. + +Playing: /home/philip/Music/sorted/Narnia/Course of a Generation/02 When the Stars Are Falling.mp3 +``` + +``` +> git annex version +git-annex version: 6.20161012 +build flags: Assistant Webapp Pairing Testsuite S3(multipartupload)(storageclasses) WebDAV Inotify DBus DesktopNotify XMPP ConcurrentOutput TorrentParser MagicMime Feeds Quvi +key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 SHA1E SHA1 MD5E MD5 WORM URL +remote types: git gcrypt S3 bup directory rsync web bittorrent webdav tahoe glacier ddar hook external +local repository version: 6 +supported repository versions: 3 5 6 +upgrade supported from repository versions: 0 1 2 3 4 5 +operating system: linux x86_64 +``` + +Any consecutive `git annex get` commands don’t notice that the file is not completely transferred and leave it in a broken state. +`git annex get --failed` does not correct the problem. + + +### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) + +Yes, it (kind of) works for keeping my music library in sync. diff --git a/doc/bugs/When_stopping___96__git_annex_get__96___files_left_broken/comment_1_9392346203c561b88f30fa2ce7540b76._comment b/doc/bugs/When_stopping___96__git_annex_get__96___files_left_broken/comment_1_9392346203c561b88f30fa2ce7540b76._comment new file mode 100644 index 000000000..4f90ddfa6 --- /dev/null +++ b/doc/bugs/When_stopping___96__git_annex_get__96___files_left_broken/comment_1_9392346203c561b88f30fa2ce7540b76._comment @@ -0,0 +1,22 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 1""" + date="2016-11-16T18:36:34Z" + content=""" +Thing is, git-annex get does not update the file in place. Only once the +entire file is downloaded, and its content is verified correct is it moved +into a place where you can access it. + +So, it seems much more likely to me that the content of the file, as +originally added to git-annex, was bad, and the it had just finished +verifying the content and moving it into place when you interruped the +command. + +Please check with `git annex fsck` on the file and see if it determines +it has the content git-annex expects it to have. + +However, I notice you're using a v6 repository. Is the file an unlocked +file? It's possible that in that specific case there could be a bug. +I've interrupted `git annex get` on a nearly daily basis for years, but +v6 is still experimental and not as well tested. +"""]] diff --git a/doc/bugs/YouTube_-_error_in_importfeed.mdwn b/doc/bugs/YouTube_-_error_in_importfeed.mdwn new file mode 100644 index 000000000..d300c621f --- /dev/null +++ b/doc/bugs/YouTube_-_error_in_importfeed.mdwn @@ -0,0 +1,74 @@ +### Please describe the problem. +When adding a YouTube channel via importfeed I get the error: + +``` + warning: bad feed content; no enclosures to download +``` + +### What steps will reproduce the problem? +1. `cd $(mktemp -d)` +2. `git init && git annex init` +3. `git annex importfeed https://www.youtube.com/feeds/videos.xml\?playlist_id\=PLoXkGkpREHNBY9KtkdqhBIfx-waIGSKet` +4. Get sad. :-( + +(URL [https://www.youtube.com/feeds/videos.xml?playlist_id=PLoXkGkpREHNBY9KtkdqhBIfx-waIGSKet](https://www.youtube.com/feeds/videos.xml?playlist_id=PLoXkGkpREHNBY9KtkdqhBIfx-waIGSKet) looks like a feed to Firefox) + + +### What version of git-annex are you using? On what operating system? +OSX (MacOS?) - installed via homebrew + + git-annex version: 6.20161210 + build flags: Assistant Webapp Pairing Testsuite S3(multipartupload)(storageclasses) WebDAV FsEvents XMPP ConcurrentOutput TorrentParser MagicMime Feeds Quvi + key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 SHA1E SHA1 MD5E MD5 WORM URL + remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav tahoe glacier ddar hook external + +Debian Jessie - installed via apt-get (ASIDE: why is the apt-get version sooooo old?) + + git-annex version: 5.20141125 + build flags: Assistant Webapp Webapp-secure Pairing Testsuite S3 WebDAV Inotify DBus DesktopNotify XMPP DNS Feeds Quvi TDFA CryptoHash + key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SKEIN256E SKEIN512E SHA256 SHA1 SHA512 SHA224 SHA384 SKEIN256 SKEIN512 WORM URL + remote types: git gcrypt S3 bup directory rsync web webdav tahoe glacier ddar hook external + + +### Additional information + +Running with `--debug` (see below) seems to indicate that the feed downloads correctly, but it is the parsing that is failing. I don't know what command is being run to parse the feed though. + + +``` shell +git annex importfeed --debug https://www.youtube.com/feeds/videos.xml\?playlist_id\=PLoXkGkpREHNBY9KtkdqhBIfx-waIGSKet +``` +results in: + +``` shell +(checking known urls...) [2016-12-19 12:39:36.387714] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","git-annex"] +[2016-12-19 12:39:36.392367] process done ExitSuccess +[2016-12-19 12:39:36.392496] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--hash","refs/heads/git-annex"] +[2016-12-19 12:39:36.396484] process done ExitSuccess +[2016-12-19 12:39:36.406716] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","ls-files","--stage","-z","--","."] +[2016-12-19 12:39:36.412674] process done ExitSuccess +importfeed https://www.youtube.com/feeds/videos.xml?playlist_id=PLoXkGkpREHNBY9KtkdqhBIfx-waIGSKet +[2016-12-19 12:39:36.413555] call: wget ["--clobber","-c","-O","/var/folders/l0/l60294_970b9fh46062znm0r0000gn/T/feed16807282475249","https://www.youtube.com/feeds/videos.xml?playlist_id=PLoXkGkpREHNBY9KtkdqhBIfx-waIGSKet","--user-agent","git-annex/6.20161210"] +--2016-12-19 12:39:36-- https://www.youtube.com/feeds/videos.xml?playlist_id=PLoXkGkpREHNBY9KtkdqhBIfx-waIGSKet +Resolving www.youtube.com... 216.58.199.78, 2404:6800:4006:806::200e +Connecting to www.youtube.com|216.58.199.78|:443... connected. +HTTP request sent, awaiting response... 200 OK +Length: unspecified [text/xml] +Saving to: ‘/var/folders/l0/l60294_970b9fh46062znm0r0000gn/T/feed16807282475249’ + +/var/folders/l0/l60294_970b9fh46062znm0r0000gn/T/f [ <=> ] 23.81K --.-KB/s in 0.02s + +2016-12-19 12:39:37 (1.22 MB/s) - ‘/var/folders/l0/l60294_970b9fh46062znm0r0000gn/T/feed16807282475249’ saved [24386] + +[2016-12-19 12:39:37.595869] process done ExitSuccess + + warning: bad feed content; no enclosures to download +ok +``` + +### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) + +Yes, for years. I donated to fund the dev and proudly display my git-annex stickers! + +> This is now fixed in feed's git repository, and will be in the next +> release of feed after the current 0.3.11.1 release. [[done]] --[[Joey]] diff --git a/doc/bugs/YouTube_-_error_in_importfeed/comment_1_3c6a60ab9c772b95ca5205199554b914._comment b/doc/bugs/YouTube_-_error_in_importfeed/comment_1_3c6a60ab9c772b95ca5205199554b914._comment new file mode 100644 index 000000000..afdff4942 --- /dev/null +++ b/doc/bugs/YouTube_-_error_in_importfeed/comment_1_3c6a60ab9c772b95ca5205199554b914._comment @@ -0,0 +1,16 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 1""" + date="2016-12-19T19:55:23Z" + content=""" +It's somewhat misleading that it complains there are no enclosures in the +feed. While importfeed mostly downloads only enclosures in podcast feeds, +it also checks link tags, which this feed contains, to see if quvi supports +downloading content from them. Quvi does support the links in this feed, +so it should work despite there being no enclosures. + +I've reproduced it not working, and it seems that the problem is this is +not quite a valid Atom feed, and the feed parsing library is failing to +parse it. Perhaps that can be improved; I filed a bug here +<https://github.com/bergmark/feed/issues/18> +"""]] diff --git a/doc/bugs/YouTube_-_error_in_importfeed/comment_2_fe28e0f76dbefb1963820011fc8fc3e7._comment b/doc/bugs/YouTube_-_error_in_importfeed/comment_2_fe28e0f76dbefb1963820011fc8fc3e7._comment new file mode 100644 index 000000000..edf4c855c --- /dev/null +++ b/doc/bugs/YouTube_-_error_in_importfeed/comment_2_fe28e0f76dbefb1963820011fc8fc3e7._comment @@ -0,0 +1,9 @@ +[[!comment format=mdwn + username="m8r-achx62@7323980ed426b7f78c85dfefe7358672bce44e98" + nickname="m8r-achx62" + avatar="http://cdn.libravatar.org/avatar/adaf4c4277529e10e32c467fa4ed4b9a" + subject="comment 2" + date="2016-12-19T22:33:13Z" + content=""" +Thanks for following this up Joey! +"""]] diff --git a/doc/bugs/__34__commitBuffer__58___invalid_argument___40__invalid_character__41____34___during___34__git_annex_sync__34__.mdwn b/doc/bugs/__34__commitBuffer__58___invalid_argument___40__invalid_character__41____34___during___34__git_annex_sync__34__.mdwn new file mode 100644 index 000000000..d4e3ad471 --- /dev/null +++ b/doc/bugs/__34__commitBuffer__58___invalid_argument___40__invalid_character__41____34___during___34__git_annex_sync__34__.mdwn @@ -0,0 +1,52 @@ +### Please describe the problem. + +In my unlocked adjusted branch, I get a lot of errors during "git annex sync". It appears to work fine otherwise (the files actually get synced). Below is what I see on the terminal. The repository is otherwise clean (no local or remote changes). +This has started to happen around a month ago, though I cannot pinpoint the exact version. This is in the same repo you used to debug the disappearing files in direct mode recently (thanks a lot btw!). + +### What version of git-annex are you using? On what operating system? + +[[!format sh """ +$ git annex version +git-annex version: 6.20161110-gd48f4ca +build flags: Assistant Webapp Pairing Testsuite S3(multipartupload)(storageclasses) WebDAV Inotify DBus DesktopNotify XMPP ConcurrentOutput TorrentParser MagicMime Feeds Quvi +key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 SHA1E SHA1 MD5E MD5 WORM URL +remote types: git gcrypt S3 bup directory rsync web bittorrent webdav tahoe glacier ddar hook external +$ lsb_release -a +No LSB modules are available. +Distributor ID: Debian +Description: Debian GNU/Linux 8.6 (jessie) +Release: 8.6 +Codename: jessie +"""]] + +### Please provide any additional information below. + +[[!format sh """ +$ git annex sync --content +commit +On branch adjusted/master(unlocked) +nothing to commit, working tree clean +ok +pull origin +remote: Counting objects: 113, done. +remote: Compressing objects: 100% (113/113), done. +remote: Total 113 (delta 112), reused 0 (delta 0) +Receiving objects: 100% (113/113), 7.16 KiB | 0 bytes/s, done. +Resolving deltas: 100% (112/112), completed with 112 local objects. +From /srv/annex/bilder + 97a4806..78cb4ef git-annex -> origin/git-annex +ok +(merging origin/git-annex into git-annex...) + +git-annex: fd:25: commitBuffer: invalid argument (invalid character) +failed + +git-annex: fd:25: commitBuffer: invalid argument (invalid character) +failed + +[...] + +git-annex: fd:25: commitBuffer: invalid argument (invalid character) +failed +git-annex: sync: 2653 failed +"""]] diff --git a/doc/bugs/add_fails_with_v6_repo_when_four_levels_deep/comment_1_e308245bf81a536db6f9a2b743d912bf._comment b/doc/bugs/add_fails_with_v6_repo_when_four_levels_deep/comment_1_e308245bf81a536db6f9a2b743d912bf._comment new file mode 100644 index 000000000..a5d988fae --- /dev/null +++ b/doc/bugs/add_fails_with_v6_repo_when_four_levels_deep/comment_1_e308245bf81a536db6f9a2b743d912bf._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 1""" + date="2016-11-16T18:49:09Z" + content=""" +I'm not able to reproduce the problem with your test case and git-annex +version 6.20161012. + +Can you still reproduce it after upgrading? +"""]] diff --git a/doc/bugs/add_fails_with_v6_repo_when_four_levels_deep/comment_2_b3998823aca4266089dcbcf325d8f8c1._comment b/doc/bugs/add_fails_with_v6_repo_when_four_levels_deep/comment_2_b3998823aca4266089dcbcf325d8f8c1._comment new file mode 100644 index 000000000..1046fb066 --- /dev/null +++ b/doc/bugs/add_fails_with_v6_repo_when_four_levels_deep/comment_2_b3998823aca4266089dcbcf325d8f8c1._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="t.z.mates" + avatar="http://cdn.libravatar.org/avatar/90f15fad216078fd08d62cc676487925" + subject="comment 2" + date="2016-11-19T04:42:25Z" + content=""" +Thanks for looking into it; I just checked again, and even on the newest version (6.20161118 binary), I'm still experiencing the behavior. However, I checked on an older OpenSuse box I have, and there it works (6.20161031 from OpenSuse repo). + +Since my two machines experiencing the problem are both running arch, it seems it's somehow related to that distro. I've checked both installing via the binary (from kitenet) and from the arch community repo, but both produce the same behavior. Further, the OpenSuse install has the same build flags as the binaries, so that doesn't seem to be it. Are there any other diagnostics I can run? + +This particular problem isn't very troublesome (it doesn't seem to have any material impact aside from error messages); however, I also occasionally experience a more serious bug. Namely, when certain (seemingly random) files are added to the repo locked, their content disappears and the symlink is broken (this is the other problem I alluded to in the description). I suspect that problem is related to this one though, since it also only affects my arch machines. I haven't yet submitted a report for that bug yet, though, since I can't reliably replicate it. +"""]] diff --git a/doc/bugs/add_fails_with_v6_repo_when_four_levels_deep/comment_3_d74835534f52c7f123b14e5d74194733._comment b/doc/bugs/add_fails_with_v6_repo_when_four_levels_deep/comment_3_d74835534f52c7f123b14e5d74194733._comment new file mode 100644 index 000000000..918bb4bac --- /dev/null +++ b/doc/bugs/add_fails_with_v6_repo_when_four_levels_deep/comment_3_d74835534f52c7f123b14e5d74194733._comment @@ -0,0 +1,7 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 3""" + date="2016-12-13T16:54:11Z" + content=""" +Perhaps it's caused by a particular/old version of git? +"""]] diff --git a/doc/bugs/add_fails_with_v6_repo_when_four_levels_deep/comment_4_f9d6dffb2617715c58216f54016de3a4._comment b/doc/bugs/add_fails_with_v6_repo_when_four_levels_deep/comment_4_f9d6dffb2617715c58216f54016de3a4._comment new file mode 100644 index 000000000..af0b2030b --- /dev/null +++ b/doc/bugs/add_fails_with_v6_repo_when_four_levels_deep/comment_4_f9d6dffb2617715c58216f54016de3a4._comment @@ -0,0 +1,13 @@ +[[!comment format=mdwn + username="t.z.mates" + avatar="http://cdn.libravatar.org/avatar/90f15fad216078fd08d62cc676487925" + subject="comment 4" + date="2016-12-20T23:08:44Z" + content=""" +Hmm, I don't think an old version of git is the cause. I'm currently running the most recent build of git (2.11.0), but have used a number of versions over the past year. + +I'm not sure if this is relevant, but this other bug reports similar behavior: [sync --content, fatal is outside repository errors](https://git-annex.branchable.com/forum/sync_--content__44___fatal_is_outside_repository_errors/). Specifically, it notes that there is an odd use of relative paths: +> The relative path ../Users is curious + +My error also appends an extra period. In particular, the path should be \"./1/2/3/4/foo\" but prints \"../1/2/3/4/foo\". +"""]] diff --git a/doc/bugs/adds_file_destined_for_annex_into_git_in___39__addurl__39__.mdwn b/doc/bugs/adds_file_destined_for_annex_into_git_in___39__addurl__39__.mdwn new file mode 100644 index 000000000..a5cdfd6d6 --- /dev/null +++ b/doc/bugs/adds_file_destined_for_annex_into_git_in___39__addurl__39__.mdwn @@ -0,0 +1,41 @@ +### Please describe the problem. + +When addurl'ing a big file with .gitattributes configured to add only some files directly into git (and 'git annex add' operating correctly), addurl adds large files straight into git. + +### What version of git-annex are you using? On what operating system? + +git-annex version: 6.20161018+gitgf3c366a-1~ndall+1 + + +### Please provide any additional information below. + +[[!format sh """ +$> cat .gitattributes +* annex.backend=MD5E +* annex.largefiles=(largerthan=100kb) +*.json annex.largefiles=nothing +*.txt annex.largefiles=nothing +*.tsv annex.largefiles=nothing +*.nii.gz annex.largefiles=(largerthan=0kb) +*.tgz annex.largefiles=(largerthan=0kb) +*.tar.gz annex.largefiles=(largerthan=0kb) +*.gz annex.largefiles=(largerthan=0kb) + +$> git annex addurl http://fcp-indi.s3.amazonaws.com/data/Projects/HBNSSI/RawDataTars/sub-0031121_baseline.tar.gz\?versionId\=7FvexHgyazWF.dUo238FA7XRiK0FWQDw. +addurl fcp_indi.s3.amazonaws.com_data_Projects_HBNSSI_RawDataTars_sub_0031121_baseline.tar.gz_versionId_7FvexHgyazWF.dUo238FA7XRiK0FWQDw. (downloading http://fcp-indi.s3.amazonaws.com/data/Projects/HBNSSI/RawDataTars/sub-0031121_baseline.tar.gz?versionId=7FvexHgyazWF.dUo238FA7XRiK0FWQDw. ...) +/mnt/btrfs/datasets/datalad/crawl-misc/indi/ 100%[==============================================================================================>] 195.44M 21.2MB/s in 12s +(non-large file; adding content to git repository) ok +(recording state in git...) +cached/staged changes: + \u2026r.gz_versionId_7FvexHgyazWF.dUo238FA7XRiK0FWQDw. | Bin 0 -> 204937338 bytes + +$> ls -l fcp_indi.s3.amazonaws.com_data_Projects_HBNSSI_RawDataTars_sub_0031121_baseline.tar.gz_versionId_7FvexHgyazWF.dUo238FA7XRiK0FWQDw. +-rw------- 1 yoh datalad 204937338 Oct 25 17:30 fcp_indi.s3.amazonaws.com_data_Projects_HBNSSI_RawDataTars_sub_0031121_baseline.tar.gz_versionId_7FvexHgyazWF.dUo238FA7XRiK0FWQDw. +cached/staged changes: + \u2026r.gz_versionId_7FvexHgyazWF.dUo238FA7XRiK0FWQDw. | Bin 0 -> 204937338 bytes + +"""]] + +[[!meta author=yoh]] + +> [[fixed|done]] --[[Joey]] diff --git a/doc/bugs/adds_file_destined_for_annex_into_git_in___39__addurl__39__/comment_1_d598317883753baf02175a3bf866e08a._comment b/doc/bugs/adds_file_destined_for_annex_into_git_in___39__addurl__39__/comment_1_d598317883753baf02175a3bf866e08a._comment new file mode 100644 index 000000000..e03e574f3 --- /dev/null +++ b/doc/bugs/adds_file_destined_for_annex_into_git_in___39__addurl__39__/comment_1_d598317883753baf02175a3bf866e08a._comment @@ -0,0 +1,20 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 1""" + date="2016-11-21T15:12:54Z" + content=""" +It's sufficient to have "* annex.largefiles=(largerthan=100kb)" +in .gitattributes. + +Even "* annex.largefiles=(largerthan=0kb)" will reproduce it. + +Ok, I see why.. It's running the largefile matcher on the destination file +before it renames the temp file to it! + +Seems to have been broken this way ever since addurl got largefiles +support. Testing didn't catch it because it only affects largefiles +expressions that need to examine the file. + +Fixed in git. Audited other checkFileMatcher calls for this problem; +the rest are ok. +"""]] diff --git a/doc/bugs/addurl_pathdepth_description_misleading.mdwn b/doc/bugs/addurl_pathdepth_description_misleading.mdwn index 531e43cbf..7bb9d582a 100644 --- a/doc/bugs/addurl_pathdepth_description_misleading.mdwn +++ b/doc/bugs/addurl_pathdepth_description_misleading.mdwn @@ -11,3 +11,7 @@ This isn't how it behaves. It would be more accurate as (emphasis on changes): > For example, adding the url http://www.example.com/dir/subdir/bigfile with --pathdepth=1 will use "**dir_subdir_bigfile**", while --pathdepth=3 will use "bigfile". For what I am doing (adding a directory tree with addurl and file:// URLs), I'd actually like the behaviour described (to recreate the tree), but I'm not sure which one was the *intended* behaviour.. + +> [[done]]; bug report didn't show what was wrong; I can see nothing wrong; +> bug reporter cannot seem to remember what was wrong. Probably user error. +> --[[Joey]] diff --git a/doc/bugs/addurl_pathdepth_description_misleading/comment_3_2744e42db662486b46e203a72c3e56c7._comment b/doc/bugs/addurl_pathdepth_description_misleading/comment_3_2744e42db662486b46e203a72c3e56c7._comment new file mode 100644 index 000000000..45be25186 --- /dev/null +++ b/doc/bugs/addurl_pathdepth_description_misleading/comment_3_2744e42db662486b46e203a72c3e56c7._comment @@ -0,0 +1,21 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 3""" + date="2016-11-16T19:04:37Z" + content=""" +Seems to work as described here: + + joey@darkstar:~/tmp/r>rm localhost__joey_blog_index.html + joey@darkstar:~/tmp/r>git annex addurl --pathdepth 2 http://localhost/~joey/blog/index.html + addurl blog/index.html (downloading http://localhost/~joey/blog/index.html ...) + /home/joey/tmp/r/ 100%[===========>] 40.70K --.-KB/s in 0s + ok + (recording state in git...) + joey@darkstar:~/tmp/r>ls + blog/ + joey@darkstar:~/tmp/r>ls blog + index.html + +It would probably help if you can provide a test case where it does not work +as described. +"""]] diff --git a/doc/bugs/addurl_pathdepth_description_misleading/comment_4_2a9eb14a8c6d06747bb5dda7ff179ec7._comment b/doc/bugs/addurl_pathdepth_description_misleading/comment_4_2a9eb14a8c6d06747bb5dda7ff179ec7._comment new file mode 100644 index 000000000..0cafc2a4d --- /dev/null +++ b/doc/bugs/addurl_pathdepth_description_misleading/comment_4_2a9eb14a8c6d06747bb5dda7ff179ec7._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="CandyAngel" + avatar="http://cdn.libravatar.org/avatar/15c0aade8bec5bf004f939dd73cf9ed8" + subject="comment 4" + date="2016-11-25T20:27:07Z" + content=""" +I really don't know what to say. I can't even figure out which computer I updated git-annex on to test if it was still happening.. let alone reproduce it anymore. It does work fine. + +I'm so sorry to bother you with this, I've done something stupid! This is exactly why you ask for a transcript of bugs occurring. (Feel free to use this as an example for why you ask for them, so some good can come of it at least..). +"""]] diff --git a/doc/bugs/android__58___cannot_link_executable/comment_2_1057c0477050e52e463c36e03fcab09d._comment b/doc/bugs/android__58___cannot_link_executable/comment_2_1057c0477050e52e463c36e03fcab09d._comment new file mode 100644 index 000000000..a8469cfe0 --- /dev/null +++ b/doc/bugs/android__58___cannot_link_executable/comment_2_1057c0477050e52e463c36e03fcab09d._comment @@ -0,0 +1,9 @@ +[[!comment format=mdwn + username="moc514@eb7af2cd9147722b29f32b6606feb2b8563dfac8" + nickname="moc514" + avatar="http://cdn.libravatar.org/avatar/c8c98fc66ef014e61c163375ca9e7422" + subject="Nexus 6p" + date="2016-12-16T02:08:21Z" + content=""" +I also have the same issue with the Nexus 6p with 7.1.1 +"""]] diff --git a/doc/bugs/annex_add_ignores_.-prefixed_directories.mdwn b/doc/bugs/annex_add_ignores_.-prefixed_directories.mdwn new file mode 100644 index 000000000..5db40f4dd --- /dev/null +++ b/doc/bugs/annex_add_ignores_.-prefixed_directories.mdwn @@ -0,0 +1,78 @@ +### Please describe the problem. + +annex add seems to ignore content under directories having . prefix. + +We thought to unify (across direct/indirect/v6) adding files to annex repository by using 'git annex add' with corresponding setting for largefiles for any addition, but it seems to ignore content under .-prefixed directories, unlike git + +### What version of git-annex are you using? On what operating system? + +6.20161122+gitg9f179ae-1~ndall+1 + +### Please provide any additional information below. + +[[!format sh """ +hopa:/tmp/datalad_temp_test_annex_add_no_dotfilesqMXck8 +$> git status +On branch master + +Initial commit + +nothing to commit (create/copy files and use "git add" to track) + +$> mkdir .dir dir; echo 123 > .dir/123; echo 124 > dir/124 + +$> git status +On branch master + +Initial commit + +Untracked files: + (use "git add <file>..." to include in what will be committed) + + .dir/ + dir/ + +nothing added to commit but untracked files present (use "git add" to track) + +$> git annex add -c 'annex.largefiles=nothing' . +add dir/124 (non-large file; adding content to git repository) ok +(recording state in git...) + +$> git status +On branch master + +Initial commit + +Changes to be committed: + (use "git rm --cached <file>..." to unstage) + + new file: dir/124 + +Untracked files: + (use "git add <file>..." to include in what will be committed) + + .dir/ + + +# and with regular git +$> git -c 'annex.largefiles=nothing' add . + +$> git status +On branch master + +Initial commit + +Changes to be committed: + (use "git rm --cached <file>..." to unstage) + + new file: .dir/123 + new file: dir/124 + + +"""]] + +Ref: https://github.com/datalad/datalad/issues/1027 + +[[!meta author=yoh]] + +[[done]]; oh -- it is RTFM: --include-dotfiles --[[yoh]] diff --git a/doc/bugs/cabal_constraints_for_aws_and_esqueleto/comment_1_d91e44573ef4a0ec6e7098cb4cd360f5._comment b/doc/bugs/cabal_constraints_for_aws_and_esqueleto/comment_1_d91e44573ef4a0ec6e7098cb4cd360f5._comment new file mode 100644 index 000000000..327b63469 --- /dev/null +++ b/doc/bugs/cabal_constraints_for_aws_and_esqueleto/comment_1_d91e44573ef4a0ec6e7098cb4cd360f5._comment @@ -0,0 +1,30 @@ +[[!comment format=mdwn + username="https://launchpad.net/~felixonmars" + nickname="felixonmars" + avatar="http://cdn.libravatar.org/avatar/17284a3bb2e4ad9d3be8fab31f49865be9c1dc22143c728de731fe800a335d38" + subject="comment 1" + date="2016-11-28T04:17:12Z" + content=""" +aws has merged a PR to support http-conduit 2.2, but git-annex itself doesn't build with the new component yet: + +``` +[ 95 of 544] Compiling Utility.Url ( Utility/Url.hs, dist/build/git-annex/git-annex-tmp/Utility/Url.o ) + +Utility/Url.hs:354:34: error: + * The constructor `StatusCodeException' should have 2 arguments, but has been given 3 + * In the pattern: StatusCodeException s _ _ + In an equation for `matchStatusCodeException': + matchStatusCodeException want e@(StatusCodeException s _ _) + | want s = Just e + | otherwise = Nothing + +Utility/Url.hs:354:34: error: + * Couldn't match expected type `HttpException' + with actual type `HttpExceptionContent' + * In the pattern: StatusCodeException s _ _ + In an equation for `matchStatusCodeException': + matchStatusCodeException want e@(StatusCodeException s _ _) + | want s = Just e + | otherwise = Nothing +``` +"""]] diff --git a/doc/bugs/cabal_constraints_for_aws_and_esqueleto/comment_2_32c45cc852a17e837f72dd8769a25781._comment b/doc/bugs/cabal_constraints_for_aws_and_esqueleto/comment_2_32c45cc852a17e837f72dd8769a25781._comment new file mode 100644 index 000000000..ada283d8b --- /dev/null +++ b/doc/bugs/cabal_constraints_for_aws_and_esqueleto/comment_2_32c45cc852a17e837f72dd8769a25781._comment @@ -0,0 +1,9 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 2""" + date="2016-12-13T15:58:25Z" + content=""" +This got fixed in the meantime. Note that posting comments to a bug that +has already been closed is a good way to get new problems not to be +noticed.. +"""]] diff --git a/doc/bugs/cannot_change_locale___40__en__95__US.UTF-8__41__/comment_5_eca31aeb974571c9cca7a399e00984a5._comment b/doc/bugs/cannot_change_locale___40__en__95__US.UTF-8__41__/comment_5_eca31aeb974571c9cca7a399e00984a5._comment new file mode 100644 index 000000000..9cf3695f5 --- /dev/null +++ b/doc/bugs/cannot_change_locale___40__en__95__US.UTF-8__41__/comment_5_eca31aeb974571c9cca7a399e00984a5._comment @@ -0,0 +1,13 @@ +[[!comment format=mdwn + username="http://svario.it/gioele" + nickname="Gioele" + avatar="http://cdn.libravatar.org/avatar/af2f2ba0dafe4650011d20f2168d43cff773aba97f55ae5b252bb873f391c1e2" + subject="compiled with GHC 8, but LOCPATH is still set" + date="2016-12-21T21:51:09Z" + content=""" +This bug does not want to die. + +The current standalone build (`6.20161211-gc3ab3c668`) has been compiled with GHC 8 but when I launch `runshell`, I still see that `LOCPATH` is set and the character encoding is messed up. + +I deduced the version of GHC used to compile git-annex with `strings ./shimmed/git-annex/git-annex | grep 'GHC [0-9]'`. +"""]] diff --git a/doc/bugs/cannot_change_locale___40__en__95__US.UTF-8__41__/comment_6_0cada5a6154438c674f01d449378ffe9._comment b/doc/bugs/cannot_change_locale___40__en__95__US.UTF-8__41__/comment_6_0cada5a6154438c674f01d449378ffe9._comment new file mode 100644 index 000000000..1942a6f52 --- /dev/null +++ b/doc/bugs/cannot_change_locale___40__en__95__US.UTF-8__41__/comment_6_0cada5a6154438c674f01d449378ffe9._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 6""" + date="2016-12-24T16:57:45Z" + content=""" +This is an old closed bug report. The recent comments are about a +completely unrelated issue, which I suspect was fixed by +[[!commit 95c8b37544c383983712c5c368dd803c51bf8eeb]]. + +Please file new bug reports if you have an issue, if the old bug report was +closed years ago. +"""]] diff --git a/doc/bugs/getFileSize_conflict_between_Utility.Directory_and_Utility.FileSize.mdwn b/doc/bugs/getFileSize_conflict_between_Utility.Directory_and_Utility.FileSize.mdwn new file mode 100644 index 000000000..75a546159 --- /dev/null +++ b/doc/bugs/getFileSize_conflict_between_Utility.Directory_and_Utility.FileSize.mdwn @@ -0,0 +1,34 @@ +### Please describe the problem. +directory 1.3.0.0 causes a conflict for "getFileSize" + +### What steps will reproduce the problem? +Build git-annex with directory 1.3.0.0 (first need to bump max directory version on concurrent-output (and aws if building with s3)) + +### What version of git-annex are you using? On what operating system? +6.20161210 on macOS 10.11 El Capitan + + +### Please provide any additional information below. + +[[!format sh """ +[23 of 34] Compiling Common ( Common.hs, dist/setup/Common.o ) + +Common.hs:3:16: error: + Conflicting exports for ‘getFileSize’: + ‘module X’ exports ‘X.getFileSize’ + imported from ‘Utility.Directory’ at Common.hs:28:1-29 + (and originally defined in ‘System.Directory’) + ‘module X’ exports ‘X.getFileSize’ + imported from ‘Utility.FileSize’ at Common.hs:34:1-28 + (and originally defined at Utility/FileSize.hs:26:1-11) +"""]] + +A fix, though possibly not best, is to make this change in Common.hs: +[[!format sh """ +import Utility.Directory as X hiding (getFileSize) +"""]] + +### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) +Yes :) + +> [[fixed|done]]; thanks for reporting! diff --git a/doc/bugs/git-annex-fsck___34__-all__34___flag_doesn__39__t_work_for_special_remote.mdwn b/doc/bugs/git-annex-fsck___34__-all__34___flag_doesn__39__t_work_for_special_remote.mdwn new file mode 100644 index 000000000..80193cd54 --- /dev/null +++ b/doc/bugs/git-annex-fsck___34__-all__34___flag_doesn__39__t_work_for_special_remote.mdwn @@ -0,0 +1,28 @@ +### Please describe the problem. +I tried to use `git-annex-fsck --all --from remote` to check files on a special remote, but git-annex did a scan of the local repo instead. If I don't use the `--all` flag, it correctly checks the files on the remote (but just the files in the current checked out branch). + +### What steps will reproduce the problem? + mkdir repo + mkdir special + cd repo + git init + git annex init + git annex initremote special type=directory directory=../special encryption=none + touch testfile + git annex add testfile + git annex copy testfile --to special + chmod -R +w ../special/* + rm -r ../special/* + git annex fsck --all --from special # should check special remote but checks local repo instead + git diff git-annex^ git-annex # activity log shows that it checked special remote + git annex fsck --from special # correctly checks special remote, identifies missing file + + +### What version of git-annex are you using? On what operating system? +6.20161012 on Ubuntu 16.10 + +### Have you had any luck using git-annex before? +Yes, it's been very helpful for managing large files between laptops, desktops, external storage, and remote storage. + +> Thanks for an excellent test case and a clear bug report. I've fixed this +> bug. [[done]] --[[Joey]] diff --git a/doc/bugs/git-annex_fromkey_barfs_on_utf-8_input.mdwn b/doc/bugs/git-annex_fromkey_barfs_on_utf-8_input.mdwn new file mode 100644 index 000000000..e544015a6 --- /dev/null +++ b/doc/bugs/git-annex_fromkey_barfs_on_utf-8_input.mdwn @@ -0,0 +1,38 @@ +### Please describe the problem. + +I'm sending a stream of keys and filenames to git-annex fromkey on stdin, and it errors out with "git-annex: <stdin>: hGetContents: invalid argument (invalid byte sequence)". On the other hand yipdw tried to reproduce this and it worked fine for him, so I must be doing something wrong. + +I have LANG=en_US.UTF-8 set in my environment, if that matters. + +### What steps will reproduce the problem? + +[[!format sh """ +echo "MD5-s3263532--0b4d070eff7baa8ef314ca330aecb71f é" | git-annex fromkey +"""]] + +### What version of git-annex are you using? On what operating system? + +[[!format sh """ +git-annex version: 6.20161118-g0a34f08 +build flags: Assistant Webapp Pairing Testsuite S3(multipartupload)(storageclasses) WebDAV Inotify DBus DesktopNotify XMPP ConcurrentOutput TorrentParser MagicMime Feeds Quvi +key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 SHA1E SHA1 MD5E MD5 WORM URL +remote types: git gcrypt S3 bup directory rsync web bittorrent webdav tahoe glacier ddar hook external +local repository version: 5 +supported repository versions: 3 5 6 +upgrade supported from repository versions: 0 1 2 3 4 5 +operating system: linux x86_64 +"""]] + +### Please provide any additional information below. + +Note that this is indeed valid utf-8: + +[[!format sh """ + db48x ~ projects IA.BAK-server echo "é" | hexdump -C +00000000 c3 a9 0a |...| +00000003 +"""]] + +> Despite my strange inability to reproduce these, there's really only one +> thing that can fix it, namely using fileEncoding. Now done for all batch +> and stdin reading stuff. [[fixed|done]] I suppose. --[[Joey]] diff --git a/doc/bugs/git-annex_fromkey_barfs_on_utf-8_input/comment_1_aa6fe46ee76dd8bfa9a56cbd5131cb8b._comment b/doc/bugs/git-annex_fromkey_barfs_on_utf-8_input/comment_1_aa6fe46ee76dd8bfa9a56cbd5131cb8b._comment new file mode 100644 index 000000000..a0409e281 --- /dev/null +++ b/doc/bugs/git-annex_fromkey_barfs_on_utf-8_input/comment_1_aa6fe46ee76dd8bfa9a56cbd5131cb8b._comment @@ -0,0 +1,55 @@ +[[!comment format=mdwn + username="alpernebbi" + avatar="http://cdn.libravatar.org/avatar/daf2abb14f39e28ad75d5f9a03fcd106" + subject="UTF-8 problems in some other commands" + date="2016-12-05T20:46:07Z" + content=""" +Running the command above gives me the same error on Xubuntu 16.04, using `git-annex-standalone` package from NeuroDebian repositories. + + git-annex version: 6.20161122+gitg9f179ae-1~ndall+1 + build flags: Assistant Webapp Pairing Testsuite S3(multipartupload)(storageclasses) WebDAV Inotify DBus DesktopNotify XMPP ConcurrentOutput TorrentParser MagicMime Feeds Quvi + key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 SHA1E SHA1 MD5E MD5 WORM URL + remote types: git gcrypt S3 bup directory rsync web bittorrent webdav tahoe glacier ddar hook external + local repository version: 5 + supported repository versions: 3 5 6 + upgrade supported from repository versions: 0 1 2 3 4 5 + operating system: linux x86_64 + +I encountered other commands that fail as well: + + $ touch u.txt ü.txt + $ git annex add + + $ git-annex calckey ü.txt + # prints key + + $ git-annex calckey --batch + ü.txt + # dies + + $ git-annex lookupkey ü.txt + # prints key + + $ git-annex lookupkey --batch + ü.txt + # dies + + $ git-annex metadata --batch --json + {\"file\":\"ü.txt\"} + # dies + + $ git-annex metadata --batch --json + {\"file\":\"u.txt\",\"fields\":{\"ü\":[\"b\"]}} + # dies + + $ git-annex metadata --batch --json + {\"file\":\"u.txt\",\"fields\":{\"a\":[\"ü\"]}} + # dies + +All those die without output, all $? are 0. +No values were recorded to metadata. +Also: + + $ git-annex-metadata --json + # entry for \"ü.txt\" has \"file\":\"��.txt\" +"""]] diff --git a/doc/bugs/git-annex_fromkey_barfs_on_utf-8_input/comment_2_37643180ecbc6c6bb0504b3acb18d1e7._comment b/doc/bugs/git-annex_fromkey_barfs_on_utf-8_input/comment_2_37643180ecbc6c6bb0504b3acb18d1e7._comment new file mode 100644 index 000000000..0b2cabdf9 --- /dev/null +++ b/doc/bugs/git-annex_fromkey_barfs_on_utf-8_input/comment_2_37643180ecbc6c6bb0504b3acb18d1e7._comment @@ -0,0 +1,31 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 2""" + date="2016-12-08T21:03:14Z" + content=""" +I'm not able to reproduce the original reported problem; it works even when +using the C locale. However, it may be that this patch would fix it: + +<pre> +diff --git a/Command/FromKey.hs b/Command/FromKey.hs +index dca63aa..6a81c1f 100644 +--- a/Command/FromKey.hs ++++ b/Command/FromKey.hs +@@ -45,7 +45,9 @@ startMass = do + next massAdd + + massAdd :: CommandPerform +-massAdd = go True =<< map (separate (== ' ')) . lines <$> liftIO getContents ++massAdd = do ++ liftIO $ fileEncoding stdin ++ go True =<< map (separate (== ' ')) . lines <$> liftIO getContents + where + go status [] = next $ return status + go status ((keyname,f):rest) | not (null keyname) && not (null f) = do +</pre> + +(The NeuroDebian git-annex-standalone may well have had its locale +installation broken by [[!commit c07981122672f6cc87ca08efb57d8a7b1e2f5725]], +which assumes that the git-annex.linux is writable by the user. +I doubt that is related to the original bug report.) +"""]] diff --git a/doc/bugs/git-annex_fromkey_barfs_on_utf-8_input/comment_3_7342e29b0d2225abc5800638e3b377ed._comment b/doc/bugs/git-annex_fromkey_barfs_on_utf-8_input/comment_3_7342e29b0d2225abc5800638e3b377ed._comment new file mode 100644 index 000000000..5011aa1fa --- /dev/null +++ b/doc/bugs/git-annex_fromkey_barfs_on_utf-8_input/comment_3_7342e29b0d2225abc5800638e3b377ed._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="alpernebbi" + avatar="http://cdn.libravatar.org/avatar/daf2abb14f39e28ad75d5f9a03fcd106" + subject="comment 3" + date="2016-12-10T07:36:04Z" + content=""" +I experienced all these with the [linux standalone](https://git-annex.branchable.com/install/Linux_standalone/) from this site as well. + +However, I couldn't reproduce them in a Debian unstable chroot where I installed the `git-annex` package from their repos. +"""]] diff --git a/doc/bugs/git-annex_won__39__t_execute_on_WD_My_Cloud_NAS/comment_8_48026cf7c187e97d53d15d35ed2c3670._comment b/doc/bugs/git-annex_won__39__t_execute_on_WD_My_Cloud_NAS/comment_8_48026cf7c187e97d53d15d35ed2c3670._comment new file mode 100644 index 000000000..493031115 --- /dev/null +++ b/doc/bugs/git-annex_won__39__t_execute_on_WD_My_Cloud_NAS/comment_8_48026cf7c187e97d53d15d35ed2c3670._comment @@ -0,0 +1,14 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 8""" + date="2016-11-16T21:48:49Z" + content=""" +The arm daily build now uses a 32kb page size. So try +<https://downloads.kitenet.net/git-annex/autobuild/armel/git-annex-standalone-armel.tar.gz> + +That has been verified to fix the problem on a Drobo 5N. + +This may still not be enough for some of the affected NAS devices, which +use a 64kb page size. Unfortunately, gold fails to link with a 64kb page +size: <http://bugs.debian.org/844467> +"""]] diff --git a/doc/bugs/git_annex_init_failed_due_to_unsupported_ssh_option.mdwn b/doc/bugs/git_annex_init_failed_due_to_unsupported_ssh_option.mdwn index eb961030c..f4339bff7 100644 --- a/doc/bugs/git_annex_init_failed_due_to_unsupported_ssh_option.mdwn +++ b/doc/bugs/git_annex_init_failed_due_to_unsupported_ssh_option.mdwn @@ -65,3 +65,5 @@ I would just like to be sure nothing else is hidden. > .git/config to remove that in order to recover from the problem, so might > as well remove `.git/annex/ssh_config` too. > --[[Joey]] + +>> Fixed more by stopping using `.git/annex/ssh_config` at all. --[[Joey]] diff --git a/doc/bugs/git_annex_init_failed_due_to_unsupported_ssh_option/comment_2_32e142afd9fe65843d53883ba2ae48cb._comment b/doc/bugs/git_annex_init_failed_due_to_unsupported_ssh_option/comment_2_32e142afd9fe65843d53883ba2ae48cb._comment new file mode 100644 index 000000000..42550d51f --- /dev/null +++ b/doc/bugs/git_annex_init_failed_due_to_unsupported_ssh_option/comment_2_32e142afd9fe65843d53883ba2ae48cb._comment @@ -0,0 +1,17 @@ +[[!comment format=mdwn + username="scottgorlin@a32946b2aad278883c1690a0753241583a9855b9" + nickname="scottgorlin" + avatar="http://cdn.libravatar.org/avatar/2dd1fc8add62bbf4ffefac081b322563" + subject="IgnoreUnknown Include considered harmful?" + date="2016-11-23T20:07:45Z" + content=""" +As noted, include appears to not work on a mac at the moment. This means git-annex silently ignores the included configs, which may be required to ssh to the remotes of interest. This is happening to me. + +My understanding is that ssh aliases are the recommended way of juggling multiple private keys amongst multiple hosts, so it is a required part of many git workflows. In this particular case, I have set up git annex on a NAS which does not allow multiple ssh users (QNAP) and the authentication is done only via key identity, not username. Thus, host aliases are necessary. + +If one config can't include another, I would prefer an early failure indicating a problem with the config file, or better, a solution where git-annex doesn't require a config. In this scenario, git fetch remote_name and git annex copy --to remotename do not resolve to the same alias definitions (the latter is missing because of the ignored config!). + +I got my setup to work only by finding and manually editing <repo>/.git/annex/ssh_config, which to my knowledge is undocumented (ie when is it written? do any commands change it?); manual mucking around inside .git to me is not a good practice, and for now I have two different alias's defined (in repo and in ~/.ssh/config) + + +"""]] diff --git a/doc/bugs/regression_due_to_usage_of_ssh_7.3___34__include__34___feature/comment_1_45003ab569c4649ca29c07877a83af29._comment b/doc/bugs/regression_due_to_usage_of_ssh_7.3___34__include__34___feature/comment_1_45003ab569c4649ca29c07877a83af29._comment new file mode 100644 index 000000000..566926a32 --- /dev/null +++ b/doc/bugs/regression_due_to_usage_of_ssh_7.3___34__include__34___feature/comment_1_45003ab569c4649ca29c07877a83af29._comment @@ -0,0 +1,15 @@ +[[!comment format=mdwn + username="palday@91f366b5178879146d2b6e1e53bfa21389ee89a8" + nickname="palday" + avatar="http://cdn.libravatar.org/avatar/077a63af75ddba159980fbf88690f401" + subject="Temporary workaround until the brew formula is updated" + date="2016-11-29T02:17:52Z" + content=""" +The homebrew formula doesn't yet this fix, but you can get around the problem in the meantime by getting a newer SSH via homebrew: + +``` +brew install homebrew/dupes/openssh +``` + +You can then choose to keep that or get rid of it when the formula for git annex is later updated. +"""]] diff --git a/doc/bugs/ssh___34__Include__34___breaks_user-specified_IgnoreUnknown.mdwn b/doc/bugs/ssh___34__Include__34___breaks_user-specified_IgnoreUnknown.mdwn new file mode 100644 index 000000000..20878cb21 --- /dev/null +++ b/doc/bugs/ssh___34__Include__34___breaks_user-specified_IgnoreUnknown.mdwn @@ -0,0 +1,30 @@ +### Please describe the problem. + +The OpenSSH client parses configuration in a "first match wins" manner, and this also applies to `IgnoreUnknown`. This means that when git-annex's `Include ~/.ssh/config` is processed, any user-specified `IgnoreUnknown` setting in the global configuration will be ignored because it has already been set. As a result, every time git-annex runs ssh, it immediately exits with an error: + +[[!format text """ +drop vol3 somefile.mkv (locking vol5...) (lockcontent failed) (checking vol5...) +/home/grawity/.ssh/config: line 217: Bad configuration option: gssapikeyexchange +/home/grawity/.ssh/config: terminating, 1 bad configuration options +failed +"""]] + +To be fair, this might be an OpenSSH bug (IgnoreUnknown ought to be merged), but it seems git-annex is triggering it unnecessarily. + +### What steps will reproduce the problem? + +1. In `~/.ssh/config`, have some unrecognized options (e.g. `GSSAPIKeyExchange`) and a corresponding `IgnoreUnknown`. + +2. Try to use a git-annex feature which directly invokes ssh, e.g. get or drop. + +### What version of git-annex are you using? On what operating system? + +6.20161210 on Arch, but I think this was introduced in a 201611* release. + +### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) + +Yes, it's been taking care of my archives for nearly a year. + +> How annoying, ssh seems to make it impossible to set this in a way that +> doesn't break some configurations. Oh well, gave up on setting it +> and removed the code to do so. [[done]] --[[Joey]] diff --git a/doc/bugs/ssh___34__Include__34___feature_broke_Android.mdwn b/doc/bugs/ssh___34__Include__34___feature_broke_Android.mdwn new file mode 100644 index 000000000..ca5b86c87 --- /dev/null +++ b/doc/bugs/ssh___34__Include__34___feature_broke_Android.mdwn @@ -0,0 +1,10 @@ +### Please describe the problem. +https://git-annex.branchable.com/bugs/git_annex_init_failed_due_to_unsupported_ssh_option/ deal with Include not being supported by pre 7.3 by using the 6.4+ IgnoreUnknown directive. + +Unfortunately, the Android apk (which I got from https://downloads.kitenet.net/git-annex/android/current/5.0/git-annex.apk) has (according to ssh -v) OpenSSH_6.0p1. + +I had to edit .git/annex/ssh.config to comment out the three offending lines and then append the contents of ~/.ssh/config to get git-annex working again. + +(This is on a Nexus 10 running stock, though I doubt it matters) + +> Reverted use of this feature for now.[[done]] --[[Joey]] diff --git a/doc/bugs/ssh___34__Include__34___feature_broke_Android/comment_1_14818629616e3daeb8293b710298ce31._comment b/doc/bugs/ssh___34__Include__34___feature_broke_Android/comment_1_14818629616e3daeb8293b710298ce31._comment new file mode 100644 index 000000000..0cf33b8b3 --- /dev/null +++ b/doc/bugs/ssh___34__Include__34___feature_broke_Android/comment_1_14818629616e3daeb8293b710298ce31._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 1""" + date="2016-12-08T21:11:31Z" + content=""" +Indeed, the ssh bundled in the apk is shockingly old by now, and needs to +get updated. +"""]] diff --git a/doc/bugs/unRAID_shares_treated_as_a_crippled_filesystem.mdwn b/doc/bugs/unRAID_shares_treated_as_a_crippled_filesystem.mdwn new file mode 100644 index 000000000..dd491e5fd --- /dev/null +++ b/doc/bugs/unRAID_shares_treated_as_a_crippled_filesystem.mdwn @@ -0,0 +1,46 @@ +### Please describe the problem. + +Running `git annex init` on an [unRAID server](https://lime-technology.com/what-is-unraid/) results in an annex created with `crippledfilesystem = true` and `direct = true`. I understand from reading [this](https://git-annex.branchable.com/design/assistant/blog/day_188__crippled_filesystem_support/) that it occurs when `git annex init` performs a probe to determine if all of the following are supported: + +1. symlinks +2. hard links +3. unix permissions + +Although unRAID disks are formatted with xfs, and therefore support all three of the above, I'm assuming that unRAID's method of combining multiple disks into one "share" is the cause of the problem (hardlinks still work on a single disk, but not on shares that span multiple disks). Symlinks and unix permissions work normally in the unRAID-created shares. + +Is there any way to allow the use of 'indirect' mode with multi-disk shares? As I mentioned, symlinks and unix permissions work normally--it's only the hardlinks that won't work across the multi-disk shares. + +I can create a 'normal' annex as long as I `cd` to a single disk drive first--what would happen if the annex was later moved onto a multi-disk share? Would it still work? Would it fail gracefully? Would it cause data loss? + +### What steps will reproduce the problem? + + cd /mnt/user/NameOfShare + git init + git annex init + +The following will result in the creation of a 'normal' indirect share: + + cd /mnt/disk1 + git init + git annex init + +### What version of git-annex are you using? On what operating system? + + git-annex version: 6.20161211-gc3ab3c668 + build flags: Assistant Webapp Pairing Testsuite S3(multipartupload)(storageclasses) WebDAV Inotify DBus DesktopNotify XMPP ConcurrentOutput TorrentParser MagicMime Feeds Quvi + key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 SHA1E SHA1 MD5E MD5 WORM URL + remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav tahoe glacier ddar hook external + +### Please provide any additional information below. + +[[!format sh """ +# If you can, paste a complete transcript of the problem occurring here. +# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log + + +# End of transcript or log. +"""]] + +### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) + +Has been working great, so far, except for the above. diff --git a/doc/bugs/unable_to_decommit_memory__58___Invalid_argument.mdwn b/doc/bugs/unable_to_decommit_memory__58___Invalid_argument.mdwn new file mode 100644 index 000000000..6cd90264c --- /dev/null +++ b/doc/bugs/unable_to_decommit_memory__58___Invalid_argument.mdwn @@ -0,0 +1,31 @@ +### Please describe the problem. +Recent builds of git-annex spew out many lines such as: + + git-annex: unable to decommit memory: Invalid argument + git-annex: unable to decommit memory: Invalid argument + git-annex: unable to decommit memory: Invalid argument + git-annex: unable to decommit memory: Invalid argument + git-annex: unable to decommit memory: Invalid argument + +### What steps will reproduce the problem? +This happens to me syncing any large repository now. + +### What version of git-annex are you using? On what operating system? +git-annex version: 6.20161118-g0a34f08 + +uname -r: 4.4.14-11.pvops.qubes.x86_64 + +/etc/system-release: Fedora release 23 (Twenty Three) + +### Please provide any additional information below. + +I found this: https://ghc.haskell.org/trac/ghc/ticket/12495 + +It looks like this is a problem that occurs only on kernels < 4.5, when ghc is built with a newer glibc, I think. + +### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) + +Git annex rocks! + +> [[fixed|done]]; fix is confirmed, all linux standalone builds are updated +> (and I pinged Yoh to update the neurodebian standalone build). --[[Joey]] diff --git a/doc/bugs/unable_to_decommit_memory__58___Invalid_argument/comment_10_b21a337256c58953e1440317c0c1db80._comment b/doc/bugs/unable_to_decommit_memory__58___Invalid_argument/comment_10_b21a337256c58953e1440317c0c1db80._comment new file mode 100644 index 000000000..15dde50f8 --- /dev/null +++ b/doc/bugs/unable_to_decommit_memory__58___Invalid_argument/comment_10_b21a337256c58953e1440317c0c1db80._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 10""" + date="2016-12-19T19:53:11Z" + content=""" +The only way the files could be in lost+found is if the system crashed or +there was a disk error etc. Can't be due to this bug. So, it may be that +this bug did not actually cause any data loss. The screwed up symlinks could +have been caused by a disk error. +"""]] diff --git a/doc/bugs/unable_to_decommit_memory__58___Invalid_argument/comment_1_b76704adf6b6aa441a35bf9458d3950d._comment b/doc/bugs/unable_to_decommit_memory__58___Invalid_argument/comment_1_b76704adf6b6aa441a35bf9458d3950d._comment new file mode 100644 index 000000000..3fe7af8dd --- /dev/null +++ b/doc/bugs/unable_to_decommit_memory__58___Invalid_argument/comment_1_b76704adf6b6aa441a35bf9458d3950d._comment @@ -0,0 +1,15 @@ +[[!comment format=mdwn + username="0xloem@0bd8a79a57e4f0dcade8fc81d162c37eae4d6730" + nickname="0xloem" + avatar="http://cdn.libravatar.org/avatar/b8c087f7c5e6a9358748f0727c077f3b" + subject="Corrupt Links Produced, Significant Data Loss" + date="2016-12-10T12:31:31Z" + content=""" +Additionally, I added a mess of files with this release of git-annex, and of the 200 files added, three of the links produced were corrupt. I'm still searching for where these files have gone to recover the data. + +The files look like this in `ls -l`, they were bup files: + + lrwxrwxrwx 1 user user 338 Jun 17 22:36 bup.git/objects/pack/pack-47b493a3bbbd22200d2b390c277e49ce713243cc.pack -> *??:?;J????????? + lrwxrwxrwx 1 user user 336 Jun 17 21:41 bup.git/objects/pack/pack-4d202b3929b187d4acaf1602526e7344eef1edc8.pack -> ?p???GWj??????ܥ??{b?#???>C??%??????~à???/hjT;?p??d?8??oyE?K?)6?uL+??h??&???SB}?'s??֫{?>^i?&?f??^{ш??aD??t4?C?sBTk>d6H???5h3?ڋ6fAa??=?r????j?????a8K??????????B?~????I͕?T7?Y??=???b?7C???鋤??8???\"?????#???M?????}z?A??9?C>?-?GD??7?ј;'P?H??ɑ??Zr?/U???W?G??3@\"??Ȧ?z?h???U??Ԇ???R??u??I????62??>@??@?a??x???}?????)d?G;(???m_?^3?????T + lrwxrwxrwx 1 user user 332 Jul 20 07:32 bup.git/objects/pack/pack-5328381f3b023c1356581c22d1e74d4eda0b46a3.idx -> c??'w??????????m?q#?ٱCO??o????ʃ?Ʃڌ??[???Ѐ??*?;.?c?N?0?????D$ o?r????8BGn?96gY?B?Z1?=???{??z?71????!aG?>?u)???i\?G[???:?Kk??%??.mu???n???K??ǚ????q&Z-?E???]??/?6???} +"""]] diff --git a/doc/bugs/unable_to_decommit_memory__58___Invalid_argument/comment_2_3469bfd3ba5e7935f3350f0bd78a0c94._comment b/doc/bugs/unable_to_decommit_memory__58___Invalid_argument/comment_2_3469bfd3ba5e7935f3350f0bd78a0c94._comment new file mode 100644 index 000000000..52932cd5e --- /dev/null +++ b/doc/bugs/unable_to_decommit_memory__58___Invalid_argument/comment_2_3469bfd3ba5e7935f3350f0bd78a0c94._comment @@ -0,0 +1,24 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 2""" + date="2016-12-10T15:13:44Z" + content=""" +I think that the "x86-32, for ancient kernels" build should avoid this +problem. <http://git-annex.branchable.com/install/Linux_standalone/> + +It's very surprising if this lead to symlinks being created that apparently +contain garbage in their link targets. Perhaps glibc is failing in a way +with the old kernel that leads to memory corruption? I have asked the GHC +developers if that could be the case in +<https://ghc.haskell.org/trac/ghc/ticket/12865> + +I hope that the content of your files is in fact somewhere under +`.git/annex/objects/` -- look around, and with some luck, you may find it. +Unfortunately, the information about, which object file goes with which +working tree has apparently been lost. (Also, you might check if these +symlinks have been staged in git; it's possible though unlikely that the +correct link target got staged in git.) + +I have filed a bug on Debian's ghc to get them to fast-track getting the +patch into ghc. <https://bugs.debian.org/847677> +"""]] diff --git a/doc/bugs/unable_to_decommit_memory__58___Invalid_argument/comment_3_886756620cdbb6ab838269fe2f00db4e._comment b/doc/bugs/unable_to_decommit_memory__58___Invalid_argument/comment_3_886756620cdbb6ab838269fe2f00db4e._comment new file mode 100644 index 000000000..b5316c0de --- /dev/null +++ b/doc/bugs/unable_to_decommit_memory__58___Invalid_argument/comment_3_886756620cdbb6ab838269fe2f00db4e._comment @@ -0,0 +1,9 @@ +[[!comment format=mdwn + username="0xloem@0bd8a79a57e4f0dcade8fc81d162c37eae4d6730" + nickname="0xloem" + avatar="http://cdn.libravatar.org/avatar/b8c087f7c5e6a9358748f0727c077f3b" + subject="comment 3" + date="2016-12-11T00:26:41Z" + content=""" +Thank you so much for the prompt response. My system wouldn't shut down cleanly after this, either, so there may have been something else screwy going on. Still, I'll be using the build for ancient kernels exclusively for the near future. Thank you. +"""]] diff --git a/doc/bugs/unable_to_decommit_memory__58___Invalid_argument/comment_3_a4499b5506c0624f01d436e14ccce909._comment b/doc/bugs/unable_to_decommit_memory__58___Invalid_argument/comment_3_a4499b5506c0624f01d436e14ccce909._comment new file mode 100644 index 000000000..7aea3d6af --- /dev/null +++ b/doc/bugs/unable_to_decommit_memory__58___Invalid_argument/comment_3_a4499b5506c0624f01d436e14ccce909._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 3""" + date="2016-12-11T19:35:21Z" + content=""" +All Linux standalone builds have been updated with a version of ghc that +has that bug fixed. Can you please upgrade and verify it's fixed? +"""]] diff --git a/doc/bugs/unable_to_decommit_memory__58___Invalid_argument/comment_4_2440227de9b6bc77ae1c73b69a36f7a5._comment b/doc/bugs/unable_to_decommit_memory__58___Invalid_argument/comment_4_2440227de9b6bc77ae1c73b69a36f7a5._comment new file mode 100644 index 000000000..801f70223 --- /dev/null +++ b/doc/bugs/unable_to_decommit_memory__58___Invalid_argument/comment_4_2440227de9b6bc77ae1c73b69a36f7a5._comment @@ -0,0 +1,9 @@ +[[!comment format=mdwn + username="0xloem@0bd8a79a57e4f0dcade8fc81d162c37eae4d6730" + nickname="0xloem" + avatar="http://cdn.libravatar.org/avatar/b8c087f7c5e6a9358748f0727c077f3b" + subject="Verification" + date="2016-12-11T00:59:50Z" + content=""" +I saw the new comment on the download page and tried running `git annex test`. I can confirm that `git annex test` eventually segfaults using the normal build on my system, whereas it passes successfully using the 'ancient kernels' build. The version strings output for the two are identical. +"""]] diff --git a/doc/bugs/unable_to_decommit_memory__58___Invalid_argument/comment_5_c5e3dc25acf0cfb98d7068fe7f83e63a._comment b/doc/bugs/unable_to_decommit_memory__58___Invalid_argument/comment_5_c5e3dc25acf0cfb98d7068fe7f83e63a._comment new file mode 100644 index 000000000..9dec23b00 --- /dev/null +++ b/doc/bugs/unable_to_decommit_memory__58___Invalid_argument/comment_5_c5e3dc25acf0cfb98d7068fe7f83e63a._comment @@ -0,0 +1,9 @@ +[[!comment format=mdwn + username="0xloem@0bd8a79a57e4f0dcade8fc81d162c37eae4d6730" + nickname="0xloem" + avatar="http://cdn.libravatar.org/avatar/b8c087f7c5e6a9358748f0727c077f3b" + subject="Nope a Fluke" + date="2016-12-11T13:26:29Z" + content=""" +Apologies. I can't reproduce the segfault running the tests again. The corruption and crashing seems to have been some horrifying fluke. +"""]] diff --git a/doc/bugs/unable_to_decommit_memory__58___Invalid_argument/comment_7_e31ee8f49bf5f73620209c524f1edb3d._comment b/doc/bugs/unable_to_decommit_memory__58___Invalid_argument/comment_7_e31ee8f49bf5f73620209c524f1edb3d._comment new file mode 100644 index 000000000..08e3364ca --- /dev/null +++ b/doc/bugs/unable_to_decommit_memory__58___Invalid_argument/comment_7_e31ee8f49bf5f73620209c524f1edb3d._comment @@ -0,0 +1,11 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 7""" + date="2016-12-12T17:30:03Z" + content=""" +Can you please check if the current builds still have the "unable to +decommit memory" problem or not? + +(What it does after that error is probably nondeterministic, fixing that +error is the crucial thing.) +"""]] diff --git a/doc/bugs/unable_to_decommit_memory__58___Invalid_argument/comment_8_038a8e39ec0e91cb04af738eaf9095e1._comment b/doc/bugs/unable_to_decommit_memory__58___Invalid_argument/comment_8_038a8e39ec0e91cb04af738eaf9095e1._comment new file mode 100644 index 000000000..6d8994155 --- /dev/null +++ b/doc/bugs/unable_to_decommit_memory__58___Invalid_argument/comment_8_038a8e39ec0e91cb04af738eaf9095e1._comment @@ -0,0 +1,9 @@ +[[!comment format=mdwn + username="0xloem@0bd8a79a57e4f0dcade8fc81d162c37eae4d6730" + nickname="0xloem" + avatar="http://cdn.libravatar.org/avatar/b8c087f7c5e6a9358748f0727c077f3b" + subject="comment 8" + date="2016-12-13T15:52:34Z" + content=""" +It looks like the errors are gone. Thank you so much. +"""]] diff --git a/doc/bugs/unable_to_decommit_memory__58___Invalid_argument/comment_9_6c3f4d165bca7a27683df286363bc19b._comment b/doc/bugs/unable_to_decommit_memory__58___Invalid_argument/comment_9_6c3f4d165bca7a27683df286363bc19b._comment new file mode 100644 index 000000000..42a14f12a --- /dev/null +++ b/doc/bugs/unable_to_decommit_memory__58___Invalid_argument/comment_9_6c3f4d165bca7a27683df286363bc19b._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="xloem" + avatar="http://cdn.libravatar.org/avatar/b8c087f7c5e6a9358748f0727c077f3b" + subject="Coda" + date="2016-12-18T14:56:17Z" + content=""" +The missing files turned up in 'lost+found' the next time I ran fsck =) +"""]] diff --git a/doc/bugs/wget_always_shows_progress_bar.mdwn b/doc/bugs/wget_always_shows_progress_bar.mdwn new file mode 100644 index 000000000..708db8906 --- /dev/null +++ b/doc/bugs/wget_always_shows_progress_bar.mdwn @@ -0,0 +1,25 @@ +### Please describe the problem. + +git annex addurl or importfeed operations were quiet on git-annex versions older than 5.20141219, if +annex.web-options was set to "--quiet". But now the --show-progress option is always passed to wget. + +In some use cases this might be nice, but I'm using GNU Parallel to update multiple podcast feeds +concurrently, and it causes wget to output the ugly "dot" indicator for every feed, which is totally +useless since parallel buffers and groups the output. Adding "--no-show-progress" to web-options +does not help (it does not override --show-progress), nor does redirecting stdout to /dev/null. +Redirecting stderr would hide possible errors. + +### What steps will reproduce the problem? + +parallel git annex importfeed --relaxed --quiet ::: http://feeds.feedburner.com/freakonomicsradio + +### What version of git-annex are you using? On what operating system? + +5.20151208-1~bpo8+1 on Debian. + +### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) + +I love git annex and use it daily. + + +> [[done]] --[[Joey]] diff --git a/doc/bugs/wget_always_shows_progress_bar/comment_1_d40883c9f9aade47112a0479ad56ed06._comment b/doc/bugs/wget_always_shows_progress_bar/comment_1_d40883c9f9aade47112a0479ad56ed06._comment new file mode 100644 index 000000000..c2e6eb53f --- /dev/null +++ b/doc/bugs/wget_always_shows_progress_bar/comment_1_d40883c9f9aade47112a0479ad56ed06._comment @@ -0,0 +1,19 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 1""" + date="2016-12-13T15:49:20Z" + content=""" +Since 2014, git-annex has used wget -q --show-progress to get a +progress bar without the several other lines of output it would normally +display. Whether a given git-annex build does this depends on what +version of wget it saw at configure time. + +Running git-annex with --quiet will disable the wget progress bar (and +other git-annex output). This seems like the thing to do if you're running +git-annex concurrently. (Of course, git-annex also has its own built-in +concurrency with -J which can display multiple download progress bars in a +nice way.) + +Still, might as well make the web-options come after the default options so +they can be overridden. Doing so. +"""]] diff --git a/doc/bugs/when_you_get_a_file_but_don__39__t_actually_have_enough_space_for_it__44___the_error_message_makes_useless_suggestions.mdwn b/doc/bugs/when_you_get_a_file_but_don__39__t_actually_have_enough_space_for_it__44___the_error_message_makes_useless_suggestions.mdwn new file mode 100644 index 000000000..19e839263 --- /dev/null +++ b/doc/bugs/when_you_get_a_file_but_don__39__t_actually_have_enough_space_for_it__44___the_error_message_makes_useless_suggestions.mdwn @@ -0,0 +1,21 @@ +The suggestion to make remotes available isn't really applicable, since the error was local. + +This is with git annex 6.20161110-gd48f4ca. + +[[!format sh """ + ../git-annex.linux/git-annex get archiveteam-fire/metro.co.uk-urls-2007-04-12-20150627/metro.co.uk-urls-2007-04-12-20150627_meta.xml +get archiveteam-fire/metro.co.uk-urls-2007-04-12-20150627/metro.co.uk-urls-2007-04-12-20150627_meta.xml + not enough free space, need 98.82 GB more (use --force to override this check or adjust annex.diskreserve) + + Unable to access these remotes: web + + Try making some of these repositories available: + 00000000-0000-0000-0000-000000000001 -- web + 9f8218c0-763f-463d-9152-ecdc56d4452c -- iabak@redwyne.jwintjen.de:~/IA.BAK/shard12 +failed +git-annex: get: 1 failed +"""]] + +### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) + +mixed success diff --git a/doc/builds.mdwn b/doc/builds.mdwn index e6c7c8082..77c9351e3 100644 --- a/doc/builds.mdwn +++ b/doc/builds.mdwn @@ -49,7 +49,7 @@ <iframe width=1024 scrolling=no frameborder=0 marginheight=0 marginwidth=0 src="https://downloads.kitenet.net/git-annex/autobuildtest/x86_64-apple-yosemite/testresult/status"> </iframe> <h2><a href="https://qa.nest-initiative.org/view/msysGit/job/msysgit-git-annex-assistant-test/">Windows</a></h2> -<a href="https://qa.nest-initiative.org/view/msysGit/job/msysgit-git-annex-assistant-test/">here</a> +<a href="http://vps.zaytsev.net:8080/">here</a> (firewalled from most locations; kite can access it) <h2><a href="https://buildd.debian.org/status/package.php?p=git-annex&suite=sid">Debian</a></h2> <iframe width=1024 scrolling=no height=500px frameborder=0 marginheight=0 marginwidth=0 src="https://buildd.debian.org/status/package.php?p=git-annex&suite=sid"> </iframe> diff --git a/doc/design/assistant/telehash.mdwn b/doc/design/assistant/telehash.mdwn index 373f1a575..788074f96 100644 --- a/doc/design/assistant/telehash.mdwn +++ b/doc/design/assistant/telehash.mdwn @@ -46,13 +46,7 @@ or [cjdns](https://github.com/cjdelisle/cjdns) or tor or i2p or [magic wormhole] * Awesome. * Easy to install, use; very well known. -* May need root to set up a hidden service. -* There's been some [haskell packages developed recently](http://www.leonmergen.com/haskell/privacy/2015/05/30/on-anonymous-networking-in-haskell-announcing-tor-and-i2p-for-haskell.html) - to communicate with tor and set up onion addresses for a service. - Could be used to make git-annex run as a hidden service. - However, that relies on tor being configured with a ControlPort, - without authentication. The normal tor configuration does not enable a - ControlPort. +* Supported in git-annex now! ## i2p status @@ -66,26 +60,25 @@ or [cjdns](https://github.com/cjdelisle/cjdns) or tor or i2p or [magic wormhole] ## general design -* Make address.log that contains (uuid, transport, address, Maybe authtoken) -* The authtoken is an additional guard, to protect against transports - where the address might be able to be guessed, or observed by the rest of - the network. -* Some addresses can be used with only the provided authtoken - from the address.log. Remotes can be auto-enabled for these. -* Other addresses have Nothing povided for the authtoken, and one - has to instead be provided during manual enabling of the remote. +* There is a generic P2P protocol, which should be usable with any P2P + system that can send messages between peers. +* A p2p remote has an url like tor-annex::fijdksajdksjfkj, which connects + to a specific peer. The peer's address may be kept private, but + the design allows the address to be public without giving access to + the peer. +* An authtoken also needs to be presented when connecting with a peer. + This is stored in local creds storage and must be kept private. * The remotedaemon runs, and/or communicates with the program implementing - the network transport. For example for tor, the remotedaemon runs - the hidden service, and also connects to the tor hidden services of - other nodes. + the P2P network. For example for tor, the remotedaemon runs the + hidden service. * The remotedaemon handles both sides of git push over the transport. * The remotedaemon may also support sending objects over the transport, depending on the transport. -## address discovery +## address exchange The address is a public key, and the authtoken is some large chunk of data, -so won't want to type that in. Need discovery. +so won't want to type that in. Need discovery or exchange for peering. * Easy way is any set of repos that are already connected can communicate them via address.log. @@ -96,32 +89,26 @@ so won't want to type that in. Need discovery. it can be read over the phone. * Users may not have a way to communicate with perfect forward secrecy. So it would be good to have a address+authtoken that can only be used - one time during pairing: - - 1. Alice uses the webapp to generate a one-time address+authtoken, - and sends it into a message to Bob. - 2. Bob enters it into his webapp. - 3. Bob's assistant contacts Alice's over the transport, presents the - one-time authtoken. (Alice's assistant accepts it, and marks it as - used so it cannot be used again.) - 4. Alice's webapp shows that it's ready to finish pairing; so does Bob's. - Both wait for their users to confirm before proceeding. - 5. Alice's assistant generates a new, permanant use authtoken, sends it - to Bob's assistant, which stores it and enables a remote using it. - 6. Bob's assistant generates a new, permanant use authtoken, sends it to - Alice's assistant, which stores it and enables a remote using it. - 7. Alice and Bob's assistants are now paired. - - Note that this exchange can be actively MITMed. If Eve can intercept - Alice's message to Bob, then Eve can pair with Alice. Or, if Eve can - forge a message from Alice to Bob, Eve can trick Bob into pairing with - her. - - If they make a phone call, it's much harder for Eve to MITM it. - Eve would need to listen to Alice reading the authtoken and enter it - before Bob does, so pairing with Alice. But as long as Alice waits - for Bob to confirm he's ready to finish pairing, this will fail, - because Bob won't get to that point if the authtoken is intercepted. + one time during pairing. +* Check out [PAKE](https://en.wikipedia.org/wiki/Password-authenticated_key_agreement) + for MITM resistance. +* Possibly use magic wormhole to exchange the address, which avoids + the users needing to exchange so much data. The magic wormhole code + is just 3 words, and it uses PAKE. + + I tried it, and opened a couple of bug reports that would be useful in + integrating it with git-annex: + + - [option to receive to a specific file](https://github.com/warner/magic-wormhole/issues/101) + - [machine readable wormhole code ](https://github.com/warner/magic-wormhole/issues/104]) + +## local lan detection + +At connection time, after authentication, the remote can send +(ip address, ssh host key). Try sshing to the ip address to check if +the host key matches. If so, can enable a ssh remote, which will +be cheaper than using the transport. Send the ssh public key back to the +remote to get it authorized. ## remotedaemon diff --git a/doc/design/roadmap.mdwn b/doc/design/roadmap.mdwn index 795cc8c84..5e636f33b 100644 --- a/doc/design/roadmap.mdwn +++ b/doc/design/roadmap.mdwn @@ -3,7 +3,6 @@ * [[design/caching_database]] for metadata views * [[assistant/deltas]] * [[assistant/gpgkeys]] management for the assistant -* [[assistant/telehash]] or similar * [[design/requests_routing]] * [[design/new_repo_versions]] diff --git a/doc/devblog/day_425__tor.mdwn b/doc/devblog/day_425__tor.mdwn new file mode 100644 index 000000000..08fe21cdd --- /dev/null +++ b/doc/devblog/day_425__tor.mdwn @@ -0,0 +1,23 @@ +Have waited too long for some next-generation encrypted P2P network, like +telehash to emerge. Time to stop waiting; tor hidden services are not as +cutting edge, but should work. Updated the [[design|design/assistant/telehash]] +and started implementation in the `tor` branch. + +Unfortunately, Tor's default configuration does not enable the ControlPort. +And, changing that in the configuration could be problimatic. This +makes it harder than it ought to be to register a tor hidden service. +So, I implemented a `git annex enable-tor` command, which can be run as root +to set it up. The webapp will probably use `su-to-root` or `gksu` to run it. +There's some Linux-specific parts in there, and it uses a socket for +communication between tor and the hidden service, which may cause problems +for Windows porting later. + +Next step will be to get `git annex remotedaemon` to run as a tor hidden +service. + +Also made a `no-xmpp` branch which removes xmpp support from the assistant. +That will remove 3000 lines of code when it's merged. Will probably wait +until after tor hidden services are working. + +Today's work was sponsored by Jake Vosloo on +[Patreon](https://www.patreon.com/joeyh/). diff --git a/doc/devblog/day_425__tor/comment_1_1dd41fa32eb3867d764f3238005b5b81._comment b/doc/devblog/day_425__tor/comment_1_1dd41fa32eb3867d764f3238005b5b81._comment new file mode 100644 index 000000000..fe609cab0 --- /dev/null +++ b/doc/devblog/day_425__tor/comment_1_1dd41fa32eb3867d764f3238005b5b81._comment @@ -0,0 +1,11 @@ +[[!comment format=mdwn + username="grawity@2ea26be48562f66fcb9b66307da72b1e2e37453f" + nickname="grawity" + avatar="http://cdn.libravatar.org/avatar/7003e967f47003bae82966aa373de8ef" + subject="comment 1" + date="2016-11-15T18:01:18Z" + content=""" +…or `pkexec`, which is present on many systems and generally integrates better with whatever DE/non-DE the user may be running. + +(OTOH, pkexec does not set up X11 access – then again, root helpers shouldn't need it.) +"""]] diff --git a/doc/devblog/day_426__grab_bag.mdwn b/doc/devblog/day_426__grab_bag.mdwn new file mode 100644 index 000000000..36e32077e --- /dev/null +++ b/doc/devblog/day_426__grab_bag.mdwn @@ -0,0 +1,63 @@ +Fixed one howler of a bug today. Turns out that +`git annex fsck --all --from remote` didn't actually check the content of +the remote, but checked the local repository. Only `--all` was buggy; +`git annex fsck --from remote` was ok. Don't think this is crash priority +enough to make a release for, since only `--all` is affected. + +Somewhat uncomfortably made `git annex sync` pass +`--allow-unrelated-histories` to git merge. While I do think that git's +recent refusal to merge unrelated histories is good in general, the +problem is that initializing a direct mode repository involves making an +empty commit. So merging from a remote into such a direct mode repository +means merging unrelated histories, while an indirect mode repository doesn't. +Seems best to avoid such inconsistencies, and the only way I could see to +do it is to always use `--allow-unrelated-histories`. May revisit this once +direct mode is finally removed. + +Using the git-annex arm standalone bundle on some WD NAS boxes used to +work, and then it seems they changed their kernel to use a nonstandard page +size, and broke it. This actually seems to be a +[bug in the gold linker](http://bugs.debian.org/844467), which defaults to an +unncessarily small page size on arm. The git-annex arm bundle is being +adjusted to try to deal with this. + +ghc 8 made `error` include some backtrace information. While it's really +nice to have backtraces for unexpected exceptions in Haskell, it turns +out that git-annex used `error` a lot with the intent of showing an error +message to the user, and a backtrace clutters up such messages. So, +bit the bullet and checked through every `error` in git-annex and made such +ones not include a backtrace. + +Also, I've been considering what protocol to use between git-annex nodes +when communicating over tor. One way would be to make it very similar to +`git-annex-shell`, using rsync etc, and possibly reusing code from +git-annex-shell. However, it can take a while to make a connection across +the tor network, and that method seems to need a new connection for each +file transfered etc. Also thought about using a http based protocol. The +servant library is great for that, you get both http client and server +implementations almost for free. Resuming interrupted transfers might +complicate it, and the hidden service side would need to listen on a unix +socket, instead of the regular http port. It might be worth it to use http +for tor, if it could be reused for git-annex http servers not on the tor +network. But, then I'd have to make the http server support git pull and +push over http in a way that's compatable with how git uses http, including +authentication. Which is a whole nother ball of complexity. So, I'm leaning +instead to using a simple custom protocol something like: + + > AUTH $localuuid $token + < AUTH-SUCCESS $remoteuuid + > SENDPACK $length + > $gitdata + < RECVPACK $length + < $gitdata + > GET $pos $key + < DATA $length + < $bytes + > SUCCESS + > PUT $key + < PUT-FROM $pos + > DATA $length + > $bytes + < SUCCESS + +Today's work was sponsored by Riku Voipio. diff --git a/doc/devblog/day_426__grab_bag/comment_1_4d01c756850032d351fa99188a3301a7._comment b/doc/devblog/day_426__grab_bag/comment_1_4d01c756850032d351fa99188a3301a7._comment new file mode 100644 index 000000000..7b5a2949a --- /dev/null +++ b/doc/devblog/day_426__grab_bag/comment_1_4d01c756850032d351fa99188a3301a7._comment @@ -0,0 +1,11 @@ +[[!comment format=mdwn + username="https://anarc.at/openid/" + nickname="anarcat" + avatar="http://cdn.libravatar.org/avatar/b36dcf65657dd36128161355d8920a99503def9461c1bb212410980fe6f07125" + subject="how about reusing the special remote protocol?" + date="2016-11-16T21:58:08Z" + content=""" +git-annex already has a custom protocol detailed in [[design/external_special_remote_protocol]]. it could be quite useful to have that protocol extended to support direct object transfer instead of having to mess around with temporary files like may remotes do, for example... + +maybe that makes no sense at all, i don't know. :) --[[anarcat]] +"""]] diff --git a/doc/devblog/day_427__free_p2p.mdwn b/doc/devblog/day_427__free_p2p.mdwn new file mode 100644 index 000000000..7c727587b --- /dev/null +++ b/doc/devblog/day_427__free_p2p.mdwn @@ -0,0 +1,51 @@ +For a Haskell programmer, and day where a big thing is implemented +without the least scrap of code that touches the IO monad is a good day. +And this was a good day for me! + +Implemented the p2p protocol for tor hidden services. Its needs are somewhat +similar to the external special remote protocol, but the two protocols are +not fully overlapping with one-another. Rather than try to unify them, and +so complicate both cases, I prefer to reuse as much code as possible between +separate protocol implementations. The generating and parsing of messages +is largely shared between them. I let the new p2p protocol otherwise +develop in its own direction. + +But, I *do* want to make this p2p protocol reusable for other types of p2p +networks than tor hidden services. This was an opportunity to use the Free +monad, which I'd never used before. It worked out great, letting me write +monadic code to handle requests and responses in the protocol, that reads +the content of files and resumes transfers and so on, all independent +of any concrete implementation. + +The whole implementation of the protocol only needed 74 lines of monadic code. +It helped that I was able to factor out functions like this one, that is used +both for handling a download, and by the remote when an upload is sent to it: + + receiveContent :: Key -> Offset -> Len -> Proto Bool + receiveContent key offset len = do + content <- receiveBytes len + ok <- writeKeyFile key offset content + sendMessage $ if ok then SUCCESS else FAILURE + return ok + +To get transcripts of the protocol in action, the Free monad can be evaluated +purely, providing the other side of the conversation: + + ghci> putStrLn $ protoDump $ runPure (put (fromJust $ file2key "WORM--foo")) [PUT_FROM (Offset 10), SUCCESS] + > PUT WORM--foo + < PUT-FROM 10 + > DATA 90 + > bytes + < SUCCESS + result: True + + ghci> putStrLn $ protoDump $ runPure (serve (toUUID "myuuid")) [GET (Offset 0) (fromJust $ file2key "WORM--foo")] + < GET 0 WORM--foo + > PROTO-ERROR must AUTH first + result: () + +Am very happy with all this pure code and that I'm finally using Free monads. +Next I need to get down the the dirty business of wiring this up to +actual IO actions, and an actual network connection. + +Today's work was sponsored by Jake Vosloo on Patreon. diff --git a/doc/devblog/day_428-429__git_push_to_hiddden_service.mdwn b/doc/devblog/day_428-429__git_push_to_hiddden_service.mdwn new file mode 100644 index 000000000..ec1bf12fd --- /dev/null +++ b/doc/devblog/day_428-429__git_push_to_hiddden_service.mdwn @@ -0,0 +1,31 @@ +The `tor` branch is coming along nicely. + +This weekend, I continued working on the P2P protocol, implementing +it for network sockets, and extending it to support connecting up +git-send-pack/git-receive-pack. + +There was a bit of a detour when I split the Free monad into two separate +ones, one for Net operations and the other for Local filesystem operations. + +This weekend's work was sponsored by Thomas Hochstein on Patreon. + +---- + +Today, implemented a `git-remote-tor-annex` command that git will +use for tor-annex:: urls, and made `git annex remotedaemon` +serve the tor hidden service. + +Now I have git push/pull working to the hidden service, for example: + + git pull tor-annex::eeaytkuhaupbarfi.onion:47651 + +That works very well, but does not yet check that the user is authorized +to use the repo, beyond knowing the onion address. And currently +it only works in git-annex repos; with some tweaks it should +also work in plain git repos. + +Next, I need to teach git-annex how to access tor-annex remotes. +And after that, an interface in the webapp for setting them up and +connecting them together. + +Today's work was sponsored by Josh Taylor on Patreon. diff --git a/doc/devblog/day_430__tor_socket_problem.mdwn b/doc/devblog/day_430__tor_socket_problem.mdwn new file mode 100644 index 000000000..7e7c8d1bd --- /dev/null +++ b/doc/devblog/day_430__tor_socket_problem.mdwn @@ -0,0 +1,13 @@ +Debian's tor daemon is very locked down in the directories it can read +from, and so I've had a hard time finding a place to put the unix socket +file for git-annex's tor hidden service. Painful details in +<http://bugs.debian.org/846275>. At least for now, I'm putting it under +/etc/tor/, which is probably a FHS violation, but seems to be the only +option that doesn't involve a lot of added complexity. + +--- + +The Windows autobuilder is moving, since +[NEST](http://nest-initiative.org/) is shutting down the server it has been +using. Yury Zaytsev has set up a new Windows autobuilder, hosted at +Dartmouth College this time. diff --git a/doc/devblog/day_431__p2p_linking.mdwn b/doc/devblog/day_431__p2p_linking.mdwn new file mode 100644 index 000000000..1e53ffefc --- /dev/null +++ b/doc/devblog/day_431__p2p_linking.mdwn @@ -0,0 +1,27 @@ +Today I finished the second-to-last big missing peice for tor hidden service +remotes. Networks of these remotes are P2P networks, and there needs to be +a way for peers to find one-another, and to authenticate with one-another. +The `git annex p2p` command sets up links between peers in such a network. + +So far it has only a basic interface that sets up a one way link between +two peers. In the first repository, run `git annex p2p --gen-address`. +That outputs a long address. In the second repository, run +`git annex p2p --link peer1`, and paste the address into it. That sets up a +git remote named "peer1" that connects back to the first repository over tor. + +That is a one-directional link, while a bi-directional link would be +much more convenient to have between peers. Worse, the address can be reused by +anyone who sees it, to link into the repository. And, the address is far +too long to communicate in any way except for pasting it. + +So I want to improve that later. What I'd really like to have is an +interface that displays a one-time-use phrase of five to ten words, that +can be read over the phone or across the room. Exchange phrases with a +friend, and get your repositories securely linked together with tor. + +But, `git annex p2p` is good enough for now. I can move on to the final +keystone of the tor support, which is file transfer over tor. +That should, fingers crossed, be relatively easy, and the `tor` branch is +close to mergeable now. + +Today's work was sponsored by Riku Voipio. diff --git a/doc/devblog/day_431__p2p_linking/comment_1_1d5f809564c25e765f82594af8e174ab._comment b/doc/devblog/day_431__p2p_linking/comment_1_1d5f809564c25e765f82594af8e174ab._comment new file mode 100644 index 000000000..9eceb71ed --- /dev/null +++ b/doc/devblog/day_431__p2p_linking/comment_1_1d5f809564c25e765f82594af8e174ab._comment @@ -0,0 +1,49 @@ +[[!comment format=mdwn + username="https://anarc.at/openid/" + nickname="anarcat" + avatar="http://cdn.libravatar.org/avatar/b36dcf65657dd36128161355d8920a99503def9461c1bb212410980fe6f07125" + subject="magic wormhole" + date="2016-11-30T22:16:19Z" + content=""" +> What I'd really like to have is an interface that displays a +> one-time-use phrase of five to ten words, that can be read over the +> phone or across the room. Exchange phrases with a friend, and get +> your repositories securely linked together with tor. + +I already mentionned the project in [[design/assistant/telehash/]], +but [magic-wormhole](https://github.com/warner/magic-wormhole) does +exactly that: + + % wormhole send README.md + Sending 7924 byte file named 'README.md' + On the other computer, please run: wormhole receive + Wormhole code is: 7-crossover-clockwork + + Sending (<-10.0.1.43:58988).. + 100%|=========================| 7.92K/7.92K [00:00<00:00, 6.02MB/s] + File sent.. waiting for confirmation + Confirmation received. Transfer complete. + +Receiver: + + % wormhole receive + Enter receive wormhole code: 7-crossover-clockwork + Receiving file (7924 bytes) into: README.md + ok? (y/n): y + Receiving (->tcp:10.0.1.43:58986).. + 100%|===========================| 7.92K/7.92K [00:00<00:00, 120KB/s] + Received file written to README.md + +While that example shows a file transfer, arbitrary data can be +transfered this way. There's a documented protocol, and it's not +completely peer-to-peer: there are relay servers to deal with NAT'd +machines. But the [PAKE +protocol](https://en.wikipedia.org/wiki/Password-authenticated_key_agreement) +(basically SPAKE2) could be a good inspiration here. + +Otherwise, I must say that, as a user, I don't mind copy-pasting a +hidden service string (if that's what it's about): i can do that over +a secure medium (email + OpenPGP or IM + OTR) easily... But I +understand it can be difficult to do for new users. + +"""]] diff --git a/doc/devblog/day_432-433__almost_there.mdwn b/doc/devblog/day_432-433__almost_there.mdwn new file mode 100644 index 000000000..b41ce3f70 --- /dev/null +++ b/doc/devblog/day_432-433__almost_there.mdwn @@ -0,0 +1,13 @@ +Friday and today were spent implementing both sides of the P2P protocol for +git-annex content transfers. + +There were some tricky cases to deal with. For example, when a file is being +sent from a direct mode repository, or v6 annex.thin repository, the +content of the file can change as it's being transferred. Including being +appended to or truncated. Had to find a way to deal with that, to avoid +breaking the protocol by not sending the indicated number of bytes of data. + +It all seems to be done now, but it's not been tested at all, and there are +probably some bugs to find. (And progress info is not wired up yet.) + +Today's work was sponsored by Trenton Cronholm on Patreon. diff --git a/doc/devblog/day_434__it_works.mdwn b/doc/devblog/day_434__it_works.mdwn new file mode 100644 index 000000000..75d096b31 --- /dev/null +++ b/doc/devblog/day_434__it_works.mdwn @@ -0,0 +1,27 @@ +Git annex transfers over Tor worked correctly the first time I tried them +today. I had been expecting protocol implementation bugs, so this was a +nice surprise! + +Of course there were some bugs to fix. I had forgotten to add UUID +discovery to `git annex p2p --link`. And, resuming interrupted transfers +was buggy. + +Spent some time adding progress updates to the Tor remote. I was curious to +see what speed transfers would run. Speed will of course vary depending on +the Tor relays being used, but this example with a 100 mb file is not bad: + + copy big4 (to peer1...) + 62% 1.5MB/s 24s + +There are still a couple of [[known bugs|todo/tor]], +but I've merged the `tor` branch into `master` already. + +---- + +Alpernebbi has built a GUI for editing git-annex metadata. +Something I always wanted! +[[Read about it here|tips/a_gui_for_metadata_operations]] + +---- + +Today's work was sponsored by Ethan Aubin. diff --git a/doc/devblog/day_435-436_post_tor_merge.mdwn b/doc/devblog/day_435-436_post_tor_merge.mdwn new file mode 100644 index 000000000..2f05e0252 --- /dev/null +++ b/doc/devblog/day_435-436_post_tor_merge.mdwn @@ -0,0 +1,20 @@ +More improvements to tor support. Yesterday, debugged a reversion that +broke push/pull over tor, and made actual useful error messages be +displayed when there were problems. Also fixed a memory leak, although I +fixed it by reorganizing code and could not figure out quite why it happened, +other than that the ghc runtime was not managing to be as lazy as I would +expect. + +Today, added git ref change notification to the +P2P protocol, and made the remotedaemon automatically fetch changes from +tor remotes. So, it should work to use the assistant to keep +repositories in sync over tor. I have not tried it yet, and linking over tor +still needs to be done at the command line, so it's not really ready for +webapp users yet. + +Also fixed a denial of service attack in git-annex-shell and git-annex when +talking to a remote git-annex-shell. It was possible to feed either a large +amount of data when they tried to read a line of data, and summon the OOM +killer. Next release will be expedited some because of that. + +Today's work was sponsored by Thomas Hochstein on Patreon. diff --git a/doc/devblog/day_437__catching_up.mdwn b/doc/devblog/day_437__catching_up.mdwn new file mode 100644 index 000000000..1deb54459 --- /dev/null +++ b/doc/devblog/day_437__catching_up.mdwn @@ -0,0 +1,20 @@ +Quite a backlog developed in the couple of weeks I was concentrating on tor +support. I've taken a first pass through it and fixed the most pressing +issues now. + +Most important was an ugly memory corruption problem in the GHC runtime +system that may have led to data corruption when using git-annex with Linux +kernels older than 4.5. All the Linux standalone builds of git-annex have +been updated to fix that issue. + +Today dealt with several more things, including fixing a buggy timestamp +issue with `metadata --batch`, reverting the ssh ServerAliveInterval +setting (broke on too many systems with old ssh or complicated ssh +configurations), making batch input not be rejected when it can't be decoded +as UTF-8, and more. + +Also, spent some time learning a little bit about Magic Wormhole and SPAKE, +as a way to exchange tor remote addresses. Using Magic Wormhole for that +seems like a reasonable plan. I did file a couple bugs on it which will +need to get fixed, and then using it is mostly a question of whether it's +easy enough to install that git-annex can rely on it. diff --git a/doc/devblog/day_438__bi-directional_p2p_links.mdwn b/doc/devblog/day_438__bi-directional_p2p_links.mdwn new file mode 100644 index 000000000..abcbed122 --- /dev/null +++ b/doc/devblog/day_438__bi-directional_p2p_links.mdwn @@ -0,0 +1,6 @@ +Improved `git annex p2p --link` to create a bi-directional link +automatically. Bi-directional links are desirable more often than not, so +it's the default behavior. + +Also continued thinking about using magic wormhole for communicating +p2p addresses for pairing. And filed some more bugs on magic wormhole. diff --git a/doc/devblog/day_439__wormhole_pairing.mdwn b/doc/devblog/day_439__wormhole_pairing.mdwn new file mode 100644 index 000000000..cc988a2db --- /dev/null +++ b/doc/devblog/day_439__wormhole_pairing.mdwn @@ -0,0 +1,51 @@ +`git annex p2p --pair` implemented, using Magic Wormhole codes +that have to be exchanged between the repositories being paired. + +It looks like this, with the same thing being done at the same time +in the other repository. + + joey@elephant:~/tmp/bench3/a>git annex p2p --pair + p2p pair peer1 (using Magic Wormhole) + + This repository's pairing code is: 1-select-bluebird + + Enter the other repository's pairing code: (here I entered 8-fascinate-sawdust) + Exchanging pairing data... + Successfully exchanged pairing data. Connecting to peer1... + ok + +And just that simply, the two repositories find one another, +Tor onion addresses and authentication data is exchanged, and a git remote +is set up connecting via Tor. + + joey@elephant:~/tmp/bench3/a>git annex sync peer1 + commit + ok + pull peer1 + warning: no common commits + remote: Counting objects: 5, done. + remote: Compressing objects: 100% (3/3), done. + remote: Total 5 (delta 0), reused 0 (delta 0) + Unpacking objects: 100% (5/5), done. + From tor-annex::5vkpoyz723otbmzo.onion:61900 + * [new branch] git-annex -> peer1/git-annex + +Very pleased with this, and also the whole thing worked on the very first +try! + +It might be slightly annoying to have to exchange two codes during pairing. +It would be possible to make this work with only one code. I decided to go +with two codes, even though it's only marginally more secure than one, +mostly for UI reasons. The pairing interface and +[[instructions for using it|tips/peer_to_peer_network_with_tor]] is simplfied +by being symmetric. + +(I also decided to revert the work I did on Friday to make `p2p --link` +set up a bidirectional link. Better to keep `--link` the simplest possible +primitive, and pairing makes bidirectional links more easily.) + +Next: Some more testing of this and the Tor hidden services, a webapp UI +for P2P peering, and then finally removing XMPP support. I hope to finish +that by New Years. + +Today's work was sponsored by Jake Vosloo on Patreon. diff --git a/doc/ekg.mdwn b/doc/ekg.mdwn index 508fd2e92..26d0128a6 100644 --- a/doc/ekg.mdwn +++ b/doc/ekg.mdwn @@ -20,11 +20,11 @@ git-annex will continue to work. For the really tricky memory leaks, here's how to make a profiling build of git-annex. -1. `cabal configure` with only the flags you really need -2. `cabal build --ghc-options="-prof -auto-all -caf-all"` +1. `cabal configure --enable-profiling` This will probably fail due to some missing profiling libraries. You have to get the profiling versions of all needed haskell libraries installed somehow. +2. `cabal build` 3. Run git-annex with the special flags `+RTS -hc -p` 4. Reproduce the memory leak problem. 5. If the assistant was run, stop it. diff --git a/doc/forum/Extra___38___missing_folders_on_remote.mdwn b/doc/forum/Extra___38___missing_folders_on_remote.mdwn new file mode 100644 index 000000000..b78961ccc --- /dev/null +++ b/doc/forum/Extra___38___missing_folders_on_remote.mdwn @@ -0,0 +1,9 @@ +I created a local annex directory that's an adjusted branch used with the assistant. On another machine, I initialized an annex directory, then made this into a full-backup ssh remote for my local. + +After the assistant pushes to the remote, and the remote runs `git annex sync`, the remote is missing some directories and has some extra directories. For example, it has the extra directory `Documents/programs/Documents/programs/`, which has different contents than `Documents/programs/`. Both directories are missing the subdirectory `graphing_experiments/`. + +From my local, `git annex whereis Documents/programs/graphing_experiments` says the directory exists on the remote. But it's not there. + +I recreated the remote from scratch and the problem persists. + +The assistant says the remote is caught up, and is keeping up with new content changes. What could cause this? diff --git a/doc/forum/GIT__95__SSH.mdwn b/doc/forum/GIT__95__SSH.mdwn new file mode 100644 index 000000000..968385d08 --- /dev/null +++ b/doc/forum/GIT__95__SSH.mdwn @@ -0,0 +1,3 @@ +Is there any way to get git-annex to respect the GIT_SSH environment variable just like git? This would be super helpful for specifying ssh options (keys in my application) + +Hacks like a ~/.ssh/config with a fake hostname to specify the key are not appropriate in my case, because keys are generated on the fly by a server-side application that may be distributed across multiple servers. So I really need to be able to specify a key file at run time and then use the normal remote URI. I can do this with git and GIT_SSH, but have not found a solution for git-annex yet. diff --git a/doc/forum/Git-annex_link_to_different_file_names.mdwn b/doc/forum/Git-annex_link_to_different_file_names.mdwn new file mode 100644 index 000000000..a7a7c2727 --- /dev/null +++ b/doc/forum/Git-annex_link_to_different_file_names.mdwn @@ -0,0 +1,41 @@ +This is a recreation of a stackexchange question, in case the community here is more knowledgeable. + +Link to stackexchange question : http://unix.stackexchange.com/questions/325753/git-annex-link-to-different-file-names + +Content : +"Maybe this is just a crazy use case that doesn't work, but I was wondering if there's a way to build a file's history from files with different file names. I'm exploring this idea because I'd like to have a git-annex system but I can't force my coworkers to adapt. + +Here's what I have in mind : + + Folder 1, managed by coworkers (On a shared disk) : + + drawing_shop_12_nov_2015.pdf + drawing_shop_13_nov_2015.pdf + drawing_asbuilt_14_nov_2015.pdf + drawing_asbuilt_rev1_15_nov_2015.pdf + +And + + Git-annex, managed by me : + + drawing.pdf + + (with a shop branch and a asbuilt branch) + +The git-annex's drawing.pdf would have an history like this : + + [shop] + | + Commit A "Initial shop drawing" + | + Commit B "Add corrections from Wizzbasket" + \ + | + [asbuilt] + Commit C "Reflect as built" + | + Commit D "Change dweezelbox block for simplicity" + +But somehow the "managed by coworkers" repo would be a direct mode repo with Commit A pointing to drawing_shop_12_nov_2015.pdf, Commit B to drawing_shop_13_nov_2015.pdf etc. + +Can this be done?" diff --git a/doc/forum/Git-annex_link_to_different_file_names/comment_1_17ab85276bcf495a656c7091753c086f._comment b/doc/forum/Git-annex_link_to_different_file_names/comment_1_17ab85276bcf495a656c7091753c086f._comment new file mode 100644 index 000000000..8085253ad --- /dev/null +++ b/doc/forum/Git-annex_link_to_different_file_names/comment_1_17ab85276bcf495a656c7091753c086f._comment @@ -0,0 +1,60 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 1""" + date="2016-12-20T19:27:31Z" + content=""" +Yes, you can do this. Effectively, you have two branches. In the master +branch, you have drawing.pdf with a single name and changes commited to it. +In the coworkers branch, you have the multiple different versions. Git has +no difficulty representing this, but it's up to you to maintain the +different branches. + +For example: + + joey@darkstar:~/tmp/shop>git commit -m 'updated drawing some more' + [master 1403dd4] updated drawing some more + 1 file changed, 1 insertion(+), 1 deletion(-) + joey@darkstar:~/tmp/shop>git checkout coworkers + Switched to branch 'coworkers' + joey@darkstar:~/tmp/shop#coworkers>git show master + commit 1403dd49b2c378e78b8b8ec82d73e295c486697b + Author: Joey Hess <joeyh@joeyh.name> + Date: Tue Dec 20 15:31:17 2016 -0400 + + updated drawing some more + + diff --git a/drawing.pdf b/drawing.pdf + index b59371e..c05ed95 120000 + --- a/drawing.pdf + +++ b/drawing.pdf + @@ -1 +1 @@ + -.git/annex/objects/55/MZ/SHA256E-s13--c5f6529491f9e6d40e893d2ffc008bc297bcc56a680040c124e4019fb5c1a94d.pdf/SHA256E-s13--c5f6529491f9e6d40e893d2ffc008bc297bcc56a680040c124e4019fb5c1a94d.pdf + \ No newline at end of file + +.git/annex/objects/xj/XF/SHA256E-s17--7786e857a89634ff9242d899245cbcc5e009736af6b0553cb7283b2daef77d16.pdf/SHA256E-s17--7786e857a89634ff9242d899245cbcc5e009736af6b0553cb7283b2daef77d16.pdf + \ No newline at end of file + joey@darkstar:~/tmp/shop#coworkers>ln -s .git/annex/objects/xj/XF/SHA256E-s17--7786e857a89634ff9242d899245cbcc5e009736af6b0553cb7283b2daef77d16.pdf/SHA256E-s17--7786e857a89634ff9242d899245cbcc5e009736af6b0553cb7283b2daef77d16.pdf drawing_rev2.pdf + joey@darkstar:~/tmp/shop#coworkers>git add drawing_rev2.pdf + joey@darkstar:~/tmp/shop#coworkers>ls + drawing.pdf@ drawing_rev2.pdf@ + joey@darkstar:~/tmp/shop#coworkers>git commit -m 'added rev2 of drawing' + [coworkers cf27781] added rev2 of drawing + 1 file changed, 1 insertion(+) + create mode 120000 drawing_rev2.pdf + +In the example, I looked at what was committed to master, and copied and +pasted the git-annex symlink into a new drawing_rev2.pdf file. + +That's the basic idea. There might be a better way to do that. Another way, +for example, would be to have 2 clones of the repo, one with master checked +out and one with coworkers checked out. You could then run, in the +coworkers checkout: + + cp -a ../master/drawing.pdf drawing_rev2.pdf + git add drawing_rev2.pdf + git commit -m 'added rev2 of drawing' + +That results in the same commit as the method I showed. + +With some scripting, you should be able to automate keeping the two +branches in sync. +"""]] diff --git a/doc/forum/Multiple_interface_to_the_same_annex/comment_1_ea9e3a987112d8bf6421be234bf61d3c._comment b/doc/forum/Multiple_interface_to_the_same_annex/comment_1_ea9e3a987112d8bf6421be234bf61d3c._comment new file mode 100644 index 000000000..df21114bc --- /dev/null +++ b/doc/forum/Multiple_interface_to_the_same_annex/comment_1_ea9e3a987112d8bf6421be234bf61d3c._comment @@ -0,0 +1,15 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 1""" + date="2016-12-13T16:55:21Z" + content=""" +Creating a separate special remote pointing to the same content is not a +good idea. This will confuse git-annex's location tracking, and in some +cases can lead to data loss, since git-annex will assume it can safely +delete a file from one of the "two" repositories, since it thinks the +"other" one will still have the content of the file. + +Instead, you need a way to configure a regular git remote, +pointing at the repository, that is read-only, or is accessed over rsync, +or whatever your requirements are. +"""]] diff --git a/doc/forum/Odd_Hybrid_Symlinks_To_Content.mdwn b/doc/forum/Odd_Hybrid_Symlinks_To_Content.mdwn new file mode 100644 index 000000000..5e3b4d8b4 --- /dev/null +++ b/doc/forum/Odd_Hybrid_Symlinks_To_Content.mdwn @@ -0,0 +1,27 @@ +I've somehow managed to get my indirect repository to symlink to literal content instead of object files. + +By this I mean literally the symlink is pointing at the contents of the file as the filename. + +So if I have a blah.txt file with this content: + +* First line +* second line + +And I ls -al to view the symlink pointer, it shows up as this: + +* blah.txt -> First line?second line + +It literally has the contents of the file as the destination filename. + +I've tried a couple things I could think of to re-symlink the files, but they don't seem to do anything as they think everything is fine: + +* git annex indirect //returns nothing +* git annex lock blah.txt //returns nothing +* git annex fix blah.txt //returns nothing +* git annex fsck //returns nothing + +I'm actually able to find several of these files hanging around by searching for all symlinks that don't point to something in the .git directory. + +Is there a way for me to replace the symlinks with correct symlinks to the objects in .git/annex? Can it even figure out which ones it was supposed to point to if the symlinks are messed up (are filenames -> content hashes stored anywhere else)? + +Else I might have to go do some manual rebasing and history editing to try to undo the bad commits manually. I've synced this repo to another direct repo so I'll need to figure out how to manually fix that repo too (using proxy). From what I can tell the annex/direct/master seems to be same as master and synced/master branches? Is there an [[internals]] page for direct branches besides [[direct_mode]] so I know what should be fixed where? diff --git a/doc/forum/Preserving_Directories_in_Metadata_Views.mdwn b/doc/forum/Preserving_Directories_in_Metadata_Views.mdwn new file mode 100644 index 000000000..dfc45cb4b --- /dev/null +++ b/doc/forum/Preserving_Directories_in_Metadata_Views.mdwn @@ -0,0 +1,47 @@ +I want to use metadata views to sort files into top-level directories based on a tag, but then preserve the directory structure underneath that. I'm having trouble with this. + +Say I have an annex at `~/annex` with a structure like this: + + $ tree + . + ├── foo + │ └── bar + │ ├── one.txt + │ ├── three.txt + │ └── two.txt + └── waldo + └── fred + ├── a.txt + ├── b.txt + └── c.txt + +I tag some of the files with `blah`: + + $ git annex metadata -t blah foo/bar/* + +Now I want to change my view to only see those files with a certain tag, but I want to maintain their directory structure, ie I want to end up with something like this: + + $ tree + . + ├── blah + │ └── foo + │ └── bar + │ ├── one.txt + │ ├── three.txt + │ └── two.txt + +If I do `git annex view blah` I see the files `one.txt`, `two.txt` and `three.txt` but they are in the top level of `~/annex`. The `foo` and `bar` directories are not present. + +If I do `git annex view blah "/=*"` then the files I present under the `foo` directory, but the `bar` subdirectory is not there. + +It would also be fine if I could just hide the files that did not have the `blah` tag, so that I ended up with this: + + $ tree + . + ├── foo + │ └── bar + │ ├── one.txt + │ ├── three.txt + │ └── two.txt + +Is something like this possible? diff --git a/doc/forum/Sending_requests_across_the_network/comment_3_9859c46db3527ad329c8e0df06edd153._comment b/doc/forum/Sending_requests_across_the_network/comment_3_9859c46db3527ad329c8e0df06edd153._comment new file mode 100644 index 000000000..c63404e0e --- /dev/null +++ b/doc/forum/Sending_requests_across_the_network/comment_3_9859c46db3527ad329c8e0df06edd153._comment @@ -0,0 +1,11 @@ +[[!comment format=mdwn + username="neocryptek@659edac901ffbc8e541a974f8f18987eeafc63bd" + nickname="neocryptek" + avatar="http://cdn.libravatar.org/avatar/d9bfdefa9b503f1ac4844a686618374e" + subject="comment 3" + date="2016-11-13T22:39:44Z" + content=""" +Thanks, that makes sense. + +All git annex repositories using the same branch will have the same (symlink) working directory right (assuming entire network has been synced eventually)? +"""]] diff --git a/doc/forum/What_is_the_assistant_up_to__63__.mdwn b/doc/forum/What_is_the_assistant_up_to__63__.mdwn new file mode 100644 index 000000000..623268f7e --- /dev/null +++ b/doc/forum/What_is_the_assistant_up_to__63__.mdwn @@ -0,0 +1,5 @@ +Is there a way to see what the assistant is doing right now, what failed, etc. ? +I am running the assistant on a remote server so the Webapp's interface is not easily available. + +I was hoping to have something that is easier to read than the daemon.log + diff --git a/doc/forum/What_is_the_assistant_up_to__63__/comment_1_9baa0e54c19105c7cce946c19c587866._comment b/doc/forum/What_is_the_assistant_up_to__63__/comment_1_9baa0e54c19105c7cce946c19c587866._comment new file mode 100644 index 000000000..f007f5d30 --- /dev/null +++ b/doc/forum/What_is_the_assistant_up_to__63__/comment_1_9baa0e54c19105c7cce946c19c587866._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="marekj" + avatar="http://cdn.libravatar.org/avatar/65a60e8f5183feeeef8cef815bf73e61" + subject="git annex info" + date="2016-12-21T12:58:37Z" + content=""" +I found that git annex info provides information on current transfers. Using -F it provides what I want. +"""]] diff --git a/doc/forum/What_to_do_if_special_remotes_refuses_drops__63__.mdwn b/doc/forum/What_to_do_if_special_remotes_refuses_drops__63__.mdwn new file mode 100644 index 000000000..512e89528 --- /dev/null +++ b/doc/forum/What_to_do_if_special_remotes_refuses_drops__63__.mdwn @@ -0,0 +1,9 @@ +I have a special remote that I would like to delete and have marked it as such in the assistant. Although this was before my myriad of problems with git annex itself wanting to repair the repo all the time. Right now if I take a loog into my daemon.log I see the following error over and over again: + +``` +drop skydrive foo.bar + This file could not be removed +failed +``` + +I checked if I can login into my account and it works just fine. So I assume that this might be a bug? Is it somehow possible to forego the cleaning out of the special remote and just mark it as deleted for good? Thanks in advance! diff --git a/doc/forum/What_to_do_if_special_remotes_refuses_drops__63__/comment_1_0b523b2b6c361346c36ad456bbbac645._comment b/doc/forum/What_to_do_if_special_remotes_refuses_drops__63__/comment_1_0b523b2b6c361346c36ad456bbbac645._comment new file mode 100644 index 000000000..cf808aac4 --- /dev/null +++ b/doc/forum/What_to_do_if_special_remotes_refuses_drops__63__/comment_1_0b523b2b6c361346c36ad456bbbac645._comment @@ -0,0 +1,15 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 1""" + date="2016-11-18T15:39:20Z" + content=""" +It could certianly be a bug in the special remote implementation. It's also +possible for some special remotes to intentionally not be able to remove +content (this is the case with the web special remote, and the bup special +remote at least). + +You can manually remove the special remote, by editing .git/config and +deleting the stanza for that remote. You may want to run `git annex dead +$remotename` first, if you don't intend to ever use that special remote +again. +"""]] diff --git a/doc/forum/how_to_disaster_recovery/comment_12_f2e570dc60a6f16e8f696d94e253775f._comment b/doc/forum/how_to_disaster_recovery/comment_12_f2e570dc60a6f16e8f696d94e253775f._comment new file mode 100644 index 000000000..7c7da0ef4 --- /dev/null +++ b/doc/forum/how_to_disaster_recovery/comment_12_f2e570dc60a6f16e8f696d94e253775f._comment @@ -0,0 +1,7 @@ +[[!comment format=mdwn + username="openmedi" + subject="comment 12" + date="2016-11-18T12:03:44Z" + content=""" +A recent update to annex via homebrew now reslolves the issue with the weird looking webapp. +"""]] diff --git a/doc/forum/more_intelligent_copy_.mdwn b/doc/forum/more_intelligent_copy_.mdwn new file mode 100644 index 000000000..1c9889a74 --- /dev/null +++ b/doc/forum/more_intelligent_copy_.mdwn @@ -0,0 +1,15 @@ +Hi, + +I noticed, that + +git annex copy --to REMOTE FILES + +and + +git annex copy --to REMOTE --not --in REMOTE FILES + +behave differently. The first does not check, whether file contents are already in the remote the latter does that. I realize that this mimics "normal" (UNIX) copy behaviour but I was not entirely certain this was desired. +Depending on the type of the remote and its configuration (encryption) the latter is considerably faster. + +Just my two cents. + diff --git a/doc/forum/more_intelligent_copy_/comment_1_526f6a007f44f389ef7c904024752541._comment b/doc/forum/more_intelligent_copy_/comment_1_526f6a007f44f389ef7c904024752541._comment new file mode 100644 index 000000000..9b5866c5c --- /dev/null +++ b/doc/forum/more_intelligent_copy_/comment_1_526f6a007f44f389ef7c904024752541._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="annexuser" + avatar="http://cdn.libravatar.org/avatar/6ae692503ee113b1ab7b329b40084d5c" + subject="comment 1" + date="2016-12-13T02:00:08Z" + content=""" +git annex copy --to REMOTE FILES --fast +"""]] diff --git a/doc/forum/more_intelligent_copy_/comment_2_7b3f5d2e9de4b13de821177db2f57bcd._comment b/doc/forum/more_intelligent_copy_/comment_2_7b3f5d2e9de4b13de821177db2f57bcd._comment new file mode 100644 index 000000000..474a068b6 --- /dev/null +++ b/doc/forum/more_intelligent_copy_/comment_2_7b3f5d2e9de4b13de821177db2f57bcd._comment @@ -0,0 +1,15 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 2""" + date="2016-12-13T16:51:03Z" + content=""" +`git annex copy --fast` has the same behavior as `--not --in REMOTE` + +The reason this is not the default behavior is that git-annex's location +tracking information can sometimes be out of date, and then those +two will not copy some files despite their content not being any longer +in the remote. This won't lead to data loss, but could result +in unexpected behavior, and so the slower, more understandable behavior +is used by default. (Although I sometimes go back and forth on switching +it.) +"""]] diff --git a/doc/forum/recover_deleted_files___63__/comment_5_29ec08578bc45e4bbdecf76d1eb33826._comment b/doc/forum/recover_deleted_files___63__/comment_5_29ec08578bc45e4bbdecf76d1eb33826._comment new file mode 100644 index 000000000..a59b07964 --- /dev/null +++ b/doc/forum/recover_deleted_files___63__/comment_5_29ec08578bc45e4bbdecf76d1eb33826._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="veron_veron@8e19f168a8da3dabcdbf28ccd3f27edfb40941ed" + nickname="veron_veron" + avatar="http://cdn.libravatar.org/avatar/eb7805af696010b4e8844aefeeb89a1b" + subject="easy" + date="2016-12-20T09:19:31Z" + content=""" +Indeed the recycle bin, and using Windows Explorer to look in folders; the programs you used are not designed to search for files; especially programs like Elements, which uses a catalog is not what you need now. Moving files confuses PS Elements organiser enough to make it seem like you lost files, while they're actually there. When things go wrong, use the basic tools provided in Windows itself (explorer, search). +For file recovery programs, I've use <a href=\"http://www.passwordmanagers.net/resources/Software-for-Recovering-Deleted-Files-71.html\">Software for Recovering Deleted Files</a> on a few occassions; it's free and it works as good as can be expected. The main thing is to stop using the external hard disk until you use this tool, and avoid writing any files to it. +"""]] diff --git a/doc/forum/sync_--content__44___fatal_is_outside_repository_errors/comment_2_09c62e4abf4ccc0d2e030ef5e1bcdf71._comment b/doc/forum/sync_--content__44___fatal_is_outside_repository_errors/comment_2_09c62e4abf4ccc0d2e030ef5e1bcdf71._comment new file mode 100644 index 000000000..fcca1b28e --- /dev/null +++ b/doc/forum/sync_--content__44___fatal_is_outside_repository_errors/comment_2_09c62e4abf4ccc0d2e030ef5e1bcdf71._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="andrew" + avatar="http://cdn.libravatar.org/avatar/acc0ece1eedf07dd9631e7d7d343c435" + subject="how to investigate" + date="2016-11-16T15:37:01Z" + content=""" +Any thoughts? I am unsure how to investigate where this problem is. I assume these files are in my git repo or git-annex objects but I can't seem to find them using any search commands. + +Thanks, + +Andrew +"""]] diff --git a/doc/forum/sync_--content__44___fatal_is_outside_repository_errors/comment_2_8f694afa77f5a835c826d29d46d44615._comment b/doc/forum/sync_--content__44___fatal_is_outside_repository_errors/comment_2_8f694afa77f5a835c826d29d46d44615._comment new file mode 100644 index 000000000..ccaeeb409 --- /dev/null +++ b/doc/forum/sync_--content__44___fatal_is_outside_repository_errors/comment_2_8f694afa77f5a835c826d29d46d44615._comment @@ -0,0 +1,30 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 2""" + date="2016-11-16T18:52:07Z" + content=""" +It would be helpful if you said what version of git-annex you are using. + +And, is your git-annex repository using the new experimental v6 format? One +user reported a similar error message with a v6 git-annex repository. See +[[bugs/assistant_crashes_in_TransferScanner]] + +Or might your repository be using direct mode? + +So, please paste in `git annex version` and `git annex info` output. + +It kind of looks like it's having difficulty determining where the top of +the git repository is, or constructing a relative path to the git +repository. + +Are there any symlinks in the path to /Users/andrew/notes ? Eg, is /Users +a symlink, or /Users/andrew a symlink, or //Users/andrew/notes itself +symlinked to elsewhere? + +Does only `git annex sync --content` fail? What if you run, eg +`git annex copy --auto --to cloud` and `git annex get --auto --from cloud`, +does that fail similarly, or does it succeed? + +You say it's only failing for some files. Do the filenames that it's +failing on contain any non-ascii characters? +"""]] diff --git a/doc/forum/sync_--content__44___fatal_is_outside_repository_errors/comment_4_a7f476aeacf88679f25badc78fad886a._comment b/doc/forum/sync_--content__44___fatal_is_outside_repository_errors/comment_4_a7f476aeacf88679f25badc78fad886a._comment new file mode 100644 index 000000000..eec45e333 --- /dev/null +++ b/doc/forum/sync_--content__44___fatal_is_outside_repository_errors/comment_4_a7f476aeacf88679f25badc78fad886a._comment @@ -0,0 +1,57 @@ +[[!comment format=mdwn + username="andrew" + avatar="http://cdn.libravatar.org/avatar/acc0ece1eedf07dd9631e7d7d343c435" + subject="git annex copy --auto --to cloud works" + date="2016-11-17T17:49:27Z" + content=""" +Yes, only `git annex sync --content` seems to fail. I am using v6 with a mix of unlocked and locked files. I did not know about the --auto flags for copy/get. + +* `git annex copy --auto --to cloud` works fine +* `git annex get --auto --from cloud` works fine + + +*Are there any symlinks in the path to /Users/andrew/notes ? Eg, is /Users a symlink, or /Users/andrew a symlink, or //Users/andrew/notes itself symlinked to elsewhere?* + +**No** + +*You say it's only failing for some files. Do the filenames that it's failing on contain any non-ascii characters?* + +**They seem normal.** + +*So, please paste in git annex version and git annex info output.* + + git-annex version: 6.20161110-gd48f4ca + build flags: Assistant Webapp Pairing Testsuite S3(multipartupload)(storageclasses) WebDAV FsEvents XMPP ConcurrentOutput TorrentParser MagicMime Feeds Quvi + key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 SHA1E SHA1 MD5E MD5 WORM URL + remote types: git gcrypt S3 bup directory rsync web bittorrent webdav tahoe glacier ddar hook external + local repository version: 6 + supported repository versions: 3 5 6 + upgrade supported from repository versions: 0 1 2 3 4 5 + operating system: darwin x86_64 + + repository mode: indirect + trusted repositories: 0 + semitrusted repositories: 10 + 00000000-0000-0000-0000-000000000001 -- web + 00000000-0000-0000-0000-000000000002 -- bittorrent + 22de57a0-c9ca-4bfe-8349-3141b3a87c8f -- Dream Objects [cloud] + 334791ca-c284-4a87-a233-fc29be00d31a -- [disc_May-2-2015_a] + 4c57ac0e-b8fe-4b4b-98d3-fb0a1b6b9657 -- MacBook Air [here] + 6a85150d-6ea2-4ba1-92ce-8f4ef575b8e0 -- prowl MacBook Mini + 896c3d52-427a-41a1-867c-d18e6740d758 -- disc_May_4_2015_1 + 96391b13-3981-430f-ac3b-6210e3d4e759 -- [disc_May-2-2015_b] + b4a41e90-2398-4bba-aaf5-d8f8cd78a5bc -- 2TB USB Drive [usbdrive] + e42b223d-ec04-4ad8-bdf7-8429a45d844c -- disc_May-2-2015_a + untrusted repositories: 0 + transfers in progress: none + available local disk space: 2.32 gigabytes (+1 megabyte reserved) + temporary object directory size: 29.47 megabytes (clean up with git-annex unused) + local annex keys: 4104 + local annex size: 10.53 gigabytes + annexed files in working tree: 6417 + size of annexed files in working tree: 80.75 gigabytes + bloom filter size: 32 mebibytes (0.8% full) + backend usage: + SHA256E: 6417 + +"""]] diff --git a/doc/forum/two-way_assistant_sync_with_ssh_special_remote.mdwn b/doc/forum/two-way_assistant_sync_with_ssh_special_remote.mdwn new file mode 100644 index 000000000..ca04e442c --- /dev/null +++ b/doc/forum/two-way_assistant_sync_with_ssh_special_remote.mdwn @@ -0,0 +1,32 @@ +I am attempting to set up automatic two-way synchronization between my laptop and a server via ssh by running assistant on both machines. I want to have both machines be non-bare and unlocked. + +On the rhel server: + + $ mkdir ~/annex + $ cd ~/annex + $ git init + $ git annex init u --version=6 + $ echo This is test file 1. >testfile1.txt + $ git annex add testfile1.txt + $ git annex sync + $ git remote add ml2 ssh://laptop/Users/username/annex + $ git annex adjust --unlock + $ git annex wanted . standard + $ git annex group . client + +On my mac laptop: + + $ cd ~/ + $ git clone ssh://server/home/username/annex + $ cd annex + $ git annex init ml2 --version=6 + $ git annex sync + $ git annex adjust --unlock + $ git annex wanted . standard + $ git annex group . client + +Everything seems to work when I manually sync. But when I run + + $ git annex assistant + +on both machines, I only get one-way automatic synchronization. Changes on the laptop are immediately propagated to the server. But changes on the server do not show up on the laptop until I manually sync. What am I doing wrong? diff --git a/doc/forum/two-way_assistant_sync_with_ssh_special_remote/comment_1_d42def5dfc1cf814fdb07f7cf808bb12._comment b/doc/forum/two-way_assistant_sync_with_ssh_special_remote/comment_1_d42def5dfc1cf814fdb07f7cf808bb12._comment new file mode 100644 index 000000000..a9db03580 --- /dev/null +++ b/doc/forum/two-way_assistant_sync_with_ssh_special_remote/comment_1_d42def5dfc1cf814fdb07f7cf808bb12._comment @@ -0,0 +1,24 @@ +[[!comment format=mdwn + username="binx" + avatar="http://cdn.libravatar.org/avatar/1c2b6fe37ed500f4b72c105e42e81ba9" + subject="comment 1" + date="2016-12-16T13:57:41Z" + content=""" +I updated git-annex on the server. I now *sometimes* get automatic two-way syncing to happen with assistant. But it is not really consistent or reliable. One thing that seems to consistently fail is when I drag and drop a file into the annex folder on the mac. When I drag filename.txt into ~/annex, and then run git annex sync, I get the message: + + $ git annex sync + commit + On branch adjusted/master(unlocked) + Your branch is up-to-date with 'origin/adjusted/master(unlocked)'. + Untracked files: + .DS_Store + filename.txt + + nothing added to commit but untracked files present + ok + pull u + ok + pull origin + ok + +"""]] diff --git a/doc/forum/uuid_mismatch___59___expected_Just___40__UUID_...___41___but_remote_gitrepo_has_UUID_.../comment_3_a681a4847acbe890c4e486288b3c81d3._comment b/doc/forum/uuid_mismatch___59___expected_Just___40__UUID_...___41___but_remote_gitrepo_has_UUID_.../comment_3_a681a4847acbe890c4e486288b3c81d3._comment new file mode 100644 index 000000000..d92f3fe17 --- /dev/null +++ b/doc/forum/uuid_mismatch___59___expected_Just___40__UUID_...___41___but_remote_gitrepo_has_UUID_.../comment_3_a681a4847acbe890c4e486288b3c81d3._comment @@ -0,0 +1,19 @@ +[[!comment format=mdwn + username="yomguy" + avatar="http://cdn.libravatar.org/avatar/03db077c04f8b753f3f504d9a2b06a29" + subject="comment 3" + date="2016-11-18T14:00:51Z" + content=""" +Hi joey, + +After modifying the gcrypt-id as you proposed, I have finally managed to clone the repo with + +`git clone gcrypt::ssh://my.domain/home/admin/` + +But now I get only unresolved symbolic links for each files, that is .git/annex/objects directory only contains .map files. + +Would you have an idea about the reason/source of this behavior? + +Thank you so much, +Guillaume +"""]] diff --git a/doc/forum/vanilla_git_repo_as_special_remote__63__.mdwn b/doc/forum/vanilla_git_repo_as_special_remote__63__.mdwn new file mode 100644 index 000000000..94fa54865 --- /dev/null +++ b/doc/forum/vanilla_git_repo_as_special_remote__63__.mdwn @@ -0,0 +1,27 @@ +Right now I have separate "normal" Git repositories and separate Git annex repositories and I would love to have Git annex track and sync everything for me. The problem I have is I'd like to use "real" Git content tracking for some data (ex: text files) where I'd like to get normal Git features with (ex: diff). I'd like to combine normal Git content tracking with Git annex location tracking and syncing if possible. Ideally the cost (ex: increased git repo size and git slowdown) of content tracking would not need to be propagated across the entire git annex network, just on repos that want it (just like git annex only copies content to clients who want it and symlink the rest). + +The largefiles config provides a mechanism to add content to git directly in git annex, but that cost would be applied across the entire network, not opt-in per client. + +Ideally I'd like this situation: + +1. Git annex tracking everything as symlinks. No content is checked into these git repos. +2. A subset of git annex content (ex: subfolder) synced to a normal remote non-annex git repository (ex: GitHub). This Git repo has content tracked in git itself. + +And I could use the git annex repos to sync everything. Somehow the git annex repo would know that the #2 remote was a "special content git remote" and push any content updates as normal git content commits. + +Or an adjusted branch that had the content tracked and I could sync that content branch around to only the remotes where I wanted the content history stored in git (since adjusted branches don't seem to annex sync by default). But master would just track the symlinks of those files and be synced around to all annexes. + +Can adjusted branches do this somehow? + +Some references: + +* [[special_remotes/external]] +* [[design/adjusted_branches]] +* [[todo/hide_missing_files]] +* [[tips/largefiles]] +* [[submodules]] +* [[forum/git-subtree_support__63__]] + +Thanks! + +-neocryptek diff --git a/doc/forum/vanilla_git_repo_as_special_remote__63__/comment_1_67e186265ae21f2cd8451750152f2a6d._comment b/doc/forum/vanilla_git_repo_as_special_remote__63__/comment_1_67e186265ae21f2cd8451750152f2a6d._comment new file mode 100644 index 000000000..65397495a --- /dev/null +++ b/doc/forum/vanilla_git_repo_as_special_remote__63__/comment_1_67e186265ae21f2cd8451750152f2a6d._comment @@ -0,0 +1,13 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 1""" + date="2016-11-16T18:30:12Z" + content=""" +You can use bup as a special remote, which will store the content in a git +repository. But, not in a form that git diff can be used with. + +[[git-annex-diffdriver]] can be used to make `git diff` work on annexed +files. For example: + + export GIT_EXTERNAL_DIFF="git-annex diffdriver -- diff -u --" +"""]] diff --git a/doc/forum/vanilla_git_repo_as_special_remote__63__/comment_2_6314256da98966f4c7d02aa0d6bf94ff._comment b/doc/forum/vanilla_git_repo_as_special_remote__63__/comment_2_6314256da98966f4c7d02aa0d6bf94ff._comment new file mode 100644 index 000000000..f46355403 --- /dev/null +++ b/doc/forum/vanilla_git_repo_as_special_remote__63__/comment_2_6314256da98966f4c7d02aa0d6bf94ff._comment @@ -0,0 +1,17 @@ +[[!comment format=mdwn + username="neocryptek@659edac901ffbc8e541a974f8f18987eeafc63bd" + nickname="neocryptek" + avatar="http://cdn.libravatar.org/avatar/d9bfdefa9b503f1ac4844a686618374e" + subject="comment 2" + date="2016-11-21T22:26:52Z" + content=""" +Right, though bup also requires installation on the server. I'm looking for a way to store content into a vanilla git repo (as I don't have permission to install anything custom on the server). + +Since I want to store the content outside of git annex, it feels like a special remote. Though ideally it would have human readable files like: + +* <https://git-annex.branchable.com/todo/dumb__44___unsafe__44___human-readable_backend/> + +But since it's git and not just a normal (single version) filesystem, it could dedupe and save previous versions. Is there an easy way to hook git up safely to the external remote protocol: + +* [[special_remotes/external]] +"""]] diff --git a/doc/git-annex-add.mdwn b/doc/git-annex-add.mdwn index b65ed5132..15bb8a6a0 100644 --- a/doc/git-annex-add.mdwn +++ b/doc/git-annex-add.mdwn @@ -15,9 +15,9 @@ If no path is specified, adds files from the current directory and below. Files that are already checked into git and are unmodified, or that git has been configured to ignore will be silently skipped. -If annex.largefiles is configured, and does not match a file, `git annex -add` will behave the same as `git add` and add the non-large file directly -to the git repository, instead of to the annex. +If annex.largefiles is configured, and does not match a file, +`git annex add` will behave the same as `git add` and add the +non-large file directly to the git repository, instead of to the annex. Large files are added to the annex in locked form, which prevents further modification of their content unless unlocked by [[git-annex-unlock]](1). diff --git a/doc/git-annex-enable-tor.mdwn b/doc/git-annex-enable-tor.mdwn new file mode 100644 index 000000000..f06966400 --- /dev/null +++ b/doc/git-annex-enable-tor.mdwn @@ -0,0 +1,36 @@ +# NAME + +git-annex enable-tor - enable tor hidden service + +# SYNOPSIS + +git annex enable-tor + +sudo git annex enable-tor $(id -u) + +# DESCRIPTION + +This command enables a tor hidden service for git-annex. + +It modifies `/etc/tor/torrc` to register the hidden service. If run as a +normal user, it will try to use sudo/su/etc to get root access to modify +that file. If you run it as root, pass it your non-root user id number, +as output by `id -u` + +After this command is run, `git annex remotedaemon` can be run to serve the +tor hidden service, and then `git-annex p2p --gen-address` can be run to +give other users access to your repository via the tor hidden service. + +# SEE ALSO + +[[git-annex]](1) + +[[git-annex-p2p-auth]](1) + +[[git-annex-remotedaemon]](1) + +# AUTHOR + +Joey Hess <id@joeyh.name> + +Warning: Automatically converted into a man page by mdwn2man. Edit with care. diff --git a/doc/git-annex-fromkey.mdwn b/doc/git-annex-fromkey.mdwn index 461f42eb6..2591e9785 100644 --- a/doc/git-annex-fromkey.mdwn +++ b/doc/git-annex-fromkey.mdwn @@ -4,14 +4,16 @@ git-annex fromkey - adds a file using a specific key # SYNOPSIS -git annex fromkey `[key file]` +git annex fromkey `[key file ...]` # DESCRIPTION This plumbing-level command can be used to manually set up a file in the git repository to link to a specified key. -If the key and file are not specified on the command line, they are +Multiple pairs of file and key can be given in a single command line. + +If no key and file pair are specified on the command line, they are instead read from stdin. Any number of lines can be provided in this mode, each containing a key and filename, separated by a single space. @@ -26,7 +28,7 @@ to do that. * `--force` Allow making a file link to a key whose content is not in the local - repository. The key may not be known to git-annex at all. + repository. The key may not be known to git-annex at all. # SEE ALSO diff --git a/doc/git-annex-map.mdwn b/doc/git-annex-map.mdwn index cf28a958e..ece26b367 100644 --- a/doc/git-annex-map.mdwn +++ b/doc/git-annex-map.mdwn @@ -10,8 +10,8 @@ git annex map Helps you keep track of your repositories, and the connections between them, by going out and looking at all the ones it can get to, and generating a -Graphviz file displaying it all. If the `dot` command is available, it is -used to display the file to your screen (using x11 backend). +Graphviz file displaying it all. If the `xdot` or `dot` command is available, +it is used to display the file to your screen. This command only connects to hosts that the host it's run on can directly connect to. It does not try to tunnel through intermediate hosts. @@ -37,7 +37,7 @@ on that host. * `--fast` - Disable using `dot` to display the generated Graphviz file. + Don't display the generated Graphviz file, but save it for later use. # SEE ALSO diff --git a/doc/git-annex-p2p.mdwn b/doc/git-annex-p2p.mdwn new file mode 100644 index 000000000..127ed9a5d --- /dev/null +++ b/doc/git-annex-p2p.mdwn @@ -0,0 +1,73 @@ +# NAME + +git-annex p2p - configure peer-2-peer links between repositories + +# SYNOPSIS + +git annex p2p [options] + +# DESCRIPTION + +This command can be used to link git-annex repositories over peer-2-peer +networks. + +Currently, the only P2P network supported by git-annex is Tor hidden +services. + +# OPTIONS + +* `--pair` + + Run this in two repositories to pair them together over the P2P network. + + This will print out a code phrase, like "3-mango-elephant", and + will prompt for you to enter the code phrase from the other repository. + + Once code phrases have been exchanged, the two repositories will + be paired. A git remote will be created for the other repository, + with a name like "peer1". + + This uses [Magic Wormhole](https://github.com/warner/magic-wormhole) + to verify the code phrases and securely communicate the P2P addresses of + the repositories, so you will need it installed on both computers that are + being paired. + +* `--gen-address` + + Generates addresses that can be used to access this git-annex repository + over the available P2P networks. The address or addresses is output to + stdout. + + Note that anyone who knows these addresses can access your + repository over the P2P networks. + +* `--link` + + Sets up a git remote that is accessed over a P2P network. + + This will prompt for an address to be entered; you should paste in the + address that was generated by --gen-address in the remote repository. + + Defaults to making the git remote be named "peer1", "peer2", + etc. This can be overridden with the `--name` option. + +* `--name` + + Specify a name to use when setting up a git remote with `--link` + or `--pair`. + +# SEE ALSO + +[[git-annex]](1) + +[[git-annex-enable-tor]](1) + +[[git-annex-remotedaemon]](1) + +wormhole(1) + +# AUTHOR + +Joey Hess <id@joeyh.name> + +Warning: Automatically converted into a man page by mdwn2man. Edit with care. diff --git a/doc/git-annex-rekey.mdwn b/doc/git-annex-rekey.mdwn index 7dbe6ae96..ce5e43d41 100644 --- a/doc/git-annex-rekey.mdwn +++ b/doc/git-annex-rekey.mdwn @@ -20,7 +20,11 @@ Multiple pairs of file and key can be given in a single command line. Allow rekeying of even files whose content is not currently available. Use with caution. -# OPTIONS +* `--batch` + + Enables batch mode, in which lines are read from stdin. + Each line should contain the file, and the new key to use for that file, + separated by a single space. # SEE ALSO diff --git a/doc/git-annex-remotedaemon.mdwn b/doc/git-annex-remotedaemon.mdwn index 69b516283..b01002dc9 100644 --- a/doc/git-annex-remotedaemon.mdwn +++ b/doc/git-annex-remotedaemon.mdwn @@ -1,6 +1,6 @@ # NAME -git-annex remotedaemon - detects when remotes have changed, and fetches from them +git-annex remotedaemon - persistent communication with remotes # SYNOPSIS @@ -8,18 +8,38 @@ git annex remotedaemon # DESCRIPTION -This plumbing-level command is used by the assistant to detect -when remotes have received git pushes, so the changes can be promptly -fetched and the local repository updated. +The remotedaemon provides persistent communication with remotes. +It detects when git branches on remotes have changes, and fetches +the changes from them. -This is a better alternative to the [[git-annex-xmppgit]](1) -hack. +The assistant runs the remotedaemon and communicates with it on +stdio using a simple textual protocol. -For the remotedaemon to work, the git remote must have -[[git-annex-shell]](1) installed, with notifychanges support. -The first version of git-annex-shell that supports it is 5.20140405. +Several types of remotes are supported: -It's normal for this process to be running when the assistant is running. +For ssh remotes, the remotedaemon tries to maintain a connection to the +remote git repository, and uses git-annex-shell notifychanges to detect +when the remote git repository has changed. For this to work, the git +remote must have [[git-annex-shell]](1) installed, with notifychanges +support. The first version of git-annex-shell that supports it is +5.20140405. + +For tor-annex remotes, the remotedaemon runs a tor hidden service, +accepting connections from other nodes and serving up the contents of the +repository. This is only done if you first run `git annex enable-tor`. +Use `git annex p2p` to configure access to tor-annex remotes. + +# OPTIONS + +* `--foreground` + +Don't fork to the background, and communicate on stdin/stdout using a +simple textual protocol. The assistant runs the remotedaemon this way. + +Commands in the protocol include LOSTNET, which tells the remotedaemon +that the network connection has been lost, and causes it to stop any TCP +connctions. That can be followed by RESUME when the network connection +comes back up. # SEE ALSO @@ -27,6 +47,10 @@ It's normal for this process to be running when the assistant is running. [[git-annex-assistant]](1) +[[git-annex-enable-tor]](1) + +[[git-annex-p2p]](1) + # AUTHOR Joey Hess <id@joeyh.name> diff --git a/doc/git-annex-rmurl.mdwn b/doc/git-annex-rmurl.mdwn index 5faf9ea39..504685a58 100644 --- a/doc/git-annex-rmurl.mdwn +++ b/doc/git-annex-rmurl.mdwn @@ -4,12 +4,20 @@ git-annex rmurl - record file is not available at url # SYNOPSIS -git annex rmurl `file url` +git annex rmurl `[file url ..]` # DESCRIPTION Record that the file is no longer available at the url. +# OPTIONS + +* `--batch` + + Enables batch mode, in which lines are read from stdin. + Each line should contain the file, and the url to remove from that file, + separated by a single space. + # SEE ALSO [[git-annex]](1) diff --git a/doc/git-annex.mdwn b/doc/git-annex.mdwn index d71076087..ca1ac3620 100644 --- a/doc/git-annex.mdwn +++ b/doc/git-annex.mdwn @@ -212,6 +212,12 @@ subdirectories). See [[git-annex-enableremote]](1) for details. +* `enable-tor` + + Sets up tor hidden service. + + See [[git-annex-enable-tor]](1) for details. + * `numcopies [N]` Configure desired number of copies. @@ -379,6 +385,12 @@ subdirectories). See [[git-annex-repair]](1) for details. +* `p2p` + + Configure peer-2-Peer links between repositories. + + See [[git-annex-p2p]](1) for details. + # QUERY COMMANDS * `find [path ...]` diff --git a/doc/git-remote-tor-annex.mdwn b/doc/git-remote-tor-annex.mdwn new file mode 100644 index 000000000..4e41de877 --- /dev/null +++ b/doc/git-remote-tor-annex.mdwn @@ -0,0 +1,36 @@ +# NAME + +git-remote-tor-annex - remote helper program to talk to git-annex over tor + +# SYNOPSIS + +git fetch tor-annex::address.onion:port + +git remote add tor tor-annex::address.onion:port + +# DESCRIPTION + +This is a git remote helper program that allows git to pull and push +over tor(1), communicating with a tor hidden service. + +The tor hidden service probably requires an authtoken to use it. +The authtoken can be provided in the environment variable +`GIT_ANNEX_P2P_AUTHTOKEN`. Or, if there is a file in +`.git/annex/creds/` matching the onion address of the hidden +service, its first line is used as the authtoken. + +# SEE ALSO + +git-remote-helpers(1) + +[[git-annex]](1) + +[[git-annex-enable-tor]](1) + +[[git-annex-remotedaemon]](1) + +# AUTHOR + +Joey Hess <id@joeyh.name> + +Warning: Automatically converted into a man page by mdwn2man. Edit with care. diff --git a/doc/how_it_works.mdwn b/doc/how_it_works.mdwn index 69e5256e3..21fa39ea7 100644 --- a/doc/how_it_works.mdwn +++ b/doc/how_it_works.mdwn @@ -1,8 +1,11 @@ -This page gives a high-level view of git-annex. For a detailed +This page gives a high-level view of how git-annex works. For a detailed low-level view, see [[the_man_page|git-annex]] and [[internals]]. You do not need to read this page to get started with using git-annex. The -[[walkthrough]] provides step-by-step instructions. +[[walkthrough]] provides step-by-step examples, and [[workflow]] discusses +different ways you can use git-annex. + +---- Still reading? Ok. Git's man page calls it "a stupid content tracker". With git-annex, git is instead "a stupid filename and metadata" diff --git a/doc/install/OSX.mdwn b/doc/install/OSX.mdwn index 10fc8bad4..082fd517d 100644 --- a/doc/install/OSX.mdwn +++ b/doc/install/OSX.mdwn @@ -18,7 +18,7 @@ several more. Handy if you don't otherwise have git installed. ## autobuilds -Thanks to Dartmouth for hosting the autobuilder. +Thanks to Dartmouth College for hosting the autobuilder. * [autobuild of git-annex.dmg](https://downloads.kitenet.net/git-annex/autobuild/x86_64-apple-yosemite/git-annex.dmg) ([build logs](https://downloads.kitenet.net/git-annex/autobuild/x86_64-apple-yosemite/)) diff --git a/doc/install/Windows.mdwn b/doc/install/Windows.mdwn index 7ea667a10..1d9599ac8 100644 --- a/doc/install/Windows.mdwn +++ b/doc/install/Windows.mdwn @@ -1,8 +1,12 @@ git-annex now does Windows! -* First, [install Git for Windows](http://git-scm.com/downloads) +* First, [install Git for Windows](http://git-scm.com/downloads) + Important: **Get the 32 bit version not the 64 bit version.** - (Note that msysgit is no longer supported.) + If you installed the 64 bit version of git, then parts of git-annex will + still run, however, some features, including tools like rsync, will + not work. + * Then, [install git-annex](https://downloads.kitenet.net/git-annex/windows/current/) This port is now in reasonably good shape for command-line use of @@ -18,9 +22,9 @@ important thing is that it should end with "All tests passed". ## autobuilds A daily build is also available, thanks to Yury V. Zaytsev and -[NEST](http://nest-initiative.org/). +Dartmouth College. -* [download](https://downloads.kitenet.net/git-annex/autobuild/windows/) ([build logs](https://qa.nest-initiative.org/view/msysGit/job/msysgit-git-annex-assistant-test/)) +* [download](https://downloads.kitenet.net/git-annex/autobuild/windows/) ## building it yourself diff --git a/doc/install/fromsource.mdwn b/doc/install/fromsource.mdwn index c46321099..7973e3dc9 100644 --- a/doc/install/fromsource.mdwn +++ b/doc/install/fromsource.mdwn @@ -83,7 +83,7 @@ Get the git-annex source code, and inside the source tree, run: To build with all features enabled, including the assistant and webapp, you will need to install several C libraries and their headers, -including libgnutls, libgsasl, libxml2, libmagic, and zlib. How to do +including libgnutls, libgsasl, libxml2, libmagic, zlib, and chrpath. How to do that for your OS is beyond the scope of this page. Once the C libraries are installed, run inside the source tree: diff --git a/doc/links/key_concepts.mdwn b/doc/links/key_concepts.mdwn index f1754e0c8..3dc2f6c5d 100644 --- a/doc/links/key_concepts.mdwn +++ b/doc/links/key_concepts.mdwn @@ -3,5 +3,6 @@ * [[git-annex man page|git-annex]] * [[how_it_works]] * [[special_remotes]] +* [[workflows|workflow]] * [[sync]] * [[direct_mode]] diff --git a/doc/metadata.mdwn b/doc/metadata.mdwn index f1f0ceab7..3b61a47d5 100644 --- a/doc/metadata.mdwn +++ b/doc/metadata.mdwn @@ -13,6 +13,8 @@ Some of the things you can do with metadata include: For example `git annex find --metadata tag=foo --or --metadata tag=bar` * Using it in [[preferred_content]] expressions. For example "metadata=tag=important or not metadata=author=me" +* Editing and viewing it with git-annex-metadata-gui, + [[tips/a_gui_for_metadata_operations]]. Each file (actually the underlying key) can have any number of metadata fields, which each can have any number of values. For example, to tag diff --git a/doc/news/version_6.20161012.mdwn b/doc/news/version_6.20161012.mdwn deleted file mode 100644 index a6cb01780..000000000 --- a/doc/news/version_6.20161012.mdwn +++ /dev/null @@ -1,30 +0,0 @@ -git-annex 6.20161012 released with [[!toggle text="these changes"]] -[[!toggleable text=""" - * Optimisations to time it takes git-annex to walk working tree and find - files to work on. Sped up by around 18%. - * Optimisations to git-annex branch query and setting, avoiding repeated - copies of the environment. Speeds up commands like - "git-annex find --in remote" by over 50%. - * Optimised git-annex branch log file timestamp parsing. - * Add "total-size" field to --json-progress output. - * Make --json-progress output be shown even when the size of a object - is not known. - * Multiple external special remote processes for the same remote will be - started as needed when using -J. This should not beak any existing - external special remotes, because running multiple git-annex commands - at the same time could already start multiple processes for the same - external special remotes. - * Linux standalone: Include locale files in the bundle, and generate - locale definition files for the locales in use when starting runshell. - (Currently only done for utf-8 locales.) - * Avoid using a lot of memory when large objects are present in the git - repository and have to be checked to see if they are a pointed to an - annexed file. Cases where such memory use could occur included, but - were not limited to: - - git commit -a of a large unlocked file (in v5 mode) - - git-annex adjust when a large file was checked into git directly - * When auto-upgrading a v3 remote, avoid upgrading to version 6, - instead keep it at version 5. - * Support using v3 repositories without upgrading them to v5. - * sync: Fix bug in adjusted branch merging that could cause recently - added files to be lost when updating the adjusted branch."""]]
\ No newline at end of file diff --git a/doc/news/version_6.20161118.mdwn b/doc/news/version_6.20161118.mdwn new file mode 100644 index 000000000..42d86282c --- /dev/null +++ b/doc/news/version_6.20161118.mdwn @@ -0,0 +1,17 @@ +git-annex 6.20161118 released with [[!toggle text="these changes"]] +[[!toggleable text=""" + * git-annex.cabal: Loosen bounds on persistent to allow 2.5, which + on Debian has been patched to work with esqueleto. + This may break cabal's resolver on non-Debian systems; + if so, either use stack to build, or run cabal with + --constraint='persistent ==2.2.4.1' + Hopefully this mess with esqueleto will be resolved soon. + * sync: Pass --allow-unrelated-histories to git merge when used with git + git 2.9.0 or newer. This makes merging a remote into a freshly created + direct mode repository work the same as it works in indirect mode. + * Avoid backtraces on expected failures when built with ghc 8; + only use backtraces for unexpected errors. + * fsck --all --from was checking the existence and content of files + in the local repository, rather than on the special remote. Oops. + * Linux arm standalone: Build with a 32kb page size, which is needed + on several ARM NAS devices, including Drobo 5N, and WD NAS."""]]
\ No newline at end of file diff --git a/doc/news/version_6.20161210.mdwn b/doc/news/version_6.20161210.mdwn new file mode 100644 index 000000000..345d4fe4c --- /dev/null +++ b/doc/news/version_6.20161210.mdwn @@ -0,0 +1,31 @@ +git-annex 6.20161210 released with [[!toggle text="these changes"]] +[[!toggleable text=""" + * Linux standalone: Updated ghc to fix its "unable to decommit memory" + bug, which may have resulted in data loss when these builds were used + with Linux kernels older than 4.5. + * enable-tor: New command, enables tor hidden service for P2P syncing. + * p2p: New command, allows linking repositories using a P2P network. + * remotedaemon: Serve tor hidden service. + * Added git-remote-tor-annex, which allows git pull and push to the tor + hidden service. + * remotedaemon: Fork to background by default. Added --foreground switch + to enable old behavior. + * addurl: Fix bug in checking annex.largefiles expressions using + largerthan, mimetype, and smallerthan; the first two always failed + to match, and the latter always matched. + * Relicense 5 source files that are not part of the webapp from AGPL to GPL. + * map: Run xdot if it's available in PATH. On OSX, the dot command + does not support graphical display, while xdot does. + * Debian: xdot is a better interactive viewer than dot, so Suggest + xdot, rather than graphviz. + * rmurl: Multiple pairs of files and urls can be provided on the + command line. + * rmurl: Added --batch mode. + * fromkey: Accept multiple pairs of files and keys. + Thanks, Daniel Brooks. + * rekey: Added --batch mode. + * add: Stage modified non-large files when running in indirect mode. + (This was already done in v6 mode and direct mode.) + * git-annex-shell, remotedaemon, git remote: Fix some memory DOS attacks. + * Fix build with http-client 0.5. + Thanks, Alper Nebi Yasak."""]] diff --git a/doc/related_software.mdwn b/doc/related_software.mdwn index bfc68186c..acd9df838 100644 --- a/doc/related_software.mdwn +++ b/doc/related_software.mdwn @@ -16,6 +16,10 @@ designed to interoperate with it. * [Magit](http://github.com/magit/magit), an Emacs mode for Git, has [an extension](https://github.com/magit/magit-annex) for git annex. +* [git-annex-metadata-gui](https://github.com/alpernebbi/git-annex-metadata-gui) + is a GUI for editing the metadata. An easier alternative to using + [[git-annex-metadata]] at the command line. + * [DataLad](http://datalad.org/) uses git-annex to provide access to scientific data available from various sources. @@ -42,4 +46,7 @@ designed to interoperate with it. * [git-annex-watcher](https://github.com/rubiojr/git-annex-watcher) is a status icon for your desktop. +* [git-annex-adaptor](https://github.com/alpernebbi/git-annex-adapter) + is a python interface to git-annex. + See also [[not]] for software that is *not* related to git-annex, but similar. diff --git a/doc/special_remotes.mdwn b/doc/special_remotes.mdwn index 2d99069be..1dc3d8705 100644 --- a/doc/special_remotes.mdwn +++ b/doc/special_remotes.mdwn @@ -18,6 +18,7 @@ They cannot be used by other git commands though. * [[tahoe]] * [[web]] * [[bittorrent]] +* [[tor]] * [[xmpp]] * [[hook]] diff --git a/doc/special_remotes/S3/comment_28_c4dafad82a898eafd6d9e3703fad2c48._comment b/doc/special_remotes/S3/comment_28_c4dafad82a898eafd6d9e3703fad2c48._comment new file mode 100644 index 000000000..864974205 --- /dev/null +++ b/doc/special_remotes/S3/comment_28_c4dafad82a898eafd6d9e3703fad2c48._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="David_K" + avatar="http://cdn.libravatar.org/avatar/09dd8544695feb9b8d8ee54e4ff0168d" + subject="comment 28" + date="2016-11-16T01:28:14Z" + content=""" +I'd like to reiterate a question that was unanswered above: + +Is there a way to tell the S3 backend to store the files as they are named locally, instead of by hashed content name? i.e., I've annexed foo/bar.txt and annex puts it in s3 as mybucket.name/foo/bar.txt instead of mybucket.name/GPGHMACSHA1-random.txt + + +"""]] diff --git a/doc/special_remotes/rsync/comment_14_2261b1b7441eff9e28ec8e1f98d77980._comment b/doc/special_remotes/rsync/comment_14_2261b1b7441eff9e28ec8e1f98d77980._comment new file mode 100644 index 000000000..4d6f0cfc2 --- /dev/null +++ b/doc/special_remotes/rsync/comment_14_2261b1b7441eff9e28ec8e1f98d77980._comment @@ -0,0 +1,9 @@ +[[!comment format=mdwn + username="davidriod@e75b369a4b1cced29c14354bce7493c61f00b1c7" + nickname="davidriod" + avatar="http://cdn.libravatar.org/avatar/d6e327bd88b88802d6f0c20c83f682a2" + subject="Sharing rsync special remote between repository" + date="2016-11-24T19:23:42Z" + content=""" +I was wondering if it is possible to share a rsync special remote between repository which are not parented in any way. The use case would be that even if these repositories are not related at all they still may contains the same binary file. It would be useful to have a single rsync remote in order to reduce space usage. I think it could work as the object names are based on their checksum, but I wonder if anyone has already try that ? +"""]] diff --git a/doc/special_remotes/rsync/comment_15_a4a0491a7dcee2e7b7786127518866af._comment b/doc/special_remotes/rsync/comment_15_a4a0491a7dcee2e7b7786127518866af._comment new file mode 100644 index 000000000..2f68c3f57 --- /dev/null +++ b/doc/special_remotes/rsync/comment_15_a4a0491a7dcee2e7b7786127518866af._comment @@ -0,0 +1,22 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 15""" + date="2016-12-13T16:43:42Z" + content=""" +@davidriod you can do things like this with special remotes, as long +as the special remotes are not encrypted. + +I don't really recommend it. With such a shared special remote R and two +disconnected git repos -- call them A and B, some confusing situations can +occur. For example, the only copies of some files may be +on special remote R and git repo B. A knows about the copy in R, so +git-annex is satisfied there is one copy of the file. But now, B can drop +the content from R, which is allowed as the content is in B. A is then left +unable to recover the content of the files at all, since they have been +removed from R. + +Better to connect the two repositories A and B, even if you do work in +two separate branches. Then if a file ends up located only on B, A will be +able to say where it is, and could even get it from B (if B was set up as a +remote). +"""]] diff --git a/doc/special_remotes/tor.mdwn b/doc/special_remotes/tor.mdwn new file mode 100644 index 000000000..12d3dfedf --- /dev/null +++ b/doc/special_remotes/tor.mdwn @@ -0,0 +1,10 @@ +git-annex can communicate over the Tor network. This allows direct +communication between git-annex repositories, no matter where they are +located. + +A git remote using tor has an url that looks like +`tor-annex::2lssjzicvsxkdc2v.onion:19984` + +To set this up, use [[git-annex-enabletor]] and [[git-annex-p2p]], +and run [[git-annex-remotedaemon]] to serve the Tor hidden service. +It's explained in detail in [[tips/peer_to_peer_network_with_tor]]. diff --git a/doc/sync.mdwn b/doc/sync.mdwn index 0250d2fef..cddccd112 100644 --- a/doc/sync.mdwn +++ b/doc/sync.mdwn @@ -42,3 +42,5 @@ repositories, but does not transfer the content of annexed files. If you want to fully synchronise two repositories content, you can use `git annex sync --content`. You can also configure [[preferred_content]] settings to make only some content be synced. + +See [[git-annex-sync]] for the command's man page. diff --git a/doc/thanks.mdwn b/doc/thanks.mdwn index 645cca732..eec468606 100644 --- a/doc/thanks.mdwn +++ b/doc/thanks.mdwn @@ -1,6 +1,6 @@ The development of git-annex was made possible by the generous donations of many people. I want to say "Thank You!" to each of -you individually, but until I meet all 1500 of you, this page will have to +you individually, but until I meet all 1500+ of you, this page will have to do. You have my most sincere thanks. --[[Joey]] You can support my git-annex development @@ -18,16 +18,8 @@ git-annex development is partially supported by the [NSF](https://www.nsf.gov/awardsearch/showAward?AWD_ID=1429999) as a part of the [DataLad project](http://datalad.org/). -Thanks also to these folks for their support: martin f. krafft, John Byrnes, -Aurélien Couderc, Jeffrey Chiu, Amitai Schlair, Andreas, Anthony DeRobertis, -Baldur Kristinsson, Boyd Stephen Smith Jr., Brock Spratlen, Christian Diller, -Denis Dzyubenko, Eskild Hustvedt, Evgeni Kunev, FBC, Fernando Jimenez, Greg -Young, Henrik RIomar, Ignacio, Jake Vosloo, James Valleroy, Jason Woofenden, -Jeff Goeke-Smith, Jim, Jochen Bartl, Johannes Schlatow, John Carr, Josh -Taylor, Josh Triplett, Kuno Woudt, Matthias Urlichs, Mattias J, Nathan Howell, -Nick Daly, Nicolas Schodet, Ole-Morten Duesund, Remy van Elst, RémiV, Thom -May, Thomas Ferris Nicolaisen, Thomas Hochstein, Tyler Cipriani, encryptio, -Øyvind A. Holm +Thanks also to these folks for their support: +[[!inline raw=yes pages="thanks/list"]] and anonymous supporters. ## 2013-2014 @@ -385,13 +377,11 @@ Tyree, Aaron Whitehouse * Rsync.net, for providing me a free account so I can make sure git-annex works well with it. * LeastAuthority.com, for providing me a free Tahoe-LAFS grid account, - so I can test git-annex with that, and back up the git-annex assistant - screencasts. + so I can test git-annex with that. +* Yury V. Zaytsev for running the Windows autobuilder. +* Kevin McKenzie for providing a OSX account for testing. * Anna and Mark, for the loan of the video camera; as well as the rest of my family, for your support. Even when I couldn't explain what I was working on. -* The Hodges, for providing such a congenial place for me to live and work - on these first world problems, while you're off helping people in the - third world. * And Mom, for stamping and stuffing so many thank you envelopes, and all the rhubarb pies. diff --git a/doc/thanks/list b/doc/thanks/list new file mode 100644 index 000000000..653466f81 --- /dev/null +++ b/doc/thanks/list @@ -0,0 +1,53 @@ +martin f. krafft, +John Byrnes, +Aurélien Couderc, +Jeffrey Chiu, +Amitai Schlair, +Andreas, +Anthony DeRobertis, +Baldur Kristinsson, +Boyd Stephen Smith Jr., +Brock Spratlen, +Christian Diller, +Denis Dzyubenko, +Eskild Hustvedt, +Evgeni Kunev, +FBC, +Fernando Jimenez, +Greg Young, +Henrik RIomar, +Ignacio, +Jake Vosloo, +James Valleroy, +Jason Woofenden, +Jeff Goeke-Smith, +Jim, +Jochen Bartl, +Johannes Schlatow, +John Carr, +Josh Taylor, +Josh Triplett, +Kuno Woudt, +Matthias Urlichs, +Mattias J, +Nathan Howell, +Nick Daly, +Nicolas Schodet, +Ole-Morten Duesund, +Remy van Elst, +RémiV, +Thom May, +Thomas Ferris Nicolaisen, +Thomas Hochstein, +Tyler Cipriani, +encryptio, +Øyvind A. Holm, +Bruno BEAUFILS, +Rémi Vanicat, +Trenton Cronholm, +Francois Marier, +Peter Hogg, +Amitai Schleier, +Brennen Bearnes, +Tim Howes, +Willard Korfhage, diff --git a/doc/tips/How_to_retroactively_annex_a_file_already_in_a_git_repo/comment_7_603db6818d33663b70b917c04fd8485b._comment b/doc/tips/How_to_retroactively_annex_a_file_already_in_a_git_repo/comment_7_603db6818d33663b70b917c04fd8485b._comment new file mode 100644 index 000000000..5527c2b43 --- /dev/null +++ b/doc/tips/How_to_retroactively_annex_a_file_already_in_a_git_repo/comment_7_603db6818d33663b70b917c04fd8485b._comment @@ -0,0 +1,30 @@ +[[!comment format=mdwn + username="https://launchpad.net/~stephane-gourichon-lpad" + nickname="stephane-gourichon-lpad" + avatar="http://cdn.libravatar.org/avatar/02d4a0af59175f9123720b4481d55a769ba954e20f6dd9b2792217d9fa0c6089" + subject=""Hmm, guyz? Are you serious with these scripts?" Well, what's the matter?" + date="2016-11-15T10:58:32Z" + content=""" +## Wow, scary + +Dilyin's comment is scary. It suggests bad things can happen, but is not very clear. + +Bloated history is one thing. +Obviously broken repo is bad but can be (slowly) recovered from remotes. +Subtly crippled history that you don't notice can be a major problem (especially once you have propagated it to all your remotes to \"recover from bloat\"). + +## More common than it seems + +There's a case probably more common than people actually report: mistakenly doing `git add` instead of `git annex add` and realizing it only after a number of commits. Doing `git annex add` at that time will have the file duplicated (regular git and annex). + +Extra wish: when doing `git annex add` of a file that is already present in git history, `git-annex` could notice and tell. + +## Simple solution? + +Can anyone elaborate on the scripts provided here, are they safe? What can happen if improperly used or in corner cases? + +* \"files are replaced with symlinks and are in the index\" -> so what ? +* \"Make sure that you don't have annex.largefiles settings that would prevent annexing the files.\" -> What would happen? Also `.gitattributes`. + +Thank you. +"""]] diff --git a/doc/tips/How_to_retroactively_annex_a_file_already_in_a_git_repo/comment_8_834410421ccede5194bd8fbaccea8d1a._comment b/doc/tips/How_to_retroactively_annex_a_file_already_in_a_git_repo/comment_8_834410421ccede5194bd8fbaccea8d1a._comment new file mode 100644 index 000000000..2c36962aa --- /dev/null +++ b/doc/tips/How_to_retroactively_annex_a_file_already_in_a_git_repo/comment_8_834410421ccede5194bd8fbaccea8d1a._comment @@ -0,0 +1,82 @@ +[[!comment format=mdwn + username="StephaneGourichon" + avatar="http://cdn.libravatar.org/avatar/8cea01af2c7a8bf529d0a3d918ed4abf" + subject="Walkthrough of a prudent retroactive annex." + date="2016-11-24T11:27:59Z" + content=""" +Been using the one-liner. Despite the warning, I'm not dead yet. + +There's much more to do than the one-liner. + +This post offers instructions. + +# First simple try: slow + +Was slow (estimated >600s for 189 commits). + +# In tmpfs: about 6 times faster + +I have cloned repository into /run/user/1000/rewrite-git, which is a tmpfs mount point. (Machine has plenty of RAM.) + +There I also did `git annex init`, git-annex found its state branches. + +On second try I also did + + git checkout -t remotes/origin/synced/master + +So that filter-branch would clean that, too. + +There, `filter-branch` operation finished in 90s first try, 149s second try. + +`.git/objects` wasn't smaller. + +# Practicing reduction on clone + +This produced no visible benefit: + +time git gc --aggressive +time git repack -a -d + +Even cloning and retrying on clone. Oh, but I should have done `git clone file:///path` as said on git-filter-branch man page's section titled \"CHECKLIST FOR SHRINKING A REPOSITORY\" + +This (as seen on https://rtyley.github.io/bfg-repo-cleaner/ ) was efficient: + + git reflog expire --expire=now --all && git gc --prune=now --aggressive + +`.git/objects` shrunk from 148M to 58M + +All this was on a clone of the repo in tmpfs. + +# Propagating cleaned up branches to origin + +This confirmed that filter-branch did not change last tree: + + git diff remotes/origin/master..master + git diff remotes/origin/synced/master synced/master + +This, expectedly, was refused: + + git push origin master + git push origin synced/master + +On origin, I checked out the hash of current master, then on tmpfs clone + + git push -f origin master + git push -f origin synced/master + +Looks good. + +I'm not doing the aggressive shrink now, because of the \"two orders of magnitude more caution than normal filter-branch\" recommended by arand. + +# Now what? Check if precious not broken + +I'm planning to do the same operation on the other repos, then : + +* if everything seems right, +* if `git annex sync` works between all those fellows +* etc, +* then I would perform the reflog expire, gc prune on some then all of them, etc. + +Joey, does this seem okay? Any comment? + +"""]] diff --git a/doc/tips/a_gui_for_metadata_operations.mdwn b/doc/tips/a_gui_for_metadata_operations.mdwn new file mode 100644 index 000000000..1e1180068 --- /dev/null +++ b/doc/tips/a_gui_for_metadata_operations.mdwn @@ -0,0 +1,13 @@ +Hey everyone. + +I wrote a GUI for git-annex metadata in Python: [git-annex-metadata-gui](https://github.com/alpernebbi/git-annex-metadata-gui). +It shows the files that are in the current branch (only those in the annex) in the respective folder hierarchy. +The keys that are in the repository, but not in the current branch are also shown in another tab. +You can view, edit or remove fields for individual files with support for multiple values for fields. +There is a file preview for image and text files as well. +I uploaded some screenshots in the repository to show it in action. + +While making it, I decided to move the git-annex calls into its own Python package, +which became [git-annex-adapter](https://github.com/alpernebbi/git-annex-adapter). + +I hope these can be useful to someone other than myself as well. diff --git a/doc/tips/a_gui_for_metadata_operations/comment_1_1ce311d8328ea370a6a3494adea0f5db._comment b/doc/tips/a_gui_for_metadata_operations/comment_1_1ce311d8328ea370a6a3494adea0f5db._comment new file mode 100644 index 000000000..2a55de0be --- /dev/null +++ b/doc/tips/a_gui_for_metadata_operations/comment_1_1ce311d8328ea370a6a3494adea0f5db._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 1""" + date="2016-12-07T19:58:11Z" + content=""" +Thank you for this, I've always wanted such a GUI, and it's been a common +user request! +"""]] diff --git a/doc/tips/peer_to_peer_network_with_tor.mdwn b/doc/tips/peer_to_peer_network_with_tor.mdwn new file mode 100644 index 000000000..0fdc34625 --- /dev/null +++ b/doc/tips/peer_to_peer_network_with_tor.mdwn @@ -0,0 +1,163 @@ +git-annex has recently gotten support for running as a +[Tor](https://torproject.org/) hidden service. This is a nice secure +and easy to use way to connect repositories in different +locations. No account on a central server is needed; it's peer-to-peer. + +## dependencies + +To use this, you need to get Tor installed and running. See +[their website](https://torproject.org/), or try a command like: + + sudo apt-get install tor + +You also need to install [Magic Wormhole](https://github.com/warner/magic-wormhole). + + sudo apt-get install magic-wormhole + +## pairing two repositories + +You have two git-annex repositories on different computers, and want to +connect them together over Tor so they share their contents. Or, you and a +friend want to connect your repositories together. Pairing is an easy way +to accomplish this. + +In each git-annex repository, run these commands: + + git annex enable-tor + git annex remotedaemon + +The enable-tor command may prompt for the root password, since it +configures Tor. Now git-annex is running as a Tor hidden service, but +it will only talk to peers after pairing with them. + +In both repositories, run this command: + + git annex p2p --pair + +This will print out a pairing code, like "11-incredible-tumeric", +and prompt for you to enter the other repository's pairing code. + +Once the pairing codes are exchanged, the two repositories will be securely +connected to one-another via Tor. Each will have a git remote, with a name +like "peer1", which connects to the other repository. + +Then, you can run commands like `git annex sync peer1 --content` to sync +with the paired repository. + +Pairing connects just two repositories, but you can repeat the process to +pair with as many other repositories as you like, in order to build up +larger networks of repositories. + +## how to exchange pairing codes + +When pairing with a friend's repository, you have to exchange +pairing codes. How to do this securely? + +The pairing codes can only be used once, so it's ok to exchange them in +a way that someone else can access later. However, if someone can overhear +your exchange of codes in real time, they could trick you into pairing +with them. + +Here are some suggestions for how to exchange the codes, +with the most secure ways first: + +* In person. +* In an encrypted message (gpg signed email, Off The Record (OTR) + conversation, etc). +* By a voice phone call. + +## starting git-annex remotedaemon on boot + +Notice the `git annex remotedaemon` being run in the above examples. +That command runs the Tor hidden service so that other peers +can connect to your repository over Tor. + +So, you may want to arrange for the remotedaemon to be started on boot. +You can do that with a simple cron job: + + @reboot cd ~/myannexrepo && git annex remotedaemon + +If you use the git-annex assistant, and have it auto-starting on boot, it +will take care of starting the remotedaemon for you. + +## speed of large transfers + +Tor prioritizes security over speed, and the Tor network only has so much +bandwidth to go around. So, distributing large quantities (gigabytes) +of data over Tor may be slow, and should probably be avoided. + +One way to avoid sending much data over tor is to set up an encrypted +[[special_remote|special_remotes]] someplace. git-annex knows that Tor is +rather expensive to use, so if a file is available on a special remote as +well as over Tor, it will download it from the special remote. + +You can contribute to the Tor network by +[running a Tor relay or bridge](https://www.torproject.org/getinvolved/relays.html.en). + +## onion addresses and authentication + +You don't need to know about this, but it might be helpful to understand +how it works. + +git-annex's Tor support uses onion address as the address of a git remote. +You can `git pull`, push, etc with those onion addresses: + + git pull tor-annnex::eeaytkuhaupbarfi.onion:4412 + git remote add peer1 tor-annnex::eeaytkuhaupbarfi.onion:4412 + +Onion addresses are semi-public. When you add a remote, they appear in your +`.git/config` file. For security, there's a second level of authentication +that git-annex uses to make sure that only people you want to can access +your repository over Tor. That takes the form of a long string of numbers +and letters, like "7f53c5b65b8957ef626fd461ceaae8056e3dbc459ae715e4". + +The addresses generated by `git annex peer --gen-addresses` +combine the onion address with the authentication data. + +When you run `git annex peer --link`, it sets up a git remote using +the onion address, and it stashes the authentication data away in a file in +`.git/annex/creds/` + +When you pair repositories, these addresses are exchanged using +[Magic Wormhole](https://github.com/warner/magic-wormhole). + +## security + +Tor hidden services can be quite secure. But this doesn't mean that using +git-annex over Tor is automatically perfectly secure. Here are some things +to consider: + +* Anyone who learns the address of a peer can connect to that peer, + download the whole history of the git repository, and any available + annexed files. They can also upload new files to the peer, and even + remove annexed files from the peer. So consider ways that the address + of a peer might be exposed. + +* While Tor can be used to anonymize who you are, git defaults to including + your name and email address in git commit messages. So if you want an + anonymous git-annex repository, you'll need to configure git not to do + that. + +* Using Tor prevents listeners from decrypting your traffic. But, they'll + probably still know you're using Tor. Also, by traffic analysis, + they may be able to guess if you're using git-annex over tor, and even + make guesses about the sizes and types of files that you're exchanging + with peers. + +* There have been past attacks on the Tor network that have exposed + who was running Tor hidden services. + <https://blog.torproject.org/blog/tor-security-advisory-relay-early-traffic-confirmation-attack> + +* An attacker who can connect to the git-annex Tor hidden service, even + without authenticating, can try to perform denial of service attacks. + +* Magic wormhole is pretty secure, but the code phrase could be guessed + (unlikely) or intercepted. An attacker gets just one chance to try to enter + the correct code phrase, before pairing finishes. If the attacker + successfully guesses/intercepts both code phrases, they can MITM the + pairing process. + + If you don't want to use magic wormhole, you can instead manually generate + addresses with `git annex p2p --gen-addresses` and send them over an + authenticated, encrypted channel (such as OTR) to a friend to add with + `git annex p2p --link`. This may be more secure, if you get it right. diff --git a/doc/tips/using_Google_Cloud_Storage/comment_8_1b4eb7e0f44865cd5ff0f8ef507d99c1._comment b/doc/tips/using_Google_Cloud_Storage/comment_8_1b4eb7e0f44865cd5ff0f8ef507d99c1._comment new file mode 100644 index 000000000..1a71f7726 --- /dev/null +++ b/doc/tips/using_Google_Cloud_Storage/comment_8_1b4eb7e0f44865cd5ff0f8ef507d99c1._comment @@ -0,0 +1,9 @@ +[[!comment format=mdwn + username="scottgorlin@a32946b2aad278883c1690a0753241583a9855b9" + nickname="scottgorlin" + avatar="http://cdn.libravatar.org/avatar/2dd1fc8add62bbf4ffefac081b322563" + subject="Coldline" + date="2016-11-21T00:49:23Z" + content=""" +Wanted to add that \"storageclass=COLDLINE\" appears to work seamlessly, both from my mac and arm NAS. As far as I can tell, this appears to be a no-brainer vs glacier - builtin git annex client, simpler/cheaper billing, and no 4 hour delay! +"""]] diff --git a/doc/todo/Long_Running_Filter_Process.mdwn b/doc/todo/Long_Running_Filter_Process.mdwn new file mode 100644 index 000000000..329abaf45 --- /dev/null +++ b/doc/todo/Long_Running_Filter_Process.mdwn @@ -0,0 +1,22 @@ +Hello, + +while reading the release notes of git 2.11 I noticed a cool new feature has been merged: + +> If the filter command (a string value) is defined via +> `filter.<driver>.process` then Git can process all blobs with a +> single filter invocation for the entire life of a single Git +> command. + +see the [git documentation][1]. + +This has been developed in the context of git-lfs (see [PR 1382] [2]). + +If I understand correctly how it works this could speed up v6 repos. Looking at the history/website +of git-annex there doesn't seem to be yet any work on this so I though it was worth calling the +attention on the feature. + +Thanks a lot for all the work on git-annex, it's a really amazing project! The more I study it the more cool features I discover :) + + +[1]: https://github.com/git/git/blob/v2.11.0/Documentation/gitattributes.txt#L384 +[2]: https://github.com/git-lfs/git-lfs/pull/1382 diff --git a/doc/todo/Long_Running_Filter_Process/comment_1_f155ffc7dbd074964dd53165274ec8a0._comment b/doc/todo/Long_Running_Filter_Process/comment_1_f155ffc7dbd074964dd53165274ec8a0._comment new file mode 100644 index 000000000..34d05d771 --- /dev/null +++ b/doc/todo/Long_Running_Filter_Process/comment_1_f155ffc7dbd074964dd53165274ec8a0._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 1""" + date="2016-12-13T15:57:05Z" + content=""" +Yes, this will make [[smudge]] faster when eg checking out a lot of working +tree changes. I will need to add support for it. +"""]] diff --git a/doc/todo/Workflow_guide/comment_4_b6f5ce361529356a77b0e6141a62c06d._comment b/doc/todo/Workflow_guide/comment_4_b6f5ce361529356a77b0e6141a62c06d._comment new file mode 100644 index 000000000..6127e3e8d --- /dev/null +++ b/doc/todo/Workflow_guide/comment_4_b6f5ce361529356a77b0e6141a62c06d._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="marekj" + avatar="http://cdn.libravatar.org/avatar/65a60e8f5183feeeef8cef815bf73e61" + subject="I took the liberty to do it" + date="2016-12-14T07:49:26Z" + content=""" +I simply copied @xloem's into a new [[workflow]] page. I have been looking for such a guide myself for quite some time. +"""]] diff --git a/doc/todo/Workflow_guide/comment_5_6ec6fb45021ba82ed6a4bb9a6f3cfceb._comment b/doc/todo/Workflow_guide/comment_5_6ec6fb45021ba82ed6a4bb9a6f3cfceb._comment new file mode 100644 index 000000000..fe30f6106 --- /dev/null +++ b/doc/todo/Workflow_guide/comment_5_6ec6fb45021ba82ed6a4bb9a6f3cfceb._comment @@ -0,0 +1,19 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 5""" + date="2016-12-20T19:04:12Z" + content=""" +Good start on the workflow page! + +I've added some links to it to make it discoverable. + +Not sure if the workflow page quite gets to what was originally requested: + +> I want to start keeping track of some files I have in a directory +> I want to copy them to a second computer. +> From a third place, I want to get them from the second computer. +> I change the files on one computer, and I want to make sure the changes get synced to the others. +> What are the commands you'd run at each step? + +Leaving this todo open for now.. +"""]] diff --git a/doc/todo/Workflow_guide/comment_6_640e5c6cdea8a6fae63c3fab6970f1f2._comment b/doc/todo/Workflow_guide/comment_6_640e5c6cdea8a6fae63c3fab6970f1f2._comment new file mode 100644 index 000000000..9eae8e911 --- /dev/null +++ b/doc/todo/Workflow_guide/comment_6_640e5c6cdea8a6fae63c3fab6970f1f2._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="joey" + subject="""comment 6""" + date="2016-12-21T18:19:07Z" + content=""" +In a way the use cases on the front page of the website are trying to +accomplish the same thing requested here. I think that section could be +moved more in the direction of listing some ways to use git-annex and +linking to walkthroughs for the different use cases. +"""]] diff --git a/doc/todo/renameremote.mdwn b/doc/todo/renameremote.mdwn new file mode 100644 index 000000000..3a92bf507 --- /dev/null +++ b/doc/todo/renameremote.mdwn @@ -0,0 +1,24 @@ +Sometimes a name has been used for a special remote, and you want to change +the name. A common reason is that the special remote has become dead, and +you want to reuse the name for a new special remote. + +Initremote prevents reusing a name when the old one exists, even if the old +one is dead. And that makes sense in general, because a dead remote can +come back sometimes, and that would leave the repo with two special remotes +with the same name, and so enableremote would need to be run with a uuid +instead of a name to specify which one to enable, which is not a desirable +state of affairs. + +So, add `git annex renameremote oldname newname`. This could also do a `git +remote rename`, or equivilant. (`git remote rename` gets confused by special +remotes not having a fetch url and fails; this can be worked around by +manually renaming the stanza in git config.) + +Implementing that would need a way to remove the old name from remote.log. +We can't remove lines from union merged files, but what we could do is +add a new line like: + + - name=oldname timestamp=<latest> + +And in parsing remote.log, if the UUID is "-", don't include the +remote with that name in the the resulting map. diff --git a/doc/todo/smudge.mdwn b/doc/todo/smudge.mdwn index 78a20fd6d..611722490 100644 --- a/doc/todo/smudge.mdwn +++ b/doc/todo/smudge.mdwn @@ -37,6 +37,10 @@ git-annex should use smudge/clean filters. And developed a patch set: <http://thread.gmane.org/gmane.comp.version-control.git/297475> +* Implement git's new `filter.<driver>.process` interface, which will + let only 1 git-annex process be started by git when processing + multiple files, and so should be faster. + * Checking out a different branch causes git to smudge all changed files, and write their content. This does not honor annex.thin. A warning message is printed in this case. diff --git a/doc/todo/tor.mdwn b/doc/todo/tor.mdwn new file mode 100644 index 000000000..734670839 --- /dev/null +++ b/doc/todo/tor.mdwn @@ -0,0 +1,23 @@ +git-annex sync over tor + +Mostly working! + +Current todo list: + +* Webapp UI to set it up. +* When a transfer can't be done because another transfer of the same + object is already in progress, the message about this is output by the + remotedaemon --debug, but not forwarded to the peer, which shows + "Connection reset by peer" +* Think about locking some more. What happens if the connection to the peer + is dropped while we think we're locking content there from being dropped? + +Eventually: + +* Windows and Android support. +* Limiting authtokens to read-only access. +* Revoking authtokens. (This and read-only need a name associated with an + authtoken, so the user can adjust its configuration after creating it.) +* friend-of-a-friend peer discovery to build more interconnected networks + of nodes +* Discovery of nodes on same LAN, and direct connection to them. diff --git a/doc/todo/xmpp_removal.mdwn b/doc/todo/xmpp_removal.mdwn index 26d452940..373c16ca1 100644 --- a/doc/todo/xmpp_removal.mdwn +++ b/doc/todo/xmpp_removal.mdwn @@ -21,6 +21,8 @@ telehash. But, can't wait on that forever.. XMPP support is already disabled by default in some builds of git-annex, notably the stack build. It's never worked on Windows. +The [[no-xmpp]] branch is ready for merging. + Next step is probably to default the flag to false by default, except for in a few builds like the Debian package and standalone builds. diff --git a/doc/walkthrough.mdwn b/doc/walkthrough.mdwn index 22c94d570..b35cf808d 100644 --- a/doc/walkthrough.mdwn +++ b/doc/walkthrough.mdwn @@ -1,4 +1,9 @@ -A walkthrough of the basic features of git-annex. +A walkthrough of some of the basic features of git-annex, using the command +line. If you don't want to use the command line, see [[assistant/quickstart]] +instead. + +What follows is only one possible [[workflow]] for using git-annex, +but following along will teach you the basic concepts from the ground up. [[!toc]] diff --git a/doc/workflow.mdwn b/doc/workflow.mdwn new file mode 100644 index 000000000..042b4bab4 --- /dev/null +++ b/doc/workflow.mdwn @@ -0,0 +1,97 @@ +Git-annex supports a wide variety of workflows, a spectrum that ranges from +completely automatic behavior where git-annex handles everything, through +manual behavior where git-annex does only what you say when you tell it to, +all the way down to internal behavior, where you have complete control and +understand how everything is stored and exactly what changes are happening. + +I will proceed to summarize all of these. I will begin at the automatic +end, hoping that this is most useful, and drill down to the low level +approaches. Note, however, that this is the opposite order of how git-annex +was developed. A list of workflows that started from manual, +commandline usage would be much more intuitive, but you'd have to be +willing to read the man page and wiki pages to get started, and that's +pretty much what's already out there anyway. + +Note that for each of these levels of interaction, all the levels following +will also work as well. So you can actually manually move annexed files +around while the webapp is running, etc. + +# 1. [[git annex webapp|git-annex-webapp]] + +The [[`git annex webapp`|git-annex-webapp]] command launches a local web +server which serves a graphical user interface and automatically manages +git annex. It will attempt to guide you through the whole process and do +everything for you. The intent is that no other commands are +needed. This should be run on every machine that may produce file changes. + +# 2. [[git annex assistant|git-annex-assistant]] without the webapp + +You could call [[`git annex assistant`|git-annex-assistant]] the +command-line version of the webapp, giving you more control over creating +and connecting your repositories, and configuring how files are moved +between them. + +The assistant, when running, will automatically watch for file changes and +synchronize them to other repositories, but you must manually create the +repositories and configure the rules for syncing. To create a repository, +use `git init` and then [[`git annex init`|git-annex-init]], and then `git +remote add` it to any other repositories. If you want more than one annex, +you can add their paths to `~/.config/git-annex/autostart` if you would +like them to automatically begin syncing when `git annex assistant +--autostart` is run, perhaps on boot or login. You can configure rules for +where files are copied using the repository setup commands such as [[git +annex preferred-content|git-annex-preferred-content]] to configure +[[content preferences|preferred content]] for what goes where, [[`git annex +numcopies`|git-annex-numcopies]] for how many [[copies]] must be kept of +each file, and [[`git config annex.largefiles`|tips/largefiles]] to define +small files that should be stored straight in git; most of the settings are +accessible in one place with [[`git annex vicfg`|git-annex-vicfg]]. + +# 3. [[git annex watch|git-annex-watch]] without the assistant + +The [[`git annex watch`|git-annex-watch]] command is like the assistant but +has no automatic network behavior, giving you complete control over when +repositories are pushed and pulled, and when files are moved between +systems. The local repository is watched, and any file changes are added to +git-annex. In order to synchronize between repositories, you must run +[[`git annex sync --content`|git-annex-sync]] in the repository with the +changes, which will merge the git history and logs with your remotes, and +automatically transfer files to match your preferred and required content +expressions. + +# 4. No background processes + +This allows you to decide when and what files are annexed. In order to tell +git-annex to manage files, you must [[`git annex add`|git-annex-add]] the +files. + +# 5. Plain [[git annex sync|git-annex-sync]] without `--content` + +This gives you fine-grained control of where copies of your files are +stored. [[`git annex sync`|git-annex-sync]] without `--content` tells +git-annex to merge git histories, but it does not automatically transfer +your large files between systems. To transfer files and directories, you +can use [[`git annex get`|git-annex-get]], [[`git annex +drop`|git-annex-drop]], [[`git annex move`|git-annex-move]], and [[`git +annex copy`|git-annex-copy]]. Git-annex will not violate a required content +expression or your numcopies setting unless you pass `--force`, so your +files are still safe. + +# 6. Manual management of git history without the synchronizer + +This allows you to control precisely what is committed to git, what commit +message is used, and how your history is merged between repositories. You +must have an understanding of git, and run `git commit` after `git annex +add` to store the change. You must manage the git history yourself, using +`git pull` and `git push`, to synchronize repositories. You may freely use +git normally side-by-side with git-annex. + +# 7. Manual management of git annex keys + +This gives you control of what and where git annex stores your files under +the hood, and how they are associated with your working tree, rather than +using the `git annex add` and `git annex get` commands which reference +files automatically. Git-annex has a variety of plumbing commands listed in +the [[man page|git-annex]] that let you directly store and retrieve data in +an annex associated with your git repository, where every datafile is +enumerated by a unique hashkey. |