1 files changed, 115 insertions, 0 deletions

diff --git a/doc/bugs/Hybrid_encryption_can__39__t_generate_the_right_key_after_moving_files.mdwn b/doc/bugs/Hybrid_encryption_can__39__t_generate_the_right_key_after_moving_files.mdwn
new file mode 100644
index 000000000..d01c3fc24
--- /dev/null
+++ b/doc/bugs/Hybrid_encryption_can__39__t_generate_the_right_key_after_moving_files.mdwn
@@ -0,0 +1,115 @@
+### Please describe the problem.
+
+I had to migrate a special remote using hybrid encryption from an Amazon Cloud Drive-hosted rclone remote to a Google Drive-hosted rclone remote. Now I am testing it by trying to fsck a particular file.
+
+I've left all of the special remote settings on the git annex side the same, and just changed my `.rclone.conf` to have "amazondrive", the rclone remote name, actually point to the new Google Drive location.
+
+When I try to fsck my file, git annex generates an HMAC key, successfully downloads it from the remote, but then fails to decrypt it. After that, it looks like it retries: it generates a **different HMAC key**, which it can't find the file for.
+
+I've compared the files in Amazon and in Google Drive and they are the same.
+
+Using the [independent decryption script](https://git-annex.branchable.com/tips/Decrypting_files_in_special_remotes_without_git-annex/), modified to use the SHA512 MAC that my repo uses, and to not use every matching line from `remote.log`, I can generate the *second* HMAC key that git annex appears to fall back to, but not the first one. I think that this means I can't generate the decryption key either; I've had no luck manually decrypting the file with the characters after the first 256 of the decrypted shared key as the GPG passphrase.
+
+How is Git Annex generating that first HMAC, if not by the process used in the decryption script? Why is it able to get an HMAC that exists but not a valid decryption key? What is the thinking behind trying multiple HMAC keys for the same file?
+
+### What steps will reproduce the problem?
+
+Have an encrypted, hybrid Git Annex repo in an Amazon Drive rclone remote.
+Copy all the files over to Google Drive, and make a Google Drive rclone remote in rclone.conf with the same name that the Amazon Drive one had.
+Try to retrieve files from the remote with Git Annex.
+
+### What version of git-annex are you using? On what operating system?
+
+`6.20170525+gitge1cf095ae-1~ndall+1` on Ubuntu 16.04
+
+### Please provide any additional information below.
+
+[[!format sh """
+# If you can, paste a complete transcript of the problem occurring here.
+# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
+
+$ git annex fsck info.txt --from=amazon --debug
+[2017-06-03 11:24:31.368563623] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","git-annex"]
+[2017-06-03 11:24:31.373358653] process done ExitSuccess
+[2017-06-03 11:24:31.373520685] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--hash","refs/heads/git-annex"]
+[2017-06-03 11:24:31.379319755] process done ExitSuccess
+[2017-06-03 11:24:31.382954929] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch"]
+[2017-06-03 11:24:31.386196477] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)"]
+[2017-06-03 11:24:31.392852081] read: git ["config","--null","--list"]
+[2017-06-03 11:24:31.397686571] process done ExitSuccess
+[2017-06-03 11:24:31.400187312] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","ls-files","--cached","-z","--","info.txt"]
+[2017-06-03 11:24:31.424951599] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","check-attr","-z","--stdin","annex.backend","annex.numcopies","annex.largefiles","--"]
+[2017-06-03 11:24:31.425544622] read: git ["--version"]
+[2017-06-03 11:24:31.428009953] process done ExitSuccess
+fsck info.txt [2017-06-03 11:24:31.479360829] chat: gpg ["--batch","--no-tty","--use-agent","--quiet","--trust-model","always","--decrypt"]
+[2017-06-03 11:24:31.503040759] process done ExitSuccess
+[2017-06-03 11:24:31.506638987] chat: /home/anovak/bin/git-annex-remote-rclone []
+[2017-06-03 11:24:31.510094668] git-annex-remote-rclone[1] --> VERSION 1
+[2017-06-03 11:24:31.510281634] git-annex-remote-rclone[1] <-- PREPARE
+[2017-06-03 11:24:31.511006098] git-annex-remote-rclone[1] --> GETCONFIG prefix
+[2017-06-03 11:24:31.511148645] git-annex-remote-rclone[1] <-- VALUE data/annex-rclone
+[2017-06-03 11:24:31.515487261] git-annex-remote-rclone[1] --> GETCONFIG target
+[2017-06-03 11:24:31.515770189] git-annex-remote-rclone[1] <-- VALUE amazondrive
+[2017-06-03 11:24:31.519503001] git-annex-remote-rclone[1] --> GETCONFIG rclone_layout
+[2017-06-03 11:24:31.519666709] git-annex-remote-rclone[1] <-- VALUE lower
+[2017-06-03 11:24:31.523726589] git-annex-remote-rclone[1] --> PREPARE-SUCCESS
+[2017-06-03 11:24:31.523902589] git-annex-remote-rclone[1] <-- CHECKPRESENT GPGHMACSHA512--4354571263cd241ac8300e0bd8fc3643c184b922500c3d9318b7ce590684e1de820d41c0b245f1595e0bd38b2844a5151192981b4f9ab41bda135d5f7a184d16
+[2017-06-03 11:24:31.530946791] git-annex-remote-rclone[1] --> DIRHASH-LOWER GPGHMACSHA512--4354571263cd241ac8300e0bd8fc3643c184b922500c3d9318b7ce590684e1de820d41c0b245f1595e0bd38b2844a5151192981b4f9ab41bda135d5f7a184d16
+[2017-06-03 11:24:31.531236912] git-annex-remote-rclone[1] <-- VALUE b95/785/
+Total objects: 1 Total size: 119 Bytes (119 Bytes) 2017/06/03 11:24:42 Transferred: 0 Bytes (0 Bytes/s) Errors: 0 Checks: 0 Transferred: 0 Elapsed time: 10.4s
+[2017-06-03 11:24:42.04576211] git-annex-remote-rclone[1] --> CHECKPRESENT-SUCCESS GPGHMACSHA512--4354571263cd241ac8300e0bd8fc3643c184b922500c3d9318b7ce590684e1de820d41c0b245f1595e0bd38b2844a5151192981b4f9ab41bda135d5f7a184d16
+
+[2017-06-03 11:24:42.049286934] git-annex-remote-rclone[1] <-- TRANSFER RETRIEVE GPGHMACSHA512--4354571263cd241ac8300e0bd8fc3643c184b922500c3d9318b7ce590684e1de820d41c0b245f1595e0bd38b2844a5151192981b4f9ab41bda135d5f7a184d16 .git/annex/tmp/GPGHMACSHA512--4354571263cd241ac8300e0bd8fc3643c184b922500c3d9318b7ce590684e1de820d41c0b245f1595e0bd38b2844a5151192981b4f9ab41bda135d5f7a184d16
+[2017-06-03 11:24:42.050298028] git-annex-remote-rclone[1] --> DIRHASH-LOWER GPGHMACSHA512--4354571263cd241ac8300e0bd8fc3643c184b922500c3d9318b7ce590684e1de820d41c0b245f1595e0bd38b2844a5151192981b4f9ab41bda135d5f7a184d16
+[2017-06-03 11:24:42.05045573] git-annex-remote-rclone[1] <-- VALUE b95/785/
+2017/06/03 11:24:52 Local file system at /tmp/tmp.ITXaCoES4i: Waiting for checks to finish
+2017/06/03 11:24:52 Local file system at /tmp/tmp.ITXaCoES4i: Waiting for transfers to finish
+2017/06/03 11:24:54 
+Transferred:    119 Bytes (9 Bytes/s)
+Errors:                 0
+Checks:                 0
+Transferred:            1
+Elapsed time:         12s
+[2017-06-03 11:24:54.083003717] git-annex-remote-rclone[1] --> TRANSFER-SUCCESS RETRIEVE GPGHMACSHA512--4354571263cd241ac8300e0bd8fc3643c184b922500c3d9318b7ce590684e1de820d41c0b245f1595e0bd38b2844a5151192981b4f9ab41bda135d5f7a184d16
+[2017-06-03 11:24:54.083554696] chat: gpg ["--batch","--no-tty","--use-agent","--quiet","--trust-model","always","--batch","--passphrase-fd","22","--decrypt"]
+gpg: decryption failed: bad key
+[2017-06-03 11:24:54.086768586] process done ExitFailure 2
+[2017-06-03 11:24:54.088931923] git-annex-remote-rclone[1] <-- TRANSFER RETRIEVE GPGHMACSHA512--7439fc103f30635cd914af0ce14370aef2f9a81f415ba7a652bc20454c90893bcd5701ebbed855f6a0285fce413e5c039c15a7049738cf519e2719c9d485ce28 .git/annex/tmp/GPGHMACSHA512--7439fc103f30635cd914af0ce14370aef2f9a81f415ba7a652bc20454c90893bcd5701ebbed855f6a0285fce413e5c039c15a7049738cf519e2719c9d485ce28
+[2017-06-03 11:24:54.089806756] git-annex-remote-rclone[1] --> DIRHASH-LOWER GPGHMACSHA512--7439fc103f30635cd914af0ce14370aef2f9a81f415ba7a652bc20454c90893bcd5701ebbed855f6a0285fce413e5c039c15a7049738cf519e2719c9d485ce28
+[2017-06-03 11:24:54.090050153] git-annex-remote-rclone[1] <-- VALUE b47/e54/
+2017/06/03 11:25:03 Local file system at /tmp/tmp.5lapDbIoZy: Waiting for checks to finish
+2017/06/03 11:25:03 Local file system at /tmp/tmp.5lapDbIoZy: Waiting for transfers to finish
+2017/06/03 11:25:03 Attempt 1/3 failed with 0 errors and: directory not found
+2017/06/03 11:25:03 Local file system at /tmp/tmp.5lapDbIoZy: Waiting for checks to finish
+2017/06/03 11:25:03 Local file system at /tmp/tmp.5lapDbIoZy: Waiting for transfers to finish
+2017/06/03 11:25:03 Attempt 2/3 failed with 0 errors and: directory not found
+2017/06/03 11:25:04 Local file system at /tmp/tmp.5lapDbIoZy: Waiting for checks to finish
+2017/06/03 11:25:04 Local file system at /tmp/tmp.5lapDbIoZy: Waiting for transfers to finish
+2017/06/03 11:25:04 Attempt 3/3 failed with 0 errors and: directory not found
+2017/06/03 11:25:04 Failed to copy: directory not found
+[2017-06-03 11:25:04.479940212] git-annex-remote-rclone[1] --> TRANSFER-FAILURE RETRIEVE GPGHMACSHA512--7439fc103f30635cd914af0ce14370aef2f9a81f415ba7a652bc20454c90893bcd5701ebbed855f6a0285fce413e5c039c15a7049738cf519e2719c9d485ce28
+
+  failed to download file from remote
+failed
+[2017-06-03 11:25:04.492702185] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","hash-object","-w","--stdin-paths","--no-filters"]
+[2017-06-03 11:25:04.494054942] feed: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","update-index","-z","--index-info"]
+[2017-06-03 11:25:04.672344301] process done ExitSuccess
+[2017-06-03 11:25:04.67255166] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--hash","refs/heads/git-annex"]
+[2017-06-03 11:25:04.68070068] process done ExitSuccess
+(recording state in git...)
+[2017-06-03 11:25:04.680983817] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","write-tree"]
+[2017-06-03 11:25:04.900456464] process done ExitSuccess
+[2017-06-03 11:25:04.900643429] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","commit-tree","a8fc26b91dab1ffac0499d3e21c0f691858d3a6a","--no-gpg-sign","-p","refs/heads/git-annex"]
+[2017-06-03 11:25:04.965122443] process done ExitSuccess
+[2017-06-03 11:25:04.965288386] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","update-ref","refs/heads/git-annex","c0b297184826849d8a24c899faa9f4f47c1f8e53"]
+[2017-06-03 11:25:04.970235879] process done ExitSuccess
+[2017-06-03 11:25:04.971493023] process done ExitSuccess
+git-annex: fsck: 1 failed
+
+
+# End of transcript or log.
+"""]]
+
+### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
+
+