diff options
Diffstat (limited to 'doc/bugs/Unnecessary_remote_transfers')
11 files changed, 0 insertions, 273 deletions
diff --git a/doc/bugs/Unnecessary_remote_transfers/comment_10_b778fbb1386f0f51bf057ffacd590ebb._comment b/doc/bugs/Unnecessary_remote_transfers/comment_10_b778fbb1386f0f51bf057ffacd590ebb._comment deleted file mode 100644 index 8c985243a..000000000 --- a/doc/bugs/Unnecessary_remote_transfers/comment_10_b778fbb1386f0f51bf057ffacd590ebb._comment +++ /dev/null @@ -1,12 +0,0 @@ -[[!comment format=mdwn - username="https://www.google.com/accounts/o8/id?id=AItOawmNu4V5fvpLlBhaCUfXXOB0MI5NXwh8SkU" - nickname="Adam" - subject="comment 10" - date="2013-11-03T02:19:04Z" - content=""" -Thanks, Joey. I didn't realize that there were two instances of the assistant running. - -Forgive my ignorance, but if I disable the assistant running in the backup repo, will the other assistant still backup files to it? - -Also, from a UI perspective, does the webapp currently not seem to support secondary internal drives, at least in a useful way? -"""]] diff --git a/doc/bugs/Unnecessary_remote_transfers/comment_11_55430eac842d0a192dc7f41d7730e4d5._comment b/doc/bugs/Unnecessary_remote_transfers/comment_11_55430eac842d0a192dc7f41d7730e4d5._comment deleted file mode 100644 index 808c5bca6..000000000 --- a/doc/bugs/Unnecessary_remote_transfers/comment_11_55430eac842d0a192dc7f41d7730e4d5._comment +++ /dev/null @@ -1,10 +0,0 @@ -[[!comment format=mdwn - username="http://joeyh.name/" - ip="209.250.56.47" - subject="comment 11" - date="2013-11-03T02:32:47Z" - content=""" -Yes, either of the assistants will still back up files to it, as long as they have it configured as a git remote. The same way that your transfer repository is (probably) not running the git-annex assistant. - -It does sound like using the removable drive UI would have saved you some trouble. I'm not sure why the webapp would not list a llvm device as a removable drive -- from inspecting the code, it seems like it would not be filtered out, as long as /proc/mounts shows the decive as /dev/something (I don't have a llvm device handy to check what it looks like). -"""]] diff --git a/doc/bugs/Unnecessary_remote_transfers/comment_1_00c18e07678dc513a02d974fe059df73._comment b/doc/bugs/Unnecessary_remote_transfers/comment_1_00c18e07678dc513a02d974fe059df73._comment deleted file mode 100644 index 7baa58489..000000000 --- a/doc/bugs/Unnecessary_remote_transfers/comment_1_00c18e07678dc513a02d974fe059df73._comment +++ /dev/null @@ -1,12 +0,0 @@ -[[!comment format=mdwn - username="http://joeyh.name/" - ip="209.250.56.47" - subject="comment 1" - date="2013-11-02T19:54:44Z" - content=""" -In what order does the webapp list your repositories? - -When a new file appears, it will always go through the list of repositories in order, and try to transfer the file to each in turn, unless the configuration indicates it shouldn't. So, if the remote transfer repo is listed before the local repo, it will first upload the file to the transfer repo, and then upload it to the local repo. (And then later on, remove it from the transfer repo, since the file has reached all clients, probably.) OTOH, if the local repo comes first, it will upload the file to it over the LAN, and then when it comes to the transfer repo, if the file has successfully been sent to all clients, the transfer repo will no longer want it, and so no expensive upload is done over the internet. - -You can re-order the repository list in the webapp by dragging them up and down. It should default to having your locally paired repos first, unless you've changed it. If you think it came up with the wrong order, paste in your .git/config before you re-order the repositories.. -"""]] diff --git a/doc/bugs/Unnecessary_remote_transfers/comment_2_2e9992dbfceabd6df535a2770626de16._comment b/doc/bugs/Unnecessary_remote_transfers/comment_2_2e9992dbfceabd6df535a2770626de16._comment deleted file mode 100644 index cc0cb067e..000000000 --- a/doc/bugs/Unnecessary_remote_transfers/comment_2_2e9992dbfceabd6df535a2770626de16._comment +++ /dev/null @@ -1,35 +0,0 @@ -[[!comment format=mdwn - username="https://www.google.com/accounts/o8/id?id=AItOawmNu4V5fvpLlBhaCUfXXOB0MI5NXwh8SkU" - nickname="Adam" - subject="comment 2" - date="2013-11-02T22:48:18Z" - content=""" -The webapp has the remote repo at the bottom of the list. But here's .git/config. It has a high repo cost for the local paired repo, and no cost listed for the remote one: - -[[!format sh \"\"\" -[core] - repositoryformatversion = 0 - filemode = true - bare = false - logallrefupdates = true -[annex] - uuid = 946f9095-588b-4421-b66a-4a1e5632ff6b - version = 3 - direct = true - diskreserve = 1 megabyte - numcopies = 1 - debug = false -[gc] - auto = 0 -[remote \"Onyx\"] - url = ssh://me@git-annex-Onyx.local-me_annex/~/annex/ - annex-uuid = 80709bc2-3cbe-434b-b7b9-306278b9a4e9 - annex-cost = 125.0 - fetch = +refs/heads/*:refs/remotes/Onyx/* -[remote \"Remote\"] - annex-rsyncurl = username@git-annex-example.net-username_annex:annex/ - annex-uuid = 33930bae-63d2-4a52-b330-58872aaeb1bf - fetch = - annex-sync = true -\"\"\"]] -"""]] diff --git a/doc/bugs/Unnecessary_remote_transfers/comment_3_a98f3091a6a658919f0562cf396439c2._comment b/doc/bugs/Unnecessary_remote_transfers/comment_3_a98f3091a6a658919f0562cf396439c2._comment deleted file mode 100644 index b6900f339..000000000 --- a/doc/bugs/Unnecessary_remote_transfers/comment_3_a98f3091a6a658919f0562cf396439c2._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="https://www.google.com/accounts/o8/id?id=AItOawmNu4V5fvpLlBhaCUfXXOB0MI5NXwh8SkU" - nickname="Adam" - subject="comment 3" - date="2013-11-02T22:59:18Z" - content=""" -Also, I just realized that it's uploading two large files to my remote transfer repo, even though I already have both files in my local paired repo on both systems; i.e. there's no need to send it to the transfer repo at all. And it was yesterday when the two files were transferred. So I have no idea why today it thinks it needs to upload them to the transfer repo. :/ -"""]] diff --git a/doc/bugs/Unnecessary_remote_transfers/comment_4_417c1e8e27ee1a1f9ebf9160560605c5._comment b/doc/bugs/Unnecessary_remote_transfers/comment_4_417c1e8e27ee1a1f9ebf9160560605c5._comment deleted file mode 100644 index b0d4c1146..000000000 --- a/doc/bugs/Unnecessary_remote_transfers/comment_4_417c1e8e27ee1a1f9ebf9160560605c5._comment +++ /dev/null @@ -1,96 +0,0 @@ -[[!comment format=mdwn - username="https://www.google.com/accounts/o8/id?id=AItOawmNu4V5fvpLlBhaCUfXXOB0MI5NXwh8SkU" - nickname="Adam" - subject="Super confused" - date="2013-11-02T23:30:41Z" - content=""" -I think now I'm double- or triple-confused. Here is me@desktop:~/annex/.git/config: - -[[!format sh \"\"\" -[core] - repositoryformatversion = 0 - filemode = true - bare = false - logallrefupdates = true -[annex] - uuid = 80709bc2-3cbe-434b-b7b9-306278b9a4e9 - version = 3 - direct = true - fscknudge = true -[gc] - auto = 0 -[remote \"laptop\"] - url = ssh://me@git-annex-laptop-me_annex/~/annex/ - annex-uuid = 946f9095-588b-4421-b66a-4a1e5632ff6b - annex-cost = 175.0 - fetch = +refs/heads/*:refs/remotes/laptop/* -[remote \"Remote\"] - annex-rsyncurl = username@example.net:annex/ - annex-uuid = 33930bae-63d2-4a52-b330-58872aaeb1bf - fetch = - annex-sync = false -[remote \"backupOndesktop\"] - url = /mnt/debian/home/me/annex-backup - annex-uuid = 86535965-6ca7-4bf3-89af-bca3a07f96f9 - fetch = +refs/heads/*:refs/remotes/backupOndesktop/* -\"\"\"]] - -What I have set up (or intended to set up) is: - -* Client: me@laptop:~/annex -* Client: me@desktop:~/annex -* Transfer: username@example.net:~/annex -* Full Backup: me@desktop:/mnt/debian/home/me/annex-backup - -What I expected was that: - -1. The two Client repos would sync directly over the LAN whenever possible. -2. The transfer repo would be used only to sync the laptop and desktop, and then only if my laptop were not on the LAN with my desktop. -3. The Full Backup repo would be synced directly with the me@desktop Client repo, within the same system, from one hard disk to the other. - -But what I'm seeing is...not quite that. - -1. It's very confusing (to me, at least) that on my desktop system I can look at my annex setup from the \"perspective\" of both local repos. When I \"switch\" to the other repo, the setup looks different: even the same repos can have different names and descriptions, and can be enabled and disabled independently depending on the \"perspective\"...but their Type is still the same. - -2. Looking from the \"perspective\" of the Full Backup repo, I'm seeing files being transferred to the Transfer repo, even though those files are already present in both Client repos, and even though the webapp says that the Full Backup repo has been synced. When I disable and reenable the Transfer repo, it starts those transfers over again, even though they aren't necessary. For example, here is the log from disabling and reenabling the Transfer repo from the \"perspective\" of the Full Backup repo: - -[[!format sh \"\"\" -[2013-11-02 18:24:35 CDT] main: Syncing with example.net_annex -(gpg) -gpg: Terminated caught ... exiting -[2013-11-02 18:24:49 CDT] call: git [\"--git-dir=/mnt/debian/home/me/annex-backup/.git\",\"--work-tree=/mnt/debian/home/me/annex-backup\",\"config\",\"remote.example.net_annex.annex-sync\",\"true\"] -[2013-11-02 18:24:49 CDT] read: git [\"config\",\"--null\",\"--list\"] -[2013-11-02 18:24:49 CDT] read: git [\"config\",\"--null\",\"--list\"] -[2013-11-02 18:24:49 CDT] read: git [\"config\",\"--null\",\"--list\"] -[2013-11-02 18:24:49 CDT] main: Syncing with example.net_annex -[2013-11-02 18:24:49 CDT] read: git [\"--git-dir=/mnt/debian/home/me/annex-backup/.git\",\"--work-tree=/mnt/debian/home/me/annex-backup\",\"symbolic-ref\",\"HEAD\"] -[2013-11-02 18:24:49 CDT] read: git [\"--git-dir=/mnt/debian/home/me/annex-backup/.git\",\"--work-tree=/mnt/debian/home/me/annex-backup\",\"show-ref\",\"refs/heads/master\"] -[2013-11-02 18:24:49 CDT] read: git [\"--git-dir=/mnt/debian/home/me/annex-backup/.git\",\"--work-tree=/mnt/debian/home/me/annex-backup\",\"show-ref\",\"git-annex\"] -[2013-11-02 18:24:49 CDT] read: git [\"--git-dir=/mnt/debian/home/me/annex-backup/.git\",\"--work-tree=/mnt/debian/home/me/annex-backup\",\"show-ref\",\"--hash\",\"refs/heads/git-annex\"] -[2013-11-02 18:24:49 CDT] read: git [\"--git-dir=/mnt/debian/home/me/annex-backup/.git\",\"--work-tree=/mnt/debian/home/me/annex-backup\",\"log\",\"refs/heads/git-annex..2c42d607c099b6ec4a20603b809f44d161e42489\",\"--oneline\",\"-n1\"] -[2013-11-02 18:24:49 CDT] read: git [\"--git-dir=/mnt/debian/home/me/annex-backup/.git\",\"--work-tree=/mnt/debian/home/me/annex-backup\",\"symbolic-ref\",\"HEAD\"] -[2013-11-02 18:24:49 CDT] read: git [\"--git-dir=/mnt/debian/home/me/annex-backup/.git\",\"--work-tree=/mnt/debian/home/me/annex-backup\",\"show-ref\",\"refs/heads/master\"] -[2013-11-02 18:24:49 CDT] TransferScanner: starting scan of [Remote { name =\"example.net_annex\" }] -[2013-11-02 18:24:49 CDT] read: git [\"--git-dir=/mnt/debian/home/me/annex-backup/.git\",\"--work-tree=/mnt/debian/home/me/annex-backup\",\"ls-files\",\"--cached\",\"-z\",\"--\"] -[2013-11-02 18:24:49 CDT] TransferScanner: queued Upload UUID \"33930bae-63d2-4a52-b330-58872aaeb1bf\" video.flv Nothing : expensive scan found missing object -[2013-11-02 18:24:49 CDT] Transferrer: Transferring: Upload UUID \"33930bae-63d2-4a52-b330-58872aaeb1bf\" video.flv Nothing -[2013-11-02 18:24:49 CDT] TransferScanner: queued Upload UUID \"33930bae-63d2-4a52-b330-58872aaeb1bf\" image.jpg Nothing : expensive scan found missing object -[2013-11-02 18:24:49 CDT] call: /home/me/.bin/git-annex.linux/git-annex [\"transferkeys\",\"--readfd\",\"35\",\"--writefd\",\"34\"] -[2013-11-02 18:24:49 CDT] TransferScanner: queued Upload UUID \"33930bae-63d2-4a52-b330-58872aaeb1bf\" linux-image-3.8.0-29-generic_3.8.0-29.42_i386.deb Nothing : expensive scan found missing object -[2013-11-02 18:24:49 CDT] TransferScanner: finished scan of [Remote { name =\"example.net_annex\" }] -[2013-11-02 18:24:49 CDT] read: git [\"--git-dir=/mnt/debian/home/me/annex-backup/.git\",\"--work-tree=/mnt/debian/home/me/annex-backup\",\"show-ref\",\"git-annex\"] -[2013-11-02 18:24:49 CDT] read: git [\"--git-dir=/mnt/debian/home/me/annex-backup/.git\",\"--work-tree=/mnt/debian/home/me/annex-backup\",\"show-ref\",\"--hash\",\"refs/heads/git-annex\"] -[2013-11-02 18:24:49 CDT] read: git [\"--git-dir=/mnt/debian/home/me/annex-backup/.git\",\"--work-tree=/mnt/debian/home/me/annex-backup\",\"log\",\"refs/heads/git-annex..2c42d607c099b6ec4a20603b809f44d161e42489\",\"--oneline\",\"-n1\"] -[2013-11-02 18:24:49 CDT] chat: git [\"--git-dir=/mnt/debian/home/me/annex-backup/.git\",\"--work-tree=/mnt/debian/home/me/annex-backup\",\"cat-file\",\"--batch\"] -[2013-11-02 18:24:49 CDT] read: git [\"config\",\"--null\",\"--list\"] -[2013-11-02 18:24:49 CDT] TransferWatcher: transfer starting: Upload UUID \"33930bae-63d2-4a52-b330-58872aaeb1bf\" video.flv Nothing -(gpg) [2013-11-02 18:24:49 CDT] chat: gpg [\"--batch\",\"--no-tty\",\"--use-agent\",\"--quiet\",\"--trust-model\",\"always\",\"--batch\",\"--passphrase-fd\",\"20\",\"--symmetric\",\"--force-mdc\",\"--no-textmode\"] -[2013-11-02 18:24:50 CDT] call: git [\"--git-dir=/mnt/debian/home/me/annex-backup/.git\",\"--work-tree=/mnt/debian/home/me/annex-backup\",\"config\",\"remote.example.net_annex.annex-sync\",\"false\"] -[2013-11-02 18:24:50 CDT] read: git [\"config\",\"--null\",\"--list\"] -[2013-11-02 18:24:50 CDT] read: git [\"config\",\"--null\",\"--list\"] -[2013-11-02 18:24:50 CDT] read: git [\"config\",\"--null\",\"--list\"] - -gpg: Terminated caught ... exiting\"\"\"]] - -I have no idea why it's doing that, because all three of those files are already in both Client repos and the Full Backup repo. -"""]] diff --git a/doc/bugs/Unnecessary_remote_transfers/comment_5_eb5a2717a1f0c7bb761d2a7866b23def._comment b/doc/bugs/Unnecessary_remote_transfers/comment_5_eb5a2717a1f0c7bb761d2a7866b23def._comment deleted file mode 100644 index 0e2ec4ec8..000000000 --- a/doc/bugs/Unnecessary_remote_transfers/comment_5_eb5a2717a1f0c7bb761d2a7866b23def._comment +++ /dev/null @@ -1,8 +0,0 @@ -[[!comment format=mdwn - username="https://www.google.com/accounts/o8/id?id=AItOawmNu4V5fvpLlBhaCUfXXOB0MI5NXwh8SkU" - nickname="Adam" - subject="Deleting repos from different perspectives" - date="2013-11-02T23:33:48Z" - content=""" -One other thing, sorry, but I think this is important: From the perspective of the Full Backup repo, I set the Transfer repo to be deleted, thinking that would prevent the desktop's Client and Full Backup repos from syncing through the Transfer repo. But then when I switch to the perspective of the desktop's Client repo, it is \"cleaning out\" the Transfer repo--but I wanted the Transfer repo to be used for syncing the two Client repos! So I set it back to Transfer instead of Unwanted...but then when I switch back to the Full Backup perspective, the Transfer repo is no longer Unwanted. -"""]] diff --git a/doc/bugs/Unnecessary_remote_transfers/comment_6_89f756db1f3f2e60a3bd1f35f55fee43._comment b/doc/bugs/Unnecessary_remote_transfers/comment_6_89f756db1f3f2e60a3bd1f35f55fee43._comment deleted file mode 100644 index 509cc3f3f..000000000 --- a/doc/bugs/Unnecessary_remote_transfers/comment_6_89f756db1f3f2e60a3bd1f35f55fee43._comment +++ /dev/null @@ -1,40 +0,0 @@ -[[!comment format=mdwn - username="https://www.google.com/accounts/o8/id?id=AItOawmNu4V5fvpLlBhaCUfXXOB0MI5NXwh8SkU" - nickname="Adam" - subject="config from backup repo" - date="2013-11-03T00:20:11Z" - content=""" -Sorry for posting over and over again, just trying to provide info when I discover it. - -Here's **/mnt/debian/home/me/annex-backup/.git/config** - -[[!format sh \"\"\" -[core] - repositoryformatversion = 0 - filemode = true - bare = false - logallrefupdates = true -[annex] - uuid = 86535965-6ca7-4bf3-89af-bca3a07f96f9 - version = 3 - direct = true - fscknudge = true - diskreserve = 100 megabyte - numcopies = 1 - debug = false -[gc] - auto = 0 -[remote \"desktop\"] - url = /home/me/annex - fetch = +refs/heads/*:refs/remotes/desktop/* - annex-uuid = 80709bc2-3cbe-434b-b7b9-306278b9a4e9 -[remote \"Remote\"] - annex-rsyncurl = username@example.net:annex/ - annex-uuid = 33930bae-63d2-4a52-b330-58872aaeb1bf - annex-sync = false - fetch =\"\"\"]] - -1. Should I remove the \"Remote\" Transfer repo from this config file to prevent it from using the Transfer repo? I only want this repo to sync directly with the other internal hard disk. - -2. Is it correct for git-annex to add this Transfer repo to this config file in the first place? -"""]] diff --git a/doc/bugs/Unnecessary_remote_transfers/comment_7_5aaf8766a7ba05c4f92715e5d5175a8f._comment b/doc/bugs/Unnecessary_remote_transfers/comment_7_5aaf8766a7ba05c4f92715e5d5175a8f._comment deleted file mode 100644 index f117f15c1..000000000 --- a/doc/bugs/Unnecessary_remote_transfers/comment_7_5aaf8766a7ba05c4f92715e5d5175a8f._comment +++ /dev/null @@ -1,13 +0,0 @@ -[[!comment format=mdwn - username="http://joeyh.name/" - ip="209.250.56.47" - subject="comment 7" - date="2013-11-03T00:45:09Z" - content=""" -* The costs you show look fine. -* Yes, deleting a repository in the webapp actually deletes it. It doesn't just remove the remote from .git/config. Most people using the webapp don't want to draw such a fine distinction, I think. -* It's not a very usual configuration to have 2 repositories on the same machine with the git-annex assistant running in both. You might just want to configure the assistant to not run in the backup repository. - -Some of the things you've said suggest that the backup repository might not be immediately noticing when changes are pushed to it. Since its location is shown as /mnt/debian/home/me, I have to wonder if that's some NFS mount or other network filesystem causing problems. - -"""]] diff --git a/doc/bugs/Unnecessary_remote_transfers/comment_8_e856b350632cc865d16d1995a6cdf065._comment b/doc/bugs/Unnecessary_remote_transfers/comment_8_e856b350632cc865d16d1995a6cdf065._comment deleted file mode 100644 index 1177205e3..000000000 --- a/doc/bugs/Unnecessary_remote_transfers/comment_8_e856b350632cc865d16d1995a6cdf065._comment +++ /dev/null @@ -1,14 +0,0 @@ -[[!comment format=mdwn - username="https://www.google.com/accounts/o8/id?id=AItOawmNu4V5fvpLlBhaCUfXXOB0MI5NXwh8SkU" - nickname="Adam" - subject="comment 8" - date="2013-11-03T01:07:31Z" - content=""" -> It's not a very usual configuration to have 2 repositories on the same machine with the git-annex assistant running in both. You might just want to configure the assistant to not run in the backup repository. - -Maybe that's the crux of the problem. But I'm not sure what I did wrong to make that happen. I have a second internal hard disk in my desktop system, and I wanted to put a Full Backup repo on it. I tried to add a \"Removable drive\" repo (since I sometimes unplug it to swap cables with a DVD drive), but since the assistant didn't detect any actual removable drives, that didn't work. So I used \"Add another repository\" and set it to a Full Backup repo. Is that the wrong way to do it? :) - -> Some of the things you've said suggest that the backup repository might not be immediately noticing when changes are pushed to it. Since its location is shown as /mnt/debian/home/me, I have to wonder if that's some NFS mount or other network filesystem causing problems. - -/mnt/debian is /dev/mapper/lvm-root, another internal disk. The disk is working fine. :) -"""]] diff --git a/doc/bugs/Unnecessary_remote_transfers/comment_9_64f831545b34b78452952cf49b5f5b05._comment b/doc/bugs/Unnecessary_remote_transfers/comment_9_64f831545b34b78452952cf49b5f5b05._comment deleted file mode 100644 index 03127c4ad..000000000 --- a/doc/bugs/Unnecessary_remote_transfers/comment_9_64f831545b34b78452952cf49b5f5b05._comment +++ /dev/null @@ -1,25 +0,0 @@ -[[!comment format=mdwn - username="http://joeyh.name/" - ip="209.250.56.47" - subject="comment 9" - date="2013-11-03T01:52:56Z" - content=""" -Ok, since we don't have NFS craziness, I'm going to put theories about it not noticing incoming syncs on hold. (I think I probably misunderstood [this comment](http://git-annex.branchable.com/bugs/Unnecessary_remote_transfers/#comment-39490c318620c141e7557b7bcba0e5c8) and anyway the assistant has some throttling so won't always immediately load config changes that have been synced to it if some other config changes were loaded a minute before.) - -I think this behavior can probably be explained without resorting to any bugs, now that I have, I think, a full picture of the repository network: - -<pre> -laptop <-> desktop <-> backup - | | | - v v v - transfer (cloud) -</pre> - -Now when a file is added to desktop, it immediately copies it to backup, which is on the same machine, so that happens quite quickly. Then it starts sending it across the LAN to laptop. - -Meanwhile, the assistant daemon running on backup wakes up, notices it's just received an object, and sees that this object is currently located on only desktop and backup, but not yet on laptop or transfer. Which means that it should send the object to transfer, from which it will eventually reach laptop. - -The only ways I can think of to avoid such an unnecessary transfer would be a) for desktop could somehow tell backup that it's in the process of sending the file to laptop or b) for a map of current state of the network to be constructed and maintained and analyzed automatically, so it it could conclude backup does not need to send files to transfer if they're already present on desktop. There is some discussion about this in [[design/assistant/syncing]]. Hard problem in general I think, although slightly less hard in this specific case since desktop and backup know they're in the same machine. - -But it should be easy to configure it so this doesn't happen. Just make backup not have the transfer repository configured as a remote (or just pause it syncing to there in the webapp). Or, don't run the assistant at all on backup (see `~/.config/git-annex/autostart`). -"""]] |