diff options
Diffstat (limited to 'doc/todo')
394 files changed, 5622 insertions, 0 deletions
diff --git a/doc/todo/A_really_simple_way_to_pair_devices_like_bittorent_sync.mdwn b/doc/todo/A_really_simple_way_to_pair_devices_like_bittorent_sync.mdwn new file mode 100644 index 000000000..93ccc083d --- /dev/null +++ b/doc/todo/A_really_simple_way_to_pair_devices_like_bittorent_sync.mdwn @@ -0,0 +1,7 @@ +I like the way btsync search for the peer. So if I need to sync my laptop and other family laptop both with a total different and changing network setup the two device found each other do NAT traversal if needed use relays but the end the two folders are synced. But its closed-source :( I like git and git-annex looks really great. I'm learning it now. + +First I thought with xmpp I can sync files without ssh/rsync or other remote access to my devices just with two jabber account. Now I know I can't. :( + +It would be just great to have some means to sync files without cloud just the two device. Without the ssh / rsync jut share some secret and the devices do the rest. :-o + +Anyway thanks for hearing. I'm looking forward to know more about git-annex. Thank you for that sw. =-<>-= diff --git a/doc/todo/A_really_simple_way_to_pair_devices_like_bittorent_sync/comment_1_d828bc374e50a49101c0b863f9b33080._comment b/doc/todo/A_really_simple_way_to_pair_devices_like_bittorent_sync/comment_1_d828bc374e50a49101c0b863f9b33080._comment new file mode 100644 index 000000000..13571e681 --- /dev/null +++ b/doc/todo/A_really_simple_way_to_pair_devices_like_bittorent_sync/comment_1_d828bc374e50a49101c0b863f9b33080._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawkEHjHAWnJ0BJzdv_hePwU1my8X4wCseh8" + nickname="Sz" + subject="comment 1" + date="2013-07-23T11:00:21Z" + content=""" +Why not use xmpp for file transfer too not only for git sync? +"""]] diff --git a/doc/todo/A_really_simple_way_to_pair_devices_like_bittorent_sync/comment_2_a4badfc248be428e6426a936212cc896._comment b/doc/todo/A_really_simple_way_to_pair_devices_like_bittorent_sync/comment_2_a4badfc248be428e6426a936212cc896._comment new file mode 100644 index 000000000..2bff793f0 --- /dev/null +++ b/doc/todo/A_really_simple_way_to_pair_devices_like_bittorent_sync/comment_2_a4badfc248be428e6426a936212cc896._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://johan.kiviniemi.name/" + ip="83.145.237.224" + subject="comment 2" + date="2013-07-23T11:37:53Z" + content=""" +Transferring the data over XMPP would almost certainly take too much XMPP server bandwidth, but using something like [libjingle](https://developers.google.com/talk/libjingle/) to set up P2P connections should work nicely. That would require libjingle (or equivalent) bindings for Haskell, though. Libjingle negotiates over XMPP to set up the P2P connection and provides a TCP-like layer for reliable, ordered communication. One could use that for both git metadata and the file transfers. +"""]] diff --git a/doc/todo/A_really_simple_way_to_pair_devices_like_bittorent_sync/comment_3_0b04089d3d33fdb48eeb46bf168e9a3c._comment b/doc/todo/A_really_simple_way_to_pair_devices_like_bittorent_sync/comment_3_0b04089d3d33fdb48eeb46bf168e9a3c._comment new file mode 100644 index 000000000..d0c000d2b --- /dev/null +++ b/doc/todo/A_really_simple_way_to_pair_devices_like_bittorent_sync/comment_3_0b04089d3d33fdb48eeb46bf168e9a3c._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawkEHjHAWnJ0BJzdv_hePwU1my8X4wCseh8" + nickname="Sz" + subject="comment 3" + date="2013-07-24T09:08:45Z" + content=""" +I'm for it! +1 +"""]] diff --git a/doc/todo/A_really_simple_way_to_pair_devices_like_bittorent_sync/comment_4_2bcab1b7998b4df08fca41b8d810f115._comment b/doc/todo/A_really_simple_way_to_pair_devices_like_bittorent_sync/comment_4_2bcab1b7998b4df08fca41b8d810f115._comment new file mode 100644 index 000000000..f3bafca89 --- /dev/null +++ b/doc/todo/A_really_simple_way_to_pair_devices_like_bittorent_sync/comment_4_2bcab1b7998b4df08fca41b8d810f115._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U" + nickname="Richard" + subject="comment 4" + date="2013-08-03T07:12:40Z" + content=""" +NAT traversal requires central infrastructure and is unreliable at best. Obviously, a for-profit entity like bittorrent can shoulder that. + +For LAN situations, [zeroconf](http://www.zeroconf.org/) along with passphrase-based pairing may be the long-term answer. Arguably, that would enhance vanilla Git almost as much as it would enhance git-annex, but that does not seem likely to happen. +"""]] diff --git a/doc/todo/A_really_simple_way_to_pair_devices_like_bittorent_sync/comment_5_677e958c3f2effec7528b484aeb6478d._comment b/doc/todo/A_really_simple_way_to_pair_devices_like_bittorent_sync/comment_5_677e958c3f2effec7528b484aeb6478d._comment new file mode 100644 index 000000000..07b001c2e --- /dev/null +++ b/doc/todo/A_really_simple_way_to_pair_devices_like_bittorent_sync/comment_5_677e958c3f2effec7528b484aeb6478d._comment @@ -0,0 +1,13 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawm5iosFbL2By7UFeViqkc6v-hoAtqILeDA" + nickname="Laszlo" + subject="comment 5" + date="2013-08-25T07:48:18Z" + content=""" +What is the problem with bittorrent protocol in general? +It is some technicality or purely philosophical? + +Best, + Laszlo + +"""]] diff --git a/doc/todo/A_really_simple_way_to_pair_devices_like_bittorent_sync/comment_6_56e53803fdede895cba717e6b6e9a1bb._comment b/doc/todo/A_really_simple_way_to_pair_devices_like_bittorent_sync/comment_6_56e53803fdede895cba717e6b6e9a1bb._comment new file mode 100644 index 000000000..41e1bda78 --- /dev/null +++ b/doc/todo/A_really_simple_way_to_pair_devices_like_bittorent_sync/comment_6_56e53803fdede895cba717e6b6e9a1bb._comment @@ -0,0 +1,23 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawmkBwMWvNKZZCge_YqobCSILPMeK6xbFw8" + nickname="develop" + subject="comment 6" + date="2013-08-25T08:39:15Z" + content=""" +I just did a cursory search on haskell torrent support. And the required pieces do seem to be be there. +https://github.com/jlouis/combinatorrent or https://github.com/astro/haskell-torrent for downloading. i'm not sure if either supports DHT, but that exists here https://github.com/aninhumer/haskell-dht + +That said, i think implementing this would require some quite major overhauls in the system. It probably won't be trivial to implement. + +Note: This is for straight \"bittorrent\", not for \"bittorrent sync\". Bittorrent sync is closed source, and while an API might come at some point, it doesn't currently exist. + +I do seem to recall joeyh talking about supporting further transport protocols(perhaps through hooks). So I'm adding the above links for future reference if this does get implemented. + +But IMHO, this doesn't seem like a trivial feature to add. It might have to take some refactoring of some core git-annex parts. Certain things have to be changed quite a bit. + +Currently a git-annex client doesn't really require anything(except rsync) to sync from a remote. With bittorrent with DHT support to share between clients, suddenly git-annex will have to maintain a constant bittorrent thread(maybe multiple) that constantly seeds all the files in the git-annex repository, while waiting for a potential remote to request data. + +So even if this happens, it is probably gonna take some time. + +Just my 2cents. +"""]] diff --git a/doc/todo/Bittorrent-like_features.mdwn b/doc/todo/Bittorrent-like_features.mdwn new file mode 100644 index 000000000..41988a422 --- /dev/null +++ b/doc/todo/Bittorrent-like_features.mdwn @@ -0,0 +1,45 @@ +There are two different possible ways git-annex could use bittorrent: + +Let's describe those one by one. + +[[!toc]] + +Downloading files from multiple git-annex sources simultaneously +================================================================ + +Having your remotes (optionally!) act like a swarm would be an awesome feature to have because you bring in a lot of new features that optimize storage, bandwidth, and overall traffic usage. This would be made a lot easier if parts of it were implemented in small steps that added a nifty feature. The best part is, each of these could be implemented by themselves, and they're all features that would be really useful. + + 1. Concurrent downloads of a file from remotes. + + This would make sense to have, it saves upload traffic on your remotes, and you also get faster DL speeds on the receiving end. + + 2. Implementing part of the super-seeding capabilities. + + You upload pieces of a file to different remotes from your laptop, and on your desktop you can download all those pieces and put them together again to get a complete file. If you really wanted to get fancy, you could build in redundancy (ala RAID) so if a remote or two gets lost, you don't lose the entire file. This would be a very efficient use of storage if you have a bunch of free cloud storage accounts (~1GB each) and some big files you want to back up. + + 3. Setting it up so that those remotes could talk to one another and share those pieces. + + This is where it gets more like bittorrent. Useful because you upload 1 copy and in a few hours, have say, 5 complete copies on 5 different remotes. You could add or remove remotes from a swarm locally, and push those changes to those remotes, which then adapt themselves to suit the new rules and share those with other remotes in the swarm (rules should be GPG-signed as a safety precaution). Also, if/when deltas get implemented, you could push that delta to the swarm and have all the remotes adopt it. This is cooler than regular bittorrent because the shared file can be updated. As a safety precaution, the delta could be GPG signed so a corrupt file doesn't contaminate the entire swarm. Each remote could have bandwidth/storage limits set in a dotfile. + +This is a high-level idea of how it might work, and it's also a HUGE set of features to add, but if implemented, you'd be saving a ton of resources, adding new use cases, and making git-annex more flexible. + +Obviously, Step 3 would only work on remotes that you have control of processes on, but if given login credentials to cloud storage remotes (potentially dangerous!) they could read/write to something like dropbox or rsync. + +Another thing, this would be completely trackerless. You just use remote groups (or create swarm definitions) and share those with your remotes. **It's completely decentralized!** + +This was originally posted [[as a forum post|forum/Wishlist:_Bittorrent-like_transfers]] by [[users/GLITTAH]]. + +Using an external client (addurl torrent support) +================================================= + +The alternative to this would be to add `addurl` support for bittorrent files. The same way we can now add Youtube videos to a git-annex repository thanks to [[quvi]], we could also simply do: + + git annex addtorrent debian-live-7.0.0-amd64-standard.iso.torrent + +or even better: + + git annex addurl http://cdimage.debian.org/debian-cd/current-live/amd64/bt-hybrid/debian-live-7.0.0-amd64-standard.iso.torrent + +This way, a torrent would just become another source for a specific file. When we `get` the file, it fires up `$YOUR_FAVORITE_TORRENT_CLIENT` to download the file. + +That way we avoid the implementation complexity of shoving a complete bittorrent client within the assistant. The `get` operation would block until the torrent is downloaded, i guess... --[[anarcat]] diff --git a/doc/todo/Bittorrent-like_features/comment_1_f4c110ef35ebf4fd89f06edf2c4f0c48._comment b/doc/todo/Bittorrent-like_features/comment_1_f4c110ef35ebf4fd89f06edf2c4f0c48._comment new file mode 100644 index 000000000..eba291af9 --- /dev/null +++ b/doc/todo/Bittorrent-like_features/comment_1_f4c110ef35ebf4fd89f06edf2c4f0c48._comment @@ -0,0 +1,13 @@ +[[!comment format=mdwn + username="Gastlag" + ip="109.190.97.30" + subject="Gittorrent" + date="2013-08-28T21:49:56Z" + content=""" +May this could interest you : few years ago somes tried to mix Git and Bittorrent. + +http://www.advogato.org/article/994.html +http://utsl.gen.nz/gittorrent/rfc.html +http://code.google.com/p/gittorrent/ +https://git.wiki.kernel.org/index.php/SoC2010Application#Did_your_organization_participate_in_past_GSoCs.3F_If_so.2C_please_summarize_your_involvement_and_the_successes_and_challenges_of_your_participation +"""]] diff --git a/doc/todo/Build_for_Synology_DSM.mdwn b/doc/todo/Build_for_Synology_DSM.mdwn new file mode 100644 index 000000000..be45ea631 --- /dev/null +++ b/doc/todo/Build_for_Synology_DSM.mdwn @@ -0,0 +1 @@ +It would be wonderful if a pre-built package would be available for Synology NAS. Basically, this is an ARM-based Linux. It has most of the required shell commands either out of the box or easily available (through ipkg). But I think it would be difficult to install the Haskell compiler and all the required modules, so it would probably be better to cross-compile targeting ARM. diff --git a/doc/todo/Build_for_Synology_DSM/comment_10_e351084d9a83db3fd6d9d983227a6410._comment b/doc/todo/Build_for_Synology_DSM/comment_10_e351084d9a83db3fd6d9d983227a6410._comment new file mode 100644 index 000000000..b62a929d7 --- /dev/null +++ b/doc/todo/Build_for_Synology_DSM/comment_10_e351084d9a83db3fd6d9d983227a6410._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawkwjBDXkP9HAQKhjTgThGOxUa1B99y_WRA" + nickname="Franck" + subject="comment 10" + date="2013-06-02T17:23:43Z" + content=""" +I updated the C program to simplify it so it uses a static path for `_chrooter`. In the previous version, I suspect that one can play with symlinks and use it to get a root shell. So, if `_chrooter` is not installed in `/opt/bin` this file has to be edited too before compilation. +"""]] diff --git a/doc/todo/Build_for_Synology_DSM/comment_11_cc67a584f5c460a6fb63cf099c20e573._comment b/doc/todo/Build_for_Synology_DSM/comment_11_cc67a584f5c460a6fb63cf099c20e573._comment new file mode 100644 index 000000000..324fa8423 --- /dev/null +++ b/doc/todo/Build_for_Synology_DSM/comment_11_cc67a584f5c460a6fb63cf099c20e573._comment @@ -0,0 +1,9 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawkwjBDXkP9HAQKhjTgThGOxUa1B99y_WRA" + nickname="Franck" + subject="comment 11" + date="2013-06-03T09:55:54Z" + content=""" +A last update and I stop spamming this thread: I've implemented access control and simplified customisation. All this has been moved to https://bitbucket.org/franckp/gasp + +"""]] diff --git a/doc/todo/Build_for_Synology_DSM/comment_12_94023593d294b9cf69090fcfd6ca0e5a._comment b/doc/todo/Build_for_Synology_DSM/comment_12_94023593d294b9cf69090fcfd6ca0e5a._comment new file mode 100644 index 000000000..39c243ec4 --- /dev/null +++ b/doc/todo/Build_for_Synology_DSM/comment_12_94023593d294b9cf69090fcfd6ca0e5a._comment @@ -0,0 +1,9 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawlJEI45rGczFAnuM7gRSj4C6s9AS9yPZDc" + nickname="Kevin" + subject="SynoCommunity" + date="2013-06-26T18:12:39Z" + content=""" +Creating an installable git-annex package available via [SynoCommunity](http://www.synocommunity.com/) would be awesome. They have created [cross-compilation tools](https://github.com/SynoCommunity/spksrc) to help build the packages and integrate the start/stop scripts with the package manager. + +"""]] diff --git a/doc/todo/Build_for_Synology_DSM/comment_13_314255fd503d125b5aeae2f62acfd592._comment b/doc/todo/Build_for_Synology_DSM/comment_13_314255fd503d125b5aeae2f62acfd592._comment new file mode 100644 index 000000000..3c54a9271 --- /dev/null +++ b/doc/todo/Build_for_Synology_DSM/comment_13_314255fd503d125b5aeae2f62acfd592._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawnrP-0DGtHDJbWSXeiyk0swNkK1aejoN3c" + nickname="sebastien" + subject="comment 13" + date="2013-08-06T12:18:35Z" + content=""" +I post an issue to github synocommunity for that, i hope somenone have some time to package this great features. +"""]] diff --git a/doc/todo/Build_for_Synology_DSM/comment_15_9525cd0d75ff4c15182d10a855774b69._comment b/doc/todo/Build_for_Synology_DSM/comment_15_9525cd0d75ff4c15182d10a855774b69._comment new file mode 100644 index 000000000..c3edf99e2 --- /dev/null +++ b/doc/todo/Build_for_Synology_DSM/comment_15_9525cd0d75ff4c15182d10a855774b69._comment @@ -0,0 +1,30 @@ +[[!comment format=mdwn + username="lorenzo" + ip="84.75.27.69" + subject="Running Debian squeeze binaries on libc 2.5 based NAS" + date="2013-10-27T23:56:26Z" + content=""" +Following the suggestions in this page I tried to run the binaries that debian provides on my Lacie NetworkSpace which is another one of these NAS devices with old libc. After uploading the binaries and required libraries and using `LD_LIBRARY_PATH` to force the loader to use the version I uploaded of the libraries I was still having a segfault (similar to what Franck was experiencing) while running git-annex in a chroot was working. + +It turns out that it is possible to solve the problem without having to use chroot by not loading the binary directly but by substituting it with a script that calls the correct `ld-linux.so.3`. Assume you have uncompressed the files from the deb packages in `/opt/git-annex`. + +First create a directory `/opt/git-annex/usr/bin/git-annex.exec` and copy the executable `/opt/git-annex/usr/bin/git-annex` there. + +Then create script `/opt/git-annex/usr/bin/git-annex` with the following contents: + + #!/bin/bash + + PREFIX=/opt/git-annex + + export GCONV_PATH=$PREFIX/usr/lib/gconv + + exec $PREFIX/lib/ld-linux.so.3 --library-path $PREFIX/lib/:$PREFIX/usr/lib/ $PREFIX/usr/bin/git-annex.exec/git-annex \"$@\" + +The `GCONV_PATH` setting is important to prevent the app from failing with the message: + + git-annex.exec: mkTextEncoding: invalid argument (Invalid argument) + +The original executable is moved to a different directory instead of being simply renamed to make sure that `$0` is correct when the executable starts. The parameter for the linker `--library-path` is used instead of the environment variable `LD_LIBRARY_PATH` to make sure that the programs exec'ed by git-annex do not have the variable set. + +Some more info about the approach: [[http://www.novell.com/coolsolutions/feature/11775.html]] +"""]] diff --git a/doc/todo/Build_for_Synology_DSM/comment_1_4059016fa8da6af7a3eba8966821e8eb._comment b/doc/todo/Build_for_Synology_DSM/comment_1_4059016fa8da6af7a3eba8966821e8eb._comment new file mode 100644 index 000000000..074ba998c --- /dev/null +++ b/doc/todo/Build_for_Synology_DSM/comment_1_4059016fa8da6af7a3eba8966821e8eb._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + nickname="joey" + subject="comment 1" + date="2013-05-24T15:55:42Z" + content=""" +There are already git-annex builds for arm available from eg, Debian. There's a good chance that, assuming you match up the arm variant (armel, armhf, etc) and that the NAS uses glibc and does not have too old a version, that the binary could just be copied in, possibly with some other libraries, and work. This is what's done for the existing Linux standalone builds. + +So, I look at this bug report as \"please add a standalone build for arm\", not as a request to support a specific NAS which I don't have ;) +"""]] diff --git a/doc/todo/Build_for_Synology_DSM/comment_2_8900c2985ab68b3b566c9f5d326471d6._comment b/doc/todo/Build_for_Synology_DSM/comment_2_8900c2985ab68b3b566c9f5d326471d6._comment new file mode 100644 index 000000000..40e6398f0 --- /dev/null +++ b/doc/todo/Build_for_Synology_DSM/comment_2_8900c2985ab68b3b566c9f5d326471d6._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawkwjBDXkP9HAQKhjTgThGOxUa1B99y_WRA" + nickname="Franck" + subject="comment 2" + date="2013-05-24T21:31:44Z" + content=""" +I tried to run the binary from the Debian package, unfortunately, after installing tons of libraries, git-annex fails complaining that GLIBC is not recent enough. Perhaps a static build for ARM (armel) can solve the problem? Thanks again for your help! +"""]] diff --git a/doc/todo/Build_for_Synology_DSM/comment_3_f2b77368473d42b7f21e9d51d6415b58._comment b/doc/todo/Build_for_Synology_DSM/comment_3_f2b77368473d42b7f21e9d51d6415b58._comment new file mode 100644 index 000000000..651edacd7 --- /dev/null +++ b/doc/todo/Build_for_Synology_DSM/comment_3_f2b77368473d42b7f21e9d51d6415b58._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + nickname="joey" + subject="comment 3" + date="2013-05-25T04:42:22Z" + content=""" +Which Debian package? Different ones link to different libcs. + +(It's not really possible to statically link something with as many dependencies as git-annex on linux anymore, unfortunately.) +"""]] diff --git a/doc/todo/Build_for_Synology_DSM/comment_4_a55fea734044c270ceb10adf9c8d9a76._comment b/doc/todo/Build_for_Synology_DSM/comment_4_a55fea734044c270ceb10adf9c8d9a76._comment new file mode 100644 index 000000000..50ae82ca0 --- /dev/null +++ b/doc/todo/Build_for_Synology_DSM/comment_4_a55fea734044c270ceb10adf9c8d9a76._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawkwjBDXkP9HAQKhjTgThGOxUa1B99y_WRA" + nickname="Franck" + subject="comment 4" + date="2013-05-25T07:40:13Z" + content=""" +I've actually tried several ones: 4.20130521 on sid, 3.20120629~bpo60+2 on squeeze-backports, 3.20120629 on wheezy and jessie, plus a package for Ubuntu 11.02. All of them try to load GLIBC 2.6/2.7 while my system has 2.5 only... I'll try a different approach: install Debian in a chroot on the NAS and extract all the required files, including all libraries. +"""]] diff --git a/doc/todo/Build_for_Synology_DSM/comment_5_59865ada057c640ac29855c65cf45dd9._comment b/doc/todo/Build_for_Synology_DSM/comment_5_59865ada057c640ac29855c65cf45dd9._comment new file mode 100644 index 000000000..725025283 --- /dev/null +++ b/doc/todo/Build_for_Synology_DSM/comment_5_59865ada057c640ac29855c65cf45dd9._comment @@ -0,0 +1,23 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawkwjBDXkP9HAQKhjTgThGOxUa1B99y_WRA" + nickname="Franck" + subject="comment 5" + date="2013-05-25T10:03:24Z" + content=""" +Unfortunately, chroot approach does not work either. While git-annex works fine when I'm in the chroot, it doesn't work any more outside. If I don't copy libc, I get a version error (just like before so this is normal): + + git-annex: /lib/libc.so.6: version `GLIBC_2.7' not found (required by /opt/share/git-annex/bin/git-annex) + git-annex: /lib/libc.so.6: version `GLIBC_2.6' not found (required by /opt/share/git-annex/bin/git-annex) + git-annex: /lib/libc.so.6: version `GLIBC_2.7' not found (required by /opt/share/git-annex/lib/libgmp.so.10) + +When I copy libc from the Debian chroot, then, it complains about libpthread: + + git-annex: relocation error: /lib/libpthread.so.0: symbol __default_rt_sa_restorer, version GLIBC_PRIVATE not defined in file libc.so.6 with link time reference + +If then I copy libpthread also, I get: + + Illegal instruction (core dumped) + +So, I'm stuck... :-( +I'll try to find a way using the version in the chroot instead of trying to export it to the host system... +"""]] diff --git a/doc/todo/Build_for_Synology_DSM/comment_6_6d860b1ad8816077b5fa596a71b12d5c._comment b/doc/todo/Build_for_Synology_DSM/comment_6_6d860b1ad8816077b5fa596a71b12d5c._comment new file mode 100644 index 000000000..417293db3 --- /dev/null +++ b/doc/todo/Build_for_Synology_DSM/comment_6_6d860b1ad8816077b5fa596a71b12d5c._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawln3ckqKx0x_xDZMYwa9Q1bn4I06oWjkog" + nickname="Michael" + subject="bind mount" + date="2013-05-25T15:55:52Z" + content=""" +You could bind-mount (e.g. mount -o bind /data /chroot/data ) your main Synology fs into the chroot for git-annex to use. +"""]] diff --git a/doc/todo/Build_for_Synology_DSM/comment_7_19ef2d293ba3bc7ece443d7278371c3f._comment b/doc/todo/Build_for_Synology_DSM/comment_7_19ef2d293ba3bc7ece443d7278371c3f._comment new file mode 100644 index 000000000..47d092331 --- /dev/null +++ b/doc/todo/Build_for_Synology_DSM/comment_7_19ef2d293ba3bc7ece443d7278371c3f._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawkwjBDXkP9HAQKhjTgThGOxUa1B99y_WRA" + nickname="Franck" + subject="comment 7" + date="2013-05-25T19:01:29Z" + content=""" +This is indeed what I'm doing. But I need to make a wrapper that will call the command in the chroot. Thanks for the tip anyway. :-) +"""]] diff --git a/doc/todo/Build_for_Synology_DSM/comment_8_609b7ad87dfbba49ec1f8c6fc2739ccd._comment b/doc/todo/Build_for_Synology_DSM/comment_8_609b7ad87dfbba49ec1f8c6fc2739ccd._comment new file mode 100644 index 000000000..8a3490956 --- /dev/null +++ b/doc/todo/Build_for_Synology_DSM/comment_8_609b7ad87dfbba49ec1f8c6fc2739ccd._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawmqz6wCn-Q1vzrsHGvEJHOt_T5ZESilxhc" + nickname="Sören" + subject="comment 8" + date="2013-05-26T13:50:31Z" + content=""" +I have a Synology NAS too, so I thought I could try to run git-annex in a Debian chroot. +As it [turns out](http://forum.synology.com/wiki/index.php/What_kind_of_CPU_does_my_NAS_have), my model (DS213+) runs on a PowerPC CPU instead of ARM. Unfortunately, it isn't compatible with PPC in Debian either because it is a different PowerPC variant. +There is an unofficial Debian port called [powerpcspe](http://wiki.debian.org/PowerPCSPEPort), but ghc doesn't build there yet for [some reason](http://buildd.debian-ports.org/status/package.php?p=git-annex&suite=sid). + +Any chance that there will be a build for this architecture at some point in the future or should I better look for another NAS? ;-) +"""]] diff --git a/doc/todo/Build_for_Synology_DSM/comment_9_d94a73b9a07c5cadf191005f817fd59a._comment b/doc/todo/Build_for_Synology_DSM/comment_9_d94a73b9a07c5cadf191005f817fd59a._comment new file mode 100644 index 000000000..c8b45fc60 --- /dev/null +++ b/doc/todo/Build_for_Synology_DSM/comment_9_d94a73b9a07c5cadf191005f817fd59a._comment @@ -0,0 +1,29 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawkwjBDXkP9HAQKhjTgThGOxUa1B99y_WRA" + nickname="Franck" + subject="comment 9" + date="2013-06-02T13:14:56Z" + content=""" +Hi, I finally succeeded! :-) + +Here are the main steps: + + 1. install `debian-chroot` on the NAS + 2. create an account `gitannex` in Debian + 3. configure git on this account (this is important otherwise git complains and fails) `git config --global user.email YOUR_EMAIL` and `git config --global user.name YOUR_NAME` + 4. install `gcc` on the NAS (using `ipkg`) + 5. download the files here: https://www.dropbox.com/sh/b7z68a730aj3mnm/95nFOzE1QP + 6. edit `_chrooter` to fit your settings (probably there is nothing to change if your Debian is freshly installed) + 7. run `make install`, everything goes to `/opt/bin`, if you change this, you should also edit line 17 in file `gasp` + 8. create an account `gitannex` on the NAS (doesn't need to be the same name as in Debian, but I feel it is easier) + 9. edit its `.ssh/authorized_keys` to prefix lines as follows `command=\"gasp\" THE_PUBLIC_KEY_AS_USUAL` + 10. it should work + 11. the repositories will be in the Debian account, but it's easy to symlink them in the NAS account if you wish + +The principle is as follows: `command=\"gasp\"` allows to launch `gasp` on SSH connexion instead of the original command given to `ssh`. This command is retrieved by `gasp` and prefixed with `chrooter-` (so, eg, running `ssh git` on the client results in running `chrooter-git` on the NAS). `chrooter-*` commands are symlinks to `chrooter`, this is a setuid root binary that launches `_chrooter`. (This intermediary binary is necessary because `_chrooter` is a script which cannot be setuid, and setuid is required for the chroot and identity change.) Finally, `_chrooter` starts the `debian-chroot` service, chroot to the target dir, changes identity and eventually launches the original command as if it was lauched directly by `gitannex` user in Debian. `_chrooter` and `gasp` are Python scripts, I did not use shell in order to avoid error-prone issues with spaces in arguments (that need to be passed around several times in the process). + +I'll try now to add command-line parameters to `gasp` in order to restrict the commands that can be run through SSH and the repositories allowed. + +Cheers, +Franck +"""]] diff --git a/doc/todo/Check_if_an_upgrade_is_available_in_the_webapp.mdwn b/doc/todo/Check_if_an_upgrade_is_available_in_the_webapp.mdwn new file mode 100644 index 000000000..e102606ca --- /dev/null +++ b/doc/todo/Check_if_an_upgrade_is_available_in_the_webapp.mdwn @@ -0,0 +1,5 @@ +Especially on Mac OSX (and Windows, and maybe Android), it would be great to be able to check in the webapp if an upgrade is available. A deeper integration with these OS would be even better: for example on Mac OSX, an icon on the status bar list available upgrades for some programs, including LibreOffice and others which are not installed by default. + +Also, it would be great to be able to download and install git-annex upgrades directly from the webapp. + +> comprehensively [[done]]; [[design/assistant/upgrading]] --[[Joey]] diff --git a/doc/todo/Check_if_an_upgrade_is_available_in_the_webapp/comment_1_c904182f6bff8b1a42070bbc038eb34e._comment b/doc/todo/Check_if_an_upgrade_is_available_in_the_webapp/comment_1_c904182f6bff8b1a42070bbc038eb34e._comment new file mode 100644 index 000000000..750e3b83a --- /dev/null +++ b/doc/todo/Check_if_an_upgrade_is_available_in_the_webapp/comment_1_c904182f6bff8b1a42070bbc038eb34e._comment @@ -0,0 +1,17 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="209.250.56.246" + subject="comment 1" + date="2013-11-15T20:51:18Z" + content=""" +I have thought about doing this, especially if there is ever a security hole in git-annex. + +All it needs is a file containing the version number to be written along-side the git-annex build, and git-annex knowing if it was built as a standalone build, and should check that. + +As for actually performing the upgrade: + +* Easy on Linux +* Not sure on OSX.. Is it possible to use hdiutil attach to replace a dmg while a program contained in it is currently running? +* Probably impossible on Android, at least not without using double the space. Probably better to get git-annex into an app store. +* Doable on Windows, but would need git-annex to be distributed in a form that was not a installer.exe. +"""]] diff --git a/doc/todo/Deleting_Unused_Files_by_Age.mdwn b/doc/todo/Deleting_Unused_Files_by_Age.mdwn new file mode 100644 index 000000000..b72768bca --- /dev/null +++ b/doc/todo/Deleting_Unused_Files_by_Age.mdwn @@ -0,0 +1,13 @@ +I periodically move unused files to one of my servers. What I would like to +do is drop any unused file that has been unused for say more than 6 months? +I would like to not drop all unused files. + +> It strikes me that this is quite similar to how git handles deleting +> stale refs with the reflog. So, if `git annex unused` were changed to +> also look at the reflog, it would keep all files referred to by all refs +> in the reflog, until the reflog expires. You could then set reflog expiry +> to 6 months, and be done. +> +> However, I think that many users expect git annex unused to be able to +> immediately find and remove a file after it's been deleted. So this +> probably needs to be a configurable behavior. --[[Joey]] diff --git a/doc/todo/Move_ssh_config_to___126____47__ssh__47__git-annex__47__config.mdwn b/doc/todo/Move_ssh_config_to___126____47__ssh__47__git-annex__47__config.mdwn new file mode 100644 index 000000000..5dc063100 --- /dev/null +++ b/doc/todo/Move_ssh_config_to___126____47__ssh__47__git-annex__47__config.mdwn @@ -0,0 +1,7 @@ +### Please describe the problem. +Instead of storing config for each remote in ~/.ssh/config, which mixes the user own config with that of git-annex-assistant, which is irritating if (like me) you store your ssh config in a vcs. Since the option -F allows the choice of the config file, it should be possible to move the config into ~/.ssh/git-annex/config. The only issue I see is according to the ssh man page on my system states that the system-wide config is ignored if a config file is specified on the command line. + +### What version of git-annex are you using? On what operating system? +I'm using git-annex 4.20130601 on a Debian Testing/Unstable/Experimental mix. + +[[!tag design/assistant]] diff --git a/doc/todo/Move_ssh_config_to___126____47__ssh__47__git-annex__47__config/comment_1_284c806e83a32af81b02aea7c7bc285a._comment b/doc/todo/Move_ssh_config_to___126____47__ssh__47__git-annex__47__config/comment_1_284c806e83a32af81b02aea7c7bc285a._comment new file mode 100644 index 000000000..5997664e0 --- /dev/null +++ b/doc/todo/Move_ssh_config_to___126____47__ssh__47__git-annex__47__config/comment_1_284c806e83a32af81b02aea7c7bc285a._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + nickname="joey" + subject="comment 1" + date="2013-06-11T14:44:47Z" + content=""" +The only interface git provides to do this is `GIT_SSH`, which would have to be set to a wrapper script that runs ssh with the desirned options. + +And if that were used, `git pull` by itself would not work on the repositories set up by the assistant. I don't consider that very nice. +"""]] diff --git a/doc/todo/Move_ssh_config_to___126____47__ssh__47__git-annex__47__config/comment_2_1f55ad6b39906458779b2d604b003ffe._comment b/doc/todo/Move_ssh_config_to___126____47__ssh__47__git-annex__47__config/comment_2_1f55ad6b39906458779b2d604b003ffe._comment new file mode 100644 index 000000000..3cf75df08 --- /dev/null +++ b/doc/todo/Move_ssh_config_to___126____47__ssh__47__git-annex__47__config/comment_2_1f55ad6b39906458779b2d604b003ffe._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + nickname="joey" + subject="comment 2" + date="2013-06-11T14:48:01Z" + content=""" +Also, if you're going to set up something like local pairing, why would you *not* want to commit that config to git along with your other ssh configs? Config files in $HOME are quite frequently edited by helper programs to configure changes, and I personally commit those changes all the time. + +Perhaps your real problem is that you have one `.ssh/config` that is shared between multiple hosts, and the git-annex settings are specific to a single host. Have you considered using [vcsh](https://github.com/RichiH/vcsh)? +"""]] diff --git a/doc/todo/Move_ssh_config_to___126____47__ssh__47__git-annex__47__config/comment_3_b00dce2374aac6968317d05d23bcfaf7._comment b/doc/todo/Move_ssh_config_to___126____47__ssh__47__git-annex__47__config/comment_3_b00dce2374aac6968317d05d23bcfaf7._comment new file mode 100644 index 000000000..3442fb2b2 --- /dev/null +++ b/doc/todo/Move_ssh_config_to___126____47__ssh__47__git-annex__47__config/comment_3_b00dce2374aac6968317d05d23bcfaf7._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawk3Wgg0XiqYFwM_Pw1RxZwlpNFi65g17sM" + nickname="James" + subject="comment 3" + date="2013-06-12T01:12:24Z" + content=""" +Ah, ok, I presumed there was an option in git to set a per-repository ssh command. I've looked at vcsh, but I'm not that confident with git remotes, so I don't use it (I use hg). If a per-repository ssh command added to git, would you consider adding this? +"""]] diff --git a/doc/todo/Move_ssh_config_to___126____47__ssh__47__git-annex__47__config/comment_4_743d0b077110c5cac1e2f47187b75333._comment b/doc/todo/Move_ssh_config_to___126____47__ssh__47__git-annex__47__config/comment_4_743d0b077110c5cac1e2f47187b75333._comment new file mode 100644 index 000000000..5a22c98f7 --- /dev/null +++ b/doc/todo/Move_ssh_config_to___126____47__ssh__47__git-annex__47__config/comment_4_743d0b077110c5cac1e2f47187b75333._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + nickname="joey" + subject="comment 4" + date="2013-06-12T19:23:50Z" + content=""" +If it were sane, I'd probably use it. + +In the meantime, I'm moving this to [[todo]]. +"""]] diff --git a/doc/todo/Not_working_on_Android-x86.mdwn b/doc/todo/Not_working_on_Android-x86.mdwn new file mode 100644 index 000000000..56f2ce962 --- /dev/null +++ b/doc/todo/Not_working_on_Android-x86.mdwn @@ -0,0 +1,19 @@ +[[!meta title="Android is only autobuilt for arm, not x86 or mips"]] + +### Please describe the problem. + +git-annex doesn't start on [Android-x86](http://www.android-x86.org) in VirtualBox (version 4.1.18-dfsg-2+deb7u1). + +On Android 4.2.2 (android-x86-4.2-20130228.iso) it starts the terminal which prints nothing but `[Terminal session finished]`. +On Android 4.3 (android-x86-4.3-20130725.iso) it starts the terminal and prints: + + In mgmain JNI_OnLoad + + [Terminal session finished] + +The browser/webapp is never started. + +### What version of git-annex are you using? On what operating system? + +Version 1.0.52 for Android. I made sure to install the correct APK files for each version of Android. + diff --git a/doc/todo/Not_working_on_Android-x86/comment_1_5eec4d7530c9df68f1bd1b1ca7021ef5._comment b/doc/todo/Not_working_on_Android-x86/comment_1_5eec4d7530c9df68f1bd1b1ca7021ef5._comment new file mode 100644 index 000000000..4bcca0ab4 --- /dev/null +++ b/doc/todo/Not_working_on_Android-x86/comment_1_5eec4d7530c9df68f1bd1b1ca7021ef5._comment @@ -0,0 +1,14 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="209.250.56.64" + subject="comment 1" + date="2013-11-22T16:37:36Z" + content=""" +git-annex is a native binary, and is currently only being built for arm android. + +Is Android on x86 a thing used on real-world hardware? I have only seen it in the context of a developer's test environment. + +The instructions for building git-annex from source for Android should work on x86. The ghc-android build would need to be tweaked to build a cross compiler targeting that architecture. This can be done by editing settings near the top of ghc-android's `build.sh`. + +I am going to move this from bugs/ to todo/ after posting this comment, because it is certiantly not a bug, but a wishlist item at best. +"""]] diff --git a/doc/todo/Option_for_browser_to_launch_webapp_with.mdwn b/doc/todo/Option_for_browser_to_launch_webapp_with.mdwn new file mode 100644 index 000000000..dae601169 --- /dev/null +++ b/doc/todo/Option_for_browser_to_launch_webapp_with.mdwn @@ -0,0 +1,7 @@ +Firefox is my default browser, but as we all know, it doesn't load quickly. If I don't have Firefox running but I want to access the git-annex webapp, I'd rather launch the webapp in some small, quick browser like QupZilla than wait for Firefox to load. + +Could git-annex have a setting, maybe a "webapp --browser" option and/or a setting in the config file, to specify the browser to launch? + +> git-annex uses the standard `git config web.browser` if you set it. +> [[done]] +> --[[Joey]] diff --git a/doc/todo/Please_abort_build_if___34__make_test__34___fails.mdwn b/doc/todo/Please_abort_build_if___34__make_test__34___fails.mdwn new file mode 100644 index 000000000..592b5e077 --- /dev/null +++ b/doc/todo/Please_abort_build_if___34__make_test__34___fails.mdwn @@ -0,0 +1,7 @@ +A failure during "make test" should be signalled to the caller by means of +a non-zero exit code. Without that signal, it's very hard to run the +regression test suite in an automated fashion. + +> git-annex used to have a Makefile that ignored make test exit status, +> but that was fixed in commit dab5bddc64ab4ad479a1104748c15d194e138847, +> in October 6th. [[done]] --[[Joey]] diff --git a/doc/todo/Please_add_support_for_monad-control_0.3.x.mdwn b/doc/todo/Please_add_support_for_monad-control_0.3.x.mdwn new file mode 100644 index 000000000..f82224991 --- /dev/null +++ b/doc/todo/Please_add_support_for_monad-control_0.3.x.mdwn @@ -0,0 +1,9 @@ +Git-annex doesn't compile with the latest version of monad-control. Would it be hard to support that new version? + +> I have been waiting for it to land in Debian before trying to +> deal with its changes. +> +> There is now a branch in git called `new-monad-control` that will build +> with the new monad-control. --[[Joey]] + +>> Now merged to master. [[done]] --[[Joey]] diff --git a/doc/todo/S3.mdwn b/doc/todo/S3.mdwn new file mode 100644 index 000000000..7e417336f --- /dev/null +++ b/doc/todo/S3.mdwn @@ -0,0 +1,24 @@ +Support Amazon S3 as a file storage backend. + +There's a haskell library that looks good. Not yet in Debian. + +Multiple ways of using S3 are possible. Currently implemented as +a special type of git remote. + +Before this can be close, I need to fix: + +## encryption + +TODO + +## unused checking + +One problem is `git annex unused`. Currently it only looks at the local +repository, not remotes. But if something is dropped from the local repo, +and you forget to drop it from S3, cruft can build up there. + +This could be fixed by adding a hook to list all keys present in a remote. +Then unused could scan remotes for keys, and if they were not used locally, +offer the possibility to drop them from the remote. + +[[done]] diff --git a/doc/todo/Show_repo_type_in_repo_list.mdwn b/doc/todo/Show_repo_type_in_repo_list.mdwn new file mode 100644 index 000000000..40fbe6537 --- /dev/null +++ b/doc/todo/Show_repo_type_in_repo_list.mdwn @@ -0,0 +1 @@ +It would be helpful to show each repo's type in the list. diff --git a/doc/todo/Show_repo_type_in_repo_list/comment_1_ac6eb1072ef902a094b79dd8e0917c4d._comment b/doc/todo/Show_repo_type_in_repo_list/comment_1_ac6eb1072ef902a094b79dd8e0917c4d._comment new file mode 100644 index 000000000..10c0c17df --- /dev/null +++ b/doc/todo/Show_repo_type_in_repo_list/comment_1_ac6eb1072ef902a094b79dd8e0917c4d._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="209.250.56.47" + subject="comment 1" + date="2013-11-02T23:45:39Z" + content=""" +Currently if you go to the edit page for the repository, it shows some information about it, including its type and often its location, at the bottom of the page. + +I tend to feel that putting anything else in the repo list would result in it being too cluttered. +"""]] diff --git a/doc/todo/Show_repo_type_in_repo_list/comment_2_6979c487f707a724a048d20e2e5744e6._comment b/doc/todo/Show_repo_type_in_repo_list/comment_2_6979c487f707a724a048d20e2e5744e6._comment new file mode 100644 index 000000000..7d4a36324 --- /dev/null +++ b/doc/todo/Show_repo_type_in_repo_list/comment_2_6979c487f707a724a048d20e2e5744e6._comment @@ -0,0 +1,14 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawmNu4V5fvpLlBhaCUfXXOB0MI5NXwh8SkU" + nickname="Adam" + subject="comment 2" + date="2013-11-02T23:59:16Z" + content=""" +I understand. Well, here's what I'm looking at right now. :) + +[[http://alphapapa.net/outbox/view.png]] + +I just think it would be very useful to say \"Client\" or \"Transfer\" or \"Full Backup\" next to each one, and there's often plenty of room, depending on the size of the window. Especially when one's trying to wrap one's head around git-annex, being reminded what each repo is for would help a lot. Otherwise you basically have to put it in the name or description yourself...so why not just go ahead and show it? :) + +Another option might be to have an icon for each type. +"""]] diff --git a/doc/todo/Show_repo_type_in_repo_list/comment_3_529254a6cc20de7259d60a3cbc5ccaf7._comment b/doc/todo/Show_repo_type_in_repo_list/comment_3_529254a6cc20de7259d60a3cbc5ccaf7._comment new file mode 100644 index 000000000..3f8b82695 --- /dev/null +++ b/doc/todo/Show_repo_type_in_repo_list/comment_3_529254a6cc20de7259d60a3cbc5ccaf7._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="209.250.56.47" + subject="comment 3" + date="2013-11-03T00:19:34Z" + content=""" +I'd be happy to put in some icons if someone finds some good (and suitably licensed) ones that cover the different types of repositories. +"""]] diff --git a/doc/todo/Slow_transfer_for_a_lot_of_small_files..mdwn b/doc/todo/Slow_transfer_for_a_lot_of_small_files..mdwn new file mode 100644 index 000000000..00cdad0fe --- /dev/null +++ b/doc/todo/Slow_transfer_for_a_lot_of_small_files..mdwn @@ -0,0 +1,20 @@ +What steps will reproduce the problem? +Sync a lot of small files. + +What is the expected output? What do you see instead? +The expected output is hopefully a fast transfer. + +But currently it seems like git-annex is only using one thread to transfer(per host or total?) + +An option to select number of transfer threads to use(possibly per host) would be very nice. + +> Opening a lot of connections to a single host is probably not desirable. +> +> I do want to do something to allow slow hosts to not hold up transfers to +> other hosts, which might involve running multiple queued transfers at +> once. The webapp already allows the user to force a given transfer to +> happen immediately. --[[Joey]] + +And maybe also an option to limit how long a queue the browser should show, it can become quite resource intensive with a long queue. + +> The queue is limited to 20 items for this reason. --[[Joey]] diff --git a/doc/todo/Sync_repo_names__63__.mdwn b/doc/todo/Sync_repo_names__63__.mdwn new file mode 100644 index 000000000..d3bb59f04 --- /dev/null +++ b/doc/todo/Sync_repo_names__63__.mdwn @@ -0,0 +1,10 @@ +It's very confusing to me that the same repo viewed from different client systems can have different names and descriptions. This implies that making changes to a remote repo from one system only affects how that system sees the repo, but it seems to affect how the entire git-annex "pair" or "network of repos" sees it. + +I think it would be good if the names and descriptions of repos were synced across clients. + +> The descriptions of repositories are synced. (They're stored in git-annex:uuid.log) +> +> git allows for the same repository to be referred to using as many different remote names as you want to set up. git-annex inherits this, +> and I can't see this changing; there are very good reasons for remotes to +> have this flexability. [[done]] +> --[[Joey]] diff --git a/doc/todo/Use_MediaScannerConnection_on_Android.mdwn b/doc/todo/Use_MediaScannerConnection_on_Android.mdwn new file mode 100644 index 000000000..afce9308d --- /dev/null +++ b/doc/todo/Use_MediaScannerConnection_on_Android.mdwn @@ -0,0 +1,7 @@ +Currently if photos or videos are copied into the Camera/DCIM directory on an Android device, or deleted the Gallery doesn't notice the changes. + +It is necessary to call MediaScannerConnection - http://developer.android.com/reference/android/media/MediaScannerConnection.html - to notify the system of the change. + +More info, and some sample Java code: http://stackoverflow.com/questions/13270789/how-to-run-media-scanner-in-android + +It'd be awesome if the assistant did this on files it has changed. Possibly just under Camera/DCIM, but perhaps it should be configurable. MediaScannerConnection is also used to notify and index new music files. diff --git a/doc/todo/Use_a_remote_as_a_sharing_site_for_files_with_obfuscated_URLs.mdwn b/doc/todo/Use_a_remote_as_a_sharing_site_for_files_with_obfuscated_URLs.mdwn new file mode 100644 index 000000000..a42a81d02 --- /dev/null +++ b/doc/todo/Use_a_remote_as_a_sharing_site_for_files_with_obfuscated_URLs.mdwn @@ -0,0 +1,7 @@ +There are times when it is handy to be able to upload a file to a web host somewhere and share a link for that file to a select few people. + +It seems to be that the assistant could handle this scenario. It could generate a directory with a random name on the remote, and transfer the file there (using the existing filename) and the appropriate URL could be displayed in the assistant webapp to allow the user to copy the URL to send it to the appropriate people. + +Note: Joey and I had a quick chat about this use case at LCA2013. + +[[!tag design/assistant]] diff --git a/doc/todo/Use_a_remote_as_a_sharing_site_for_files_with_obfuscated_URLs/comment_1_1a1f34f4f389267d67e79409c0ca8b1d._comment b/doc/todo/Use_a_remote_as_a_sharing_site_for_files_with_obfuscated_URLs/comment_1_1a1f34f4f389267d67e79409c0ca8b1d._comment new file mode 100644 index 000000000..35f735191 --- /dev/null +++ b/doc/todo/Use_a_remote_as_a_sharing_site_for_files_with_obfuscated_URLs/comment_1_1a1f34f4f389267d67e79409c0ca8b1d._comment @@ -0,0 +1,9 @@ +[[!comment format=mdwn + username="http://edheil.wordpress.com/" + ip="99.54.57.201" + subject="comment 1" + date="2013-02-09T22:38:56Z" + content=""" +This would be an extremely cool feature to rip off from Dropbox. :) + +"""]] diff --git a/doc/todo/Wishlist:_additional_environment_variables_for_hooks.mdwn b/doc/todo/Wishlist:_additional_environment_variables_for_hooks.mdwn new file mode 100644 index 000000000..6e852b9f2 --- /dev/null +++ b/doc/todo/Wishlist:_additional_environment_variables_for_hooks.mdwn @@ -0,0 +1,14 @@ +It would be nice if a couple of additional environment variables to be set for hook uses. + +In particular: + + GIT_ANNEX_DIRECT=`git config annex.direct` + +and + + GIT_TOP_LEVEL=`git rev-parse --show-toplevel` + + +I've made some changes to flickrannex to allow the sub-directories above the uploaded image to be added as tags. This change has been merged into trunk: [[https://github.com/TobiasTheViking/flickrannex]] + +What I needed was both the environment variables mentioned above. One is set as part of the annex-hook and the other I guestimate from the file path. If it was set in git-annex it would be much cleaner (and accurate). So...I think this info would be useful for other hook. diff --git a/doc/todo/Wishlist:_sanitychecker_fix_wrong_UUID__47__duplicate_remote.mdwn b/doc/todo/Wishlist:_sanitychecker_fix_wrong_UUID__47__duplicate_remote.mdwn new file mode 100644 index 000000000..ff7773d3e --- /dev/null +++ b/doc/todo/Wishlist:_sanitychecker_fix_wrong_UUID__47__duplicate_remote.mdwn @@ -0,0 +1,7 @@ +In certain situations different client annexes might get the same remote repository added, but before being synced. + +Once the two clients sync they will both have two remotes with the same name. But only one UUID will have any content(Assuming only one client pushed). + +It would be nice to have some (automatic?) way to resolve this conflict. + +Not sure if anything sane can be done if both clients have pushed? diff --git a/doc/todo/__96__git_annex_import_--lazy__96___--_Delete_everything_that__39__s_in_the_source_directory_and_also_in_the_target_annex.mdwn b/doc/todo/__96__git_annex_import_--lazy__96___--_Delete_everything_that__39__s_in_the_source_directory_and_also_in_the_target_annex.mdwn new file mode 100644 index 000000000..996c03461 --- /dev/null +++ b/doc/todo/__96__git_annex_import_--lazy__96___--_Delete_everything_that__39__s_in_the_source_directory_and_also_in_the_target_annex.mdwn @@ -0,0 +1,29 @@ +As per IRC + + 22:13:10 < RichiH> joeyh: btw, i have been pondering a `git annex import --lazy` or some such which basically goes through a directory and deletes everything i find in the annex it run from + 22:50:39 < joeyh> not sure of the use case + 23:41:06 < RichiH> joeyh: the use case is "i have important a ton of data into my annexes. now, i am going through the usual crud of cp -ax'ed, rsync'ed, and other random 'new disk, move stuff around and just put a full dump over there' file dumps and would like to delete everything that's annexed already" + 23:41:33 < RichiH> joeyh: that would allow me to spend time on dealing with the files which are not yet annexed + 23:41:54 < RichiH> instead of verifying file after file which has been imported already + 23:43:19 < joeyh> have you tried just running git annex import in a subdirectory and then deleting the dups? + 23:45:34 < joeyh> or in a separate branch for that matter, which you could then merge in, etc + 23:54:08 < joeyh> Thinking anout it some more, it would need to scan the whole work tree to see what keys were there, and populate a lookup table. I prefer to avoid things that need git-annex to do such a large scan and use arbitrary amounts of memory. + 00:58:11 < RichiH> joeyh: that would force everything into the annex, though + 00:58:20 < RichiH> a plain import, that is + 00:58:53 < RichiH> in a usual data dump directory, there's tons of stuff i will never import + 00:59:00 < RichiH> i want to delete large portions of it + 00:59:32 < RichiH> but getting rid of duplicates first allows me to spend my time focused on stuff humans are good at: deciding + 00:59:53 < RichiH> whereas the computer can focus on stuff it's good at: mindless comparision of bits + 01:00:15 < RichiH> joeyh: as you're saying this is complex, maybe i need to rephrase + 01:01:40 < RichiH> what i envision is git annex import --foo to 1) decide what hashing algorithm should be used for this file 2) hash that file 3) look into the annex if that hash is annexed 3a) optionally verify numcopies within the annex 4) delete the file in the source directory + 01:01:47 < RichiH> and then move on to the next file + 01:02:00 < RichiH> if the hash does not exist in the annex, leave it alone + 01:02:50 < RichiH> if the hash exists in annex, but numcopies is not fulfilled, just import it as a normal import would + 01:03:50 < RichiH> that sounds quite easy, to me; in fact i will prolly script it if you decide not to implement it + 01:04:07 < RichiH> but i think it's useful for a _lot_ of people who migrate tons of data into annexes + 01:04:31 < RichiH> thus i would rather see this upstream and not hacked locally + +The only failure mode I see in the above is "file has been dropped elsewhere, numcopies not fulfilled, but that info is not synched to the local repo, yet" -- This could be worked around by always importing the data. + +> [[done]] as `git annex import --deduplicate`. +> --[[Joey]] diff --git a/doc/todo/__96__git_annex_import_--lazy__96___--_Delete_everything_that__39__s_in_the_source_directory_and_also_in_the_target_annex/comment_1_0cc16eb17151309113cec6d1cccf203d._comment b/doc/todo/__96__git_annex_import_--lazy__96___--_Delete_everything_that__39__s_in_the_source_directory_and_also_in_the_target_annex/comment_1_0cc16eb17151309113cec6d1cccf203d._comment new file mode 100644 index 000000000..0b4e22e7c --- /dev/null +++ b/doc/todo/__96__git_annex_import_--lazy__96___--_Delete_everything_that__39__s_in_the_source_directory_and_also_in_the_target_annex/comment_1_0cc16eb17151309113cec6d1cccf203d._comment @@ -0,0 +1,20 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U" + nickname="Richard" + subject="comment 1" + date="2013-08-06T14:22:03Z" + content=""" +To expand a bit on the use case: + +I have several migration directories which I simply moved to new systems or disks with the help of `cp -ax` or `rsync`. +As I don't _need_ the data per se and merely want to hold on to it in case I ever happen to need it again and as disk space is laughably cheap, I have a lot of duplicates. +While I can at least detect bit flips with the help of checksum lists, cleaning those duplicates of duplicated duplicates is quite some effort. +To make things worse, photos, music, videos, letter and whatnot are thrown into the same container directories. + +All in all, getting data out of those data dumps and into a clean structure is quite an effort. +`git annex import --lazy` would help with this effort as I could start with the first directory, sort stuff by hand, and annex it. +As soon as data lives in any of my annexes, I could simply run `git annex import --lazy` to get rid of all duplicates while retaining the unannexed files. +Iterating through this process a few times, I will be left with clean annexes on the one hand and stuff I can simply delete on the other hand. + +I could script all this by hand on my own machine, but I am _certain_ that others would find easy, integrated, and unit tested support for whittling down data dumps over time useful. +"""]] diff --git a/doc/todo/__96__git_annex_status__47__version__96___should_print_the_local_OS.mdwn b/doc/todo/__96__git_annex_status__47__version__96___should_print_the_local_OS.mdwn new file mode 100644 index 000000000..9c2afecb4 --- /dev/null +++ b/doc/todo/__96__git_annex_status__47__version__96___should_print_the_local_OS.mdwn @@ -0,0 +1,6 @@ +That would make assessing weird reports like [[bugs/Should_UUID__39__s_for_Remotes_be_case_sensitive__63__/]] easier and quicker. + +> No, if people want to file a bug report, it's up to them to tell me +> relevant details about their OS. I'm not going down the rathole +> of making git-annex muck about trying to gather such information. +> [[done]] --[[Joey]] diff --git a/doc/todo/__96__git_annex_sync_--auto__96___or___96__git_annex_auto__96___--_synchronize_symlinks__44___tracking_info__44___and_actual_data.mdwn b/doc/todo/__96__git_annex_sync_--auto__96___or___96__git_annex_auto__96___--_synchronize_symlinks__44___tracking_info__44___and_actual_data.mdwn new file mode 100644 index 000000000..d48b4426f --- /dev/null +++ b/doc/todo/__96__git_annex_sync_--auto__96___or___96__git_annex_auto__96___--_synchronize_symlinks__44___tracking_info__44___and_actual_data.mdwn @@ -0,0 +1,3 @@ +As per DebConf13: Introduce a one-shot command to synchronize everything, including data, with the other remotes. + +Especially useful for the debconf annex. diff --git a/doc/todo/add_--exclude_option_to_git_annex_find.mdwn b/doc/todo/add_--exclude_option_to_git_annex_find.mdwn new file mode 100644 index 000000000..a797e97f5 --- /dev/null +++ b/doc/todo/add_--exclude_option_to_git_annex_find.mdwn @@ -0,0 +1,4 @@ +Seems pretty self-explanatory. + +> This was already implemented, the --exclude option can be used +> for find as well as most any other subcommand. --[[Joey]] [[done]] diff --git a/doc/todo/add_-all_option.mdwn b/doc/todo/add_-all_option.mdwn new file mode 100644 index 000000000..2f25759c2 --- /dev/null +++ b/doc/todo/add_-all_option.mdwn @@ -0,0 +1,22 @@ +`--all` would make git-annex operate on either every key with content +present (or in some cases like `get` and `copy --from` on +every keys with content not present). + +This would be useful when a repository has a history with deleted files +whose content you want to keep (so you're not using `dropunused`). +Or when you have a lot of branches and just want to be able to fsck +every file referenced in any branch (or indeed, any file referenced in any +ref). It could also be useful (or even a +good default) in a bare repository. + +A problem with the idea is that `.gitattributes` values for keys not +currently in the tree would not be available (without horrific anounts of +grubbing thru history to find where/when the key used to exist). So +`numcopies` set via `.gitattributes` would not work. This would be a +particular problem for `drop` and for `--auto`. + +--[[Joey]] + +> [[done]]. The .gitattributes problem was solved simply by not +> supporting `drop --all`. `--auto` also cannot be mixed with --all for +> similar reasons. --[[Joey]] diff --git a/doc/todo/add_a_git_backend.mdwn b/doc/todo/add_a_git_backend.mdwn new file mode 100644 index 000000000..2b224710e --- /dev/null +++ b/doc/todo/add_a_git_backend.mdwn @@ -0,0 +1,18 @@ +There should be a backend where the file content is stored.. in a git +repository! + +This way, you know your annexed content is safe & versioned, but you only +have to deal with the pain of git with large files in one place, and can +use all of git-annex's features everywhere else. + +> Speaking as a future user, do very, very much want. -- RichiH + +>> Might also be interesting to use `bup` in the git backend, to work +>> around git's big file issues there. So git-annex would pull data out +>> of the git backend using bup. --[[Joey]] + +>>> Very much so. Generally speaking, having one or more versioned storage back-ends with current data in the local annexes sounds incredibly useful. Still being able to get at old data in via the back-end and/or making offline backups of the full history are excellent use cases. -- RichiH + +[[done]], the bup special remote type is written! --[[Joey]] + +> Yay! -- RichiH diff --git a/doc/todo/add_an_icon_for_the_.desktop_file.mdwn b/doc/todo/add_an_icon_for_the_.desktop_file.mdwn new file mode 100644 index 000000000..3be158a0a --- /dev/null +++ b/doc/todo/add_an_icon_for_the_.desktop_file.mdwn @@ -0,0 +1 @@ +Maybe add the icon /usr/share/doc/git-annex/html/logo.svg to the .desktp file. diff --git a/doc/todo/add_metadata_to_annexed_files.mdwn b/doc/todo/add_metadata_to_annexed_files.mdwn new file mode 100644 index 000000000..0fc3e8953 --- /dev/null +++ b/doc/todo/add_metadata_to_annexed_files.mdwn @@ -0,0 +1,5 @@ +I would like to attach metadata to annexed files (objects) without cluttering the workdir with files containing this metadata. A common use case would be to add titles to my photo collection that could than end up in a generated photo album. + +Depending on the implementation it might also be possible to use the metadata facility for a threaded commenting system. + +The first question is whether the metadata is attached to the objects and thus shared by all paths pointing to the same data object or to paths in the worktree. I've no preference here at this point. diff --git a/doc/todo/add_metadata_to_annexed_files/comment_1_38af9b352020194e9ace34d7dde188cf._comment b/doc/todo/add_metadata_to_annexed_files/comment_1_38af9b352020194e9ace34d7dde188cf._comment new file mode 100644 index 000000000..8460300a7 --- /dev/null +++ b/doc/todo/add_metadata_to_annexed_files/comment_1_38af9b352020194e9ace34d7dde188cf._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.154.0.63" + subject="comment 1" + date="2013-08-24T19:58:54Z" + content=""" +I don't know if git-annex is the right vehicle to fix this. It seems that a more generic fix that would work in non-git-annex repos would be better. + +I can answer your question though: The metadata such as urls and locations that git-annex stores in the git-annex branch is attached to objects, and not to work tree paths. +"""]] diff --git a/doc/todo/assistant_cannot_set_up_remote_repo_via_an_ssh_alias_or_an_ip_address.mdwn b/doc/todo/assistant_cannot_set_up_remote_repo_via_an_ssh_alias_or_an_ip_address.mdwn new file mode 100644 index 000000000..a7b30ded5 --- /dev/null +++ b/doc/todo/assistant_cannot_set_up_remote_repo_via_an_ssh_alias_or_an_ip_address.mdwn @@ -0,0 +1,48 @@ +What steps will reproduce the problem? + +Using the assistant, create an SSH remote. Try to use an alias as the name +of the remote (e.g. I have a server which I have aliased to "homeworld" in +my .ssh/config. When I'm at home, that is an alias for 192.168.1.253. +When I'm not at home, I edit .ssh/config so that "homeworld" becomes an +alias for a hostname at no-ip.com.) Despite the fact that "homeworld" is a +viable ssh target because of the alias, the assistant doesn't recognize it +as a valid host to ssh to. + +I had trouble with an ip address the first time I tried it but just tried +it again and it worked fine, so please disregard that part of the title of +this bug report. + + +What is the expected output? What do you see instead? + +expected output = move to the "create a repository -- rsync or regular" page. +observed output = "cannot resolve host name" + + +What version of git-annex are you using? On what operating system? + + Version: 3.20130102 OS X Lion + + +Please provide any additional information below. + +I realize this is kind of a power user whine. Using an ssh alias which +does not correspond to an actual resolvable hostname (and cannot, because +it's supposed to be a layer of indirection over the hostname) is not an +everyday problem for an average user. + +> The assistant tries to resolve the hostname explicitly +> to catch user's typos, and also expands it to a FQDN, to make +> it more likely to be able to reach the host when roaming to other +> networks. +> +> Also, the assistant sets up it *own* .ssh/config hostname alias, +> in order to make it use the special ssh key that it generates for the host. +> So that is not compatable with using a ssh host alias you've set up. +> Even if it knew about your alias, it would set up a new hostname alias, and +> whatever machinery you have to update the alias would not work. +> +> You can, of course, add git remotes using any ssh alias you like, by +> hand, and restart the assistant and it will use them. --[[Joey]] + +[[!tag /design/assistant]] diff --git a/doc/todo/assistant_git_sync_laddering.mdwn b/doc/todo/assistant_git_sync_laddering.mdwn new file mode 100644 index 000000000..7dbc6480a --- /dev/null +++ b/doc/todo/assistant_git_sync_laddering.mdwn @@ -0,0 +1,10 @@ +When the [[design/assistant]] is running on a pair of remotes, I've seen +them get out of sync, such that every pull and merge results in a conflict, +that then has to be auto-resolved. + +This seems similar to the laddering problem described in this old bug: +[[bugs/making_annex-merge_try_a_fast-forward]] + +--[[Joey]] + +Think I've fixed this. [[done]] --[[Joey]] diff --git a/doc/todo/assistant_parallel_file_transfers.txt b/doc/todo/assistant_parallel_file_transfers.txt new file mode 100644 index 000000000..aafddf038 --- /dev/null +++ b/doc/todo/assistant_parallel_file_transfers.txt @@ -0,0 +1,15 @@ +Hi and thank you for an incredible piece of software and great work! + +I've noticed that when I add new files to a repository and I have my USB drive connected, the assistant alternate it's transfers of files. And only transfers one queued file at the time. + +file1 -->> Internet offsite computer +file1 -->> USB drive +file2 -->> Internet offsite computer +file2 -->> USB drive + + +I would prefer a logic where the assistant transfer files in parallel to my different repositories. I know that it might not be a good thing doing that with network accessed repositories, but when I have "low cost", locally attached USB drives it would be great if the transfers could be done in parallel. + + +Is there a configuration option for this already? + diff --git a/doc/todo/assistant_smarter_archive_directory_handling.mdwn b/doc/todo/assistant_smarter_archive_directory_handling.mdwn new file mode 100644 index 000000000..fa5a3e4fc --- /dev/null +++ b/doc/todo/assistant_smarter_archive_directory_handling.mdwn @@ -0,0 +1,31 @@ +Client repos do not want files in archive directories. This can turn +out to be confusing to users who are using archive directories for their +own purposes and not aware of this special case in the assistant. It can +seem like the assistant is failing to sync their files. + +I thought, first, that it should have a checkbox to enable the archive +directory behavior. + +However, I think I have a better idea. Change the preferred content +expression for clients, so they want files in archive directories, *until* +those files land in an archive. + +This way, only users who set up an archive repo get this behavior. And they +asked for it by setting up that repo! + +Also, the new behavior will mean that files in archive directories still +propigate around to clients. Consider this topology: + + client A ---- client B ---- archive + +If a file is created in client A, and moved to an archive directory before +it syncs to B, it will never get to the archive, and will continue wasting +space on A. With the new behavior, A and B serve as effectively, transfer +repositories for archived content. + +Something vaguely like this should work as the preferred content +expression for the clients: + + exclude=archive/* or (include=archive/* and (not (copies=archive:1 or copies=smallarchive:1))) + +> [[done]] --[[Joey]] diff --git a/doc/todo/assistant_threaded_runtime.mdwn b/doc/todo/assistant_threaded_runtime.mdwn new file mode 100644 index 000000000..03ba66acf --- /dev/null +++ b/doc/todo/assistant_threaded_runtime.mdwn @@ -0,0 +1,40 @@ +The [[design/assistant]] would be better if git-annex used ghc's threaded +runtime (`ghc -threaded`). + +Currently, whenever the assistant code runs some external command, all +threads are blocked waiting for it to finish. + +For transfers, the assistant works around this problem by forking separate +upload processes, and not waiting on them until it sees an indication that +they have finished the transfer. While this works, it's messy.. threaded +would be better. + +When pulling, pushing, and merging, the assistant runs external git +commands, and this does block all other threads. The threaded runtime would +really help here. + +[[done]]; the assistant now builds with the threaded runtime. +Some work still remains to run certian long-running external git commands +in their own threads to prevent them blocking things, but that is easy to +do, now. --[[Joey]] + +--- + +Currently, git-annex seems unstable when built with the threaded runtime. +The test suite tends to hang when testing add. `git-annex` occasionally +hangs, apparently in a futex lock. This is not the assistant hanging, and +git-annex does not otherwise use threads, so this is surprising. --[[Joey]] + +> I've spent a lot of time debugging this, and trying to fix it, in the +> "threaded" branch. There are still deadlocks. --[[Joey]] + +>> Fixed, by switching from `System.Cmd.Utils` to `System.Process` +>> --[[Joey]] + +--- + +It would be possible to not use the threaded runtime. Instead, we could +have a child process pool, with associated continuations to run after a +child process finishes. Then periodically do a nonblocking waitpid on each +process in the pool in turn (waiting for any child could break anything not +using the pool!). This is probably a last resort... diff --git a/doc/todo/auto_remotes.mdwn b/doc/todo/auto_remotes.mdwn new file mode 100644 index 000000000..715dea720 --- /dev/null +++ b/doc/todo/auto_remotes.mdwn @@ -0,0 +1,29 @@ +It should be possible for clones to learn about how to contact +each other without remotes needing to always be explicitly set +up. Say that `.git-annex/remote.log` is maintained by git-annex +to contain: + + UUID hostname URI + +The URI comes from configured remotes and maybe from +`file://$(pwd)`, or even `ssh://$(hostname -f)` +for the current repo. This format will merge without +conflicts or data loss. + +Then when content is belived to be in a UUID, and no +configured remote has it, the remote.log can be consulted and +URIs that look likely tried. (file:// ones if the hostname +is the same (or maybe always -- a removable drive might tend +to be mounted at the same location on different hosts), +otherwise ssh:// ones.) + +Question: When should git-annex update the remote.log? +(If not just on init.) Whenever it reads in a repo's remotes? + +> This sounds useful and the log should be updated every time any remote is being accessed. A counter or timestamp (yes, distributed times may be wrong/different) could be used to auto-prune old entries via a global and per-remote config setting. -- RichiH + +--- + +I no longer think I'd use this myself, I find that my repositories quickly +grow the paths I actually use, somewhat organically. Unofficial paths +across university quads come to mind. [[done]] --[[Joey]] diff --git a/doc/todo/auto_remotes/discussion.mdwn b/doc/todo/auto_remotes/discussion.mdwn new file mode 100644 index 000000000..b9e1522a8 --- /dev/null +++ b/doc/todo/auto_remotes/discussion.mdwn @@ -0,0 +1,7 @@ +Remotes log should probably be stored in ".git/annex/remote.log" +instead of ".git-annex/remote.log" to prevent leaking credentials. + +> The idea is to distribute the info between repositories, which is +> why it'd go in `.git-annex`. Of course that does mean that repository +> location information would be included, and if that'd not desirable +> this feature would need to be turned off. --[[Joey]] diff --git a/doc/todo/automatic_bookkeeping_watch_command.mdwn b/doc/todo/automatic_bookkeeping_watch_command.mdwn new file mode 100644 index 000000000..28b02aff2 --- /dev/null +++ b/doc/todo/automatic_bookkeeping_watch_command.mdwn @@ -0,0 +1,15 @@ +A "git annex watch" command would help make git-annex usable by users who +don't know how to use git, or don't want to bother typing the git commands. +It would run, in the background, watching via inotify for changes, and +automatically annexing new files, etc. + +The blue sky goal would be something automated like dropbox, except fully +distributed. All files put into the repository would propagate out +to all the other clones of it, as network links allow. Note that while +dropbox allows modifying files, git-annex freezes them upon creation, +so this would not be 100% equivalent to dropbox. --[[Joey]] + +This is a big project with its own [[design pages|design/assistant]]. + +> [[done]].. at least, we have a watch command an an assistant, which +> is still being developed. --[[Joey]] diff --git a/doc/todo/automatic_merge_of_synced_branches_upon___34__git_annex_sync__34__.mdwn b/doc/todo/automatic_merge_of_synced_branches_upon___34__git_annex_sync__34__.mdwn new file mode 100644 index 000000000..361585a78 --- /dev/null +++ b/doc/todo/automatic_merge_of_synced_branches_upon___34__git_annex_sync__34__.mdwn @@ -0,0 +1,16 @@ +When maintaining several replica of the same git-annex repo "git annex sync" is quite handy. +But it would be even handier if "git annex sync" would also perform automatic "git merge synced/*" actions on all remotes. + +Clearly, this is beneficial when the user wants to keep all working copies synchronized. +This is likely the case in git annex assistant like scenarios. And it's always the case in my day to day scenarios :-) +I'm not sure about other use cases that I've hard time imagining... + +As just discussed on IRC (#vcs-home/OFTC), this could be implemented in various ways: + +1) By doing ssh on each remote and running the appropriate "git merge ..." commands there. + The drawback of this is that quite often it won't be permitted to ssh on the remote and run arbitrary commands there. + +2) Having a default post-receive hook, created at the time of "git annex init" that automatically does the merges when contacted by other remotes as a consequence of "git annex sync". + + +Thanks for git-annex! diff --git a/doc/todo/avoid_unnecessary_union_merges.mdwn b/doc/todo/avoid_unnecessary_union_merges.mdwn new file mode 100644 index 000000000..5cd4b6437 --- /dev/null +++ b/doc/todo/avoid_unnecessary_union_merges.mdwn @@ -0,0 +1,20 @@ +Some commands cause a union merge unnecessarily. For example, `git annex add` +modifies the location log, which first requires reading the current log (if +any), which triggers a merge. + +Would be good to avoid these unnecessary union merges. First because it's +faster and second because it avoids a possible delay when a user might +ctrl-c and leave the repo in an inconsistent state. In the case of an add, +the file will be in the annex, but no location log will exist for it (fsck +fixes that). + +It may be that all that's needed is to modify Annex.Branch.change +to read the current value, without merging. Then commands like `get`, that +query the branch, will still cause merges, and commands like `add` that +only modify it, will not. Note that for a command like `get`, the merge +occurs before it has done anything, so ctrl-c should not be a problem +there. + +This is a delicate change, I need to take care.. --[[Joey]] + +> [[done]] (assuming I didn't miss any cases where this is not safe!) --[[Joey]] diff --git a/doc/todo/backendSHA1.mdwn b/doc/todo/backendSHA1.mdwn new file mode 100644 index 000000000..8c16b75ad --- /dev/null +++ b/doc/todo/backendSHA1.mdwn @@ -0,0 +1,7 @@ +This backend is not finished. + +In particular, while files can be added using it, git-annex will not notice +when their content changes, and will not create a new key for the new sha1 +of the net content. + +[[done]]; use unlock subcommand and commit changes with git diff --git a/doc/todo/branching.mdwn b/doc/todo/branching.mdwn new file mode 100644 index 000000000..ad7ece6f1 --- /dev/null +++ b/doc/todo/branching.mdwn @@ -0,0 +1,159 @@ +[[done]] !!! + +The use of `.git-annex` to store logs means that if a repo has branches +and the user switched between them, git-annex will see different logs in +the different branches, and so may miss info about what remotes have which +files (though it can re-learn). + +An alternative would be to store the log data directly in the git repo +as `pristine-tar` does. Problem with that approach is that git won't merge +conflicting changes to log files if they are not in the currently checked +out branch. + +It would be possible to use a branch with a tree like this, to avoid +conflicts: + +key/uuid/time/status + +As long as new files are only added, and old timestamped files deleted, +there would be no conflicts. + +A related problem though is the size of the tree objects git needs to +commit. Having the logs in a separate branch doesn't help with that. +As more keys are added, the tree object size will increase, and git will +take longer and longer to commit, and use more space. One way to deal with +this is simply by splitting the logs amoung subdirectories. Git then can +reuse trees for most directories. (Check: Does it still have to build +dup trees in memory?) + +Another approach would be to have git-annex *delete* old logs. Keep logs +for the currently available files, or something like that. If other log +info is needed, look back through history to find the first occurance of a +log. Maybe even look at other branches -- so if the logs were on master, +a new empty branch could be made and git-annex would still know where to +get keys in that branch. + +Would have to be careful about conflicts when deleting and bringing back +files with the same name. And would need to avoid expensive searching thru +all history to try to find an old log file. + +## fleshed out proposal + +Let's use one branch per uuid, named git-annex/$UUID. + +- I came to realize this would be a good idea when thinking about how + to upgrade. Each individual annex will be upgraded independantly, + so each will want to make a branch, and if the branches aren't distinct, + they will merge conflict for sure. +- TODO: What will need to be done to git to make it push/pull these new + branches? +- A given repo only ever writes to its UUID branch. So no conflicts. + - **problem**: git annex move needs to update log info for other repos! + (possibly solvable by having git-annex-shell update the log info + when content is moved using it) +- (BTW, UUIDs probably don't compress well, and this reduces the bloat of having + them repeated lots of times in the tree.) +- Per UUID branches mean that if it wants to find a file's location + amoung configured remotes, it can examine only their branches, if + desired. +- It's important that the per-repo branches propigate beyond immediate + remotes. If there is a central bare repo, that means push --all. Without + one, it means that when repo B pulls from A, and then C pulls from B, + C needs to get A's branch -- which means that B should have a tracking + branch for A's branch. + +In the branch, only one file is needed. Call it locationlog. git-annex +can cache location log changes and write them all to locationlog in +a single git operation on shutdown. + +- TODO: what if it's ctrl-c'd with changes pending? Perhaps it should + collect them to .git/annex/locationlog, and inject that file on shutdown? +- This will be less overhead than the current staging of all the log files. + +The log is not appended to, so in git we have a series of commits each of +which replaces the log's entire contens. + +To find locations of a key, all (or all relevant) branches need to be +examined, looking backward through the history of each until a log +with a indication of the presense/absense of the key is found. + +- This will be less expensive for files that have recently been added + or transfered. +- It could get pretty slow when digging deeper. +- Only 3 places in git-annex will be affected by any slowdown: move --from, + get and drop. (Update: Now also unused, whereis, fsck) + +## alternate + +As above, but use a single git-annex branch, and keep the per-UUID +info in their own log files. Hope that git can auto-merge as long as +each observing repo only writes to its own files. (Well, it can, but for +non-fast-forward merges, the git-annex branch would need to be checked out, +which is problimatic.) + +Use filenames like: + + <observing uuid>/<location uuid> + +That allows one repo to record another's state when doing a +`move`. + +## outside the box approach + +If the problem is limited to only that the `.git-annex/` files make +branching difficult (and not to the related problem that commits to them +and having them in the tree are sorta annoying), then a simple approach +would be to have git-annex look in other branches for location log info +too. + +The problem would then be that any locationlog lookup would need to look in +all other branches (any branch could have more current info after all), +which could get expensive. + +## way outside the box approach + +Another approach I have been mulling over is keeping the log file +branch checked out in .git/annex/logs/ -- this would be a checkout of a git +repository inside a git repository, using "git fake bare" techniques. This +would solve the merge problem, since git auto merge could be used. It would +still mean all the log files are on-disk, which annoys some. It would +require some tighter integration with git, so that after a pull, the log +repo is updated with the data pulled. --[[Joey]] + +> Seems I can't use git fake bare exactly. Instead, the best option +> seems to be `git clone --shared` to make a clone that uses +> `.git/annex/logs/.git` to hold its index etc, but (mostly) uses +> objects from the main repo. There would be some bloat, +> as commits to the logs made in there would not be shared with the main +> repo. Using `GIT_OBJECT_DIRECTORY` might be a way to avoid that bloat. + +## notes + +Another approach could be to use git-notes. It supports merging branches +of notes, with union merge strategy (a hook would have to do this after +a pull, it's not done automatically). + +Problem: Notes are usually attached to git +objects, and there are no git objects corresponding to git-annex keys. + +Problem: Notes are not normally copied when cloning. + +------ + +## elminating the merge problem + +Most of the above options are complicated by the problem of how to merge +changes from remotes. It should be possible to deal with the merge +problem generically. Something like this: + +* We have a local branch `B`. +* For remotes, there are also `origin/B`, `otherremote/B`, etc. +* To merge two branches `B` and `foo/B`, construct a merge commit that + makes each file have all lines that were in either version of the file, + with duplicates removed (probably). Do this without checking out a tree. + -- now implemented as git-union-merge +* As a `post-merge` hook, merge `*/B` into `B`. This will ensure `B` + is always up-to-date after a pull from a remote. +* When pushing to a remote, nothing need to be done, except ensure + `B` is either successfully pushed, or the push fails (and a pull needs to + be done to get the remote's changes merged into `B`). diff --git a/doc/todo/cache_key_info.mdwn b/doc/todo/cache_key_info.mdwn new file mode 100644 index 000000000..d4352ccf7 --- /dev/null +++ b/doc/todo/cache_key_info.mdwn @@ -0,0 +1,37 @@ +Most of git-annex is designed to be fast no matter how many other files are +in the annex. Things like add/get/drop/move/fsck have good locality; +they will only operate on as many files as you need them to. + +(git commit can get a little slow with a great deal of files, +but that's out of scope -- and recent git-annex versions use queuing +to save git add from piling up too much in the index.) + +But currently two git-annex commands are quite slow when annexes become large +in quantity of files. These are unused and status. +(Both have --fast versions that don't do as much). +> (Update: status has become acceptably fast; most of its slowdown was due to using a bad data structure; scanning the tree is not particularly slow and it no longer looks at the git-annex branch.) + +unused is slow because it needs two pieces of information that are not +quick to look up, and require examining the whole repo, very seekily: + +1. The keys present in the annex. Found by looking thru .git/annex/objects +2. The keys referenced by files in git. Found by finding every file + in git, and looking at its symlink. + +Of these, the first is less expensive (typically, an annex does not have every +key in it). It could be optimized fairly simply, by adding a database +of keys present in the annex that is optimised to list them all. The +database would be updated by the few functions that move content in and +out. + +The second is harder to optimise, because the user can delete, revert, +copy, add, etc files in git at will, and git-annex does not have a good way +to watch that and maintain a database of what keys are being referenced. + +It could use a post-commit hook and examine files changed by commits, etc. +But then staged files would be left out. It might be sufficient to +make --fast trust the database... except unused will suggest *deleting* +data if nothing references it. Or maybe it could be required to have a +clean tree with nothing staged before running git-annex unused. + +Anyway, this is a semi-longterm item for me. --[[Joey]] diff --git a/doc/todo/cache_key_info/comment_1_578df1b3b2cbfdc4aa1805378f35dc48._comment b/doc/todo/cache_key_info/comment_1_578df1b3b2cbfdc4aa1805378f35dc48._comment new file mode 100644 index 000000000..086e7f3e8 --- /dev/null +++ b/doc/todo/cache_key_info/comment_1_578df1b3b2cbfdc4aa1805378f35dc48._comment @@ -0,0 +1,11 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U" + nickname="Richard" + subject="comment 1" + date="2011-05-17T07:27:02Z" + content=""" +Sounds like a good idea. + +* git annex fsck (or similar) should check/rebuild the caches +* I would simply require a clean tree with a verbose error. 80/20 rule and defaulting to save actions. +"""]] diff --git a/doc/todo/checkout.mdwn b/doc/todo/checkout.mdwn new file mode 100644 index 000000000..50da2d62e --- /dev/null +++ b/doc/todo/checkout.mdwn @@ -0,0 +1,23 @@ +The checkout subcommand replaces the symlink that normally points at a +file's content, with a copy of the file. Once you've checked a file out, +you can edit it, and `git commit` it. On commit, git-annex will detect +if the file has been changed, and if it has, `add` its content to the +annex. + +> Internally, this will need to store the original symlink to the file, in +> `.git/annex/checkedout/$filename`. +> +> * git-annex uncheckout moves that back +> * git-annex pre-commit hook checks each file being committed to see if +> it has a symlink there, and if so, removes the symlink and adds the new +> content to the annex. +> +> And it seems the file content should be copied, not moved or hard linked: +> +> * Makes sure other annexes can find it if transferring it from +> this annex. +> * Ensures it's always available for uncheckout. +> * Avoids the last copy of a file's content being lost when +> the checked out file is modified. + +[[done]] diff --git a/doc/todo/checksum_verification_on_transfer.mdwn b/doc/todo/checksum_verification_on_transfer.mdwn new file mode 100644 index 000000000..c9d505aec --- /dev/null +++ b/doc/todo/checksum_verification_on_transfer.mdwn @@ -0,0 +1,7 @@ +Since most file transfers, particularly to/from encrypted special remotes involve git-annex streaming through the contents of the file anyway, it should be possible to add a verification of the checksum nearly for free. The main thing needed is probably a faster haskell checksum library than Data.Digest.Pure.Sha, which is probably slow enough to be annoying. + +I have not verified if an upload could be aborted before sending the data to the remote if a checksum failure is detected. It may be dependent on the individual special remote implementations. Some probably stream the encrypted data directly out the wire, while others need to set up a temp file to run a command on. It would certainly be possible to at least make the upload abort and fail if a bad checksum was detected. + +Doing the same for downloads is less useful, because the data is there locally to be fscked. The real advantage would be doing the check for uploads, to ensure that hard-to-detect corrupted files don't reach special remotes. + +--[[Joey]] diff --git a/doc/todo/checksum_verification_on_transfer/comment_1_30f77e631608b9751f9032f97d58cc30._comment b/doc/todo/checksum_verification_on_transfer/comment_1_30f77e631608b9751f9032f97d58cc30._comment new file mode 100644 index 000000000..5de1251da --- /dev/null +++ b/doc/todo/checksum_verification_on_transfer/comment_1_30f77e631608b9751f9032f97d58cc30._comment @@ -0,0 +1,17 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U" + nickname="Richard" + subject="comment 1" + date="2013-09-09T11:50:05Z" + content=""" +Doing this during downloads would still be nice. + +While the files are easier to fsck, users will need to actually do this. If it happenend automatically, it would increase safety and reduce disk i/o. + +Of course, this will not detect degradation during/after writing. + +If you don't make it the default, please at least make it optional for us bordering on OCD when it comes to data storage. + + +Richard +"""]] diff --git a/doc/todo/direct_mode_guard.mdwn b/doc/todo/direct_mode_guard.mdwn new file mode 100644 index 000000000..bb7f90897 --- /dev/null +++ b/doc/todo/direct_mode_guard.mdwn @@ -0,0 +1,105 @@ +Currently [[/direct_mode]] allows the user to point many normally safe +git commands at his foot and pull the trigger. At LCA2013, a git-annex +user suggested modifying direct mode to make this impossible. + +One way to do it would be to move the .git directory. Instead, make there +be a .git-annex directory in direct mode repositories. git-annex would know +how to use it, and would be extended to support all known safe git +commands, passing parameters through, and in some cases verifying them. + +So, for example, `git annex commit` would run `git commit --git-dir=.git-annex` + +However, `git annex commit -a` would refuse to run, or even do something +intelligent that does not involve staging every direct mode file. + +---- + +One source of problems here is that there is some overlap between git-annex +and git commands. Ie, `git annex add` cannot be a passthrough for `git +add`. The git wrapper could instead be another program, or it could be +something like `git annex git add` + +--[[Joey]] + +---- + +Or, no git wrapper could be provided. Limit the commands to only git-annex +commands. This should be all that is needed to manage a direct mode +repository simply, and if the user is doing something complicated that +needs git access, they can set `GIT_DIR=.git-annex` and be careful not to +shoot off their foot. (Or can just switch to indirect mode!) + +This wins on simplicity, and if it's the wrong choice a git wrapper +can be added later. --[[Joey]] + +--- + +Implementation: Pretty simple really. Already did the hard lifting to +support `GIT_DIR`, so only need to override the default git directory +in direct mode when that's not set to `.git-annex`. + +A few things hardcode ".git", including Assistant.Threads.Watcher.ignored +and `Seek.withPathContents`, and parts of `Git.Construct`. + +--- + +Transition: git-annex should detect when it's in a direct mode repository +with a .git directory and no .git-annex directory, and transparently +do the move to transition to the new scheme. (And remember that `git annex +indirect` needs to move it back.) + +# alternative approach: move index + +Rather than moving .git, maybe move .git/index? + +This would cause git to think that all files in the tree were deleted. +So git commit -a would make a commit that removes them from git history. +But, the files in the work tree are not touched by this. + +Also, git checkout, git merge, and other things that manipulate the work +tree refuse to do anything if they'd change a file that they think is +untracked. + +Hmm, this does't solve the user accidentially running git add on an annexed +file; the whole file still gets added. + +# alternative approach: fake bare repo + +Set core.bare to true. This prevents all work tree operations, +so prevents any foot shooting. It still lets the user run commands like +git log, even on files in the tree, and git fetch, and push, and git +config, etc. + +Even better, it integrates with other tools, like `mr`, so they know +it's a git repo. + +This seems really promising. But of course, git-annex has its own set of +behaviors in a bare repo, so will need to recognise that this repo is not +really bare, and avoid them. + +> [[done]]!! --[[Joey]] + +(Git may also have some bare repo behaviors that are unwanted. One example +is that git allows pushes to the current branch in a bare repo, +even when `receive.denyCurrentBranch` is set.) + +> This is indeed a problem. Indeed, `git annex sync` successfully +> pushes changes to the master branch of a fake bare direct mode repo. +> +> And then, syncing in the repo that was pushed to causes the changes +> that were pushed to the master branch to get reverted! This happens +> because sync commits; commit sees that files are staged in index +> differing from the (pushed) master, and commits the "changes" +> which revert it. +> +> Could fix this using an update hook, to reject the updated of the master +> branch. However, won't work on crippled filesystems! (No +x bit) +> +> Could make git annex sync detect this. It could reset the master +> branch to the last one committed, before committing. Seems very racy +> and hard to get right! +> +> Could make direct mode operate on a different branch, like +> `annex/direct/master` rather than `master`. Avoid pushing to that +> branch (`git annex sync` can map back from it to `master` and push there +> instead). A bit clumsy, but works. diff --git a/doc/todo/direct_mode_guard/comment_1_431b6e1577bbd30b07dce9002a8fe1a2._comment b/doc/todo/direct_mode_guard/comment_1_431b6e1577bbd30b07dce9002a8fe1a2._comment new file mode 100644 index 000000000..01cddc8a3 --- /dev/null +++ b/doc/todo/direct_mode_guard/comment_1_431b6e1577bbd30b07dce9002a8fe1a2._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawn-KDr_Z4CMkjS0v_TxQ08SzAB5ecHG3K0" + nickname="Glen" + subject="This sounds good" + date="2013-06-25T10:30:07Z" + content=""" +I think we might have been talking about this feature.. Seems like a good idea to me. + +Glen +"""]] diff --git a/doc/todo/direct_mode_guard/comment_2_85bdb9dc601b87bd7c77150d7b0a5cde._comment b/doc/todo/direct_mode_guard/comment_2_85bdb9dc601b87bd7c77150d7b0a5cde._comment new file mode 100644 index 000000000..7cf37a917 --- /dev/null +++ b/doc/todo/direct_mode_guard/comment_2_85bdb9dc601b87bd7c77150d7b0a5cde._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawm7AuSfii_tCkLyspL6Mr0ATlO6OxLNYOo" + nickname="Georg" + subject="comment 2" + date="2013-09-20T11:29:04Z" + content=""" +Maybe make a git sub-namespace of commands. Yeah, I know, something like git annex git-add sounds a bit on the verbose side, but it would allow access to possibly all git commands regardless of name clashes. +"""]] diff --git a/doc/todo/done.mdwn b/doc/todo/done.mdwn new file mode 100644 index 000000000..e7c98081b --- /dev/null +++ b/doc/todo/done.mdwn @@ -0,0 +1,4 @@ +recently fixed [[todo]] items. + +[[!inline pages="./* and link(./done) and !*/Discussion" sort=mtime show=10 +archive=yes]] diff --git a/doc/todo/dumb_plaindir_remote___40__e.g._for_NAS_mounts__41__.mdwn b/doc/todo/dumb_plaindir_remote___40__e.g._for_NAS_mounts__41__.mdwn new file mode 100644 index 000000000..8ce910ac3 --- /dev/null +++ b/doc/todo/dumb_plaindir_remote___40__e.g._for_NAS_mounts__41__.mdwn @@ -0,0 +1,5 @@ +I've an external USB hard disc attached to my (fritzbox) router that is only accessible through SMB/CIFS. I'd like have all my annexed files on this drive in kind of direct-mode so that I can also access the files without git-annex. + +I tried to put a direct-mode repo on the drive but this is painfully slow. The git-annex process than runs on my desktop and accesses the repo over SMB over the slow fritzbox over USB. + +I'd wish that git-annex could be told to just use a (mounted) folder as a direct-mode remote. diff --git a/doc/todo/dumb_plaindir_remote___40__e.g._for_NAS_mounts__41__/comment_1_5ed9a2336b432b842c1805add6d96509._comment b/doc/todo/dumb_plaindir_remote___40__e.g._for_NAS_mounts__41__/comment_1_5ed9a2336b432b842c1805add6d96509._comment new file mode 100644 index 000000000..bff1b2fcd --- /dev/null +++ b/doc/todo/dumb_plaindir_remote___40__e.g._for_NAS_mounts__41__/comment_1_5ed9a2336b432b842c1805add6d96509._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="209.250.56.64" + subject="comment 1" + date="2013-11-23T19:03:58Z" + content=""" +It's not clear to me what you are requesting here. + +You seem to say that running git-annex inside a mountpoint is slow. Ok. So, what possible changes to git-annex could make it fast, given that the bottleneck is the SMB/USB? +"""]] diff --git a/doc/todo/dumb_plaindir_remote___40__e.g._for_NAS_mounts__41__/comment_2_e6ba58c5c31ba23a4575f1189689946f._comment b/doc/todo/dumb_plaindir_remote___40__e.g._for_NAS_mounts__41__/comment_2_e6ba58c5c31ba23a4575f1189689946f._comment new file mode 100644 index 000000000..d60cb2d4d --- /dev/null +++ b/doc/todo/dumb_plaindir_remote___40__e.g._for_NAS_mounts__41__/comment_2_e6ba58c5c31ba23a4575f1189689946f._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawnR6E5iUghMWdUGlbA9CCs8DKaoigMjJXw" + nickname="Efraim" + subject="comment 2" + date="2013-11-26T09:26:53Z" + content=""" +perhaps he's looking to be able to expand the addurl option to include file://path/to/video.mp4, or for over smb://... , to import a file without changing its location to being inside the annex. +"""]] diff --git a/doc/todo/exclude_files_on_a_given_remote.mdwn b/doc/todo/exclude_files_on_a_given_remote.mdwn new file mode 100644 index 000000000..e8bb357d3 --- /dev/null +++ b/doc/todo/exclude_files_on_a_given_remote.mdwn @@ -0,0 +1,18 @@ +Say I have some files on remote A. But I'm away from it, and transferring +files from B to C. I'd like to avoid transferring any files I already have +on A. + +Something like: + + git annex copy --to C --exclude-on A + +This would not contact A, just use its cached location log info. + +I suppose I might also sometime want to only act on files that are +thought/known to be on A. + + git annex drop --only-on A + +--[[Joey]] + +[[done]] diff --git a/doc/todo/faster_gnupg_cipher.mdwn b/doc/todo/faster_gnupg_cipher.mdwn new file mode 100644 index 000000000..f5ff062d2 --- /dev/null +++ b/doc/todo/faster_gnupg_cipher.mdwn @@ -0,0 +1,9 @@ +Apparently newer gnupg has support for hardware-accelerated AES-NI. It +would be good to have an option to use that. I also wonder if using the +same symmetric key for many files presents a security issues (and whether +using GPG keys directly would be more secure). + +> [[done]]; you can now use encryption=pubkey when setting up a special +> remote to use pure public keys without the hybrid symmetric key scheme. +> Which you choose is up to you. Also, annex.gnupg-options can configure +> the ciphers used. --[[Joey]] diff --git a/doc/todo/faster_gnupg_cipher/comment_1_8f61f7c724a8224e61c015be68f43db7._comment b/doc/todo/faster_gnupg_cipher/comment_1_8f61f7c724a8224e61c015be68f43db7._comment new file mode 100644 index 000000000..1bf550cdf --- /dev/null +++ b/doc/todo/faster_gnupg_cipher/comment_1_8f61f7c724a8224e61c015be68f43db7._comment @@ -0,0 +1,14 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.152.108.145" + subject="comment 1" + date="2013-08-01T17:10:56Z" + content=""" +There is a remote.name.annex-gnupg-options git-config setting that can be used to pass options to gpg on a per-remote basis. + +> also wonder if using the same symmetric key for many files presents a security issues (and whether using GPG keys directly would be more secure). + +I am not a cryptographer, but I have today run this question by someone with a good amount of crypo knowledge. My understanding is that reusing a symmetric key is theoretically vulnerable to eg known-plaintext or chosen-plaintext attacks. And that modern ciphers like AES and CAST (gpg default) are designed to resist such attacks. + +If someone was particularly concerned about these attack vectors, it would be pretty easy to add a mode where git-annex uses public key encryption directly. With the disadvantage, of course, that once a file was sent to a special remote and encrypted for a given set of public keys, other keys could not later be granted access to it. +"""]] diff --git a/doc/todo/faster_gnupg_cipher/comment_2_36e1f227a320527653500b445f7c001c._comment b/doc/todo/faster_gnupg_cipher/comment_2_36e1f227a320527653500b445f7c001c._comment new file mode 100644 index 000000000..08f69d6b8 --- /dev/null +++ b/doc/todo/faster_gnupg_cipher/comment_2_36e1f227a320527653500b445f7c001c._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U" + nickname="Richard" + subject="comment 2" + date="2013-08-02T07:21:50Z" + content=""" +Using symmetric keys is significantly cheaper, computation-wise. + +The scheme of encrypting symmetric keys with asymmetric ones is ancient, well-proven, and generally accepted as a good approach. + +Using per-key files makes access control more fine-grained and is only a real performance issue once while creating the private key and a little bit every time more than one file needs to be decrypted as more than one symmetric key needs to be taken care of. +"""]] diff --git a/doc/todo/faster_gnupg_cipher/comment_3_bd0c975494333dfe558de048d888ace8._comment b/doc/todo/faster_gnupg_cipher/comment_3_bd0c975494333dfe558de048d888ace8._comment new file mode 100644 index 000000000..d0b98b7f6 --- /dev/null +++ b/doc/todo/faster_gnupg_cipher/comment_3_bd0c975494333dfe558de048d888ace8._comment @@ -0,0 +1,17 @@ +[[!comment format=mdwn + username="guilhem" + ip="129.16.20.209" + subject="comment 3" + date="2013-08-19T13:44:35Z" + content=""" +AES-NI acceleration will be used by default providing you're using +the new modularized GnuPG (v2.x) and libgcrypt ≥ 1.5.0. Of course it +only speeds up AES encryption, while GnuPG uses CAST by default; you can +either set `personal-cipher-preferences` to AES or AES256 in your +`gpg.conf` or, as joeyh hinted at, set `remote.<name>.annex-gnupg-options` +as described in the manpage. + +By the way, I observed a significant speed up when using `--compress-algo none`. +Image, music and video files are typically hard to compress further, and it seems +that's where gpg spent most of its time, at least on the few files I benchmarked. +"""]] diff --git a/doc/todo/faster_rsync_remotes.mdwn b/doc/todo/faster_rsync_remotes.mdwn new file mode 100644 index 000000000..8c40b2816 --- /dev/null +++ b/doc/todo/faster_rsync_remotes.mdwn @@ -0,0 +1,4 @@ +Using an rsync remote is currently very slow when there are a lot of files, since rsync appears to be called for each file copied. It would be awesome if each call to rsync was amortized to copy many files; rsync is very good at copying many small files quickly. + +> [[done]]; bug submitter was apparently not using a version +> with rsync connection caching. --[[Joey]] diff --git a/doc/todo/faster_rsync_remotes/comment_1_0bc3ee0ae563357675eeccf42461e59a._comment b/doc/todo/faster_rsync_remotes/comment_1_0bc3ee0ae563357675eeccf42461e59a._comment new file mode 100644 index 000000000..2f320fee2 --- /dev/null +++ b/doc/todo/faster_rsync_remotes/comment_1_0bc3ee0ae563357675eeccf42461e59a._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.152.108.145" + subject="comment 1" + date="2013-08-01T16:06:42Z" + content=""" +I cannot see a way to do this using rsync's current command-line interface. Ideas how to do it welcomed. +"""]] diff --git a/doc/todo/faster_rsync_remotes/comment_2_ccf6f75450c89ca498c8130054f8d32d._comment b/doc/todo/faster_rsync_remotes/comment_2_ccf6f75450c89ca498c8130054f8d32d._comment new file mode 100644 index 000000000..67b5feab0 --- /dev/null +++ b/doc/todo/faster_rsync_remotes/comment_2_ccf6f75450c89ca498c8130054f8d32d._comment @@ -0,0 +1,24 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawln4uCaqZRd5_nRQ-iLcJyGctIdw8ebUiM" + nickname="Edward" + subject="Just put multiple source files" + date="2013-08-01T16:29:04Z" + content=""" +It seems like you can just put multiple source files on the command line: + + ed@ed-Ubu64 /tmp$ touch a b c d + ed@ed-Ubu64 /tmp$ mkdir test + ed@ed-Ubu64 /tmp$ rsync -avz a b c d test + sending incremental file list + a + b + c + d + + sent 197 bytes received 88 bytes 570.00 bytes/sec + total size is 0 speedup is 0.00 + ed@ed-Ubu64 /tmp$ ls test + a b c d + +It also appears to work with remote transfers too. +"""]] diff --git a/doc/todo/faster_rsync_remotes/comment_3_2f6a9d23cb8351fbf0f60ed93752e76e._comment b/doc/todo/faster_rsync_remotes/comment_3_2f6a9d23cb8351fbf0f60ed93752e76e._comment new file mode 100644 index 000000000..1911048be --- /dev/null +++ b/doc/todo/faster_rsync_remotes/comment_3_2f6a9d23cb8351fbf0f60ed93752e76e._comment @@ -0,0 +1,14 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.152.108.145" + subject="comment 3" + date="2013-08-01T16:58:49Z" + content=""" +git-annex needs to build a specific directory structure on the rsync remote though. It seems it would need to build the whole tree locally, containing only the files it wants to send. + +When using encryption, it would need to encrypt all the files it's going to send and store them locally until it's built the tree. That could use a lot of disk space. + +Also, there's the problem of checking which files are already present in the remote, to avoid re-encrypting and re-sending them. Currently this is done by running rsync with the url of the file, and checking its exit code. rsync does not seem to have an interface that would allow checking multiple files in one call. So any optimisation of the number of rsync calls would only eliminate 1/2 of the current number. + +When using ssh:// urls, the rsync special remote already uses ssh connection caching, which I'd think would eliminate most of the overhead. (If you have a version of git-annex older than 4.20130417, you should upgrade to get this feature.) It should not take very long to start up a new rsync over a cached ssh connection. rsync:// is probably noticably slower. +"""]] diff --git a/doc/todo/faster_rsync_remotes/comment_4_3a2f45defebae3dde336ee5f40c26d7e._comment b/doc/todo/faster_rsync_remotes/comment_4_3a2f45defebae3dde336ee5f40c26d7e._comment new file mode 100644 index 000000000..44d7d5511 --- /dev/null +++ b/doc/todo/faster_rsync_remotes/comment_4_3a2f45defebae3dde336ee5f40c26d7e._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawln4uCaqZRd5_nRQ-iLcJyGctIdw8ebUiM" + nickname="Edward" + subject="Thanks" + date="2013-08-01T17:03:23Z" + content=""" +I am using an old version of git-annex. I'll try the newer one and see if the connection caching helps! +"""]] diff --git a/doc/todo/file_copy_progress_bar.mdwn b/doc/todo/file_copy_progress_bar.mdwn new file mode 100644 index 000000000..847c1d1eb --- /dev/null +++ b/doc/todo/file_copy_progress_bar.mdwn @@ -0,0 +1,5 @@ +Find a way to copy a file with a progress bar, while still preserving +stat. Easiest way might be to use pv and fix up the permissions etc +after? + +[[done]] diff --git a/doc/todo/free_space_checking_for_local_special_remotes.mdwn b/doc/todo/free_space_checking_for_local_special_remotes.mdwn new file mode 100644 index 000000000..0fd2eac59 --- /dev/null +++ b/doc/todo/free_space_checking_for_local_special_remotes.mdwn @@ -0,0 +1,4 @@ +Should be possible to configure an annex.diskreserve setting for local +special remotes, such as a directory special remote or possibly a bup +special remote. (Although bup's deltas will make storing some versions of +files take less space than git-annex would have to assume it would take.) diff --git a/doc/todo/free_space_checking_for_local_special_remotes/comment_1_47c254cec58cbbb3ea84c93ef8282f01._comment b/doc/todo/free_space_checking_for_local_special_remotes/comment_1_47c254cec58cbbb3ea84c93ef8282f01._comment new file mode 100644 index 000000000..00c4c5625 --- /dev/null +++ b/doc/todo/free_space_checking_for_local_special_remotes/comment_1_47c254cec58cbbb3ea84c93ef8282f01._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.154.7.235" + subject="comment 1" + date="2013-07-11T16:19:26Z" + content=""" +Directory special remotes do already honor annex.diskreserve, it turns out. Let's repurpose this bug to be about adding a per-remote configuration, in case the main annex.diskreserve is not appropriate. +"""]] diff --git a/doc/todo/fsck.mdwn b/doc/todo/fsck.mdwn new file mode 100644 index 000000000..1dcaad9a5 --- /dev/null +++ b/doc/todo/fsck.mdwn @@ -0,0 +1,11 @@ +add a git annex fsck that finds keys that have no referring file + +(done) + +* Need per-backend fsck support. sha1 can checksum all files in the annex. + WORM can check filesize. + +* Both can check that annex.numcopies is satisfied. Probably only + querying the locationlog, not doing an online verification. + +[[done]] diff --git a/doc/todo/fsck_special_remotes.mdwn b/doc/todo/fsck_special_remotes.mdwn new file mode 100644 index 000000000..7196baafe --- /dev/null +++ b/doc/todo/fsck_special_remotes.mdwn @@ -0,0 +1,13 @@ +`git annex fsck --from remote` + +Basically, this needs to receive each file in turn from the remote, to a +temp file, and then run the existing fsck code on it. Could be quite +expensive, but sometimes you really want to check. + +An unencrypted directory special remote could be optimised, by not actually +copying the file, just dropping a symlink, etc. + +The WORM backend doesn't care about file content, so it would be nice to +avoid transferring the content at all, and only send the size. + +> [[done]] --[[Joey]] diff --git a/doc/todo/git-annex-shell.mdwn b/doc/todo/git-annex-shell.mdwn new file mode 100644 index 000000000..a9e3b43ed --- /dev/null +++ b/doc/todo/git-annex-shell.mdwn @@ -0,0 +1,15 @@ +[[done]] + +I've been considering adding a `git-annex-shell` command. This would +be similar to `git-shell` (and in fact would pass unknown commands off to +`git-shell`). + +## Reasons + +* Allows locking down an account to only be able to use git-annex (and + git). +* Avoids needing to construct complex shell commands to run on the remote + system. (Mostly already avoided by the plumbing level commands.) +* Could possibly allow multiple things to be done with one ssh connection + in future. +* Allows expanding `~` and `~user` in repopath on the remote system. diff --git a/doc/todo/git-annex_unused_eats_memory.mdwn b/doc/todo/git-annex_unused_eats_memory.mdwn new file mode 100644 index 000000000..760a6ccf5 --- /dev/null +++ b/doc/todo/git-annex_unused_eats_memory.mdwn @@ -0,0 +1,32 @@ +`git-annex unused` has to compare large sets of data +(all keys with content present in the repository, +with all keys used by files in the repository), and so +uses more memory than git-annex typically needs. + +It used to be a lot worse (hundreds of megabytes). + +Now it only needs enough memory to store a Set of all Keys that currently +have content in the annex. On a lightly populated repository, it runs in +quite low memory use (like 8 mb) even if the git repo has 100 thousand +files. On a repository with lots of file contents, it will use more. + +Still, I would like to reduce this to a purely constant memory use, +as running in constant memory no matter the repo size is a git-annex design +goal. + +One idea is to use a bloom filter. +For example, construct a bloom filter of all keys used by files in +the repository. Then for each key with content present, check if it's +in the bloom filter. Since there can be false positives, this might +miss finding some unused keys. The probability/size of filter +could be tunable. + +> Fixed in `bloom` branch in git. --[[Joey]] +>> [[done]]! --[[Joey]] + +Another way might be to scan the git log for files that got removed +or changed what key they pointed to. Correlate with keys with content +currently present in the repository (possibly using a bloom filter again), +and that would yield a shortlist of keys that are probably not used. +Then scan thru all files in the repo to make sure that none point to keys +on the shortlist. diff --git a/doc/todo/git_annex_init_:_include_repo_description_and__47__or_UUID_in_commit_message.mdwn b/doc/todo/git_annex_init_:_include_repo_description_and__47__or_UUID_in_commit_message.mdwn new file mode 100644 index 000000000..be7e2dacc --- /dev/null +++ b/doc/todo/git_annex_init_:_include_repo_description_and__47__or_UUID_in_commit_message.mdwn @@ -0,0 +1,13 @@ +Would help alot when having to add large(ish) amounts of remotes. + +Maybe detect this kind of commit message and ask user whether to automatically add them? See [[auto_remotes]]: +> Question: When should git-annex update the remote.log? (If not just on init.) Whenever it reads in a repo's remotes? + +---- + +I'm not sure that the above suggestion is going down a path that really +makes sense. If you want a list of repository UUIDs and descriptions, +it's there in machine-usable form in `.git-annex/uuid.log`, there is no +need to try to pull this info out of git commit messages. --[[Joey]] + +[[done]] diff --git a/doc/todo/gitolite_and_gitosis_support.mdwn b/doc/todo/gitolite_and_gitosis_support.mdwn new file mode 100644 index 000000000..2fca83986 --- /dev/null +++ b/doc/todo/gitolite_and_gitosis_support.mdwn @@ -0,0 +1,39 @@ +gitosis and gitolite should support git-annex being used to send/receive +files from the repositories they manage. Users with read-only access +could only get files, while users with write access could also put and drop +files. + +Doing this right requires modifying both programs, to add [[git-annex-shell]] +to the list of things they can run, and only allow through appropriate +git-annex-shell subcommands to read-only users. + +I have posted an RFC for modifying gitolite to the +[gitolite mailing list](http://groups.google.com/group/gitolite?lnk=srg). + +> I have not developed a patch yet, but all that git-annex needs is a way +> to ssh to the server and run the git-annex-shell command there. +> git-annex-shell is very similar to git-shell. So, one way to enable +> it is simply to set GL_ADC_PATH to a directory containing git-annex-shell. +> +> But, that's not optimal, since git-annex-shell will send off receive-pack +> commands to git, which would bypass gitolite's permissions checking. +> Also, it makes sense to limit readonly users to only download, not +> upload/delete files from git-annex. Instead, I suggest adding something +> like this to gitolite's config: + + # If set, users with W access can write file contents into the git-annex, + # and users with R access can read file contents from the git-annex. + $GL_GIT_ANNEX = 0; + +> If this makes sense, I'm sure I can put a patch together for your +> review. It would involve modifying gl-auth-command so it knows how +> to run git-annex-shell, and how to parse out the "verb" from a +> git-annex-shell command line, and modifying R_COMMANDS and W_COMMANDS. + +As I don't write python, someone else is needed to work on gitosis. +--[[Joey]] + +> [[done]]; support for gitolite is in its `pu` branch, and some changes +> made to git-annefor gitolite is in its `pu` branch, and some changes +> made to git-annex. Word is gitosis is not being maintained so I won't +> worry about try to support it. --[[Joey]] diff --git a/doc/todo/gitrm.mdwn b/doc/todo/gitrm.mdwn new file mode 100644 index 000000000..e41c33462 --- /dev/null +++ b/doc/todo/gitrm.mdwn @@ -0,0 +1,5 @@ +how to handle git rm file? (should try to drop keys that have no +referring file, if it seems safe..) + +[[done]] -- I think that git annex unused and dropunused are the best +solution to this. diff --git a/doc/todo/hidden_files.mdwn b/doc/todo/hidden_files.mdwn new file mode 100644 index 000000000..191e9c328 --- /dev/null +++ b/doc/todo/hidden_files.mdwn @@ -0,0 +1,30 @@ +Add a `git annex hide $file` that behaves like drop, checking counter info +and updating location log to say the current repo no longer has a file -- +but does not actually remove the content. + +Then `git annex unused` can be used to clean it up later. And in the +meantime, it's still locally accessible. This can be useful if you're +planning to need to free up space later, but want to hold onto the content +for a while. Possibly you'll be disconnected later, so it's easier to push +out that intent now. + +-- + +TODO: + +* Make 100% sure this is safe. Drop, etc should never check content files + are present on other repos if the location log doesn't say the repo + has the content. + +* What will `git annex get` do if it's asked to get a file that has been + hidden? + +> Unless I am missing something: Make sure the data is correct (for SHA1 or other tracking) and restore locally. If that's not the case, delete and restore from remote. -- RichiH + +---- + +Is 'unused' a good name? 'clean' and 'autoclean' would make more sense, imo. 'clean' deletes everything, whereas an optional 'autoclean' could try to be smart based on disk usage and/or SHA1, etc. -- RichiH + +> Nah, `git annex unused/dropunused` already exist. --[[Joey]] + +>> OK, in that case forget what I said. No idea about your internal policy, but feel free to delete this part of the page, then. -- RichiH diff --git a/doc/todo/http_git_annex_404_retry.mdwn b/doc/todo/http_git_annex_404_retry.mdwn new file mode 100644 index 000000000..38ab860bb --- /dev/null +++ b/doc/todo/http_git_annex_404_retry.mdwn @@ -0,0 +1,16 @@ +A repository like http://annex.debconf.org/debconf-share/ has a git repo +published via http. When getting files from such a repo, git-annex tries +two urls. One url would be used by a bare repo, and the other by a non-bare +repo. (This is due to the directory hashing change.) Result is every file +download from a non-bare http repo starts with a 404 and then it retries +with the right url. + +Since git-annex already downloads the .git/config to find the uuid of the +http repo, it could also look at it to see if the repo is bare. If not, +set a flag, and try the two urls in reverse order, which would almost +always avoid this 404 problem. + +(The real solution is probably to flag day and get rid of the old-style +directory hashing, but that's been discussed elsewhere.) + +--[[Joey]] diff --git a/doc/todo/http_headers.mdwn b/doc/todo/http_headers.mdwn new file mode 100644 index 000000000..9f61bdc93 --- /dev/null +++ b/doc/todo/http_headers.mdwn @@ -0,0 +1,8 @@ +The IA would find it useful to be able to control the http headers +git-annex get, addurl, etc uses. This will allow setting cookies, for +example. + +* annex-web-headers=blah +* Perhaps also annex-web-headers-command=blah + +[[done]] diff --git a/doc/todo/immutable_annexed_files.mdwn b/doc/todo/immutable_annexed_files.mdwn new file mode 100644 index 000000000..b26838e95 --- /dev/null +++ b/doc/todo/immutable_annexed_files.mdwn @@ -0,0 +1,8 @@ +> josh: Do you do anything in git-annex to try to make the files immutable? +> For instance, removing write permission, or even chattr? +> joey: I don't, but that's a very good idea +> josh: Oh, I just thought of another slightly crazy but handy idea. +> josh: I'd hate to run into a program which somehow followed the symlink and then did an unlink to replace the file. +> josh: To break that, you could create a new directory under annex's internal directory for each file, and make the directory have no write permission. + +[[done]] and done --[[Joey]] diff --git a/doc/todo/importfeed:_allow___36____123__itemdate__125___with_--template.mdwn b/doc/todo/importfeed:_allow___36____123__itemdate__125___with_--template.mdwn new file mode 100644 index 000000000..46d9de34f --- /dev/null +++ b/doc/todo/importfeed:_allow___36____123__itemdate__125___with_--template.mdwn @@ -0,0 +1,5 @@ +It would be great to be able to use the pubDate of the entries with the --template option of importfeed. + +Text.Feed.Query has a getItemPublishDate (and a getFeedPubDate, if we want some kind of ${feeddate}). + +The best would be to allow a reformating of the date(s) with (for example) %Y-%m-%D diff --git a/doc/todo/importfeed:_allow___36____123__itemdate__125___with_--template/comment_1_62752c760fc12eca0c34d67d58753d00._comment b/doc/todo/importfeed:_allow___36____123__itemdate__125___with_--template/comment_1_62752c760fc12eca0c34d67d58753d00._comment new file mode 100644 index 000000000..cc3d85faf --- /dev/null +++ b/doc/todo/importfeed:_allow___36____123__itemdate__125___with_--template/comment_1_62752c760fc12eca0c34d67d58753d00._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="gueux" + ip="2a01:240:fe6d:0:7986:3659:a8bd:64f1" + subject="syntax" + date="2013-09-12T14:05:16Z" + content=""" +use \"itemdate\" and \"feeddate\" as names? + +use ${itemdate=%Y-%m-%D} syntax option? +"""]] diff --git a/doc/todo/importfeed:_allow___36____123__itemdate__125___with_--template/comment_2_21672360060f48bc2eacfa535ff4c94d._comment b/doc/todo/importfeed:_allow___36____123__itemdate__125___with_--template/comment_2_21672360060f48bc2eacfa535ff4c94d._comment new file mode 100644 index 000000000..c8770ec6e --- /dev/null +++ b/doc/todo/importfeed:_allow___36____123__itemdate__125___with_--template/comment_2_21672360060f48bc2eacfa535ff4c94d._comment @@ -0,0 +1,11 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.154.2.134" + subject="comment 2" + date="2013-09-13T19:53:52Z" + content=""" +getItemPublishDate returns a String, which can contain any of several date formats. Deferred until the feed library has something more sane. +Upstream bug: <https://github.com/sof/feed/issues/6> + +As for how to format the date in the feed, I would be ok with having itemdate (YYYYMMDD), itemyear (YYYY), itemmonth (MM) and itemday (DD). Full date formatting seems like overkill here. +"""]] diff --git a/doc/todo/incremental_fsck.mdwn b/doc/todo/incremental_fsck.mdwn new file mode 100644 index 000000000..7c56328b9 --- /dev/null +++ b/doc/todo/incremental_fsck.mdwn @@ -0,0 +1,24 @@ +Justin Azoff realized git-annex should have an incremental fsck. + +This requires storing the last fsck time of each object. + +I would not be strongly opposed to sqlite, but I think there are other +places the data could be stored. One possible place is the mode or mtime +of the .git/annex/objects/xx/yy/$key directories (the parent directories +of where the content is stored). Perhaps the sticky bit could be used to +indicate the content has been fsked, and the mtime indicate the time +of last fsck. Anything that dropped or put in content would need to +clear the sticky bit. --[[Joey]] + +> Basic incremental fsck is done now. +> +> Some enhancements would include: +> +> * --max-age=30d Once the incremental fsck completes and was started 30 days ago, +> start a new one. +> * --time-limit --size-limit --file-limit: Limit how long the fsck runs. + +>> Calling this [[done]]. The `--incremental-schedule` option +>> allows scheduling time between incremental fscks. `--time-limit` is +>> done. I implemented `--smallerthan` independently. Not clear what +>> `--file-limit` would be. --[[Joey]] diff --git a/doc/todo/incremental_fsck/comment_1_609b21141dd5686b2c0eaef2b8d63229._comment b/doc/todo/incremental_fsck/comment_1_609b21141dd5686b2c0eaef2b8d63229._comment new file mode 100644 index 000000000..709ba078c --- /dev/null +++ b/doc/todo/incremental_fsck/comment_1_609b21141dd5686b2c0eaef2b8d63229._comment @@ -0,0 +1,14 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawmBUR4O9mofxVbpb8JV9mEbVfIYv670uJo" + nickname="Justin" + subject="comment 1" + date="2012-09-20T14:11:57Z" + content=""" +I have a [proof of concept written in python](https://github.com/JustinAzoff/git-annex-background-fsck/blob/master/git-annex-background-fsck). + +You can run it and point it the root of an annex or to a subdirectory. In my brief testing it seems to work :-) + +the goal would be to have options like + + git annex fsck /data/annex --check-older-than 1w --check-for 2h --max-load-avg 0.5 +"""]] diff --git a/doc/todo/keep_annexed_files_for_a_while.mdwn b/doc/todo/keep_annexed_files_for_a_while.mdwn new file mode 100644 index 000000000..cf85b11f3 --- /dev/null +++ b/doc/todo/keep_annexed_files_for_a_while.mdwn @@ -0,0 +1,8 @@ +I don't want files that I dropped to immediately disappear from my local or all of my remotes repos on the next sync. Especially in situations where changes to the git-annex repo get automatically and immediately replicated to remote repos, I want a configurable "grace" period before files in .git/annex/objects get really deleted. + +This has similarities to the "trash" on a desktop. It might also be nice to + +* configure a maximum amount of space of the "trash" +* have a way to see the contents of the trash to easily recover deleted files + +Maybe it would make sense to just move dropped files to the desktops trash? "git annex trash" as an alternative to drop? diff --git a/doc/todo/link_file_to_remote_repo_feature.mdwn b/doc/todo/link_file_to_remote_repo_feature.mdwn new file mode 100644 index 000000000..d6b41e805 --- /dev/null +++ b/doc/todo/link_file_to_remote_repo_feature.mdwn @@ -0,0 +1,52 @@ +I have two repos, using SHA1 backend and both using git. +The first one is a laptop, the second one is a usb drive. + +When I drop a file on the laptop repo, the file is not available on that repo until I run *git annex get* +But when the usb drive is plugged in the file is actually available. + +How about adding a feature to link some/all files to the remote repo? + +e.g. +We have *railscasts/196-nested-model-form-part-1.mp4* file added to git, and only available on the usb drive: + + $ git annex whereis 196-nested-model-form-part-1.mp4 + whereis 196-nested-model-form-part-1.mp4 (1 copy) + a7b7d7a4-2a8a-11e1-aebc-d3c589296e81 -- origin (Portable usb drive) + +I can see the link with: + + $ cd railscasts + $ ls -ls 196* + 8 lrwxr-xr-x 1 framallo staff 193 Dec 20 05:49 196-nested-model-form-part-1.mp4 -> ../.git/annex/objects/Wz/6P/SHA256-s16898930--43679c67cd968243f58f8f7fb30690b5f3f067574e318d609a01613a2a14351e/SHA256-s16898930--43679c67cd968243f58f8f7fb30690b5f3f067574e318d609a01613a2a14351e + +I save this in a variable just to make the example more clear: + + ID=".git/annex/objects/Wz/6P/SHA256-s16898930--43679c67cd968243f58f8f7fb30690b5f3f067574e318d609a01613a2a14351e/SHA256-s16898930--43679c67cd968243f58f8f7fb30690b5f3f067574e318d609a01613a2a14351e" + +The file doesn't exist on the local repo: + + $ ls ../$ID + ls: ../$ID: No such file or directory + +however I can create a link to access that file on the remote repo. +First I create a needed dir: + + $ mkdir ../.git/annex/objects/Wz/6P/SHA256-s16898930--43679c67cd968243f58f8f7fb30690b5f3f067574e318d609a01613a2a14351e/ + +Then I link to the remote file: + + $ ln -s /mnt/usb_drive/repo_folder/$ID ../$ID + +now I can open the file in the laptop repo. + + +I think it could be easy to implement. Maybe It's a naive approach, but looks apealing. +Checking if it's a real file or a link shouldn't impact on performance. +The limitation is that it would work only with remote repos on local dirs + +Also allows you to have one directory structure like AFS or other distributed FS. If the file is not local I go to the remote server. +Which is great for apps like Picasa, Itunes, and friends that depends on the file location. + +> This is a duplicate of [[union_mounting]]. So closing it: [[done]]. +> +> It's a good idea, but making sure git-annex correctly handles these links in all cases is a subtle problem that has not yet been tackled. --[[Joey]] diff --git a/doc/todo/makefile:_respect___36__PREFIX.mdwn b/doc/todo/makefile:_respect___36__PREFIX.mdwn new file mode 100644 index 000000000..12d7274b9 --- /dev/null +++ b/doc/todo/makefile:_respect___36__PREFIX.mdwn @@ -0,0 +1,25 @@ +The `Makefile` should respect a `PREFIX` passed on the commandline so git-annex can be installed in (say) `$HOME`. + +Simple patch: + +[[!format diff """ +diff --git a/Makefile b/Makefile +index b8995b2..5b1a6d4 100644 +--- a/Makefile ++++ b/Makefile +@@ -3,7 +3,7 @@ all=git-annex $(mans) docs + + GHC?=ghc + GHCMAKE=$(GHC) $(GHCFLAGS) --make +-PREFIX=/usr ++PREFIX?=/usr + CABAL?=cabal # set to "./Setup" if you lack a cabal program + + # Am I typing :make in vim? Do a fast build. +"""]] + +--[[anarcat]] + +> [[done]] --[[Joey]] + +> > [[thanks]]! ;) --[[anarcat]] diff --git a/doc/todo/mdwn2man:_make_backticks_bold.mdwn b/doc/todo/mdwn2man:_make_backticks_bold.mdwn new file mode 100644 index 000000000..21707a309 --- /dev/null +++ b/doc/todo/mdwn2man:_make_backticks_bold.mdwn @@ -0,0 +1,22 @@ +The traditionnal way of marking commandline flags in a manpage is with a `.B` (for Bold, I guess). It doesn't seem to be used by mdwn2man, which makes the manpage look a little more dull than it could. + +The following patch makes those options come out more obviously: + +[[!format diff """ +diff --git a/Build/mdwn2man b/Build/mdwn2man +index ba5919b..7f819ad 100755 +--- a/Build/mdwn2man ++++ b/Build/mdwn2man +@@ -8,6 +8,7 @@ print ".TH $prog $section\n"; + + while (<>) { + s{(\\?)\[\[([^\s\|\]]+)(\|[^\s\]]+)?\]\]}{$1 ? "[[$2]]" : $2}eg; ++ s/\`([^\`]*)\`/\\fB$1\\fP/g; + s/\`//g; + s/^\s*\./\\&./g; + if (/^#\s/) { +"""]] + +I tested it against the git-annex manpage and it seems to work well. --[[anarcat]] + +> [[done]], thanks --[[Joey]] diff --git a/doc/todo/network_remotes.mdwn b/doc/todo/network_remotes.mdwn new file mode 100644 index 000000000..42efa832f --- /dev/null +++ b/doc/todo/network_remotes.mdwn @@ -0,0 +1,5 @@ +Support for remote git repositories (ssh:// specifically can be made to +work, although the other end probably needs to have git-annex +installed..) + +[[done]], at least get and put work.. diff --git a/doc/todo/nicer_whereis_output.mdwn b/doc/todo/nicer_whereis_output.mdwn new file mode 100644 index 000000000..871eee01a --- /dev/null +++ b/doc/todo/nicer_whereis_output.mdwn @@ -0,0 +1,100 @@ +We had some informal discussions on IRC about improving the output of the `whereis` command. + +[[!toc levels=2]] + +First version: columns +====================== + +[[mastensg]] started by implementing a [simple formatter](https://gist.github.com/mastensg/6500982) that would display things in columns [screenshot](http://www.ping.uio.no/~mastensg/whereis.png) + +Second version: Xs +================== + +After some suggestions from [[joey]], [[mastensg]] changed the format slightly ([screenshot](http://www.ping.uio.no/~mastensg/whereis2.png)): + +[[!format txt """ +17:01:34 <joeyh> foo +17:01:34 <joeyh> |bar +17:01:34 <joeyh> ||baz (untrusted) +17:01:34 <joeyh> ||| +17:01:34 <joeyh> XXx 3? img.png +17:01:36 <joeyh> _X_ 1! bigfile +17:01:37 <joeyh> XX_ 2 zort +17:01:39 <joeyh> __x 1?! maybemissing +17:02:09 * joeyh does a s/\?/+/ in the above +17:02:24 <joeyh> and decrements the counters for untrusted +17:03:37 <joeyh> __x 0+! maybemissing +"""]] + +Third version: incremental +========================== + +Finally, [[anarcat]] worked on making it run faster on large repositories, in a [fork](https://gist.github.com/anarcat/6502988) of that first gist. Then paging was added (so headers are repeated). + +Fourth version: tuning and blocked +================================== + +[[TobiasTheViking]] provided some bugfixes, and the next step was to implement the trusted/untrusted detection, and have a counter. + +This required more advanced parsing of the remotes, and instead of starting to do some JSON parsing, [[anarcat]] figured it was time to learn some Haskell instead. + +Current status: needs merge +=========================== + +So right now, the most recent version of the python script is in [anarcat's gist](https://gist.github.com/anarcat/6502988) and works reasonably well. However, it doesn't distinguish between trusted and untrusted repos and so on. + +Furthermore, we'd like to see this factored into the `whereis` command directly. A [raw.hs](http://codepad.org/miVJb5oK) file has been programmed by `mastensg`, and is now available in the above gist. It fits the desired output and prototypes, and has been `haskellized` thanks to [[guilhem]]. + +Now we just need to merge those marvelous functions in `Whereis.hs` - but I can't quite figure out where to throw that code, so I'll leave it to someone more familiar with the internals of git-annex. The most recent version is still in [anarcat's gist](https://gist.github.com/anarcat/6502988). --[[anarcat]] + +Desired output +-------------- + +The output we're aiming for is: + + foo + |bar + ||baz (untrusted) + ||| + XXx 2+ img.png + _X_ 1! bigfile + XX_ 2 zort + __x 0+! maybemissing + +Legend: + + * `_` - file missing from repo + * `x` - file may be present in untrusted repo + * `X` - file is present in trusted repo + * `[0-9]` - number of copies present in trusted repos + * `+` - indicates there may be more copies present + * `!` - indicates only one copy is left + +Implementation notes +-------------------- + +[[!format txt """ +20:48:18 <joeyh> if someone writes me a headerWhereis :: [(RemoteName, TrustLevel)] -> String and a formatWhereis :: [(RemoteName, TrustLevel, UUID)] -> [UUD] -> FileName -> String , I can do the rest ;) +20:49:22 <joeyh> make that second one formatWhereis :: [(RemoteName, TrueLevel, Bool)] -> FileName -> String +20:49:37 <joeyh> gah, typos +20:49:45 <joeyh> suppose you don't need the RemoteName either +"""]] + +> So, I incorporated this, in a new remotes command. +> Showing all known repositories seemed a bit much +> (I have 30-some known repositories in some cases), +> so just showing configured remotes seems a good simplification. +> [[done]] +> --[[Joey]] + +> > I would have prefered this to be optional since I don't explicitely configure all remotes in git, especially if I can't reach them all the time (e.g. my laptop). It seems to me this should at least be an option, but I am confused as to why `Remote.List.remoteList` doesn't list all remotes the same way `Remote.remote_list` does... Also, it's unfortunate that the +/!/count flags have been dropped, it would have been useful... Thanks for the merge anyways! --[[done]] +> > +> > The more I look at this, the more i think there are a few things wrong with the new `remotes` command. +> > +> > 1. the name is confusing: being a git addict, I would expect the `git annex remote` command to behave like the `git remote` command: list remotes, add remotes, remove remotes and so on. it would actually be useful to have such a command (which would replace `initremote`, I guess). i recommend replacing the current `whereis` command, even if enabled through a special flag +> > +> > 2. its behavior is inconsistent with other git annex commands: `git annex status`, for example, lists information about all remotes, regardless of whether they are configured in git. `remotes` (whatever it's called), should do the same, or at least provide an option to allow the user to list files on all remotes. The way things stand, there is no way to list files on non-git remotes, even if they are added explicitely as a remote, if the remote is not actually reachable: the files are just marked as absent (even thought `whereis` actually finds them). i recommend showing all remotes regardless, either opt-in or opt-out using a flag. +> > +> > 3. having the `!` flag, at least, would be useful because it would allow users to intuitively grep for problematic files without having to learn extra syntax. same with + and having an explicit count. +> > +> > thanks. --[[anarcat]] diff --git a/doc/todo/object_dir_reorg_v2.mdwn b/doc/todo/object_dir_reorg_v2.mdwn new file mode 100644 index 000000000..49666ddc7 --- /dev/null +++ b/doc/todo/object_dir_reorg_v2.mdwn @@ -0,0 +1,25 @@ +Several things suggest now would be a good time to reorgaize the object +directory. This would be annex.version=2. It will be slightly painful for +all users, so this should be the *last* reorg in the forseeable future. + +1. Remove colons from filenames, for [[bugs/fat_support]] + +2. Add hashing, since some filesystems do suck (like er, fat at least :) + [[forum/hashing_objects_directories]] + (Also, may as well hash .git-annex/* while at it -- that's what + really gets big.) + +3. Add filesize metadata for [[bugs/free_space_checking]]. (Currently only + present in WORM, and in an ad-hoc way.) + +4. Perhaps use a generic format that will allow further metadata to be + added later. For example, + "bSHA1,s101111,kf3101c30bb23467deaec5d78c6daa71d395d1879" + + (Probably everything after ",k" should be part of the key, even if it + contains the "," separator character. Otherwise an escaping mechanism + would be needed.) + +[[done]] now! + +Although [[bugs/free_space_checking]] is not quite there --[[Joey]] diff --git a/doc/todo/object_dir_reorg_v2/comment_1_ba03333dc76ff49eccaba375e68cb525._comment b/doc/todo/object_dir_reorg_v2/comment_1_ba03333dc76ff49eccaba375e68cb525._comment new file mode 100644 index 000000000..261c2a51f --- /dev/null +++ b/doc/todo/object_dir_reorg_v2/comment_1_ba03333dc76ff49eccaba375e68cb525._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U" + nickname="Richard" + subject="comment 1" + date="2011-03-16T01:16:48Z" + content=""" +If you support generic meta-data, keep in mind that you will need to do conflict resolution. Timestamps may not be synched across all systems, so keeping a log of old metadata could be used, sorting by history and using the latest. Which leaves the situation of two incompatible changes. This would probably mean manual conflict resolution. You will probably have thought of this already, but I still wanted to make sure this is recorded. -- RichiH +"""]] diff --git a/doc/todo/object_dir_reorg_v2/comment_2_81276ac309959dc741bc90101c213ab7._comment b/doc/todo/object_dir_reorg_v2/comment_2_81276ac309959dc741bc90101c213ab7._comment new file mode 100644 index 000000000..9785f1989 --- /dev/null +++ b/doc/todo/object_dir_reorg_v2/comment_2_81276ac309959dc741bc90101c213ab7._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U" + nickname="Richard" + subject="comment 2" + date="2011-03-16T01:19:25Z" + content=""" +Hmm, I added quite a few comments at work, but they are stuck in moderation. Maybe I forgot to log in before adding them. I am surprised this one appeared immediately. -- RichiH +"""]] diff --git a/doc/todo/object_dir_reorg_v2/comment_3_79bdf9c51dec9f52372ce95b53233bb2._comment b/doc/todo/object_dir_reorg_v2/comment_3_79bdf9c51dec9f52372ce95b53233bb2._comment new file mode 100644 index 000000000..886941be7 --- /dev/null +++ b/doc/todo/object_dir_reorg_v2/comment_3_79bdf9c51dec9f52372ce95b53233bb2._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U" + nickname="Richard" + subject="comment 1" + date="2011-03-15T14:08:41Z" + content=""" +What is the potential time-frame for this change? As I am not using git-annex for production yet, I can see myself waiting to avoid any potential hassle. + +Supporting generic metadata seems like a great idea. Though if you are going this path, wouldn't it make sense to avoid metastore for mtime etc and support this natively without outside dependencies? + +-- RichiH +"""]] diff --git a/doc/todo/object_dir_reorg_v2/comment_4_93aada9b1680fed56cc6f0f7c3aca5e5._comment b/doc/todo/object_dir_reorg_v2/comment_4_93aada9b1680fed56cc6f0f7c3aca5e5._comment new file mode 100644 index 000000000..475359abb --- /dev/null +++ b/doc/todo/object_dir_reorg_v2/comment_4_93aada9b1680fed56cc6f0f7c3aca5e5._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="http://joey.kitenet.net/" + nickname="joey" + subject="comment 4" + date="2011-03-16T03:22:45Z" + content=""" +Well, I spent a few hours playing this evening in the 'reorg' branch in git. It seems to be shaping up pretty well; type-based refactoring in haskell makes these kind of big systematic changes a matter of editing until it compiles. And it compiles and test suite passes. But, so far I've only covered 1. 3. and 4. on the list, and have yet to deal with upgrades. + +I'd recommend you not wait before using git-annex. I am committed to provide upgradability between annexes created with all versions of git-annex, going forward. This is important because we can have offline archival drives that sit unused for years. Git-annex will upgrade a repository to current standard the first time it sees it, and I hope the upgrade will be pretty smooth. It was not bad for the annex.version 0 to 1 upgrade earlier. The only annoyance with upgrades is that it will result in some big commits to git, as every symlink in the repo gets changed, and log files get moved to new names. + +(The metadata being stored with keys is data that a particular backend can use, and is static to a given key, so there are no merge issues (and it won't be used to preserve mtimes, etc).) +"""]] diff --git a/doc/todo/object_dir_reorg_v2/comment_5_821c382987f105da72a50e0a5ce61fdc._comment b/doc/todo/object_dir_reorg_v2/comment_5_821c382987f105da72a50e0a5ce61fdc._comment new file mode 100644 index 000000000..2032bce3c --- /dev/null +++ b/doc/todo/object_dir_reorg_v2/comment_5_821c382987f105da72a50e0a5ce61fdc._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U" + nickname="Richard" + subject="comment 5" + date="2011-03-16T15:51:30Z" + content=""" +Hashing & segmenting seems to be around the corner, which is nice :) + +Is there a chance that you will optionally add mtime to your native metadata store? If yes, I'd rather wait for v2 to start with the native system from the start. If not, I will probably set it up tonight. + +PS: While posting from work, my comments are held for moderation once again. I am somewhat confused as to why this happens when I can just submit directly from home. And yes, I am using the same auth provider and user in both cases. +"""]] diff --git a/doc/todo/object_dir_reorg_v2/comment_6_8834c3a3f1258c4349d23aff8549bf35._comment b/doc/todo/object_dir_reorg_v2/comment_6_8834c3a3f1258c4349d23aff8549bf35._comment new file mode 100644 index 000000000..ff86e3970 --- /dev/null +++ b/doc/todo/object_dir_reorg_v2/comment_6_8834c3a3f1258c4349d23aff8549bf35._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://joey.kitenet.net/" + nickname="joey" + subject="comment 6" + date="2011-03-16T16:32:52Z" + content=""" +The mtime cannot be stored for all keys. Consider a SHA1 key. The mtime is irrelevant; 2 files with different mtimes, when added to the SHA1 backend, should get the same key. + +Probably our spam filter doesn't like your work IP. +"""]] diff --git a/doc/todo/object_dir_reorg_v2/comment_7_42501404c82ca07147e2cce0cff59474._comment b/doc/todo/object_dir_reorg_v2/comment_7_42501404c82ca07147e2cce0cff59474._comment new file mode 100644 index 000000000..fc866c57a --- /dev/null +++ b/doc/todo/object_dir_reorg_v2/comment_7_42501404c82ca07147e2cce0cff59474._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U" + nickname="Richard" + subject="comment 7" + date="2011-03-16T21:05:38Z" + content=""" +Ah, OK. I assumed the metadata would be attached to a key, not part of the key. This seems to make upgrades/extensions down the line harder than they need to be, but you are right that this way, merges are not, and never will be, an issue. + +Though with the SHA1 backend, changing files can be tracked. This means that tracking changes in mtime or other is possible. It also means that there are potential merge issues. But I won't argue the point endlessly. I can accept design decisions :) + +The prefix at work is from a university netblock so yes, it might be on a few hundred proxy lists etc. +"""]] diff --git a/doc/todo/optimise_git-annex_merge.mdwn b/doc/todo/optimise_git-annex_merge.mdwn new file mode 100644 index 000000000..91d18ebd7 --- /dev/null +++ b/doc/todo/optimise_git-annex_merge.mdwn @@ -0,0 +1,23 @@ +Typically `git-annex merge` is fast, but it could still be sped up. + +`git-annex merge` runs `git-hash-object` once per file that needs to be +merged. Elsewhere in git-annex, `git-hash-object` is used in a faster mode, +reading files from disk via `--stdin-paths`. But here, the data is not +in raw files on disk, and I doubt writing them is the best approach. +Instead, I'd like a way to stream multiple objects into git using stdin. +Sometime, should look at either extending git-hash-object to support that, +or possibly look at using git-fast-import instead. + +--- + +`git-annex merge` also runs `git show` once per file that needs to be +merged. This could be reduced to a single call to `git-cat-file --batch`, +There is already a Git.CatFile library that can do this easily. --[[Joey]] + +> This is now done, part above remains todo. --[[Joey]] + +--- + +Merging used to use memory proportional to the size of the diff. It now +streams data, running in constant space. This probably sped it up a lot, +as there's much less allocation and GC action. --[[Joey]] diff --git a/doc/todo/optinally_transfer_file_unencryptedly.mdwn b/doc/todo/optinally_transfer_file_unencryptedly.mdwn new file mode 100644 index 000000000..ef27dc521 --- /dev/null +++ b/doc/todo/optinally_transfer_file_unencryptedly.mdwn @@ -0,0 +1,6 @@ +I have a git-annex repository on a NSLU 2, and transfers are much slower over ssh compared to unencrypted transfers (no wonder at that CPU speed). For the files that I am transferring, no encryption would be necessary. Unfortunately, ssh in Debian does not support "-c none" to disable encryption. + +It would be nice if git-annex would have a way of conveniently transferring files in another way than SSH. I’m not sure what a good way would be – maybe launching a one-shot HTTP-server on the sending end? Haskell libraries for that would be available... Of course it is not always the case that the host reachable with "ssh foo" is also reachable via TCP at "foo:1234"... And there are surely more problem. But still, it would be nice :-) + +> Setting `remote.name.annex-rsync-transport = rsh` will now +> make rsync special remotes use rsh instead of ssh. [[done]] diff --git a/doc/todo/optinally_transfer_file_unencryptedly/comment_1_4be47e7ac85d0f4e7029a96b615545a7._comment b/doc/todo/optinally_transfer_file_unencryptedly/comment_1_4be47e7ac85d0f4e7029a96b615545a7._comment new file mode 100644 index 000000000..948845b23 --- /dev/null +++ b/doc/todo/optinally_transfer_file_unencryptedly/comment_1_4be47e7ac85d0f4e7029a96b615545a7._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="guilhem" + ip="129.16.20.212" + subject="rsh?" + date="2013-04-09T16:11:50Z" + content=""" +I don't use it myself, but rsync can be used with others remote shells, among which rsh supports unencrypted streams. You probably want to set up a secure authorization mechanism to deny access to intruders, and for that kerberos comes to the rescue :-) I didn't try the combination, but it should work over git-annex already. +"""]] diff --git a/doc/todo/parallel_possibilities.mdwn b/doc/todo/parallel_possibilities.mdwn new file mode 100644 index 000000000..9c0e69e29 --- /dev/null +++ b/doc/todo/parallel_possibilities.mdwn @@ -0,0 +1,13 @@ +One of my reasons for using haskell was that it provides the possibility of +some parallell processing. Although since git-annex hits the filesystem +heavily and mostly runs other git commands, maybe not a whole lot. + +Anyway, each git-annex command is broken down into a series of independant +actions, which has some potential for parallelism. + +Each action has 3 distinct phases, basically "check", "perform", and +"cleanup". The perform actions are probably parellizable; the cleanup may be +(but not if it has to run git commands to stage state; it can queue +commands though); the check should be easily parallelizable, although they +may access the disk or run minor git query commands, so would probably not +want to run too many of them at once. diff --git a/doc/todo/parallel_possibilities/comment_1_d8e34fc2bc4e5cf761574608f970d496._comment b/doc/todo/parallel_possibilities/comment_1_d8e34fc2bc4e5cf761574608f970d496._comment new file mode 100644 index 000000000..4aceb3abd --- /dev/null +++ b/doc/todo/parallel_possibilities/comment_1_d8e34fc2bc4e5cf761574608f970d496._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawkptNW1PzrVjYlJWP_9e499uH0mjnBV6GQ" + nickname="Christian" + subject="comment 1" + date="2011-04-08T12:41:43Z" + content=""" +I also think, that fetching keys via rsync can be done by one rsync process, when the keys are fetched from one host. This would avoid establishing a new TCP connection for every file. +"""]] diff --git a/doc/todo/parallel_possibilities/comment_2_adb76f06a7997abe4559d3169a3181c3._comment b/doc/todo/parallel_possibilities/comment_2_adb76f06a7997abe4559d3169a3181c3._comment new file mode 100644 index 000000000..6ecce52c4 --- /dev/null +++ b/doc/todo/parallel_possibilities/comment_2_adb76f06a7997abe4559d3169a3181c3._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="http://ertai.myopenid.com/" + nickname="npouillard" + subject="comment 2" + date="2011-05-20T20:14:15Z" + content=""" +I agree with Christian. + +One should first make a better use of connections to remotes before exploring parallel possibilities. One should pipeline the requests and answers. + +Of course this could be implemented using parallel&concurrency features of Haskell to do this. +"""]] diff --git a/doc/todo/parallel_possibilities/comment_3_145fb974f45da99b7d4b117a3699cccf._comment b/doc/todo/parallel_possibilities/comment_3_145fb974f45da99b7d4b117a3699cccf._comment new file mode 100644 index 000000000..a72a1456c --- /dev/null +++ b/doc/todo/parallel_possibilities/comment_3_145fb974f45da99b7d4b117a3699cccf._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.154.4.90" + subject="comment 3" + date="2013-07-17T19:59:50Z" + content=""" +Note that git-annex now uses locks to communicate amoung multiple processes, so it's now possible to eg run two `git annex get` processes, and one will skip over the file the other is downloading and go on to the next file, and so on. + +This is an especially nice speedup when downloading encrypted data, since the decryption of one file will tend to happen while the other process is downloading the next file (assuming files of approximately the same size, and that decryption takes approxiately as long as downloading). + +The only thing preventing this being done by threads in one process, enabled by a -jN option, is that the output would be a jumbled mess. +"""]] diff --git a/doc/todo/pushpull.mdwn b/doc/todo/pushpull.mdwn new file mode 100644 index 000000000..6828b35b2 --- /dev/null +++ b/doc/todo/pushpull.mdwn @@ -0,0 +1,4 @@ +--push/--pull should take a reponame and files, and push those files + to that repo; dropping them from the current repo + +[[done]] (move --from/--to) diff --git a/doc/todo/quvi_0.9_support.mdwn b/doc/todo/quvi_0.9_support.mdwn new file mode 100644 index 000000000..36280452a --- /dev/null +++ b/doc/todo/quvi_0.9_support.mdwn @@ -0,0 +1,8 @@ +quvi 0.9 has a completely different interface; git-annex needs to be made +to detect and use it. + +I have already fixed the worst problem, which caused git-annex addurl to +think that every url was a valid quvi url. --[[Joey]] + +> [[done]], 0.9 is supported, although the version is detected at build +> time. --[[Joey]] diff --git a/doc/todo/redundancy_stats_in_status.mdwn b/doc/todo/redundancy_stats_in_status.mdwn new file mode 100644 index 000000000..56095fd33 --- /dev/null +++ b/doc/todo/redundancy_stats_in_status.mdwn @@ -0,0 +1,23 @@ +Currently, `git annex status` only shows the size of 1 copy of each file. +If numcopies is being used for redundancy, much more disk can actually be +in use than status shows. + +One idea: + + known annex size: 2 terabytes (plus 4 terabytes of redundant copies) + +But, to get that number, it would have to walk every location log, +counting how many copies currently exist of each file. That would make +status a lot slower than it is. + +One option is to just put it at the end of the status: + + redundancy: 300% (4 terabytes of copies) + +And ctrl-c if it's taking too long. + +Hmm, fsck looks at that same info. Maybe it could cache the redundancy +level it discovers? Since fsck can be run incrementally, it would be tricky +to get an overall number. And the number would tend to be stale, but +then again it might also be nice if status shows how long ago the last fsck +was. diff --git a/doc/todo/redundancy_stats_in_status/comment_1_9f1c10f8cea4fa60a99cbcc8306dd5de._comment b/doc/todo/redundancy_stats_in_status/comment_1_9f1c10f8cea4fa60a99cbcc8306dd5de._comment new file mode 100644 index 000000000..801c1da03 --- /dev/null +++ b/doc/todo/redundancy_stats_in_status/comment_1_9f1c10f8cea4fa60a99cbcc8306dd5de._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U" + nickname="Richard" + subject="comment 1" + date="2012-10-30T08:09:13Z" + content=""" +I like the idea of using fsck as a pre-run for status. + +Basically, it's the same as `updatedb` and `locate`; locate will warn the user if it considers its cache to be too old, as well. +"""]] diff --git a/doc/todo/redundancy_stats_in_status/comment_2_686ced0684d10511caf07953c64cd5b6._comment b/doc/todo/redundancy_stats_in_status/comment_2_686ced0684d10511caf07953c64cd5b6._comment new file mode 100644 index 000000000..a4711b2a3 --- /dev/null +++ b/doc/todo/redundancy_stats_in_status/comment_2_686ced0684d10511caf07953c64cd5b6._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.152.108.220" + subject="comment 2" + date="2013-09-25T18:03:55Z" + content=""" +`git annex status .` or otherwise running it with a directory has recently started walking all the location logs for all files in the directory in order to display variance from configured numcopies. It would be easy to add a redundancy counter to that. + +It would slow down the global status when not passed a directory to add redundancy info there. Maybe local is enough? +"""]] diff --git a/doc/todo/resuming_encrypted_uploads.mdwn b/doc/todo/resuming_encrypted_uploads.mdwn new file mode 100644 index 000000000..b3aaa7f96 --- /dev/null +++ b/doc/todo/resuming_encrypted_uploads.mdwn @@ -0,0 +1,22 @@ +Resuming interrupted uploads to encrypted special remotes is not currently +possible, because gpg does not produce consistent output. Special remotes +that could support resuming include rsync and glacier. + +Without consistent output, git-annex would need to locally cache the encrypted +file, and reuse that cache when resuming an upload. This would make +encrypted uploads more expensive in terms of both file IO and disk space +used. + +[It would be possible to write to the cache at the same time the special +remote is being fed data, and if the special remote upload fails, continue +writing the rest of the file. That would avoid half the overhead, since +the file would not need to be read from, just written to. (Although OS +caching may accomplish the same thing.)] + +Also, `git annex unused` would need to show temp files for uploads, +the same as it currently shows temp files for downloads, and users would +sometimes need to manually dropunused old uploads, that never completed. + +The question, then, is whether resuming uploads is useful enough to add +this overhead and user-visible complexity. +--[[Joey]] diff --git a/doc/todo/resuming_encrypted_uploads/comment_1_1832a6fb78e8ad7c838582f46731ac3b._comment b/doc/todo/resuming_encrypted_uploads/comment_1_1832a6fb78e8ad7c838582f46731ac3b._comment new file mode 100644 index 000000000..cf35de049 --- /dev/null +++ b/doc/todo/resuming_encrypted_uploads/comment_1_1832a6fb78e8ad7c838582f46731ac3b._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://phil.0x539.de/" + nickname="Philipp Kern" + subject="comment 1" + date="2012-12-28T23:23:29Z" + content=""" +Doesn't the encryption part already write an encrypted version completely to disk before starting to upload? (At least with the rsync remote?) So the disk space and I/O usage is already there. (Except that it's probably a temporary file that's deleted after it was created, so that it's not kept around by the filesystem when certain destructive events strike.) +"""]] diff --git a/doc/todo/resuming_encrypted_uploads/comment_2_2ecc8e782f49e90ed1549e9179eb1a1e._comment b/doc/todo/resuming_encrypted_uploads/comment_2_2ecc8e782f49e90ed1549e9179eb1a1e._comment new file mode 100644 index 000000000..a2bab9244 --- /dev/null +++ b/doc/todo/resuming_encrypted_uploads/comment_2_2ecc8e782f49e90ed1549e9179eb1a1e._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawln3ckqKx0x_xDZMYwa9Q1bn4I06oWjkog" + nickname="Michael" + subject="comment 2" + date="2012-12-29T08:00:56Z" + content=""" +Being able to resume transfers of encrypted files would absolutely be useful! Disk space is cheap, but bandwidth is not. +"""]] diff --git a/doc/todo/rsync.mdwn b/doc/todo/rsync.mdwn new file mode 100644 index 000000000..3353f19c4 --- /dev/null +++ b/doc/todo/rsync.mdwn @@ -0,0 +1,4 @@ +Transferring a file from a ssh:// remote should use rsync to allow resuming +of a prior transfer. + +[[done]] diff --git a/doc/todo/smudge.mdwn b/doc/todo/smudge.mdwn new file mode 100644 index 000000000..6103ffa61 --- /dev/null +++ b/doc/todo/smudge.mdwn @@ -0,0 +1,162 @@ +git-annex should use smudge/clean filters. + +---- + +Update: Currently, this does not look likely to work. In particular, +the clean filter needs to consume all stdin from git, which consists of the +entire content of the file. It cannot optimise by directly accessing +the file in the repository, because git may be cleaning a different +version of the file during a merge. + +So every `git status` would need to read the entire content of all +available files, and checksum them, which is too expensive. + +> Update from GitTogether: Peff thinks a new interface could be added to +> git to handle this sort of case in an efficient way.. just needs someone +> to do the work. --[[Joey]] + +---- + +The clean filter is run when files are staged for commit. So a user could copy +any file into the annex, git add it, and git-annex's clean filter causes +the file's key to be staged, while its value is added to the annex. + +The smudge filter is run when files are checked out. Since git annex +repos have partial content, this would not git annex get the file content. +Instead, if the content is not currently available, it would need to do +something like return empty file content. (Sadly, it cannot create a +symlink, as git still wants to write the file afterwards.) + +So the nice current behavior of unavailable files being clearly missing due +to dangling symlinks, would be lost when using smudge/clean filters. +(Contact git developers to get an interface to do this?) + +Instead, we get the nice behavior of not having to remeber to `git annex +add` files, and just being able to use `git add` or `git commit -a`, +and have it use git-annex when .gitattributes says to. Also, annexed +files can be directly modified without having to `git annex unlock`. + +### design + +In .gitattributes, the user would put something like "* filter=git-annex". +This way they could control which files are annexed vs added normally. + +(git-annex could have further controls to allow eg, passing small files +through to regular processing. At least .gitattributes is a special case, +it should never be annexed...) + +For files not configured this way, git-annex could continue to use +its symlink method -- this would preserve backwards compatability, +and even allow mixing the two methods in a repo as desired. + +To find files in the repository that are annexed, git-annex would do +`ls-files` as now, but would check if found files have the appropriate +filter, rather than the current symlink checks. To determine the key +of a file, rather than reading its symlink, git-annex would need to +look up the git blob associated with the file -- this can be done +efficiently using the existing code in `Branch.catFile`. + +The clean filter would inject the file's content into the annex, and hard +link from the annex to the file. Avoiding duplication of data. + +The smudge filter can't do that, so to avoid duplication of data, it +might always create an empty file. To get the content, `git annex get` +could be used (which would hard link it). A `post-checkout` hook might +be used to set up hard links for all currently available content. + +#### clean + +The trick is doing it efficiently. Since git a2b665d, v1.7.4.1, +something like this works to provide a filename to the clean script: + + git config --global filter.huge.clean huge-clean %f + +This could avoid it needing to read all the current file content from stdin +when doing eg, a git status or git commit. Instead it is passed the +filename that git is operating on, in the working directory. +(Update: No, doesn't work; git may be cleaning a different file content +than is currently on disk, and git requires all stdin be consumed too.) + +So, WORM could just look at that file and easily tell if it is one +it already knows (same mtime and size). If so, it can short-circuit and +do nothing, file content is already cached. + +SHA1 has a harder job. Would not want to re-sha1 the file every time, +probably. So it'd need a local cache of file stat info, mapped to known +objects. + +But: Even with %f, git actually passes the full file content to the clean +filter, and if it fails to consume it all, it will crash (may only happen +if the file is larger than some chunk size; tried with 500 mb file and +saw a SIGPIPE.) This means unnecessary works needs to be done, +and it slows down *everything*, from `git status` to `git commit`. +**showstopper** I have sent a patch to the git mailing list to address +this. <http://marc.info/?l=git&m=131465033512157&w=2> (Update: apparently +can't be fixed.) + +#### smudge + +The smudge script can also be provided a filename with %f, but it +cannot directly write to the file or git gets unhappy. + +### dealing with partial content availability + +The smudge filter cannot be allowed to fail, that leaves the tree and +index in a weird state. So if a file's content is requested by calling +the smudge filter, the trick is to instead provide dummy content, +indicating it is not available (and perhaps saying to run "git-annex get"). + +Then, in the clean filter, it has to detect that it's cleaning a file +with that dummy content, and make sure to provide the same identifier as +it would if the file content was there. + +I've a demo implementation of this technique in the scripts below. + +---- + +### test files + +huge-smudge: + +<pre> +#!/bin/sh +read f +file="$1" +echo "smudging $f" >&2 +if [ -e ~/$f ]; then + cat ~/$f # possibly expensive copy here +else + echo "$f not available" +fi +</pre> + +huge-clean: + +<pre> +#!/bin/sh +file="$1" +cat >/tmp/file +# in real life, this should be done more efficiently, not trying to read +# the whole file content! +if grep -q 'not available' /tmp/file; then + awk '{print $1}' /tmp/file # provide what we would if the content were avail! + exit 0 +fi +echo "cleaning $file" >&2 +# XXX store file content here +echo $file +</pre> + +.gitattributes: + +<pre> +*.huge filter=huge +</pre> + +in .git/config: + +<pre> +[filter "huge"] + clean = huge-clean %f + smudge = huge-smudge %f +<pre> diff --git a/doc/todo/smudge/comment_1_4ea616bcdbc9e9a6fae9f2e2795c31c9._comment b/doc/todo/smudge/comment_1_4ea616bcdbc9e9a6fae9f2e2795c31c9._comment new file mode 100644 index 000000000..a4eb3cf23 --- /dev/null +++ b/doc/todo/smudge/comment_1_4ea616bcdbc9e9a6fae9f2e2795c31c9._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://christian.amsuess.com/chrysn" + nickname="chrysn" + subject="git-add instead of git-annex-add" + date="2011-02-26T21:43:21Z" + content=""" +would, with these modifications in place, there still be a way to *really* git-add a file? (my main repository contains both normal git and git-annex files.) +"""]] diff --git a/doc/todo/smudge/comment_2_e04b32caa0d2b4c577cdaf382a3ff7f6._comment b/doc/todo/smudge/comment_2_e04b32caa0d2b4c577cdaf382a3ff7f6._comment new file mode 100644 index 000000000..3a223e1c7 --- /dev/null +++ b/doc/todo/smudge/comment_2_e04b32caa0d2b4c577cdaf382a3ff7f6._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="http://dieter-be.myopenid.com/" + nickname="dieter" + subject="symlinks" + date="2011-04-03T20:30:21Z" + content=""" +> (Sadly, it cannot create a symlink, as git still wants to write the file afterwards. +> So the nice current behavior of unavailable files being clearly missing due to dangling symlinks, would be lost when using smudge/clean filters. (Contact git developers to get an interface to do this?) + +Have you checked what the smudge filter sees when the input is a symlink? Because git supports tracking symlinks, so it should also support pushing symlinks through a smudge filter, right? +Either way: yes, contact the git devs, one can only ask and hope. And if you can demonstrate the awesomeness of git-annex they might get more 1interested :) +"""]] diff --git a/doc/todo/smudge/comment_3_4e7c25fe24a1e71f58a8354fa64f41c2._comment b/doc/todo/smudge/comment_3_4e7c25fe24a1e71f58a8354fa64f41c2._comment new file mode 100644 index 000000000..cd64b7001 --- /dev/null +++ b/doc/todo/smudge/comment_3_4e7c25fe24a1e71f58a8354fa64f41c2._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawn1QhtPvsRBV7pfaDW_ZTPFv_ZIxSzQ8Rg" + nickname="Paul Léo" + subject="comment 3" + date="2013-11-13T20:41:52Z" + content=""" +> SHA1 has a harder job. Would not want to re-sha1 the file every time, probably. So it'd need a local cache of file stat info, mapped to known objects. + +I think that is not true? If gits wants the file to be cleaned, it thinks that the file was changed. So you would have to SHA1 it anyway if you don't want rely on WORM (which git already does in the first step anyway). +"""]] diff --git a/doc/todo/special_remote_for_amazon_glacier.mdwn b/doc/todo/special_remote_for_amazon_glacier.mdwn new file mode 100644 index 000000000..9b8b9d74e --- /dev/null +++ b/doc/todo/special_remote_for_amazon_glacier.mdwn @@ -0,0 +1,30 @@ +Amazon's new glacier service would be a nice special remote to support for +long-term archival. + +The main difficulty is that glacier is organized into vaults, and accessing +a file in a vault takes ~4 hours. A naive implementation would make `git +annex get` wait for 4 hours, which is certainly not reasonable. + +One approach I am pondering is to make each glacier vault a separate +special remote. You could then request git-annex to spin up a remote, and +come back later, and be able to access the data stored in it (need to check +if glacier would also allow adding new data to it then). This is +conceptually similar to using git-annex with offline removable drives, +except with glacier, you have a controllable robot to get them plugged in. :) + +Ideally, git-annex would arrange for glacier to send it a message when the +vault becomes available, and the user could queue a list of commands to +run, or files to transfer, at that point. + +--[[Joey]] + +> [[done]]! --[[Joey]] + +----- + +> In the coming months, Amazon S3 will introduce an option that will allow customers to seamlessly move data between Amazon S3 and Amazon Glacier based on data lifecycle policies. + +-- <http://aws.amazon.com/glacier/faqs/#How_should_I_choose_between_Amazon_Glacier_and_Amazon_S3> + +>> They did, but it's IMHO not very useful for git-annex. It's rather +>> intended to allow aging S3 storage out to Glacier. --[[Joey]] diff --git a/doc/todo/special_remote_for_amazon_glacier/comment_1_68f129441eefcbfebf7a9db680f52759._comment b/doc/todo/special_remote_for_amazon_glacier/comment_1_68f129441eefcbfebf7a9db680f52759._comment new file mode 100644 index 000000000..68593be42 --- /dev/null +++ b/doc/todo/special_remote_for_amazon_glacier/comment_1_68f129441eefcbfebf7a9db680f52759._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://mike.magin.org/" + nickname="mmagin" + subject="comment 1" + date="2012-09-14T04:19:53Z" + content=""" +When I first heard about Glacier, it sounded great for a cheap backup copy, and I was thinking about writing a \"hook\" remote, but once I read some better analysis of the pricing (e.g. [[http://www.daemonology.net/blog/2012-09-04-thoughts-on-glacier-pricing.html]]) I rapidly lost interest. +"""]] diff --git a/doc/todo/special_remote_for_amazon_glacier/comment_2_c5eeaf8ceee414fa0379831ca52e290c._comment b/doc/todo/special_remote_for_amazon_glacier/comment_2_c5eeaf8ceee414fa0379831ca52e290c._comment new file mode 100644 index 000000000..701047f91 --- /dev/null +++ b/doc/todo/special_remote_for_amazon_glacier/comment_2_c5eeaf8ceee414fa0379831ca52e290c._comment @@ -0,0 +1,7 @@ +[[!comment format=mdwn + username="basak" + subject="comment 2" + date="2012-09-21T22:21:04Z" + content=""" +I've created a glacier command line interface that integrates with git-annex [here](https://github.com/basak/glacier-cli), currently using the hook special remote mechanism. To get around the time delay, operations which require a job submission will submit the job and then fail. Retrying again four hours later should then succeed. It seems to work pretty well with git-annex. +"""]] diff --git a/doc/todo/speed_up_fsck.mdwn b/doc/todo/speed_up_fsck.mdwn new file mode 100644 index 000000000..5d5e867f8 --- /dev/null +++ b/doc/todo/speed_up_fsck.mdwn @@ -0,0 +1,40 @@ +moving to the git-annex branch has slowed down fsck worse than most +commands. Actually, some commands have sped up, while others like get +are slightly slower but are swamped by the normal runtime. + +For fsck though, it has to pull each file's location log info out of git. +And, it's typically run on the entire tree. + +Another slow one in `git annex copy --from`. + +It would be possible to run a single `git cat-file --batch` and pass it +sha1s of location logs for file that is going to be fsked (gotten via +`read-tree`). Then just read its output until the next requested sha1 to +chunk it, and pass this in to fsck in a closure. + +The difficulty, besides writing that is that everything that works with +location logs now reads them out of git, would need to find a way to +provide the info on a side channel of some sort. + +If this is implemented, the same infrastructure could be used for other +commands like whereis and add. --[[Joey]] + +> Updated plan: +> +> Run `git ls-file --batch`, and cache its stdin and out handles in Branch +> state. +> +> To see a git-annex branch file, send it something like +> "git-annex:uuid.log", and read the content fron stdout handle. +> +> To detect the end of content, send "TOKEN\n", and look for +> "TOKEN missing" in its output. A good choice for TOKEN is anything +> that will never exist in the repo; 40 0's would be a fairly good choice, +> but even better seems to be something completely invalid and impossible +> to have as a sha1 or filename or ref: "". +> +> Hmm, except that's actually an error message sent to stderr. Unless +> stderr is connected to stdout, it might be better to look for a known, +> empty object. Could just add a git-annex:empty file to that end. + +[[done]] --[[Joey]] diff --git a/doc/todo/stream_feature__63__.mdwn b/doc/todo/stream_feature__63__.mdwn new file mode 100644 index 000000000..860edfc81 --- /dev/null +++ b/doc/todo/stream_feature__63__.mdwn @@ -0,0 +1,23 @@ +I am just asking myself, is it stupid to think that streaming in git annex would be a good idea and wouldnt it be totaly easy to implement? + +Ok just tried to link to files over ssh, it creates a link but you cant open with it its content ^^ + +But at least the files you have access over some filesystem as example samba/sshfs or just a other directory or usb-drive you could stream instead of "get" + +you could add another mode like direct and indirect, like symbolic-links or something like that? + +Sadly linux is to stupid to allow direct ssh links ( thats maybe one of the biggest features hurd has over linux ) but you could mount with sshfs readonly (checking first if sshfs is installed) to mount the files there and then map the links there. + +ok I am not so shure how hard it would be and how much bug potentials it creates, but it would be great I guess. + +git annex is a bit like a telephone book, where you get a list of where the targets are. So using it to call the persons so that they drive to you to talk with you is nice. But I think the better feature would be if you just talk with the guy over the telephone directly bevore he comes to you (streaming) + +I mean you did one great thing, you did make cloudy thing peer to peer, like git is targeted too but for smaller files, yes there are may use cases without this feature, but I would be really glad if it could do that too, if I give annex 5 locations on other pcs usb-sticks etc, I find it stupid to additionaly do setup all this sources again a second time for streaming, and then I have maybe even 2 different file names because you map stuff in git. + +So sorry its late here, I am a bit tired so I maybe dont know what I am talking right now, my english isnt the best, too, but I think it would be a great feature. + +I mean on your setup, with slow internet, you maybe always make a get command, but here, if I link to youtube, I have no problem to stream it, or even on internal network between my pcs I have gb-lan, I start directly movies streaming, I would only use get, in rare cases where I need them on a train, the normal thing is to stream stuff. + +So I have to go sleep now + +bye diff --git a/doc/todo/support-non-utf8-locales.mdwn b/doc/todo/support-non-utf8-locales.mdwn new file mode 100644 index 000000000..da40118d5 --- /dev/null +++ b/doc/todo/support-non-utf8-locales.mdwn @@ -0,0 +1,26 @@ +Currenty, git-annex forces output, particularly of filenames, in a utf-8 +locale. + +Note that this does not mean it cannot be used with filenames in other +encodings. git-annex is entirely encoding agnostic when it comes to +manipulating filenames. It just *displays* their names always converted to +utf-8, which may not look right when you have a non-utf8 locale. + +This had to be done to work around some bugs with haskell's handling +of filename encodings. In particular, + +* [[bugs/unhappy_without_UTF8_locale]]: haskell crashes when told to output + a string with characters > 255 in a non-utf8 locale. +* [[bugs/problems_with_utf8_names]]: On many OSs, haskell expects + non-decoded raw char8 in FilePaths. In order to display a filename, + though, it needs to first be decoded, and git-annex currently assumes + it was encoded as utf8. + +git-annex's behavior is unlikely to improve much until haskell's +support for utf8 filenames improves. --[[Joey]] + +> [[done]] -- I just turned off all encoding handling on stdout and stderr, +> which avoids these problems nicely. Git-annex now displays just what it +> input, at least on platforms where haskell does not decode unicode in +> FilePaths. This will later be a problem when it gets localized, but for +> now works great. --[[Joey]] diff --git a/doc/todo/support_S3_multipart_uploads.mdwn b/doc/todo/support_S3_multipart_uploads.mdwn new file mode 100644 index 000000000..711ac41b2 --- /dev/null +++ b/doc/todo/support_S3_multipart_uploads.mdwn @@ -0,0 +1,14 @@ +Did not know of this when I wrote S3 support. Ability to resume large +uploads would be good. + +<http://aws.typepad.com/aws/2010/11/amazon-s3-multipart-upload.html> + +Also allows supporting files > 5 gb, a S3 limit I was not aware of. + +NB: It would work just as well to split the object and upload the N parts +to S3, but not bother with S3's paperwork to rejoin them into one object. +Only reasons not to do that are a) backwards compatability with +the existing S3 remote and b) this would not allow accessing the content +in S3 w/o using git-annex, which could be useful in some scenarios. + +--[[Joey]] diff --git a/doc/todo/support_for_lossy_remotes.mdwn b/doc/todo/support_for_lossy_remotes.mdwn new file mode 100644 index 000000000..23083b2d7 --- /dev/null +++ b/doc/todo/support_for_lossy_remotes.mdwn @@ -0,0 +1,11 @@ +I'm curious if there's a possibility to support lossy remotes. It may be handy to support syncing to special remotes that do lossy compression on the files (e.g., videos and images). For example, one could imagine having a YouTube special remote that only syncs video files. The original files wouldn't be available for download due to the transcoding and compression that YouTube does, so they wouldn't count towards the number of copies. In this YouTube example, the user gains: + +1. an online place that their videos are available from +2. a worst-case scenario "backup" +3. a remote that they could download smaller video files + +> [[done]]; lossy remotes are supported as seen with `git annex addurl +> --fast` and also with the new addurl support for using quvi to get +> videos fro youtube. Just make a key with a URL or something in it, and +> no size or checksum, and any content will be assumed to be the right +> content. --[[Joey]] diff --git a/doc/todo/support_for_lossy_remotes/comment_1_f5cd9f9deab13ab2d2290ad763906dd3._comment b/doc/todo/support_for_lossy_remotes/comment_1_f5cd9f9deab13ab2d2290ad763906dd3._comment new file mode 100644 index 000000000..1e895944c --- /dev/null +++ b/doc/todo/support_for_lossy_remotes/comment_1_f5cd9f9deab13ab2d2290ad763906dd3._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.154.4.90" + subject="comment 1" + date="2013-07-16T19:16:54Z" + content=""" +There is already one example of a lossy remote: If you use `git annex addurl --relaxed` it generates a key that just uses the url, without its size. When retreiving such a key, any content will be accepted. +"""]] diff --git a/doc/todo/support_for_writing_external_special_remotes.mdwn b/doc/todo/support_for_writing_external_special_remotes.mdwn new file mode 100644 index 000000000..2d7cd9d15 --- /dev/null +++ b/doc/todo/support_for_writing_external_special_remotes.mdwn @@ -0,0 +1,25 @@ +It would be good to have something in between the hook special remote and +the built-in special remotes. The hook is easy to set up, but its simple +interface misses some features that a more full-features interface could +provide to a third-party program that wants to provide the best possible +special remote it can w/o being written in haskell: + +* No way to send progress updates when uploading, so no progress bars for uploads from hook special remotes in the webapp. +* No way to verify the `initremote` parameters include all needed configuration, and do any initalization needed. +* No way to query and/or set the remote.log while it's running. (Well, technically, `git annex enableremote` can set values..) +* No way to store per-key information to the git-annex branch. +* No (easy) way to split files into chunks. + +Some of these features could be added to git-annex as subcommands. Which would +improve the general programmability and flexability of git-annex. OTOH, +running `git-annex upload-progress` repeatedly is pretty ugly. Or the +interface could provide a channel for the program and git-annex to +communicate back and forth on. Maybe a mix of the two? + +A final feature such an interface should provide is the ability to drop a +program into PATH and have it just work, without the user needing to do any +configuration beyond `initremote`. So, `git annex initremote foo type=$bar` +should look for `git-annex-remote-$bar` in PATH if that's not a built-in +special remote name. + +--[[Joey]] diff --git a/doc/todo/support_fsck_in_bare_repos.mdwn b/doc/todo/support_fsck_in_bare_repos.mdwn new file mode 100644 index 000000000..32ced467e --- /dev/null +++ b/doc/todo/support_fsck_in_bare_repos.mdwn @@ -0,0 +1,17 @@ +What is says on the tin: + + 22:56:54 < RichiH> joeyh_: by the way, i have been thinking about fsck on bare repos + 22:57:37 < RichiH> joeyh_: the best i could come with is to have a bare and a non-bare access the same repo store + 22:58:00 < RichiH> joeyh_: alternatively, with the SHA* backend, you have all the information to verify that the local data is correct + 22:58:41 < RichiH> and verifying that would already be a plus. if there really _is_ a problem, having the SHA is enough to track issues down + 23:09:50 < joeyh_> oh, I think I have code that fsck could use on bare repos already.. just a matter of wiring it up + 23:10:42 < joeyh_> feel free to reopen a bug or whatever so I remember.. the unused command's branch content enumeration could be used in a bare repo + 23:14:51 < joeyh_> unused/dropunused could work in bare repos too btw + +> Also `status`'s total annex keys/size could be handled for bare repos. --[[Joey]] + +>> Fsck is done. Rest not done yet. --[[Joey]] + +>>> all [[done]]! --[[Joey]] + +[[!meta title="support unused, dropunused in bare repos"]] diff --git a/doc/todo/symlink_farming_commit_hook.mdwn b/doc/todo/symlink_farming_commit_hook.mdwn new file mode 100644 index 000000000..3e93cb34b --- /dev/null +++ b/doc/todo/symlink_farming_commit_hook.mdwn @@ -0,0 +1,14 @@ +TODO: implement below + +git-annex does use a lot of symlinks. Specicially, relative symlinks, +that are checked into git. To allow you to move those around without +annoyance, git-annex can run as a post-commit hook. This way, you can `git mv` +a symlink to an annexed file, and as soon as you commit, it will be fixed +up. + +`git annex init` tries to set up a post-commit hook that is itself a symlink +back to git-annex. If you want to have your own shell script in the post-commit +hook, just make it call `git annex` with no parameters. git-annex will detect +when it's run from a git hook and do the necessary fixups. + +[[done]] diff --git a/doc/todo/sync_my_local_git-annex_from_a_dump_remote.mdwn b/doc/todo/sync_my_local_git-annex_from_a_dump_remote.mdwn new file mode 100644 index 000000000..524782bc7 --- /dev/null +++ b/doc/todo/sync_my_local_git-annex_from_a_dump_remote.mdwn @@ -0,0 +1,6 @@ +As discussed on debconf, I have the following use case: + +* I have a dump remote, a folder on my webserver where files are uploaded through the web app. I don't have git on the webserver, just a plain folder. +* I have git-annex repo on a development server. The development server polls the webserver (ssh/ftp) once in an hour and synchronizes the state of the local git-annex repo with the state found on the webserver and commits that. +* This is not meant to be backup facility. I just want to be able to have a state on my development machine that is very likely to the state on the webserver. + diff --git a/doc/todo/sync_my_local_git-annex_from_a_dump_remote/comment_1_81d63854f89f00855cda5ace0fc8262a._comment b/doc/todo/sync_my_local_git-annex_from_a_dump_remote/comment_1_81d63854f89f00855cda5ace0fc8262a._comment new file mode 100644 index 000000000..d9abb3a3c --- /dev/null +++ b/doc/todo/sync_my_local_git-annex_from_a_dump_remote/comment_1_81d63854f89f00855cda5ace0fc8262a._comment @@ -0,0 +1,14 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="2001:4978:f:21a::2" + subject="comment 1" + date="2013-08-13T21:44:17Z" + content=""" +We had a conversation about this IRL. At the time, I thought I understood what you wanted. Reading the above, I am not so sure. + +What I thought you wanted was something like `git annex mirror --from remote`, which would, for each object known to git-annex that the location log said was present on the remote, make sure that the local repo had the object too, and for each object that the location log said was not present on the remote, drop it from the local repo (if numcopies etc allowed). + +`git annex mirror --to remote` could also be used as the complement of the above. + +If that's not the sort of thing you meant, let me know. +"""]] diff --git a/doc/todo/sync_my_local_git-annex_from_a_dump_remote/comment_2_66822b72b1450e79e8edd0c6c21d5aa6._comment b/doc/todo/sync_my_local_git-annex_from_a_dump_remote/comment_2_66822b72b1450e79e8edd0c6c21d5aa6._comment new file mode 100644 index 000000000..3d459371f --- /dev/null +++ b/doc/todo/sync_my_local_git-annex_from_a_dump_remote/comment_2_66822b72b1450e79e8edd0c6c21d5aa6._comment @@ -0,0 +1,14 @@ +[[!comment format=mdwn + username="http://thkoch2001.myopenid.com/" + nickname="thkoch" + subject="pseudocode" + date="2013-08-14T04:58:22Z" + content=""" +lets say my local annex is in direct mode, then the following might already do what I want: + +cd $LOCAL_ANNEX +rsync --recursive --delete $REMOTE . +git annex add && git commit + +It would however be nice if I could do the same with an annex in indirect mode or even a bare annex. +"""]] diff --git a/doc/todo/sync_my_local_git-annex_from_a_dump_remote/comment_3_b9f73375e2c732e798495f8ee6970c7c._comment b/doc/todo/sync_my_local_git-annex_from_a_dump_remote/comment_3_b9f73375e2c732e798495f8ee6970c7c._comment new file mode 100644 index 000000000..df4be033b --- /dev/null +++ b/doc/todo/sync_my_local_git-annex_from_a_dump_remote/comment_3_b9f73375e2c732e798495f8ee6970c7c._comment @@ -0,0 +1,14 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.154.0.63" + subject="comment 3" + date="2013-08-24T16:35:33Z" + content=""" +Seems to me that this could easily be dealt with by installing git-annex on the webserver, making the directory there a git repository, and using either a cron job or `git annex watch` to commit files as they were changed there. + +Then you can make a direct mode, indirect mode, or even a bare clone on your local machine and use git-annex to get the files. + +Maybe you have good reasons for not wanting to go that route. And rsync on a direct mode repository should work, provided to tell it to not delete `.git`. :P I don't see any way to make rsync work in an indirect mode repository. As for trying to make git-annex handle this import over rsync itself in a way that would work in an indirect mode repository, let alone a bare repository -- I don't see a good way to do it and it seems quite special case and likely to get quite complicated to implement. + +In the meantime, I did implement `git annex mirror`, which I think is a much more interesting and generally useful tool to have. And could even be used in my recommended solution above. +"""]] diff --git a/doc/todo/tahoe_lfs_for_reals.mdwn b/doc/todo/tahoe_lfs_for_reals.mdwn new file mode 100644 index 000000000..9019767eb --- /dev/null +++ b/doc/todo/tahoe_lfs_for_reals.mdwn @@ -0,0 +1,21 @@ +[[forum/tips:_special__95__remotes__47__hook_with_tahoe-lafs]] is a good +start, but Zooko points out that using Tahoe's directory translation layer +incurs O(N^2) overhead as the number of objects grows. Also, making +hash subdirectories in Tahoe is expensive. Instead it would be good to use +it as a key/value store directly. The catch is that doing so involves +sending the content to Tahoe, and getting back a key identifier. + +This would be fairly easy to do as a [[backend|backends]], which can assign its +own key names (although typically done before data is stored in it), +but a tahoe-lafs special remote would be more flexible. + +To support a special remote, a mapping is needed from git-annex keys to +Tahoe keys. + +The best place to store this mapping is perhaps as a new field in the +location log: + + date present repo-uuid newfields + +This way, each remote can store its own key-specfic data in the same place +as other key-specific data, with minimal overhead. diff --git a/doc/todo/tahoe_lfs_for_reals/comment_1_0a4793ce6a867638f6e510e71dd4bb44._comment b/doc/todo/tahoe_lfs_for_reals/comment_1_0a4793ce6a867638f6e510e71dd4bb44._comment new file mode 100644 index 000000000..16ef882a4 --- /dev/null +++ b/doc/todo/tahoe_lfs_for_reals/comment_1_0a4793ce6a867638f6e510e71dd4bb44._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="zooko" + ip="97.118.97.117" + subject="performance" + date="2011-05-17T19:20:39Z" + content=""" +Hm... O(N^2)? I think it just takes O(N). To read an entry out of a directory you have to download the entire directory (and store it in RAM and parse it). The constants are basically \"too big to be good but not big enough to be prohibitive\", I think. jctang has reported that his special remote hook performs well enough to use, but it would be nice if it were faster. + +The Tahoe-LAFS folks are working on speeding up mutable files, by the way, after which we would be able to speed up directories. +"""]] diff --git a/doc/todo/tahoe_lfs_for_reals/comment_2_80b9e848edfdc7be21baab7d0cef0e3a._comment b/doc/todo/tahoe_lfs_for_reals/comment_2_80b9e848edfdc7be21baab7d0cef0e3a._comment new file mode 100644 index 000000000..6dba86c47 --- /dev/null +++ b/doc/todo/tahoe_lfs_for_reals/comment_2_80b9e848edfdc7be21baab7d0cef0e3a._comment @@ -0,0 +1,13 @@ +[[!comment format=mdwn + username="http://joey.kitenet.net/" + nickname="joey" + subject="comment 2" + date="2011-05-17T19:57:33Z" + content=""" +Whoops! You'd only told me O(N) twice before.. + +So this is not too high priority. I think I would like to get the per-remote storage sorted out anyway, since probably it will be the thing needed to convert the URL backend into a special remote, which would then allow ripping out the otherwise unused pluggable backend infrastructure. + +Update: Per-remote storage is now sorted out, so this could be implemented +if it actually made sense to do so. +"""]] diff --git a/doc/todo/union_mounting.mdwn b/doc/todo/union_mounting.mdwn new file mode 100644 index 000000000..c42a05502 --- /dev/null +++ b/doc/todo/union_mounting.mdwn @@ -0,0 +1,10 @@ +It should be possible to union mount annexes. So if multiple drives have +content, an annex mounting them both would have available all the +content from all the drives. + +This could be done by just making .git/annex/KEY link to the actual content +on the mounted annex. + +(Need to make sure the [[copy_tracking|copies]] code does not +confused and think the symlink is a copy of the content.. Also need to make +sure that code that writes to .git/annex does not follow symlinks.)) diff --git a/doc/todo/union_mounting/comment_1_cb08435812dd7766de26199c73f38e8b._comment b/doc/todo/union_mounting/comment_1_cb08435812dd7766de26199c73f38e8b._comment new file mode 100644 index 000000000..3fadf6fa3 --- /dev/null +++ b/doc/todo/union_mounting/comment_1_cb08435812dd7766de26199c73f38e8b._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawln3ckqKx0x_xDZMYwa9Q1bn4I06oWjkog" + nickname="Michael" + subject="comment 1" + date="2013-03-01T01:26:36Z" + content=""" +This would indeed be very helpful when remotely mounting a photo/video collection over samba. +"""]] diff --git a/doc/todo/union_mounting/comment_2_240b1736f6bd4fbf87c372d3a46e661b._comment b/doc/todo/union_mounting/comment_2_240b1736f6bd4fbf87c372d3a46e661b._comment new file mode 100644 index 000000000..08901ee17 --- /dev/null +++ b/doc/todo/union_mounting/comment_2_240b1736f6bd4fbf87c372d3a46e661b._comment @@ -0,0 +1,9 @@ +[[!comment format=mdwn + username="http://edheil.wordpress.com/" + ip="173.162.44.162" + subject="comment 2" + date="2013-03-01T04:50:28Z" + content=""" ++1 this would be sweet as hell + +"""]] diff --git a/doc/todo/untracked_remotes.mdwn b/doc/todo/untracked_remotes.mdwn new file mode 100644 index 000000000..883b5acff --- /dev/null +++ b/doc/todo/untracked_remotes.mdwn @@ -0,0 +1,9 @@ +Seems that a fairly common desire in some use cases is to be able to make a +clone of a repository and be able to get files, without updating the +location tracking information. (And without even recording a uuid in the +remote.log.) Use cases include wanting to have temporary +clones without cluttering history, and centralized development where the +developers don't care to know about one-another's systems. + +It seems that such an untracked repository would need to automatically +consider itself untrusted. Is that enough to avoid losing data? diff --git a/doc/todo/use_cp_reflink.mdwn b/doc/todo/use_cp_reflink.mdwn new file mode 100644 index 000000000..39518abf1 --- /dev/null +++ b/doc/todo/use_cp_reflink.mdwn @@ -0,0 +1,7 @@ +The unlock command needs to copy a file, and it would be great to use this: + cp --reflink=auto src dst + +O(1) overhead on BTRFS. Needs coreutils 7.6; and remember that git-annex +may be used on systems without coreutils.. + +[[done]] diff --git a/doc/todo/using_url_backend.mdwn b/doc/todo/using_url_backend.mdwn new file mode 100644 index 000000000..1f3cd5628 --- /dev/null +++ b/doc/todo/using_url_backend.mdwn @@ -0,0 +1,11 @@ +There is no way to `git annex add` a file using the URL [[backend|backends]]. + +For now, we have to manually make the symlink. Something like this: + + ln -s .git/annex/URL:http:%%www.example.com%foo.tar.gz + +Note the escaping of slashes. + +A `git annex register <url>` command could do this.. + +[[done]] diff --git a/doc/todo/whishlist:_git_annex_drop_--dry-run.mdwn b/doc/todo/whishlist:_git_annex_drop_--dry-run.mdwn new file mode 100644 index 000000000..6bbfd7a4d --- /dev/null +++ b/doc/todo/whishlist:_git_annex_drop_--dry-run.mdwn @@ -0,0 +1,5 @@ +It'd be useful to be able to see what `git annex drop` would do *before* asking it to drop any files. + +For example, I just set up my preferred contents expressions, and I don't know if I got them right. Before dropping anything from this repo, it'd be nice to check what would happen. I know git annex drop will only drop files that are above their minimum numcopies, but I'd still like to avoid heavyweight copying in case I got my preferred contents expressions wrong. + +> [[done]]; added --want-get and --want-drop. --[[Joey]] diff --git a/doc/todo/whishlist:_git_annex_drop_--dry-run/comment_1_20ecfa8ffa52c238d21b904076ac69a2._comment b/doc/todo/whishlist:_git_annex_drop_--dry-run/comment_1_20ecfa8ffa52c238d21b904076ac69a2._comment new file mode 100644 index 000000000..098d399e3 --- /dev/null +++ b/doc/todo/whishlist:_git_annex_drop_--dry-run/comment_1_20ecfa8ffa52c238d21b904076ac69a2._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="209.250.56.47" + subject="comment 1" + date="2013-10-28T17:11:04Z" + content=""" +It would be nice, but it adds quite a lot of complexity to have a --dry-run, and if I add it to just drop, the next bug is going to ask for get to have it.. + +I feel that the right approach is to add a --wanted, which could then be used with find to find files that are and are not wanted, according to the preferred content settings. To see what it would want to get: `git annex find --wanted --not --in .` To see what it would want to drop: `git annex find --not --wanted --in .` +"""]] diff --git a/doc/todo/whishlist:_git_annex_drop_--dry-run/comment_2_d19bc268c9467d24baa8d8f77a6e5e09._comment b/doc/todo/whishlist:_git_annex_drop_--dry-run/comment_2_d19bc268c9467d24baa8d8f77a6e5e09._comment new file mode 100644 index 000000000..3f1102985 --- /dev/null +++ b/doc/todo/whishlist:_git_annex_drop_--dry-run/comment_2_d19bc268c9467d24baa8d8f77a6e5e09._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawnWwEEA3CurHkBjIYaJsJzFc4jtY2SCkrQ" + nickname="Diego" + subject="comment 2" + date="2013-10-29T00:28:51Z" + content=""" +That makes sense, and thanks for adding this feature so quickly! +"""]] diff --git a/doc/todo/windows_support.mdwn b/doc/todo/windows_support.mdwn new file mode 100644 index 000000000..a8cbd6db8 --- /dev/null +++ b/doc/todo/windows_support.mdwn @@ -0,0 +1,23 @@ +The git-annex Windows port is not ready for prime time. But it does exist +now! --[[Joey]] + +## status + +* Does not work with Cygwin's build of git (that git does not consistently + support use of DOS style paths, which git-annex uses on Windows). + Must use the upstream build of git for Windows. +* rsync special remotes are known buggy. +* Bad file locking, it's probably not safe to run more than one git-annex + process at the same time on Windows. +* Ssh connection caching does not work on Windows, so `git annex get` + has to connect twice to the remote system over ssh per file, which + is much slower than on systems supporting connection caching. +* Webapp doesn't build yet. +* `git annex watch` builds, but does not quite work. +* `git annex assistant` builds, but has not been tested, and is known + to not download any files. (transferrer doesn't built yet) +* watch and assistant cannot be built with cabal. Possibly due to too many + files overflowing command line length limit at link stage. + To build a binary with them: + `ghc --make git-annex.hs -threaded -XPackageImports -DWITH_ASSISTANT=1 -DWITH_WIN32NOTIFY=1` + (should add all the other flags cabal uses) diff --git a/doc/todo/windows_support/comment_10_394127e34e07ab3dc0e7b94ee6898866._comment b/doc/todo/windows_support/comment_10_394127e34e07ab3dc0e7b94ee6898866._comment new file mode 100644 index 000000000..fb061962e --- /dev/null +++ b/doc/todo/windows_support/comment_10_394127e34e07ab3dc0e7b94ee6898866._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.152.108.145" + subject="comment 10" + date="2013-08-04T18:23:00Z" + content=""" +Encryption is now working on Windows. +"""]] diff --git a/doc/todo/windows_support/comment_1_3cc26ad8101a22e95a8c60cf0c4dedcc._comment b/doc/todo/windows_support/comment_1_3cc26ad8101a22e95a8c60cf0c4dedcc._comment new file mode 100644 index 000000000..fd5b6f5cd --- /dev/null +++ b/doc/todo/windows_support/comment_1_3cc26ad8101a22e95a8c60cf0c4dedcc._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawkRITTYYsN0TFKN7G5sZ6BWGZOTQ88Pz4s" + nickname="Zoltán" + subject="cygwin" + date="2012-05-15T00:14:08Z" + content=""" +What about [Cygwin](http://cygwin.com/)? It emulates POSIX fairly well under Windows (including signals, forking, fs (also things like /dev/null, /proc), unix file permissions), has all standard gnu utilities. It also emulates symlinks, but they are unfortunately incompatible with NTFS symlinks introduced in Vista [due to some stupid restrictions on Windows](http://cygwin.com/ml/cygwin/2009-10/msg00756.html). + +If git-annex could be modified to not require symlinks to work, the it would be a pretty neat solution (and you get a real shell, not some command.com on drugs (aka cmd.exe)) +"""]] diff --git a/doc/todo/windows_support/comment_2_8acae818ce468967499050bbe3c532ea._comment b/doc/todo/windows_support/comment_2_8acae818ce468967499050bbe3c532ea._comment new file mode 100644 index 000000000..e37a55575 --- /dev/null +++ b/doc/todo/windows_support/comment_2_8acae818ce468967499050bbe3c532ea._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawk5cj-itfFHq_yhJHdzk3QOPp-PNW_MjPU" + nickname="Michael" + subject="+1 Cygwin" + date="2012-05-23T19:30:21Z" + content=""" +Windows support is a must. In my experience, binary file means proprietary editor, which means Windows. + +Unfortunately, there's not much overlap between people who use graphical editors in Windows all day vs. people who are willing to tolerate Cygwin's setup.exe, compile a Haskell program, learn git and git-annex's 90-odd subcommands, and use a mintty terminal to manage their repository, especially now that there's a sexy GitHub app for Windows. + +That aside, I think Windows-based content producers are still *the* audience for git-annex. First Windows support, then a GUI, then the world. +"""]] diff --git a/doc/todo/windows_support/comment_3_bd0a12f4c9b884ab8a06082842381a01._comment b/doc/todo/windows_support/comment_3_bd0a12f4c9b884ab8a06082842381a01._comment new file mode 100644 index 000000000..0b48db750 --- /dev/null +++ b/doc/todo/windows_support/comment_3_bd0a12f4c9b884ab8a06082842381a01._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://xolus.net/openid/max" + nickname="B0FH" + subject="What about NTFS support ?" + date="2012-08-02T17:45:10Z" + content=""" +Has git-annex been tested with an NTFS-formatted disk under Linux ? NTFS is supposed to be case-sensitive and to allow symlinks, and these are supposed to work with ntfs3g. +"""]] diff --git a/doc/todo/windows_support/comment_4_ad06b98b2ddac866ffee334e41fee6a8._comment b/doc/todo/windows_support/comment_4_ad06b98b2ddac866ffee334e41fee6a8._comment new file mode 100644 index 000000000..66f9ca71f --- /dev/null +++ b/doc/todo/windows_support/comment_4_ad06b98b2ddac866ffee334e41fee6a8._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawlc1og3PqIGudOMkFNrCCNg66vB7s-jLpc" + nickname="Paul" + subject="Re: What about NTFS support?" + date="2012-08-16T19:30:38Z" + content=""" +I successfully use git-annex on an NTFS formatted external USB drive, so yes, it is possible and works well. +"""]] diff --git a/doc/todo/windows_support/comment_5_444fc7251f57db241b6e80abae41851c._comment b/doc/todo/windows_support/comment_5_444fc7251f57db241b6e80abae41851c._comment new file mode 100644 index 000000000..8f76ee258 --- /dev/null +++ b/doc/todo/windows_support/comment_5_444fc7251f57db241b6e80abae41851c._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="https://me.yahoo.com/a/dASECLNzvckz4VwqUGYsvthiplY.#d2c27" + nickname="A. D. Sicks" + subject="comment 5" + date="2012-09-09T23:48:21Z" + content=""" +Haskell has C++ binding, so it should be possible to port it to .net/Mono with a VB GUI for Windows users. Windows has a primitive form of symlinks called shortcuts. Perhaps shortcut support in Windows could replace the use of symlinks. I've used shortcuts since XP to put my home Windows directory on another partition and never had a hitch... + +If anyone is interested in working on this, hit me up. I would like to use this in my XP vbox to have access to files on my host OS...I have a student edition of Visual Studio 2005 to do an open source port... +"""]] diff --git a/doc/todo/windows_support/comment_6_34f1f60b570c389bb1e741b990064a7e._comment b/doc/todo/windows_support/comment_6_34f1f60b570c389bb1e741b990064a7e._comment new file mode 100644 index 000000000..bf9f86f41 --- /dev/null +++ b/doc/todo/windows_support/comment_6_34f1f60b570c389bb1e741b990064a7e._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawnlwEMhiNYv__mEUABW4scn83yMraC3hqE" + nickname="Sean" + subject="NTFS symlinks" + date="2013-01-11T21:44:21Z" + content=""" +It seems that NTFS (from Vista forward) has full POSIX support for symlinks. At least, Wikipedia [seems to think so.](http://en.wikipedia.org/wiki/NTFS_symbolic_link). What about doing like GitHub and using MinGW for compatibility? Cygwin absolutely blows in terms of installation size and compatability with the rest of Windows. +"""]] diff --git a/doc/todo/windows_support/comment_7_a5ca56c487257434650420acfa60e39f._comment b/doc/todo/windows_support/comment_7_a5ca56c487257434650420acfa60e39f._comment new file mode 100644 index 000000000..0ee09ac5a --- /dev/null +++ b/doc/todo/windows_support/comment_7_a5ca56c487257434650420acfa60e39f._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawlpSOjMH7Iaz56v6Pr9KCFSpbvMXvg-y9o" + nickname="Dominik" + subject="So close :-)" + date="2013-06-30T12:46:40Z" + content=""" +I was fighting my way forward until I read here that special remote with ssh+rsync and encryption doesn't work. Interestingly I got everything working so far, ssh login is keybased, gpg -k works and the remote setup also correctly cooperated with gpg... but it just didn't sync. Any ideas how complex it is to get this last missing piece moving? +"""]] diff --git a/doc/todo/windows_support/comment_8_61214de7d967740d42905f3823ce2f65._comment b/doc/todo/windows_support/comment_8_61214de7d967740d42905f3823ce2f65._comment new file mode 100644 index 000000000..fe193f7e0 --- /dev/null +++ b/doc/todo/windows_support/comment_8_61214de7d967740d42905f3823ce2f65._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.154.4.193" + subject="comment 8" + date="2013-06-30T17:58:08Z" + content=""" +It should be easy to fix whatever's wrong the the rsync special remote. Just a matter of debugging that. + +Adding encryption support on Windows is stuck at a roadblock I don't know the way around. To drive gpg, git-annex uses the `--passphrase-fd` option, and sends the \"passphrase\" (really a big block of binary foo!) over a file descriptor of a pipe that it set up. + +Windows, AFAIK, doesn't have file descriptors, or at least there is no equivilant to them that I have access to in the Haskell POSIX compatability layer for Windows. I am reluctant to fall back to using `--passphrase-file` on Windows, since that would be a massive security hole (as would passing the passphrase as a parameter via `--passphrase=`). +"""]] diff --git a/doc/todo/windows_support/comment_9_259a0b1a6f4d8d1944173380adc5e7c8._comment b/doc/todo/windows_support/comment_9_259a0b1a6f4d8d1944173380adc5e7c8._comment new file mode 100644 index 000000000..9ae337886 --- /dev/null +++ b/doc/todo/windows_support/comment_9_259a0b1a6f4d8d1944173380adc5e7c8._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawlpSOjMH7Iaz56v6Pr9KCFSpbvMXvg-y9o" + nickname="Dominik" + subject="comment 9" + date="2013-07-31T10:29:51Z" + content=""" +The tradeoff for me is a \"local\" security hole (where I can secure my own laptop) vs. a remotely exploitable thing... If it needs to go through a file, so be it -- it would however be good if that file would be overwritten with garbage before being deleted :-) +"""]] diff --git a/doc/todo/wishlist:_Add_to_Android_version_to_Google_Play.mdwn b/doc/todo/wishlist:_Add_to_Android_version_to_Google_Play.mdwn new file mode 100644 index 000000000..f9016fb4d --- /dev/null +++ b/doc/todo/wishlist:_Add_to_Android_version_to_Google_Play.mdwn @@ -0,0 +1,9 @@ +If possible a frequently updated daily build in separate package for those more adventurous of us. + +It would make installing and testing much easier and no need to change configuration settings to allow untrusted source. + +> While it's vaid to wish that someone might put the apk into Google Play, +> I a) don't feel it's ready b) don't know if I want to go through +> the rigamarole required to use that service and c) don't feel this +> bug tracker is an appropriate place to track what is effectively a +> nontechnical request. [[done]] --[[Joey]] diff --git a/doc/todo/wishlist:_Advanced_settings_for_xmpp_and_webdav.mdwn b/doc/todo/wishlist:_Advanced_settings_for_xmpp_and_webdav.mdwn new file mode 100644 index 000000000..96552eecc --- /dev/null +++ b/doc/todo/wishlist:_Advanced_settings_for_xmpp_and_webdav.mdwn @@ -0,0 +1,7 @@ +It would be very nice with an "advanced settings" for jabber and webdav support. + +Currently XMPP fails if you use a google apps account. Since the domain provided in the email is not the same as the XMPP server. + +Same goes for webdav support. If i have my own webdav server somewhere on the internet there is no way to set it up in the assistant. + +[[!tag /design/assistant]] diff --git a/doc/todo/wishlist:_Advanced_settings_for_xmpp_and_webdav/comment_1_11c7444ab4988c60732af505b52bde3c._comment b/doc/todo/wishlist:_Advanced_settings_for_xmpp_and_webdav/comment_1_11c7444ab4988c60732af505b52bde3c._comment new file mode 100644 index 000000000..61cd3fc2e --- /dev/null +++ b/doc/todo/wishlist:_Advanced_settings_for_xmpp_and_webdav/comment_1_11c7444ab4988c60732af505b52bde3c._comment @@ -0,0 +1,20 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawmWg4VvDTer9f49Y3z-R0AH16P4d1ygotA" + nickname="Tobias" + subject="Hooks too" + date="2013-05-22T10:40:33Z" + content=""" +It would actually be very nice if this could be done with hooks too. + +Especially with the new hook method. + +Take this hook + + mega-hook = /usr/bin/python2 ~/sources/megaannex/megaannex.py + +git-annex could make a request with either the parameter(or environment variable) \"getsettingsobject\" that could return. {\"username\": \"\", \"password\": \"\", \"folder\": \"\"}. + +The point being git-annex can request from the hooks program what settings it takes, and give a web interface to set it. Then store the information in the creds folder(ew ew, that folder is unencrypted, oh well) and pass it to the hook on run. + +The advantage being that users wouldn't have to edit a settings file manually (this is currently also the case for the IMAP special remote, which also requires a settings file). +"""]] diff --git a/doc/todo/wishlist:_An_--all_option_for_dropunused.mdwn b/doc/todo/wishlist:_An_--all_option_for_dropunused.mdwn new file mode 100644 index 000000000..bd35c0e55 --- /dev/null +++ b/doc/todo/wishlist:_An_--all_option_for_dropunused.mdwn @@ -0,0 +1,4 @@ +Cleaning out a repository is presently a fairly manual process. Am I missing a UI trick? "dropunsed" with no arguments prints nothing at all; I think in that case it should display the list of what could be dropped. + +> [[done]]; comments seem satisfactory and I see no reason to complicate +> dropunused to output something unused already outputs. --[[Joey]] diff --git a/doc/todo/wishlist:_An_--all_option_for_dropunused/comment_1_d8726d108b3b40116b4ec3c9935f2dff._comment b/doc/todo/wishlist:_An_--all_option_for_dropunused/comment_1_d8726d108b3b40116b4ec3c9935f2dff._comment new file mode 100644 index 000000000..6c728a5d0 --- /dev/null +++ b/doc/todo/wishlist:_An_--all_option_for_dropunused/comment_1_d8726d108b3b40116b4ec3c9935f2dff._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.154.0.23" + subject="comment 1" + date="2012-10-22T15:35:30Z" + content=""" +`git annex unused` prints the list +"""]] diff --git a/doc/todo/wishlist:_An_--all_option_for_dropunused/comment_2_578248f7686ba2d80d7dc8b17c0cdf52._comment b/doc/todo/wishlist:_An_--all_option_for_dropunused/comment_2_578248f7686ba2d80d7dc8b17c0cdf52._comment new file mode 100644 index 000000000..a87a367d6 --- /dev/null +++ b/doc/todo/wishlist:_An_--all_option_for_dropunused/comment_2_578248f7686ba2d80d7dc8b17c0cdf52._comment @@ -0,0 +1,16 @@ +[[!comment format=mdwn + username="http://hands.com/~phil/" + nickname="hands" + subject="and you can specify ranges to dropunused" + date="2012-11-02T09:07:48Z" + content=""" +so having run: + + git annex unused + +you can then run: + + git annex dropunused 1-10000 + +or whatever, and it deletes the items in that range from the most recent <tt>unused</tt> invocation +"""]] diff --git a/doc/todo/wishlist:_An_option_like_--git-dir.mdwn b/doc/todo/wishlist:_An_option_like_--git-dir.mdwn new file mode 100644 index 000000000..cb9d374b3 --- /dev/null +++ b/doc/todo/wishlist:_An_option_like_--git-dir.mdwn @@ -0,0 +1,3 @@ +I'm currently integrating git-annex support into a filesystem synchronization tool that I use, and I have a use case where I'd like to run "git annex sync' on a local directory, and then automatically ssh over to remote hosts and run "git annex sync" in the related annex on that remote host. However, while I can easily "cd" on the local, there is no really easy way to "cd" on the remote without a hack. + +If I could say: git annex --annex-dir=PATH sync, where PATH is the annex directory, it would solve all my problems, and would also provide a nice correlation to the --git-dir option used by most Git commands. The basic idea is that I shouldn't have to be IN the directory to run git-annex commands, I should be able to tell git-annex which directory to apply its commands to. diff --git a/doc/todo/wishlist:_An_option_like_--git-dir/comment_1_5d877d90b8bdf21d4b8649744d229efd._comment b/doc/todo/wishlist:_An_option_like_--git-dir/comment_1_5d877d90b8bdf21d4b8649744d229efd._comment new file mode 100644 index 000000000..8e7c3c03e --- /dev/null +++ b/doc/todo/wishlist:_An_option_like_--git-dir/comment_1_5d877d90b8bdf21d4b8649744d229efd._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawmBUR4O9mofxVbpb8JV9mEbVfIYv670uJo" + nickname="Justin" + subject="What about..." + date="2012-10-16T16:43:29Z" + content=""" + ssh remotehost \"cd /path/to/annex && git annex sync\" +"""]] diff --git a/doc/todo/wishlist:_An_option_like_--git-dir/comment_2_462264821cbc48a433330cbf7ec6044d._comment b/doc/todo/wishlist:_An_option_like_--git-dir/comment_2_462264821cbc48a433330cbf7ec6044d._comment new file mode 100644 index 000000000..980658dc6 --- /dev/null +++ b/doc/todo/wishlist:_An_option_like_--git-dir/comment_2_462264821cbc48a433330cbf7ec6044d._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="2001:4978:f:21a::2" + subject="comment 2" + date="2012-10-17T18:31:58Z" + content=""" +You can use `GIT_DIR`. It would not be hard to add a --git-dir option, the only catch is how to communicate that state on to where it constructs its git repository data structure. (I suppose it could just set GIT_DIR..) +"""]] diff --git a/doc/todo/wishlist:_An_option_like_--git-dir/comment_3_0c3709b07a0a1091ceeee73b69e0f7ac._comment b/doc/todo/wishlist:_An_option_like_--git-dir/comment_3_0c3709b07a0a1091ceeee73b69e0f7ac._comment new file mode 100644 index 000000000..a76c42d9d --- /dev/null +++ b/doc/todo/wishlist:_An_option_like_--git-dir/comment_3_0c3709b07a0a1091ceeee73b69e0f7ac._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://me.yahoo.com/a/2grhJvAC049fJnvALDXek.6MRZMTlg--#eec89" + nickname="John" + subject="Response" + date="2012-10-20T05:21:13Z" + content=""" +@Justin If you have full shell access on the remote your solution works fine, but not if git-annex is the only binary you are allowed to execute. +"""]] diff --git a/doc/todo/wishlist:_Freeing_X_space_on_remote_Y.mdwn b/doc/todo/wishlist:_Freeing_X_space_on_remote_Y.mdwn new file mode 100644 index 000000000..5fec39d98 --- /dev/null +++ b/doc/todo/wishlist:_Freeing_X_space_on_remote_Y.mdwn @@ -0,0 +1 @@ +As suggested during the first Gitify BoF during DebConf13: Adding a way to have on-demand dropping of content in a given remote would allow a user to quickly free up disk space on demand while still heeding numcopies etc. diff --git a/doc/todo/wishlist:_GnuPG_options.mdwn b/doc/todo/wishlist:_GnuPG_options.mdwn new file mode 100644 index 000000000..2cadf8213 --- /dev/null +++ b/doc/todo/wishlist:_GnuPG_options.mdwn @@ -0,0 +1,16 @@ +[Maybe I should have extented [[wishlist:_simpler_gpg_usage/]], but I thought I'd make my own since it's perhaps too old.] + +I second Justin and [[his idea|wishlist:_simpler_gpg_usage/#comment-e120f8ede0d4cffce17cbf84564211c1]] of having per-remote GnuPG options. I'd even go one step further, and propose the option in the <tt>.gitattributes</tt> file. Indeed by default GnuPG compresses the data before encryption, which doesn't make a lot of sense for git-annex (in my use-case at least); My work-around to save this waste of CPU cycles was to customize my <tt>gpg.conf</tt>, but it's somewhat dirty since I do want to use compression in general. + +Here is how I envision the <tt>.git/config</tt>: +<pre> <code>[annex] + gnupg-options = --s2k-cipher-algo AES256 --s2k-digest-algo SHA512 --s2k-count 8388608 --cipher-algo AES256 --compress-algo none +</code></pre> + +And compression could be enabled on say, text files, with a suitable wildcard in the <tt>.gitattributes</tt> file. +<pre> <code>*.txt annex.gnupg-options="--s2k-cipher-algo AES256 --s2k-digest-algo SHA512 --s2k-count 8388608 --cipher-algo AES256 --compress-algo zlib" +</code></pre> + +This is something I could probably hack on if you think it'd be a worthwhile option ;-) + +> Done, and [[done]]! --[[Joey]] diff --git a/doc/todo/wishlist:_GnuPG_options/comment_1_6662e8a71ce20acc62147ef41ecffa50._comment b/doc/todo/wishlist:_GnuPG_options/comment_1_6662e8a71ce20acc62147ef41ecffa50._comment new file mode 100644 index 000000000..b756eccad --- /dev/null +++ b/doc/todo/wishlist:_GnuPG_options/comment_1_6662e8a71ce20acc62147ef41ecffa50._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + nickname="joey" + subject="comment 1" + date="2013-03-09T01:54:30Z" + content=""" +I'd be happy to apply a patch implementing annex.gnupg-options and/or per-remote remote.annex-gnupg-options, and I don't think it would be very hard to do. + +The gitattributes thing would be harder to do efficiently, and seems overkill. + + +"""]] diff --git a/doc/todo/wishlist:_Have_a_preview_of_download_or_upload_size.mdwn b/doc/todo/wishlist:_Have_a_preview_of_download_or_upload_size.mdwn new file mode 100644 index 000000000..12688951d --- /dev/null +++ b/doc/todo/wishlist:_Have_a_preview_of_download_or_upload_size.mdwn @@ -0,0 +1,10 @@ +When using SSH remote repository, git-annex uses rsync to download or upload files one at a time. I would like to have a preview of the overall transfer size so that I can estimate the transfer duration. + +This could be done as an option of get, move or copy, or as a separated command. + +If part of get, move or copy, git-annex could print how much has been done or how much left between every files. + +Thanks. + +> [[done]]; `git-annex status .` seems to cover the requested use case. +> --[[Joey]] diff --git a/doc/todo/wishlist:_Have_a_preview_of_download_or_upload_size/comment_1_019a2457e07377510feaa089a93bd76c._comment b/doc/todo/wishlist:_Have_a_preview_of_download_or_upload_size/comment_1_019a2457e07377510feaa089a93bd76c._comment new file mode 100644 index 000000000..4a59f37f1 --- /dev/null +++ b/doc/todo/wishlist:_Have_a_preview_of_download_or_upload_size/comment_1_019a2457e07377510feaa089a93bd76c._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.152.108.193" + subject="comment 1" + date="2013-06-25T17:26:25Z" + content=""" +git-annex is designed to work with really large trees of files, and so it processes files one at a time in a stream. To get an overall estimate of the size, it would need to traverse the whole directory to get the total, and then traverse it again to perform the transfer. This would make no-op transfers take twice as long, which is why I'm unlikely to implement it. +"""]] diff --git a/doc/todo/wishlist:_Have_a_preview_of_download_or_upload_size/comment_3_29a154699339bf040af0ee8aa24034f1._comment b/doc/todo/wishlist:_Have_a_preview_of_download_or_upload_size/comment_3_29a154699339bf040af0ee8aa24034f1._comment new file mode 100644 index 000000000..9f0c04017 --- /dev/null +++ b/doc/todo/wishlist:_Have_a_preview_of_download_or_upload_size/comment_3_29a154699339bf040af0ee8aa24034f1._comment @@ -0,0 +1,15 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawnHRhCe3qwVKQ8_NOGGSYJnAMW6FFyKbOc" + nickname="Holger" + subject="comment 3" + date="2013-07-02T04:05:06Z" + content=""" +What do you think of the following simpler variant: + + % git annex size myfile1.zip + myfile1.zip is 1329 MB + % git annex size mydir/ + mydir contains 2803 files totaling 328 GB. + +If you're worried about running over the tree twice, it may be a good idea to store the size of a subtree along with the other metadata. +"""]] diff --git a/doc/todo/wishlist:_Have_a_preview_of_download_or_upload_size/comment_3_8f7e1c4a5c714cbd719ee170354d79fa._comment b/doc/todo/wishlist:_Have_a_preview_of_download_or_upload_size/comment_3_8f7e1c4a5c714cbd719ee170354d79fa._comment new file mode 100644 index 000000000..fa9fdfb56 --- /dev/null +++ b/doc/todo/wishlist:_Have_a_preview_of_download_or_upload_size/comment_3_8f7e1c4a5c714cbd719ee170354d79fa._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.154.4.193" + subject="comment 3" + date="2013-07-02T17:04:47Z" + content=""" +You can get info like that by running `git annex status .` + +This can also be used to find out how big a download is before starting it. For example, to find all files that are not present locally before running git-annex get: + +`git annex status . --not --in here` +"""]] diff --git a/doc/todo/wishlist:_Have_a_preview_of_download_or_upload_size/comment_4_c7335f757e5546aa841cab38fffe7605._comment b/doc/todo/wishlist:_Have_a_preview_of_download_or_upload_size/comment_4_c7335f757e5546aa841cab38fffe7605._comment new file mode 100644 index 000000000..b9212a24d --- /dev/null +++ b/doc/todo/wishlist:_Have_a_preview_of_download_or_upload_size/comment_4_c7335f757e5546aa841cab38fffe7605._comment @@ -0,0 +1,19 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawnHRhCe3qwVKQ8_NOGGSYJnAMW6FFyKbOc" + nickname="Holger" + subject="comment 4" + date="2013-07-02T21:09:03Z" + content=""" +That's so cool, thanks! + +Do you think it'd be a major change to the repository format if the size of any directory was stored there so that this kind of status lookup becomes a constant time operation? The two most important operations are probably: + +* The total size of a directory, counting only files present here +* The total size of a directory, counting all files present at any location + +Of course, if the above two were constant time operations, you get --not here for free, too. + +To implement this, each node in the directory tree could have two additional 64 bit fields that hold the number of bytes in all files present anywhere (and this set of numbers is synchronized between all repositories), and the number of bytes in all files present here (only kept locally). This is only a small storage overhead (<16 MB if you have a million nodes) and suffices for repositories of size at most 2^64 bytes = 16 exabytes (probably more since most users will be ok with float accuracy). The numbers can be updated in logarithmic time every time a file changes. Instead of two numbers, it may not be that costly to store k numbers where k is the number of locations that a repository is connected to, since k is typically pretty small. + +The number of files can be stored in a similar way. +"""]] diff --git a/doc/todo/wishlist:_Have_a_preview_of_download_or_upload_size/comment_5_d2a845354f23d07880612740cf99ddd4._comment b/doc/todo/wishlist:_Have_a_preview_of_download_or_upload_size/comment_5_d2a845354f23d07880612740cf99ddd4._comment new file mode 100644 index 000000000..7e160bebf --- /dev/null +++ b/doc/todo/wishlist:_Have_a_preview_of_download_or_upload_size/comment_5_d2a845354f23d07880612740cf99ddd4._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawnHRhCe3qwVKQ8_NOGGSYJnAMW6FFyKbOc" + nickname="Holger" + subject="comment 5" + date="2013-07-03T02:43:32Z" + content=""" +Btw, this would also provide a cheap test for whether we need to recurse into the folder in certain copy or get actions (e.g., if number of bytes present here equals number of bytes globally present, we don't need to recurse). +"""]] diff --git a/doc/todo/wishlist:_Option_to_specify_max_transfer_rate.mdwn b/doc/todo/wishlist:_Option_to_specify_max_transfer_rate.mdwn new file mode 100644 index 000000000..3ecb42197 --- /dev/null +++ b/doc/todo/wishlist:_Option_to_specify_max_transfer_rate.mdwn @@ -0,0 +1,3 @@ +A big part of my online use is done via a low-speed connection over my mobile phone, this is limited to 16KB/sec because I always use up my 500MB quota the very first day of the month. `;-/` So when I need to download big files, I first download them to my online server, then transfer the files to my laptop with git-annex. If I'm connected via GSM, this occupies all the bandwidth and everything else moves like a heavily sedated slug. So if I want to work via VNC or SSH, I have to terminate ongoing transfers with Ctrl-C and then hopefully remember to restart it when I work locally. I know git-annex is robust enough to handle this gracefully, but it would be really nice to have a continuous connection going on in the background, limited to a value I choose. + +rsync(1) has a `--bwlimit` (bandwidth limit) where you can specify max download/upload speed in kilobytes/sec. It would be great if a similar option was integrated into git-annex. Thanks in advance. diff --git a/doc/todo/wishlist:_Option_to_specify_max_transfer_rate/comment_1_4fd870e14b5b70c8a6ade41406294387._comment b/doc/todo/wishlist:_Option_to_specify_max_transfer_rate/comment_1_4fd870e14b5b70c8a6ade41406294387._comment new file mode 100644 index 000000000..78ca76939 --- /dev/null +++ b/doc/todo/wishlist:_Option_to_specify_max_transfer_rate/comment_1_4fd870e14b5b70c8a6ade41406294387._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawmBUR4O9mofxVbpb8JV9mEbVfIYv670uJo" + nickname="Justin" + subject="trickle" + date="2013-01-22T22:44:52Z" + content=""" +not exactly integrated, but you can easily use trickle for this. + + trickle -d 50 git annex ... +"""]] diff --git a/doc/todo/wishlist:_Option_to_specify_max_transfer_rate/comment_2_dd854f297ad6a94b54be9f3edfd0f766._comment b/doc/todo/wishlist:_Option_to_specify_max_transfer_rate/comment_2_dd854f297ad6a94b54be9f3edfd0f766._comment new file mode 100644 index 000000000..70f04c616 --- /dev/null +++ b/doc/todo/wishlist:_Option_to_specify_max_transfer_rate/comment_2_dd854f297ad6a94b54be9f3edfd0f766._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://sunny256.sunbase.org/" + nickname="sunny256" + subject="Yay, trickle works" + date="2013-01-23T01:36:21Z" + content=""" +Justin, thanks a lot! trickle(1) works great. I didn't know about this program, but I'm not surprised that such a program is available. It never ceases to amaze me what's possible in a *NIX environment. +"""]] diff --git a/doc/todo/wishlist:_Option_to_specify_max_transfer_rate/comment_3_a8b7e90a473d5937807cc7eb456efe33._comment b/doc/todo/wishlist:_Option_to_specify_max_transfer_rate/comment_3_a8b7e90a473d5937807cc7eb456efe33._comment new file mode 100644 index 000000000..a5f8f6a1b --- /dev/null +++ b/doc/todo/wishlist:_Option_to_specify_max_transfer_rate/comment_3_a8b7e90a473d5937807cc7eb456efe33._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="2001:4978:f:21a::2" + subject="comment 3" + date="2013-01-24T01:55:10Z" + content=""" +In addition to trickle, the git-annex man page has examples of how to make rsync use --bwlimit + +Something like trickle is needed to limit rates for remotes not using rsync, however. +"""]] diff --git a/doc/todo/wishlist:_Prevent_repeated_password_prompts_for_one_command.mdwn b/doc/todo/wishlist:_Prevent_repeated_password_prompts_for_one_command.mdwn new file mode 100644 index 000000000..341a9afa4 --- /dev/null +++ b/doc/todo/wishlist:_Prevent_repeated_password_prompts_for_one_command.mdwn @@ -0,0 +1,45 @@ +Simple, when performing various git annex command over ssh, in particular a multi-file get, and using password authentication, git annex will prompt more than once for a user password. This makes batch updates very inconvenient. + +> I'd suggest using ssh-agent, or a passwordless ssh key. Possibly in +> combination with [[git-annex-shell]] if you want to lock down a +> particular ssh key to only being able to use git-annex and git-daemon. +> +> Combining multiple operations into a single ssh is on the todo list, but +> very far down it. --[[Joey]] + +>> OTOH, automatically running ssh in ControlMaster mode (and stopping it +>> at exit) would be useful and not hard thing for git-annex to do. +>> +>> It'd just need to set the appropriate config options, setting +>> ControlPath to a per-remote socket location that includes git-annex's +>> pid. Then at shutdown, run `ssh -O exit` on each such socket. +>> +>> Complicated slightly by not doing this if the user has already set up +>> more broad ssh connection caching. +>> +>> [[done]]! --[[Joey]] + +--- + +Slightly more elaborate design for using ssh connection caching: + +* Per-uuid ssh socket in `.git/annex/ssh/user@host.socket` +* Can be shared amoung concurrent git-annex processes as well as ssh + invocations inside the current git-annex. +* Also a lock file, `.git/annex/ssh/user@host.lock`. + Open and take shared lock before running ssh; store lock in lock pool. + (Not locking socket directly, because ssh might want to.) +* Run ssh like: `ssh -S .git/annex/ssh/user@host.socket -o ControlMaster=auto -o ControlPersist=yes user@host` +* At shutdown, enumerate all existing sockets, and on each: + 1. Drop any shared lock. + 2. Attempt to take an exclusive lock (non-blocking). + 3. `ssh -q -S .git/annex/ssh/user@host.socket -o ControlMaster=auto -o ControlPersist=yes -O stop user@host` + (Will exit nonzero if ssh is not running on that socket.) + 4. And then remove the socket and the lock file. +* Do same *at startup*. Why? In case an old git-annex was interrupted + and left behind a ssh. May have moved to a different network + in the meantime, etc, and be stalled waiting for a response from the + network, or talking to the wrong interface or something. + (Ie, the reason why I don't use ssh connection caching by default.) +* User should be able to override this, to use their own preferred + connection caching setup. `annex.sshcaching=false` diff --git a/doc/todo/wishlist:_Prevent_repeated_password_prompts_for_one_command/comment_1_3f9c0d08932c2ede61c802a91261a1f7._comment b/doc/todo/wishlist:_Prevent_repeated_password_prompts_for_one_command/comment_1_3f9c0d08932c2ede61c802a91261a1f7._comment new file mode 100644 index 000000000..2801d8e68 --- /dev/null +++ b/doc/todo/wishlist:_Prevent_repeated_password_prompts_for_one_command/comment_1_3f9c0d08932c2ede61c802a91261a1f7._comment @@ -0,0 +1,14 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U" + nickname="Richard" + subject="comment 1" + date="2011-05-06T18:30:02Z" + content=""" +Unless you are forced to use a password, you should really be using a ssh key. + + ssh-keygen + #put local .ssh/id_?sa.pub into remote .ssh/authorized_keys (which needs to be chmod 600) + ssh-add + git annex whatever + +"""]] diff --git a/doc/todo/wishlist:_Provide_a___34__git_annex__34___command_that_will_skip_duplicates.mdwn b/doc/todo/wishlist:_Provide_a___34__git_annex__34___command_that_will_skip_duplicates.mdwn new file mode 100644 index 000000000..933653578 --- /dev/null +++ b/doc/todo/wishlist:_Provide_a___34__git_annex__34___command_that_will_skip_duplicates.mdwn @@ -0,0 +1,28 @@ +(Hi, this is paulproteus@debian, AKA Asheesh). + +I've been enjoying using git-annex to archive my data. + +It's great that, by using git-annex and the SHA1 backend, I get a space-saving kind of deduplication through the symbolic links. + +I'm looking for the ability to filter files, before they get added to the annex, so that I don't add new files whose content is already in the annex.look That would help me in terms of personal file organization. + +It seems there is not, so this is a wishlist bug filed so that maybe such a thing might exist. What I would really like to do is: + +* $ git annex add --no-add-if-already-present . +* $ git commit -m "Slurping in some photos I found on my old laptop hard drive" + +And then I'd do something like: + +* $ git clean -f + +to remove the files that didn't get annexed in this run. That way, only one filename would ever point to a particular SHA1. + +I want this because I have copies of various of mine (photos, in particular) scattered across various hard disks. If this feature existed, I could comfortably toss them all into one git annex that grew, bit by bit, to store all of these files exactly once. + +(I would be even happier for "git annex add --unlink-duplicates .") + +(Another way to do this would be to "git annex add" them all, and then use a "git annex remove-duplicates" that could prompt me about which files are duplicates of each other, and then I could pipe that command's output into xargs git rm.) + +(As I write this, I realize it's possible to parse the destination of the symlink in a way that does this..) + +> [[done]]; see [[tips/finding_duplicate_files]] --[[Joey]] diff --git a/doc/todo/wishlist:_Provide_a___34__git_annex__34___command_that_will_skip_duplicates/comment_10_d78d79fb2f3713aa69f45d2691cf8dfe._comment b/doc/todo/wishlist:_Provide_a___34__git_annex__34___command_that_will_skip_duplicates/comment_10_d78d79fb2f3713aa69f45d2691cf8dfe._comment new file mode 100644 index 000000000..5dbb66cf6 --- /dev/null +++ b/doc/todo/wishlist:_Provide_a___34__git_annex__34___command_that_will_skip_duplicates/comment_10_d78d79fb2f3713aa69f45d2691cf8dfe._comment @@ -0,0 +1,68 @@ +[[!comment format=mdwn + username="http://adamspiers.myopenid.com/" + nickname="Adam" + subject="comment 10" + date="2011-12-23T17:22:11Z" + content=""" +> Your perl script is not O(n). Inserting into perl hash tables has +> overhead of minimum O(n log n). + +What's your source for this assertion? I would expect an amortized +average of `O(1)` per insertion, i.e. `O(n)` for full population. + +> Not counting the overhead of resizing hash tables, +> the grevious slowdown if the bucket size is overcome by data (it +> probably falls back to a linked list or something then), and the +> overhead of traversing the hash tables to get data out. + +None of which necessarily change the algorithmic complexity. However +real benchmarks are far more useful here than complexity analysis, and +[the dangers of premature optimization](http://c2.com/cgi/wiki?PrematureOptimization) +should not be forgotten. + +> Your memory size calculations ignore the overhead of a hash table or +> other data structure to store the data in, which will tend to be +> more than the actual data size it's storing. I estimate your 50 +> million number is off by at least one order of magnitude, and more +> likely two; + +Sure, I was aware of that, but my point still stands. Even 500k keys +per 1GB of RAM does not sound expensive to me. + +> in any case I don't want git-annex to use 1 gb of ram. + +Why not? What's the maximum it should use? 512MB? 256MB? +32MB? I don't see the sense in the author of a program +dictating thresholds which are entirely dependent on the context +in which the program is *run*, not the context in which it's *written*. +That's why systems have files such as `/etc/security/limits.conf`. + +You said you want git-annex to scale to enormous repositories. If you +impose an arbitrary memory restriction such as the above, that means +avoiding implementing *any* kind of functionality which requires `O(n)` +memory or worse. Isn't it reasonable to assume that many users use +git-annex on repositories which are *not* enormous? Even when they do +work with enormous repositories, just like with any other program, +they would naturally expect certain operations to take longer or +become impractical without sufficient RAM. That's why I say that this +restriction amounts to throwing out the baby with the bathwater. +It just means that those who need the functionality would have to +reimplement it themselves, assuming they are able, which is likely +to result in more wheel reinventions. I've already shared +[my implementation](https://github.com/aspiers/git-config/blob/master/bin/git-annex-finddups) +but how many people are likely to find it, let alone get it working? + +> Little known fact: sort(1) will use a temp file as a buffer if too +> much memory is needed to hold the data to sort. + +Interesting. Presumably you are referring to some undocumented +behaviour, rather than `--batch-size` which only applies when merging +multiple files, and not when only sorting STDIN. + +> It's also written in the most efficient language possible and has +> been ruthlessly optimised for 30 years, so I would be very surprised +> if it was not the best choice. + +It's the best choice for sorting. But sorting purely to detect +duplicates is a dismally bad choice. +"""]] diff --git a/doc/todo/wishlist:_Provide_a___34__git_annex__34___command_that_will_skip_duplicates/comment_11_4316d9d74312112dc4c823077af7febe._comment b/doc/todo/wishlist:_Provide_a___34__git_annex__34___command_that_will_skip_duplicates/comment_11_4316d9d74312112dc4c823077af7febe._comment new file mode 100644 index 000000000..286487eee --- /dev/null +++ b/doc/todo/wishlist:_Provide_a___34__git_annex__34___command_that_will_skip_duplicates/comment_11_4316d9d74312112dc4c823077af7febe._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joey.kitenet.net/" + nickname="joey" + subject="comment 11" + date="2011-12-23T17:52:21Z" + content=""" +I don't think that [[tips/finding_duplicate_files]] is hard to find, and the multiple different ways it shows to deal with the duplicate files shows the flexability of the unix pipeline approach. +"""]] diff --git a/doc/todo/wishlist:_Provide_a___34__git_annex__34___command_that_will_skip_duplicates/comment_12_ed6d07f16a11c6eee7e3d5005e8e6fa3._comment b/doc/todo/wishlist:_Provide_a___34__git_annex__34___command_that_will_skip_duplicates/comment_12_ed6d07f16a11c6eee7e3d5005e8e6fa3._comment new file mode 100644 index 000000000..909beed83 --- /dev/null +++ b/doc/todo/wishlist:_Provide_a___34__git_annex__34___command_that_will_skip_duplicates/comment_12_ed6d07f16a11c6eee7e3d5005e8e6fa3._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joey.kitenet.net/" + nickname="joey" + subject="comment 12" + date="2011-12-23T18:02:24Z" + content=""" +BTW, sort -S '90%' benchmarks consistently 2x as fast as perl's hashes all the way up to 1 million files. Of course the pipeline approach allows you to swap in perl or whatever else is best for you at scale. +"""]] diff --git a/doc/todo/wishlist:_Provide_a___34__git_annex__34___command_that_will_skip_duplicates/comment_1_fd213310ee548d8726791d2b02237fde._comment b/doc/todo/wishlist:_Provide_a___34__git_annex__34___command_that_will_skip_duplicates/comment_1_fd213310ee548d8726791d2b02237fde._comment new file mode 100644 index 000000000..094e4526e --- /dev/null +++ b/doc/todo/wishlist:_Provide_a___34__git_annex__34___command_that_will_skip_duplicates/comment_1_fd213310ee548d8726791d2b02237fde._comment @@ -0,0 +1,29 @@ +[[!comment format=mdwn + username="http://joey.kitenet.net/" + nickname="joey" + subject="comment 1" + date="2011-01-27T18:29:44Z" + content=""" +Hey Asheesh, I'm happy you're finding git-annex useful. + +So, there are two forms of duplication going on here. There's duplication of the content, and duplication of the filenames +pointing at that content. + +Duplication of the filenames is probably not a concern, although it's what I thought you were talking about at first. It's probably info worth recording that backup-2010/some_dir/foo and backup-2009/other_dir/foo are two names you've used for the same content in the past. If you really wanted to remove backup-2009/foo, you could do it by writing a script that looks at the basenames of the symlink targets and removes files that point to the same content as other files. + +Using SHA1 ensures that the same key is used for identical files, so generally avoids duplication of content. But if you have 2 disks with an identical file on each, and make them both into annexes, then git-annex will happily retain both +copies of the content, one per disk. It generally considers keeping copies of content a good thing. :) + +So, what if you want to remove the unnecessary copies? Well, there's a really simple way: + +<pre> +cd /media/usb-1 +git remote add other-disk /media/usb-0 +git annex add +git annex drop +</pre> + +This asks git-annex to add everything to the annex, but then remove any file contents that it can safely remove. What can it safely remove? Well, anything that it can verify is on another repository such as \"other-disk\"! So, this will happily drop any duplicated file contents, while leaving all the rest alone. + +In practice, you might not want to have all your old backup disks mounted at the same time and configured as remotes. Look into configuring [[trust]] to avoid needing do to that. If usb-0 is already a trusted disk, all you need is a simple \"git annex drop\" on usb-1. +"""]] diff --git a/doc/todo/wishlist:_Provide_a___34__git_annex__34___command_that_will_skip_duplicates/comment_2_4394bde1c6fd44acae649baffe802775._comment b/doc/todo/wishlist:_Provide_a___34__git_annex__34___command_that_will_skip_duplicates/comment_2_4394bde1c6fd44acae649baffe802775._comment new file mode 100644 index 000000000..04d58a459 --- /dev/null +++ b/doc/todo/wishlist:_Provide_a___34__git_annex__34___command_that_will_skip_duplicates/comment_2_4394bde1c6fd44acae649baffe802775._comment @@ -0,0 +1,18 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawkjvjLHW9Omza7x1VEzIFQ8Z5honhRB90I" + nickname="Asheesh" + subject="I actually *do* want to avoid duplication of filenames" + date="2011-01-28T07:30:05Z" + content=""" +I really do want just one filename per file, at least for some cases. + +For my photos, there's no benefit to having a few filenames point to the same file. As I'm putting them all into the git-annex, that is a good time to remove the pure duplicates so that I don't e.g. see them twice when browsing the directory as a gallery. Also, I am uploading my photos to the web, and I want to avoid uploading the same photo (by content) twice. + +I hope that makes things clearer! + +For now I'm just doing this: + +* paulproteus@renaissance:/mnt/backups-terabyte/paulproteus/sd-card-from-2011-01-06/sd-cards/DCIM/100CANON $ for file in *; do hash=$(sha1sum \"$file\"); if ls /home/paulproteus/Photos/in-flickr/.git-annex | grep -q \"$hash\"; then echo already annexed ; else flickr_upload \"$file\" && mv \"$file\" \"/home/paulproteus/Photos/in-flickr/2011-01-28/from-some-nested-sd-card-bk\" && (cd /home/paulproteus/Photos/in-flickr/2011-01-28/from-some-nested-sd-card-bk && git annex add . && git commit -m ...) ; fi; done + +(Yeah, Flickr for my photos for now. I feel sad about betraying the principle of autonomo.us-ness.) +"""]] diff --git a/doc/todo/wishlist:_Provide_a___34__git_annex__34___command_that_will_skip_duplicates/comment_3_076cb22057583957d5179d8ba9004605._comment b/doc/todo/wishlist:_Provide_a___34__git_annex__34___command_that_will_skip_duplicates/comment_3_076cb22057583957d5179d8ba9004605._comment new file mode 100644 index 000000000..d11119bc3 --- /dev/null +++ b/doc/todo/wishlist:_Provide_a___34__git_annex__34___command_that_will_skip_duplicates/comment_3_076cb22057583957d5179d8ba9004605._comment @@ -0,0 +1,18 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawkjvjLHW9Omza7x1VEzIFQ8Z5honhRB90I" + nickname="Asheesh" + subject="Duplication of the filenames is what I am concerned about" + date="2011-04-29T11:48:22Z" + content=""" +For what it's worth, yes, I want to actually forget I ever had the same file in the filesystem with a duplicated name. I'm not just aiming to clean up the disk's space usage; I'm also aiming to clean things up so that navigating the filesystem is easier. + +I can write my own script to do that based on the symlinks' target (and I wrote something along those lines), but I still think it'd be nicer if git-annex supported this use case. + +Perhaps: + +<pre>git annex drop --by-contents</pre> + +could let me remove a file from git-annex if the contents are available through a different name. (Right now, \"git annex drop\" requires the name *and* contents match.) + +-- Asheesh. +"""]] diff --git a/doc/todo/wishlist:_Provide_a___34__git_annex__34___command_that_will_skip_duplicates/comment_4_f120d1e83c1a447f2ecce302fc69cf74._comment b/doc/todo/wishlist:_Provide_a___34__git_annex__34___command_that_will_skip_duplicates/comment_4_f120d1e83c1a447f2ecce302fc69cf74._comment new file mode 100644 index 000000000..a218ee3d5 --- /dev/null +++ b/doc/todo/wishlist:_Provide_a___34__git_annex__34___command_that_will_skip_duplicates/comment_4_f120d1e83c1a447f2ecce302fc69cf74._comment @@ -0,0 +1,35 @@ +[[!comment format=mdwn + username="http://adamspiers.myopenid.com/" + nickname="Adam" + subject="List the duplicate filenames, then let the user decide what to do" + date="2011-12-22T12:31:29Z" + content=""" +I have the same use case as Asheesh but I want to be able to see which filenames point to the same objects and then decide which of the duplicates to drop myself. I think + + git annex drop --by-contents + +would be the wrong approach because how does git-annex know which ones to drop? There's too much potential for error. + +Instead it would be great to have something like + + git annex finddups + +While it's easy enough to knock up a bit of shell or Perl to achieve this, that relies on knowledge of the annex symlink structure, so I think really it belongs inside git-annex. + +If this command gave output similar to the excellent `fastdup` utility: + + Scanning for files... 672 files in 10.439 seconds + Comparing 2 sets of files... + + 2 files (70.71 MB/ea) + /home/adam/media/flat/tour/flat-tour.3gp + /home/adam/videos/tour.3gp + + Found 1 duplicate of 1 file (70.71 MB wasted) + Scanned 672 files (1.96 GB) in 11.415 seconds + +then you could do stuff like + + git annex finddups | grep /home/adam/media/flat | xargs rm + +"""]] diff --git a/doc/todo/wishlist:_Provide_a___34__git_annex__34___command_that_will_skip_duplicates/comment_5_5c30294b3c59fdebb1eef0ae5da4cd4f._comment b/doc/todo/wishlist:_Provide_a___34__git_annex__34___command_that_will_skip_duplicates/comment_5_5c30294b3c59fdebb1eef0ae5da4cd4f._comment new file mode 100644 index 000000000..e48a4a9b3 --- /dev/null +++ b/doc/todo/wishlist:_Provide_a___34__git_annex__34___command_that_will_skip_duplicates/comment_5_5c30294b3c59fdebb1eef0ae5da4cd4f._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://adamspiers.myopenid.com/" + nickname="Adam" + subject="Here's a Perl version" + date="2011-12-22T15:43:51Z" + content=""" +https://github.com/aspiers/git-config/blob/master/bin/git-annex-finddups + +but it would be better in git-annex itself ... +"""]] diff --git a/doc/todo/wishlist:_Provide_a___34__git_annex__34___command_that_will_skip_duplicates/comment_6_f24541ada1c86d755acba7e9fa7cff24._comment b/doc/todo/wishlist:_Provide_a___34__git_annex__34___command_that_will_skip_duplicates/comment_6_f24541ada1c86d755acba7e9fa7cff24._comment new file mode 100644 index 000000000..5d8ac8e61 --- /dev/null +++ b/doc/todo/wishlist:_Provide_a___34__git_annex__34___command_that_will_skip_duplicates/comment_6_f24541ada1c86d755acba7e9fa7cff24._comment @@ -0,0 +1,16 @@ +[[!comment format=mdwn + username="http://joey.kitenet.net/" + nickname="joey" + subject="comment 6" + date="2011-12-22T16:39:24Z" + content=""" +My main concern with putting this in git-annex is that finding duplicates necessarily involves storing a list of every key and file in the repository, and git-annex is very carefully built to avoid things that require non-constant memory use, so that it can scale to very big repositories. (The only exception is the `unused` command, and reducing its memory usage is a continuing goal.) + +So I would rather come at this from a different angle.. like providing a way to output a list of files and their associated keys, which the user can then use in their own shell pipelines to find duplicate keys: + + git annex find --include '*' --format='${file} ${key}\n' | sort --key 2 | uniq --all-repeated --skip-fields=1 + +Which is implemented now! + +(Making that pipeline properly handle filenames with spaces is left as an exercise for the reader..) +"""]] diff --git a/doc/todo/wishlist:_Provide_a___34__git_annex__34___command_that_will_skip_duplicates/comment_7_c39f1bb7c61a89b238c61bee1c049767._comment b/doc/todo/wishlist:_Provide_a___34__git_annex__34___command_that_will_skip_duplicates/comment_7_c39f1bb7c61a89b238c61bee1c049767._comment new file mode 100644 index 000000000..a33700280 --- /dev/null +++ b/doc/todo/wishlist:_Provide_a___34__git_annex__34___command_that_will_skip_duplicates/comment_7_c39f1bb7c61a89b238c61bee1c049767._comment @@ -0,0 +1,54 @@ +[[!comment format=mdwn + username="http://adamspiers.myopenid.com/" + nickname="Adam" + subject="comment 7" + date="2011-12-22T20:04:14Z" + content=""" +> My main concern with putting this in git-annex is that finding +> duplicates necessarily involves storing a list of every key and file +> in the repository + +Only if you want to search the *whole* repository for duplicates, and if +you do, then you're necessarily going to have to chew up memory in +some process anyway, so what difference whether it's git-annex or +(say) a Perl wrapper? + +> and git-annex is very carefully built to avoid things that require +> non-constant memory use, so that it can scale to very big +> repositories. + +That's a worthy goal, but if everything could be implemented with an +O(1) memory footprint then we'd be in much more pleasant world :-) +Even O(n) isn't that bad ... + +That aside, I like your `--format=\"%f %k\n\"` idea a lot. That opens +up the \"black box\" of `.git/annex/objects` and makes nice things +possible, as your pipeline already demonstrates. However, I'm not +sure why you think `git annex find | sort | uniq` would be more +efficient. Not only does the sort require the very thing you were +trying to avoid (i.e. the whole list in memory), but it's also +O(n log n) which is significantly slower than my O(n) Perl script +linked above. + +More considerations about this pipeline: + +* Doesn't it only include locally available files? Ideally it should + spot duplicates even when the backing blob is not available locally. +* What's the point of `--include '*'` ? Doesn't `git annex find` + with no arguments already include all files, modulo the requirement + above that they're locally available? +* Any user using this `git annex find | ...` approach is likely to + run up against its limitations sooner rather than later, because + they're already used to the plethora of options `find(1)` provides. + Rather than reinventing the wheel, is there some way `git annex find` + could harness the power of `find(1)` ? + +Those considerations aside, a combined approach would be to implement + + git annex find --format=... + +and then alter my Perl wrapper to `popen(2)` from that rather than using +`File::Find`. But I doubt you would want to ship Perl wrappers in the +distribution, so if you don't provide a Haskell equivalent then users +who can't code are left high and dry. +"""]] diff --git a/doc/todo/wishlist:_Provide_a___34__git_annex__34___command_that_will_skip_duplicates/comment_8_221ed2e53420278072a6d879c6f251d1._comment b/doc/todo/wishlist:_Provide_a___34__git_annex__34___command_that_will_skip_duplicates/comment_8_221ed2e53420278072a6d879c6f251d1._comment new file mode 100644 index 000000000..5ac292afe --- /dev/null +++ b/doc/todo/wishlist:_Provide_a___34__git_annex__34___command_that_will_skip_duplicates/comment_8_221ed2e53420278072a6d879c6f251d1._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://adamspiers.myopenid.com/" + nickname="Adam" + subject="How much memory would it actually use anyway?" + date="2011-12-22T20:15:22Z" + content=""" +Another thought - an SHA1 digest is 20 bytes. That means you can fit over 50 million keys into 1GB of RAM. Granted you also need memory to store the values (pathnames) which in many cases will be longer, and some users may also choose more expensive backends than SHA1 ... but even so, it seems to me that you are at risk of throwing the baby out with the bath water. +"""]] diff --git a/doc/todo/wishlist:_Provide_a___34__git_annex__34___command_that_will_skip_duplicates/comment_9_aecfa896c97b9448f235bce18a40621d._comment b/doc/todo/wishlist:_Provide_a___34__git_annex__34___command_that_will_skip_duplicates/comment_9_aecfa896c97b9448f235bce18a40621d._comment new file mode 100644 index 000000000..82c6921eb --- /dev/null +++ b/doc/todo/wishlist:_Provide_a___34__git_annex__34___command_that_will_skip_duplicates/comment_9_aecfa896c97b9448f235bce18a40621d._comment @@ -0,0 +1,14 @@ +[[!comment format=mdwn + username="http://joey.kitenet.net/" + nickname="joey" + subject="comment 9" + date="2011-12-23T16:07:39Z" + content=""" +Adam, to answer a lot of points breifly.. + +* --include='*' makes find list files whether their contents are present or not +* Your perl script is not O(n). Inserting into perl hash tables has overhead of minimum O(n log n). Not counting the overhead of resizing hash tables, the grevious slowdown if the bucket size is overcome by data (it probably falls back to a linked list or something then), and the overhead of traversing the hash tables to get data out. +* I think that git-annex's set of file matching options is coming along nicely, and new ones can easily be added, so see no need to pull in unix find(1). +* Your memory size calculations ignore the overhead of a hash table or other data structure to store the data in, which will tend to be more than the actual data size it's storing. I estimate your 50 million number is off by at least one order of magnitude, and more likely two; in any case I don't want git-annex to use 1 gb of ram. +* Little known fact: sort(1) will use a temp file as a buffer if too much memory is needed to hold the data to sort. It's also written in the most efficient language possible and has been ruthlessly optimised for 30 years, so I would be very surprised if it was not the best choice. +"""]] diff --git a/doc/todo/wishlist:_Restore_s3_files_moved_to_Glacier.mdwn b/doc/todo/wishlist:_Restore_s3_files_moved_to_Glacier.mdwn new file mode 100644 index 000000000..85fc2785c --- /dev/null +++ b/doc/todo/wishlist:_Restore_s3_files_moved_to_Glacier.mdwn @@ -0,0 +1,7 @@ +I would like to use the automated AWS lifecycle rules to move the git annex files store on S3 to Glacier after a bit of time. Git annex need must support this kind of S3 files explicitly in order for it to work. + +This is different from the adding a Glacier remote to git annex because of the reasons explained in <http://aws.typepad.com/aws/2012/11/archive-s3-to-glacier.html>. + +Basically, the files moved by AWS from S3 to Glacier are not available under the normal Glacier API. In fact, the moved S3 files are listed as available but under the `GLACIER` storage class and need a RESTORE request before they can be GET like other S3 files. Trying to GET an S3 file that has been moved to Glacier will not restore it from Glacier and will result in an 403 error. + +I suppose DELETE needs special care as well. diff --git a/doc/todo/wishlist:_Tell_git_annex___40__assistant__41___which_files___40__not__41___to_annex_via_.gitattributes.mdwn b/doc/todo/wishlist:_Tell_git_annex___40__assistant__41___which_files___40__not__41___to_annex_via_.gitattributes.mdwn new file mode 100644 index 000000000..f65a95f45 --- /dev/null +++ b/doc/todo/wishlist:_Tell_git_annex___40__assistant__41___which_files___40__not__41___to_annex_via_.gitattributes.mdwn @@ -0,0 +1,9 @@ +Title says it all. + +It would be nice if I could tell git annex (assistant) which files (not) to annex (automatically). + +[[!tag /design/assistant]] + +> [[done]]; use `annex.largefiles` git config to configure criteria for +> which files should be annexed. The rest will be added to git as normal +> files. --[[Joey]] diff --git a/doc/todo/wishlist:___34__git_annex_add__34___multiple_processes.mdwn b/doc/todo/wishlist:___34__git_annex_add__34___multiple_processes.mdwn new file mode 100644 index 000000000..a04af05b4 --- /dev/null +++ b/doc/todo/wishlist:___34__git_annex_add__34___multiple_processes.mdwn @@ -0,0 +1,10 @@ +Hello, + +i'm in the process of managing my music collection with git-annex. The initial "git annex add" using the sha1 banckend is quite long an i was wondering that it could be nice to launch multiple "sha1sum" processes in parallel to speed up things. + +Anyway, thanks for this wonderful piece of software. + +Jean-Baptiste + +> closing as dup of [[parallel possibilities]] (also see comments below) +> [[done]] --[[Joey]] diff --git a/doc/todo/wishlist:___34__git_annex_add__34___multiple_processes/comment_1_85b14478411a33e6186a64bd41f0910d._comment b/doc/todo/wishlist:___34__git_annex_add__34___multiple_processes/comment_1_85b14478411a33e6186a64bd41f0910d._comment new file mode 100644 index 000000000..2364b7fb8 --- /dev/null +++ b/doc/todo/wishlist:___34__git_annex_add__34___multiple_processes/comment_1_85b14478411a33e6186a64bd41f0910d._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://joey.kitenet.net/" + nickname="joey" + subject="comment 1" + date="2011-02-25T19:12:42Z" + content=""" +I'd expect the checksumming to be disk bound, not CPU bound, on most systems. + +I suggest you start off on the WORM backend, and then you can run a job later to [[migrate|walkthrough#index14h2]] to the SHA1 backend. +"""]] diff --git a/doc/todo/wishlist:___34__git_annex_add__34___multiple_processes/comment_2_82e857f463cfdf73c70f6c0a9f9a31d6._comment b/doc/todo/wishlist:___34__git_annex_add__34___multiple_processes/comment_2_82e857f463cfdf73c70f6c0a9f9a31d6._comment new file mode 100644 index 000000000..9b8240658 --- /dev/null +++ b/doc/todo/wishlist:___34__git_annex_add__34___multiple_processes/comment_2_82e857f463cfdf73c70f6c0a9f9a31d6._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joey.kitenet.net/" + nickname="joey" + subject="comment 2" + date="2011-02-25T19:54:28Z" + content=""" +But, see [[todo/parallel_possibilities]] +"""]] diff --git a/doc/todo/wishlist:___34__git_annex_add__34___multiple_processes/comment_3_8af85eba7472d9025c6fae4f03e3ad75._comment b/doc/todo/wishlist:___34__git_annex_add__34___multiple_processes/comment_3_8af85eba7472d9025c6fae4f03e3ad75._comment new file mode 100644 index 000000000..ee769f0dd --- /dev/null +++ b/doc/todo/wishlist:___34__git_annex_add__34___multiple_processes/comment_3_8af85eba7472d9025c6fae4f03e3ad75._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="jbd" + ip="89.158.228.148" + subject="comment 3" + date="2011-02-26T10:26:12Z" + content=""" +Thank your for your answer and the link ! +"""]] diff --git a/doc/todo/wishlist:___34__quiet__34___annex_get_for_centralized_use_case.mdwn b/doc/todo/wishlist:___34__quiet__34___annex_get_for_centralized_use_case.mdwn new file mode 100644 index 000000000..d53fa56ab --- /dev/null +++ b/doc/todo/wishlist:___34__quiet__34___annex_get_for_centralized_use_case.mdwn @@ -0,0 +1,14 @@ +We're using git-annex to manage large files as part of a team. + +We have a central repository of large files that everyone grabs from. + +We would like to be able to get the files without updating the `git-annex` branch. This way it doesn't pollute the history with essentially always unreachable locations. + +I think the easiest way would be to just add an option to not update the `git-annex` branch when `annex get` is executed. + +Thoughts? + +> See [[untracked_remotes]] for a todo item that will probably +> be useful in this sitation. Since that describes better what +> this bug report seems to be asking for, I am closing this one. +> [[closed|done]] --[[Joey]] diff --git a/doc/todo/wishlist:___34__quiet__34___annex_get_for_centralized_use_case/comment_1_5c8812973cf91b046e7fc44d7e86c78e._comment b/doc/todo/wishlist:___34__quiet__34___annex_get_for_centralized_use_case/comment_1_5c8812973cf91b046e7fc44d7e86c78e._comment new file mode 100644 index 000000000..61d82e2ae --- /dev/null +++ b/doc/todo/wishlist:___34__quiet__34___annex_get_for_centralized_use_case/comment_1_5c8812973cf91b046e7fc44d7e86c78e._comment @@ -0,0 +1,14 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.154.6.48" + subject="comment 1" + date="2013-07-17T23:23:50Z" + content=""" +I'm interested to hear that your team is using git-annex. + +Have you tried `git config annex.alwayscommit false`? This will avoid committing, and just store the info in a local journal -- so even `git annex fsck` will still work. + +Hmm, perhaps you want to update the branch when running `git annex copy` to put files onto the server, but not when getting them? A switch to disable updating the branch would then make sense. Any use of fsck would notice the inconsistency though, and commit a fix to the git-annex branch -- unless you also used the new switch when running fsck. + +But what happens if someone makes a change, pushes to the server, but forgets to `git annex copy` the file? Everyone would then be left with a missing file that git annex doesn't know where it is. One of the reasons it tracks the locations is so that if necessary you know which repository such a misplaced fail is stored in. And can go track down that person's laptop and apply a cluebat. ;) Do you do something on the server to prevent this scenario? +"""]] diff --git a/doc/todo/wishlist:___34__quiet__34___annex_get_for_centralized_use_case/comment_2_f36b6a5b128423211aac91a252ecf85f._comment b/doc/todo/wishlist:___34__quiet__34___annex_get_for_centralized_use_case/comment_2_f36b6a5b128423211aac91a252ecf85f._comment new file mode 100644 index 000000000..3379742d2 --- /dev/null +++ b/doc/todo/wishlist:___34__quiet__34___annex_get_for_centralized_use_case/comment_2_f36b6a5b128423211aac91a252ecf85f._comment @@ -0,0 +1,18 @@ +[[!comment format=mdwn + username="http://caust1c.myopenid.com/" + nickname="asbraithwaite" + subject="comment 2" + date="2013-07-17T23:42:25Z" + content=""" +What would happen if we wanted to copy a file to the server +with that set? or even when running `git annex add` +You understood me exactly though. +We'd like to be able to get the files without a commit, but copy them +with a commit of the changes. +The way we operate, if somebody makes a change with regards to +largefiles, they should also add a test for it. +Ideally, the test suite would catch it. That or people would +realize that theres a new big file and send around an email to +see who was trying to a largefile when they couldn't reach it. + +"""]] diff --git a/doc/todo/wishlist:___34__quiet__34___annex_get_for_centralized_use_case/comment_3_ad1569b2405acacd2e37f42b82f24c88._comment b/doc/todo/wishlist:___34__quiet__34___annex_get_for_centralized_use_case/comment_3_ad1569b2405acacd2e37f42b82f24c88._comment new file mode 100644 index 000000000..14ab5f65a --- /dev/null +++ b/doc/todo/wishlist:___34__quiet__34___annex_get_for_centralized_use_case/comment_3_ad1569b2405acacd2e37f42b82f24c88._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.154.6.48" + subject="comment 3" + date="2013-07-18T00:12:39Z" + content=""" +Actually, when a file is sent to a git repository on the server, it will update the location log on that side. So even if the client has alwayscommit=false, other clients will learn the file has reached the server. + +So that might just work for you. +"""]] diff --git a/doc/todo/wishlist:___34__quiet__34___annex_get_for_centralized_use_case/comment_4_8aba90150fe178ce9712ad951628f3d6._comment b/doc/todo/wishlist:___34__quiet__34___annex_get_for_centralized_use_case/comment_4_8aba90150fe178ce9712ad951628f3d6._comment new file mode 100644 index 000000000..c1f148fdb --- /dev/null +++ b/doc/todo/wishlist:___34__quiet__34___annex_get_for_centralized_use_case/comment_4_8aba90150fe178ce9712ad951628f3d6._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawln3ckqKx0x_xDZMYwa9Q1bn4I06oWjkog" + nickname="Michael" + subject="comment 4" + date="2013-07-18T18:03:00Z" + content=""" +If you can have developers only add large files on the central server, avoiding using git annex sync and only pulling from git repo should work, I think. +"""]] diff --git a/doc/todo/wishlist:___34__quiet__34___annex_get_for_centralized_use_case/comment_5_6f42d240e0021f4dfa37146bea3f5d7e._comment b/doc/todo/wishlist:___34__quiet__34___annex_get_for_centralized_use_case/comment_5_6f42d240e0021f4dfa37146bea3f5d7e._comment new file mode 100644 index 000000000..aa547d790 --- /dev/null +++ b/doc/todo/wishlist:___34__quiet__34___annex_get_for_centralized_use_case/comment_5_6f42d240e0021f4dfa37146bea3f5d7e._comment @@ -0,0 +1,16 @@ +[[!comment format=mdwn + username="http://caust1c.myopenid.com/" + nickname="asbraithwaite" + subject="comment 5" + date="2013-07-18T21:17:45Z" + content=""" +Wow. This worked better than expected. I'm still trying to understand the implementation (I'm not familiar with Haskell), but there is some magic going on. + +To explain: + +I didn't expect that when I added a file (Locally and to the annex remote) that when somebody else did a pull, git-annex would recognize this and update the working copy to download and include that file. + +I'm not sure if this is intended behavior or not (since I thought all commands had to go through git-annex) but I like it nonetheless. + +Thanks again for the tips! +"""]] diff --git a/doc/todo/wishlist:___34__quiet__34___annex_get_for_centralized_use_case/comment_6_5fda455febf728b079f26fe42bf7bcab._comment b/doc/todo/wishlist:___34__quiet__34___annex_get_for_centralized_use_case/comment_6_5fda455febf728b079f26fe42bf7bcab._comment new file mode 100644 index 000000000..3f10e48ad --- /dev/null +++ b/doc/todo/wishlist:___34__quiet__34___annex_get_for_centralized_use_case/comment_6_5fda455febf728b079f26fe42bf7bcab._comment @@ -0,0 +1,16 @@ +[[!comment format=mdwn + username="http://caust1c.myopenid.com/" + nickname="asbraithwaite" + subject="comment 6" + date="2013-07-18T21:23:58Z" + content=""" +Disregard that. I was testing it with a particular file that didn't exactly play nice. + +Basically, to test this before production I used `dd bs=1024 count=100000 if=/dev/zero of=bigfile`. + +When I did a `git pull`, it showed the symlink as being valid. + +When I changed the command to `dd bs=1024 count=100000 if=/dev/urandom of=bigfile` then it showed the symlink as bad after pulling. + +Weird! +"""]] diff --git a/doc/todo/wishlist:___34__quiet__34___annex_get_for_centralized_use_case/comment_7_f1052ab997f1a2cccbabfd1533fc0a59._comment b/doc/todo/wishlist:___34__quiet__34___annex_get_for_centralized_use_case/comment_7_f1052ab997f1a2cccbabfd1533fc0a59._comment new file mode 100644 index 000000000..deb25a9ce --- /dev/null +++ b/doc/todo/wishlist:___34__quiet__34___annex_get_for_centralized_use_case/comment_7_f1052ab997f1a2cccbabfd1533fc0a59._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawln3ckqKx0x_xDZMYwa9Q1bn4I06oWjkog" + nickname="Michael" + subject="comment 7" + date="2013-07-18T21:48:06Z" + content=""" +If you wanted to auto-get files on git pull, you could trying putting git annex get into .git/hooks/post-merge (needs to be marked as executable). +"""]] diff --git a/doc/todo/wishlist:___34__quiet__34___annex_get_for_centralized_use_case/comment_8_07804647b6023436878756bd97a23f32._comment b/doc/todo/wishlist:___34__quiet__34___annex_get_for_centralized_use_case/comment_8_07804647b6023436878756bd97a23f32._comment new file mode 100644 index 000000000..4901a2d97 --- /dev/null +++ b/doc/todo/wishlist:___34__quiet__34___annex_get_for_centralized_use_case/comment_8_07804647b6023436878756bd97a23f32._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.154.0.140" + subject="comment 8" + date="2013-07-20T20:07:53Z" + content=""" +`dd bs=1024 count=100000 if=/dev/zero of=bigfile` obviously creates the same data each time, so if that same size empty file has previously been in a repository, and the content is still present there, the symlink will automatically point to it when it gets added. In other words, git-annex performs automatic deduplication of file contents, and so it was able to save you a download in this case. +"""]] diff --git a/doc/todo/wishlist:___39__get__39___queue_and_schedule..mdwn b/doc/todo/wishlist:___39__get__39___queue_and_schedule..mdwn new file mode 100644 index 000000000..8919bae3f --- /dev/null +++ b/doc/todo/wishlist:___39__get__39___queue_and_schedule..mdwn @@ -0,0 +1,30 @@ +During the campaign adding a chunking feature to obscure filesize for encrypted files was added to the roadmap. But there is still one potentially valuable set* of data that git-annex can help obscure: when you access your remotes. + +This data can be used to determine when a user is actively using a remote, but if a remote is always accessed at the same time that data becomes less useful. Somebody could still monitor total traffic over a long period and figure out that a remote was more active in a given week or month, but scheduling reduces the resolution of your access times and their data. Maybe this isn't the most important feature to add, but it would be nice to have, and could possibly be built on top of the existing git-annex scheduler. It could work by queuing inter-remote transactions ('get', 'copy', 'sync', etc.), so that jobs start at the same time every day, or even the same time and day every week. + +Possible syntax examples: +###Setting up the schedule: +git annex queue schedule Monday:1730 (starts every monday at 5:30PM) + +git annex queue schedule 1400 (starts every day at 2PM) + +###Queuing git-annex commands: +git annex queue prepend sync (pretends 'sync' to the very front of the queue) + +git annex queue append get file.ISO (appends to queue file.ISO for retrieval from a repo) + +###Viewing/editing queue: +git annex queue view (view the current queue, jobs displayed with corresponding numbers) + +git annex queue rm 20 (removes job 20 from queue) + + +\*The four I can think of are: + +* File contents (solved by crypto) + +* File size (solved on the remote by chunking, but total traffic usage can't be helped) + +* User IP/Remote IP (solved by VPN - outside scope of git-annex, unless someone writes a plugin) + +* Access times (obscured by a queue and scheduling) diff --git a/doc/todo/wishlist:___39__whereis__39___support_in_the_webapp.mdwn b/doc/todo/wishlist:___39__whereis__39___support_in_the_webapp.mdwn new file mode 100644 index 000000000..c074988b0 --- /dev/null +++ b/doc/todo/wishlist:___39__whereis__39___support_in_the_webapp.mdwn @@ -0,0 +1,4 @@ +I've looked for this feature in the webapp but can't find it... + +I mainly use the webapp and have been wondering 'how many copies of file X are there' and 'where are the copies of file Y'? +This is available in the commandline interface but it would be nice to have this in the webapp too. diff --git a/doc/todo/wishlist:___96__git_annex_drop_--relaxed__96__.mdwn b/doc/todo/wishlist:___96__git_annex_drop_--relaxed__96__.mdwn new file mode 100644 index 000000000..626f5a03f --- /dev/null +++ b/doc/todo/wishlist:___96__git_annex_drop_--relaxed__96__.mdwn @@ -0,0 +1,5 @@ +Also suggested during the first Gitify BoF during DebConf13: + +`git annex drop` deletes immediately. In some situations a mechanism to tell git-annex "I would like to hold onto this data if possible, but if you need the space, please delete it" could be nice. + +An obvious question would be how to do cleanups. With the assistant, that's easy. On CLI, at the very least `git annex fsck` should list, and optionally delete, that data. diff --git a/doc/todo/wishlist:___96__git_annex_drop_--relaxed__96__/comment_1_c83a6cddd0ce222205a149cfa41ca395._comment b/doc/todo/wishlist:___96__git_annex_drop_--relaxed__96__/comment_1_c83a6cddd0ce222205a149cfa41ca395._comment new file mode 100644 index 000000000..32d0c0112 --- /dev/null +++ b/doc/todo/wishlist:___96__git_annex_drop_--relaxed__96__/comment_1_c83a6cddd0ce222205a149cfa41ca395._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://cstork.org/" + nickname="Chris Stork" + subject="How should this interact with the trust model and location tracking?" + date="2013-10-04T11:13:11Z" + content=""" +This could become complicated. AFAIU, right now git-annex keeps track of files as either present or absent. With this feature it's tempting to introduce a third state 'potentially dropped' (or 'dropped in a relaxed fashion') but do you then treat them as if they were dropped depending in wether they are on a trusted or untrusted repo? Or maybe a potentially dropped file in a trusted repo is treated as a file in a semitrusted repo? This becomes convoluted. You also need a command to undrop a file in case you decide that you really want to keep it and in order to do this you need a command to see which files are up for relaxed dropping.... + +As an alternative approach maybe it makes sense to extend [[preferred content]] expressions to take file sizes and disk usage into account. +"""]] diff --git a/doc/todo/wishlist:___96__git_annex_drop_--relaxed__96__/comment_2_353fbc2bcc40cb8c9af42907a34c6e5a._comment b/doc/todo/wishlist:___96__git_annex_drop_--relaxed__96__/comment_2_353fbc2bcc40cb8c9af42907a34c6e5a._comment new file mode 100644 index 000000000..40f299fa0 --- /dev/null +++ b/doc/todo/wishlist:___96__git_annex_drop_--relaxed__96__/comment_2_353fbc2bcc40cb8c9af42907a34c6e5a._comment @@ -0,0 +1,11 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.152.108.243" + subject="comment 2" + date="2013-10-04T20:17:07Z" + content=""" +I don't think that a third state would be necessary. Actually dropping the file when it happens would need to do the same numcopies verification that `git annex drop` does now. + +I agree it might be simpler to first improve the power of preferred content expressions. Unfortunately one thing that cannot be put in them is anything that probes the current state of the system. This is because repo A on machine X needs to be able to calculate the preferred content of repo B on machine Y. +But I could certainly add file size as a preferred content term, since that info is known throughout the network. +"""]] diff --git a/doc/todo/wishlist:___96__git_annex_fsck_--checksums__96___--_verify_checksums_but_disregard_annex.numcopies.mdwn b/doc/todo/wishlist:___96__git_annex_fsck_--checksums__96___--_verify_checksums_but_disregard_annex.numcopies.mdwn new file mode 100644 index 000000000..67a7e13e1 --- /dev/null +++ b/doc/todo/wishlist:___96__git_annex_fsck_--checksums__96___--_verify_checksums_but_disregard_annex.numcopies.mdwn @@ -0,0 +1,12 @@ +As the title says, I would like to see an option where git-annex verifies +that all checksums are OK but not that the required number of copies or +other possible metrics are fulfilled. + +-- RichiH + +> --numcopies is provided for times when you want to temporarily override +> annex.numcopies. So, `git annex fsck --numcopies=0` +> +> I don't see any reason to want to disable the other checks and fixups +> fsck does (of bad location log data, for example). So, [[done]] +> --[[Joey]] diff --git a/doc/todo/wishlist:___96__git_annex_fsck_--checksums__96___--_verify_checksums_but_disregard_annex.numcopies/comment_1_6bcf067e4860bdfeb1d7b9fd1702a43a._comment b/doc/todo/wishlist:___96__git_annex_fsck_--checksums__96___--_verify_checksums_but_disregard_annex.numcopies/comment_1_6bcf067e4860bdfeb1d7b9fd1702a43a._comment new file mode 100644 index 000000000..c8f40caf7 --- /dev/null +++ b/doc/todo/wishlist:___96__git_annex_fsck_--checksums__96___--_verify_checksums_but_disregard_annex.numcopies/comment_1_6bcf067e4860bdfeb1d7b9fd1702a43a._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U" + nickname="Richard" + subject="comment 1" + date="2013-07-10T23:09:31Z" + content=""" +As a side note, --numcopies was broken, but it's been fixed with 4.20130709. +"""]] diff --git a/doc/todo/wishlist:___96__git_annex_import__96___--_An_easy_way_to_get_data_into_an_annex.mdwn b/doc/todo/wishlist:___96__git_annex_import__96___--_An_easy_way_to_get_data_into_an_annex.mdwn new file mode 100644 index 000000000..cd679485b --- /dev/null +++ b/doc/todo/wishlist:___96__git_annex_import__96___--_An_easy_way_to_get_data_into_an_annex.mdwn @@ -0,0 +1,13 @@ +`git annex import` would copy data over from external places into the annex. It would be run from within the annex and in the target location where the files need to end up. + +Two basic modes of operation: + +* If run on a normal directory, e.g. an SD card, it would simply copy over and `git annex add $newstuff` + +* If run on another indirect annex, it would copy over the symlinks, copy over the object data, verify that the checksums are OK and add to the annex + +An optional `git annex import --copy-only` would copy over and verify the data, but not yet add it. That would allow the user to import into a decent data structure. If run on non-annexed data, `git annex import --copy-only` would ideally calculate checksums and create symlinks already; thus ensuring data integrity as early as possible. + +-- RichiH + +> [[done]] --[[Joey]] in 2012 diff --git a/doc/todo/wishlist:___96__git_annex_import__96___--_An_easy_way_to_get_data_into_an_annex/comment_1_b9fd1bfaf9a3d238fdb7bc9c2d75fe5f._comment b/doc/todo/wishlist:___96__git_annex_import__96___--_An_easy_way_to_get_data_into_an_annex/comment_1_b9fd1bfaf9a3d238fdb7bc9c2d75fe5f._comment new file mode 100644 index 000000000..ff9030c99 --- /dev/null +++ b/doc/todo/wishlist:___96__git_annex_import__96___--_An_easy_way_to_get_data_into_an_annex/comment_1_b9fd1bfaf9a3d238fdb7bc9c2d75fe5f._comment @@ -0,0 +1,22 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.153.254.222" + subject="comment 1" + date="2013-07-07T18:09:20Z" + content=""" +This is such a good idea that I went into the time machine and arranged for it to be implemented in June 2012: + +<pre> + import [path ...] + Moves files from somewhere outside the git work‐ + ing copy, and adds them to the annex. Individual + files to import can be specified. If a direc‐ + tory is specified, all files in it are imported, + and any subdirectory structure inside it is pre‐ + served. + + git annex import /media/camera/DCIM/ +</pre> + +I don't see much use for `--copy-only` though. so did not implement it them (also I needed to spend some of my time at the race track). It seems to me that using `--copy-only` as you describe it would do everything except for add the files to git. You can get the same behavior by using `git annex import`, which only stages the new files but does not commit them, and then moving files around and running `git annex add` on them, followed by committing. +"""]] diff --git a/doc/todo/wishlist:___96__git_annex_import__96___--_An_easy_way_to_get_data_into_an_annex/comment_2_56f6972413c6f0d9f414245b6f4d27b9._comment b/doc/todo/wishlist:___96__git_annex_import__96___--_An_easy_way_to_get_data_into_an_annex/comment_2_56f6972413c6f0d9f414245b6f4d27b9._comment new file mode 100644 index 000000000..ccdf2e704 --- /dev/null +++ b/doc/todo/wishlist:___96__git_annex_import__96___--_An_easy_way_to_get_data_into_an_annex/comment_2_56f6972413c6f0d9f414245b6f4d27b9._comment @@ -0,0 +1,62 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U" + nickname="Richard" + subject="comment 2" + date="2013-07-10T23:25:52Z" + content=""" +Ugh, learn to read, etc... + +It's not possible to import from other annexes, though. Importing just the files from an indirect repo does nothing: + + ~/test-annex--foo % git annex status + supported backends: SHA256E SHA1E SHA512E SHA224E SHA384E SHA256 SHA1 SHA512 SHA224 SHA384 WORM URL + supported remote types: git S3 bup directory rsync web webdav glacier hook + repository mode: indirect + trusted repositories: 0 + semitrusted repositories: 2 + 00000000-0000-0000-0000-000000000001 -- web + 8105410b-d8ca-4de4-bb6a-91b9772250dc -- here (richih@adamantium:~/test-annex--foo) + untrusted repositories: 0 + transfers in progress: none + available local disk space: 52 gigabytes (+1 megabyte reserved) + local annex keys: 1 + local annex size: 4 bytes + known annex keys: 1 + known annex size: 4 bytes + bloom filter size: 16 mebibytes (0% full) + backend usage: + SHA256E: 2 + ~/test-annex--foo % ls -l + total 4 + lrwxrwxrwx 1 richih richih 178 Jul 11 01:21 bar -> .git/annex/objects/g7/9v/SHA256E-s4--7d865e959b2466918c9863afca942d0fb89d7c9ac0c99bafc3749504ded97730/SHA256E-s4--7d865e959b2466918c9863afca942d0fb89d7c9ac0c99bafc3749504ded97730 + ~/test-annex--foo % cd ../test-annex + ~/test-annex % git annex status + supported backends: SHA256E SHA1E SHA512E SHA224E SHA384E SHA256 SHA1 SHA512 SHA224 SHA384 WORM URL + supported remote types: git S3 bup directory rsync web webdav glacier hook + repository mode: indirect + trusted repositories: (merging synced/git-annex into git-annex...) + 0 + semitrusted repositories: 3 + 00000000-0000-0000-0000-000000000001 -- web + 7c50da8c-dc76-4b4a-b46d-8dd16385691a -- here (richih@adamantium:~/test-annex) + d4104a13-a2eb-4f5c-ba54-990ece5c81df -- richih@adamantium:~/test-annex-2 + untrusted repositories: 0 + transfers in progress: none + available local disk space: 52 gigabytes (+1 megabyte reserved) + local annex keys: 1 + local annex size: 4 bytes + known annex keys: 1 + known annex size: 4 bytes + bloom filter size: 16 mebibytes (0% full) + backend usage: + SHA256E: 2 + ~/test-annex % ls -l + total 4 + lrwxrwxrwx 1 richih richih 178 Jul 11 01:20 foo -> .git/annex/objects/91/9x/SHA256E-s4--b5bb9d8014a0f9b1d61e21e796d78dccdf1352f23cd32812f4850b878ae4944c/SHA256E-s4--b5bb9d8014a0f9b1d61e21e796d78dccdf1352f23cd32812f4850b878ae4944c + ~/test-annex % git annex import ../test-annex--foo/bar + ~/test-annex % ls -l + total 4 + lrwxrwxrwx 1 richih richih 178 Jul 11 01:20 foo -> .git/annex/objects/91/9x/SHA256E-s4--b5bb9d8014a0f9b1d61e21e796d78dccdf1352f23cd32812f4850b878ae4944c/SHA256E-s4--b5bb9d8014a0f9b1d61e21e796d78dccdf1352f23cd32812f4850b878ae4944c + ~/test-annex % + +"""]] diff --git a/doc/todo/wishlist:___96__git_annex_import__96___--_An_easy_way_to_get_data_into_an_annex/comment_3_2c094bef802a2182de4fcd0def1ad29b._comment b/doc/todo/wishlist:___96__git_annex_import__96___--_An_easy_way_to_get_data_into_an_annex/comment_3_2c094bef802a2182de4fcd0def1ad29b._comment new file mode 100644 index 000000000..d3870e7c9 --- /dev/null +++ b/doc/todo/wishlist:___96__git_annex_import__96___--_An_easy_way_to_get_data_into_an_annex/comment_3_2c094bef802a2182de4fcd0def1ad29b._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U" + nickname="Richard" + subject="comment 3" + date="2013-07-11T11:54:25Z" + content=""" +To expand on this a bit: + +* I meant \"ugh, me\" not \"ugh, anyone else\"; I simply did not see it... +* `git annex import` on a separate annex should copy over the symlinks and the objects behind it and then run `git annex add`, thus verifying, fixing symlinks, etc, imo. +* Something that may not be said often enough: Thanks :) +"""]] diff --git a/doc/todo/wishlist:___96__git_annex_import__96___--_An_easy_way_to_get_data_into_an_annex/comment_4_14915c43001f7f72c9fe5119a104ef5c._comment b/doc/todo/wishlist:___96__git_annex_import__96___--_An_easy_way_to_get_data_into_an_annex/comment_4_14915c43001f7f72c9fe5119a104ef5c._comment new file mode 100644 index 000000000..4d4c1fbc5 --- /dev/null +++ b/doc/todo/wishlist:___96__git_annex_import__96___--_An_easy_way_to_get_data_into_an_annex/comment_4_14915c43001f7f72c9fe5119a104ef5c._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U" + nickname="Richard" + subject="comment 4" + date="2013-07-13T15:25:56Z" + content=""" +After more deliberation, there should probably be an option to not have `git annex import` run `git annex add` automagically (but still call `git annex fix`) so manual shuffling around of files is still possible. + +See [[doc/bugs/__96__git_annex_fix__96___run_on_non-annexed_files_is_no-op]] for more of the rationale on this. +"""]] diff --git a/doc/todo/wishlist:___96__git_annex_sync_-m__96__.mdwn b/doc/todo/wishlist:___96__git_annex_sync_-m__96__.mdwn new file mode 100644 index 000000000..92b5dee27 --- /dev/null +++ b/doc/todo/wishlist:___96__git_annex_sync_-m__96__.mdwn @@ -0,0 +1,10 @@ +Similar to how + + git commit -m 'foo' + +works, if I run + + git annex sync -m 'my hovercraft is full of eels' + +git annex should use that commit message instead of the default ones. That way, I could use [[sync]] directly and not be forced to commit prior to syncing just to make sure I have a useful commit message. + diff --git a/doc/todo/wishlist:_a_spec.remote_for_network_directories_that_would_mount_them_whenever_needed___40__e.g.__44___with_WebDAV__41__.mdwn b/doc/todo/wishlist:_a_spec.remote_for_network_directories_that_would_mount_them_whenever_needed___40__e.g.__44___with_WebDAV__41__.mdwn new file mode 100644 index 000000000..f2c4254ad --- /dev/null +++ b/doc/todo/wishlist:_a_spec.remote_for_network_directories_that_would_mount_them_whenever_needed___40__e.g.__44___with_WebDAV__41__.mdwn @@ -0,0 +1,25 @@ +I've seen the [[tips/using box.com as a special remote]] for using mounted WebDAV remote directory for storage of the tracked files. + +It's quite close to a scenario familiar to me, although with a difference. + +Let me describe my situation. + +I worked on a supplement to a set of textbooks. My work was dependent on the revision of the textbooks (for correct references, etc.). The textbooks were being edited by the author, and published in a WebDAV directory. + +So I set up a Git repo for my work, and also a branch to track the revisions of the textbooks which [I updated by copying them from the WebDAV directory (after mounting it)](http://unix.stackexchange.com/q/25015/4319). The branch where my work was in was either based on the textbooks branch (and rebased/merged and edited to reflect the changes in the new revisions of the textbooks when needed) or contained the repo with textbooks as a submodule. + +The textbooks were large files, so I didn't want them be a part of the Git repo with my supplement work when I publish the repo. But I wanted for those who looked into my public repo to understand how to get the textbooks I'm referring to. + +I haven't solved this problem for myself completely. Now I see I could use git-annex for this. I t would track the state of the textbooks in the repo, without actually storing them there, and whenever one would need to get the missing textbooks in a clone of the repo, git-annex could handle the download from the WebDAV directory for him, right? + +I simply wrote down the rules for reproducing these downloading and saving operations, together with source URL (as a [Makefile](https://gitorious.org/primary-school-informatics-problems/received_2011-11-mathinf-initial-edition/blobs/RULES/Makefile)): + +whenever I wanted to update the revisions of the textbooks (or to download them the first time), I would checkout the branch which included this Makefile and was for holding the textbooks, and the run: + + make get + +-- this target had the temporary mountpoint for the remote directory as prerequisite, and there was a rule to create it, and mount the specified URL at it; then it would sync the files, and I could use Git to track the changes. After I was done inspecting the remote directory, I had to clean up the temporary mountpoint fby unmounting and deleting it. I didn't make it do this automatically after a `get` operation for performance reasons (caching of the remote directory would help if I wanted to access it once again). + +So, this differs from [[tips/using box.com as a special remote]] in that the tip for WebDAV suggest to handle the mounting manually, and git-annex knows nothing about the WebDAV URL where the content comes from. + +So here's my wish: a [[special remote|special remotes]] to track the WebDAV URLs in the repo, and mount the remote directory automatically under the hood, whenever one wants to get a file from there. (Then I assume it should also unmount it immediately in order to clean up after itself, despite possible inefficiencies). diff --git a/doc/todo/wishlist:_a_spec.remote_for_network_directories_that_would_mount_them_whenever_needed___40__e.g.__44___with_WebDAV__41__/comment_1_f46b0c9b49607e9f4f7a27f5a331ce83._comment b/doc/todo/wishlist:_a_spec.remote_for_network_directories_that_would_mount_them_whenever_needed___40__e.g.__44___with_WebDAV__41__/comment_1_f46b0c9b49607e9f4f7a27f5a331ce83._comment new file mode 100644 index 000000000..32dd9c039 --- /dev/null +++ b/doc/todo/wishlist:_a_spec.remote_for_network_directories_that_would_mount_them_whenever_needed___40__e.g.__44___with_WebDAV__41__/comment_1_f46b0c9b49607e9f4f7a27f5a331ce83._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.153.14.141" + subject="comment 1" + date="2012-09-25T22:56:22Z" + content=""" +Note that git-annex already has the git configs `remote.<name>.annex-start-command` and `remote.<name>.annex-stop-command` which can be used to handle mounting and umounting. +"""]] diff --git a/doc/todo/wishlist:_a_spec.remote_for_network_directories_that_would_mount_them_whenever_needed___40__e.g.__44___with_WebDAV__41__/comment_2_1b34e1dd72011c65e881dec2543a0373._comment b/doc/todo/wishlist:_a_spec.remote_for_network_directories_that_would_mount_them_whenever_needed___40__e.g.__44___with_WebDAV__41__/comment_2_1b34e1dd72011c65e881dec2543a0373._comment new file mode 100644 index 000000000..80c5ed2a9 --- /dev/null +++ b/doc/todo/wishlist:_a_spec.remote_for_network_directories_that_would_mount_them_whenever_needed___40__e.g.__44___with_WebDAV__41__/comment_2_1b34e1dd72011c65e881dec2543a0373._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="http://lj.rossia.org/users/imz/" + ip="79.165.56.162" + subject="comment 2" + date="2012-09-25T23:33:23Z" + content=""" +I see, thanks for pointing at these config options! Perhaps, that'll be enough. + +I'll have to see whether the information on how to access the remote copy (the source URL and how to mount it) saved in config variables will be transferred to the clones of the repo. + +AFAIU [[location tracking]], usually, git-annex would transfer the information on where to look for copies from one repo to another. +"""]] diff --git a/doc/todo/wishlist:_add_systemd_services_file_samples_for_assistant_and_webapp.mdwn b/doc/todo/wishlist:_add_systemd_services_file_samples_for_assistant_and_webapp.mdwn new file mode 100644 index 000000000..85c8874fc --- /dev/null +++ b/doc/todo/wishlist:_add_systemd_services_file_samples_for_assistant_and_webapp.mdwn @@ -0,0 +1,6 @@ +It could be useful for Linux users and package maintainers to have systemd .service file samples to launch the assistant and the webapp. + +For multi-user systems, we could have a git-annex-webapp@username.service . + +See [systemd documentation](http://www.freedesktop.org/software/systemd/man/systemd.service.html) for informations about those files. +See [archlinux wiki](https://wiki.archlinux.org/index.php/Systemd/Services) for .service examples. diff --git a/doc/todo/wishlist:_add_systemd_services_file_samples_for_assistant_and_webapp/comment_1_b89e90f9a70748c95aaf81740a40b76e._comment b/doc/todo/wishlist:_add_systemd_services_file_samples_for_assistant_and_webapp/comment_1_b89e90f9a70748c95aaf81740a40b76e._comment new file mode 100644 index 000000000..5e8c8a374 --- /dev/null +++ b/doc/todo/wishlist:_add_systemd_services_file_samples_for_assistant_and_webapp/comment_1_b89e90f9a70748c95aaf81740a40b76e._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="64.134.31.139" + subject="comment 1" + date="2013-10-19T15:33:18Z" + content=""" +Wouldn't this involve running it as root or a dedicated user? Neither is ideal. git-annex already uses .desktop files to auto-start when the user who is using it logs in. +"""]] diff --git a/doc/todo/wishlist:_add_systemd_services_file_samples_for_assistant_and_webapp/comment_2_d64361380cb18b98ddb43ada1c9f540a._comment b/doc/todo/wishlist:_add_systemd_services_file_samples_for_assistant_and_webapp/comment_2_d64361380cb18b98ddb43ada1c9f540a._comment new file mode 100644 index 000000000..5dfeedf37 --- /dev/null +++ b/doc/todo/wishlist:_add_systemd_services_file_samples_for_assistant_and_webapp/comment_2_d64361380cb18b98ddb43ada1c9f540a._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawmVE2R20dyfPQIzdi6urVUUAXtD6eeBsr0" + nickname="Benoît" + subject="comment 2" + date="2013-10-19T15:43:36Z" + content=""" + Not necessarily. I have a systemd instance handling my userland services. See [archlinux wiki systemd/User](https://wiki.archlinux.org/index.php/Systemd/User). +"""]] diff --git a/doc/todo/wishlist:_addurl_https:.mdwn b/doc/todo/wishlist:_addurl_https:.mdwn new file mode 100644 index 000000000..0a62eda6d --- /dev/null +++ b/doc/todo/wishlist:_addurl_https:.mdwn @@ -0,0 +1,11 @@ +It would be nice if "git annex addurl" allowed https: urls, rather than just http:. +To give an example, here is a PDF file: + + https://www.fbo.gov/utils/view?id=59ba4c8aa59101a09827ab7b9a787b05 + +If you switch the https: to http: it redirects you back to https:. + +As more sites provide https: for non-secret traffic, this becomes more of an issue. + +> I've gotten rid of the use of the HTTP library, now it just uses curl. +> [[done]] --[[Joey]] diff --git a/doc/todo/wishlist:_addurl_https:/comment_1_4e8f5e1fc52c3000eb2a1dad0624906e._comment b/doc/todo/wishlist:_addurl_https:/comment_1_4e8f5e1fc52c3000eb2a1dad0624906e._comment new file mode 100644 index 000000000..fa500b1dd --- /dev/null +++ b/doc/todo/wishlist:_addurl_https:/comment_1_4e8f5e1fc52c3000eb2a1dad0624906e._comment @@ -0,0 +1,14 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="2001:4978:f:21a::2" + subject="comment 1" + date="2013-01-26T08:44:38Z" + content=""" +This works fine with \"git annex addurl\". + +However, with --fast, it fails: + + git-annex: user error (https not supported) + +This is because the Haskell HTTP library doesn't support https yet. <https://github.com/haskell/HTTP/issues/17> Seems to be very little momentum on fixing that, perhaps I need to switch the code to use http-enumerator, which does. +"""]] diff --git a/doc/todo/wishlist:_allow_configuration_of_downloader_for_addurl.mdwn b/doc/todo/wishlist:_allow_configuration_of_downloader_for_addurl.mdwn new file mode 100644 index 000000000..81f7ee4c0 --- /dev/null +++ b/doc/todo/wishlist:_allow_configuration_of_downloader_for_addurl.mdwn @@ -0,0 +1,3 @@ +It would be neat if Git annex addurl allowed a configuration option for a download manager command to do the actual download in place of wget/curl with a placeholder for the file name to save to & URL to get from (if that's all annex needs). That would allow the user to choose a graphical download manager if desired to make progress easier to monitor. The specific circumstance I'm seeing is with [[wishlist:_an___34__assistant__34___for_web-browsing_--_tracking_the_sources_of_the_downloads]]. I found that the existing Firefox addon [FlashGot](http://flashgot.net/) can run any command with arbitrary arguments including placeholders. Right now I've got a [script](https://gist.github.com/andyg0808/5342434) that changes to a user-selected directory and then runs git-annex addurl in it with the provided url. It works fine as a download manager for FlashGot. The issue is that there is no progress information for large file downloads. If git-annex could start a separate download manager to do the actual download, then the user would be able to check status at any time, even though the git-annex command was run from a GUI and not a terminal. + +> [[done]], you can use `annex.web-download-command` now. --[[Joey]] diff --git a/doc/todo/wishlist:_allow_the_same_remote_to_be_accissable_via_different_methods.mdwn b/doc/todo/wishlist:_allow_the_same_remote_to_be_accissable_via_different_methods.mdwn new file mode 100644 index 000000000..849e73cc3 --- /dev/null +++ b/doc/todo/wishlist:_allow_the_same_remote_to_be_accissable_via_different_methods.mdwn @@ -0,0 +1,5 @@ +This is a wishlist item: + +Please allow the same remote to be available via different remotes. So in my LAN my remote is available using a ssh-connection, and when I travel with my laptop, the git-annex can also reach this remote using the Jabber transport. + +> [[done]]; this has always been fully supported. --[[Joey]] diff --git a/doc/todo/wishlist:_allow_the_same_remote_to_be_accissable_via_different_methods/comment_1_abb6263f3807160222bba1122475c89c._comment b/doc/todo/wishlist:_allow_the_same_remote_to_be_accissable_via_different_methods/comment_1_abb6263f3807160222bba1122475c89c._comment new file mode 100644 index 000000000..a95ba56f9 --- /dev/null +++ b/doc/todo/wishlist:_allow_the_same_remote_to_be_accissable_via_different_methods/comment_1_abb6263f3807160222bba1122475c89c._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="2001:4978:f:21a::2" + subject="comment 1" + date="2013-08-07T16:09:26Z" + content=""" +You can have as many git remotes as you like all pointing at the same repository via different paths. git-annex fully supports this AFAIK. Are you having some problem with it? +"""]] diff --git a/doc/todo/wishlist:_allow_users_to_provide_UUID_when_running___96__git_annex_init__96__.mdwn b/doc/todo/wishlist:_allow_users_to_provide_UUID_when_running___96__git_annex_init__96__.mdwn new file mode 100644 index 000000000..0dc9ec08a --- /dev/null +++ b/doc/todo/wishlist:_allow_users_to_provide_UUID_when_running___96__git_annex_init__96__.mdwn @@ -0,0 +1,5 @@ +As there's no way to permanently hide remotes and I have to recreate two repos now, I would love to be able to re-use the old UUIDs to remove clutter. + +> git-annex already provides a way to do this: Copy `.git/config` from the +> original repo (or use `git-config` to set `annex.uuid`) *before* running +> `git annex init`. [[done]] --[[Joey]] diff --git a/doc/todo/wishlist:_an___34__assistant__34___for_web-browsing_--_tracking_the_sources_of_the_downloads.mdwn b/doc/todo/wishlist:_an___34__assistant__34___for_web-browsing_--_tracking_the_sources_of_the_downloads.mdwn new file mode 100644 index 000000000..c910ace83 --- /dev/null +++ b/doc/todo/wishlist:_an___34__assistant__34___for_web-browsing_--_tracking_the_sources_of_the_downloads.mdwn @@ -0,0 +1,28 @@ +A replacement for a web-browser's downloads menu that uses git-annex internally ([[`addurl`|tips/using the web as a special remote]] command for the download, and `drop` or `git rm` for the clean up) would be quite helpful: + +say, when working on a topic, writing a paper or similar things, I usually have a Git repo with my text, all relevant data and processing code, and possibly other background information. It's nice to store the literature you needed at the same place where you work. (So that it is easy to catch up with what I was doing and thinking over when I left this work aside for a while, perhaps after cloning the repo from another location.) + +When I find an interesting literature, I save the file to the directory with my work, and read it. Then I might return to it to re-read it. There might be references to this document from my work, so I'd like them to work as links (perhaps pointing at the local file, but also at the source URL for the case when my document is read by someone else not on my system). + +I need to keep track of the source URLs for the documents I have saved which I read and use. + +That's a task that fits well git-annex. + +Note that doing the dull work of copying and pasting the URL and the downloading it and then opening it for reading is a pain to do every time I'm interested in a document I have found on the web. (Of course, I would need to fill out the bibliogrphic information for this document if I want to refer to it, but that can be done later. Initially, I wish I just don't lose the source URL of a document at the moment when I get interested in it and start reading.) + +So, I could be assisted by a replacement of the "downloads" menu of, say, Firefox: whenever I want to open a file for viewing (like a PDF), it should ask me where to save it, and I'd choose the directory with my work, then it should register it with git-annex (so that the source URL is saved, and perhaps it should also write down the referring page's URL somewhere nearby automatically), download it, and open with a viewer for reading. + +Then I'll have the interesting literature there when I'm offline; the source URLs would be saved, so that they can be put into the references. Also, if I distribute this work with Git, at another location git-annex can be used to easily get all the literature again. + +(Hmmm... probably, the browser that this will be simplest for me to implement for is emacs-w3m; simply, some functions calling git-annex...) + +> This seems fairly doable to implement since the git-annex [[design/assistant]] +> already has a webapp. So a javascript toolbar thing could be made that +> submits the current url (or maybe links dragged into it?) to the webapp +> for adding to the annex. +> +> The only wrinkle is that the webapp runs under a new url each time +> it starts, due to using a high port and embedding some auth token in the +> url. --[[Joey]] + +[[!tag /design/assistant]] diff --git a/doc/todo/wishlist:_an___34__assistant__34___for_web-browsing_--_tracking_the_sources_of_the_downloads/comment_1_36ae3c75053d5ec278b5e6eb2084d57a._comment b/doc/todo/wishlist:_an___34__assistant__34___for_web-browsing_--_tracking_the_sources_of_the_downloads/comment_1_36ae3c75053d5ec278b5e6eb2084d57a._comment new file mode 100644 index 000000000..4c20561c7 --- /dev/null +++ b/doc/todo/wishlist:_an___34__assistant__34___for_web-browsing_--_tracking_the_sources_of_the_downloads/comment_1_36ae3c75053d5ec278b5e6eb2084d57a._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawmBUR4O9mofxVbpb8JV9mEbVfIYv670uJo" + nickname="Justin" + subject="comment 1" + date="2012-09-25T23:55:44Z" + content=""" +You know about `git annex addurl` right? It doesn't help with the browser integration (though I bet there are existing download manager extensions you could re-use for this) but it takes care of the other use cases. +"""]] diff --git a/doc/todo/wishlist:_an___34__assistant__34___for_web-browsing_--_tracking_the_sources_of_the_downloads/comment_3_be8eb800523db8cf7a2c68a28fbf5ea5._comment b/doc/todo/wishlist:_an___34__assistant__34___for_web-browsing_--_tracking_the_sources_of_the_downloads/comment_3_be8eb800523db8cf7a2c68a28fbf5ea5._comment new file mode 100644 index 000000000..c958fd08f --- /dev/null +++ b/doc/todo/wishlist:_an___34__assistant__34___for_web-browsing_--_tracking_the_sources_of_the_downloads/comment_3_be8eb800523db8cf7a2c68a28fbf5ea5._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://m-f-k.myopenid.com/" + ip="93.207.162.28" + subject="+1" + date="2012-10-05T20:00:31Z" + content=""" +Copy+Paste+Open is to much for lazy people like me;) I like the idea of a direct browser download manager integration. This would make it so much easier to find find the original source of a downloaded file when you're to lazy to write it down somewhere in the first place … +"""]] diff --git a/doc/todo/wishlist:_an___34__assistant__34___for_web-browsing_--_tracking_the_sources_of_the_downloads/comment_3_d9f725de41a8572c85e4c6d9b4bcc927._comment b/doc/todo/wishlist:_an___34__assistant__34___for_web-browsing_--_tracking_the_sources_of_the_downloads/comment_3_d9f725de41a8572c85e4c6d9b4bcc927._comment new file mode 100644 index 000000000..30515a49b --- /dev/null +++ b/doc/todo/wishlist:_an___34__assistant__34___for_web-browsing_--_tracking_the_sources_of_the_downloads/comment_3_d9f725de41a8572c85e4c6d9b4bcc927._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://lj.rossia.org/users/imz/" + ip="79.165.56.162" + subject="the scheme to follow to use the addurl command is longer" + date="2012-09-26T00:07:11Z" + content=""" +Justin, yes, I know. To use [[addurl|tips/using the web as a special remote]], I should force myself not to click on a link to view it, but rather to copy it, paste into the command line with addurl (then it will be downloaded, and I can run a command to view it...). Not terrible, probably learnable. +"""]] diff --git a/doc/todo/wishlist:_an___34__assistant__34___for_web-browsing_--_tracking_the_sources_of_the_downloads/comment_4_f52492e4cc6f965515800bd1c0e05c90._comment b/doc/todo/wishlist:_an___34__assistant__34___for_web-browsing_--_tracking_the_sources_of_the_downloads/comment_4_f52492e4cc6f965515800bd1c0e05c90._comment new file mode 100644 index 000000000..7efd583c0 --- /dev/null +++ b/doc/todo/wishlist:_an___34__assistant__34___for_web-browsing_--_tracking_the_sources_of_the_downloads/comment_4_f52492e4cc6f965515800bd1c0e05c90._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="andy" + ip="99.48.75.171" + subject="I've just written a script that does this (with a Firefox download manager)" + date="2013-04-09T02:51:26Z" + content=""" +So I've just written a two-line shell script that makes [FlashGot](http://flashgot.net/) able to do this. It's available on GitHub at <https://gist.github.com/andyg0808/5342434>. + +This could be even neater if Git-annex had the ability to set the download manager... Wishlist created: [[wishlist:_allow_configuration_of_downloader_for_addurl]] +"""]] diff --git a/doc/todo/wishlist:_an___34__assistant__34___for_web-browsing_--_tracking_the_sources_of_the_downloads/comment_5_5b36656fc5fa124e763f06711d9da32b._comment b/doc/todo/wishlist:_an___34__assistant__34___for_web-browsing_--_tracking_the_sources_of_the_downloads/comment_5_5b36656fc5fa124e763f06711d9da32b._comment new file mode 100644 index 000000000..494aab7a8 --- /dev/null +++ b/doc/todo/wishlist:_an___34__assistant__34___for_web-browsing_--_tracking_the_sources_of_the_downloads/comment_5_5b36656fc5fa124e763f06711d9da32b._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + nickname="joey" + subject="comment 5" + date="2013-04-09T03:36:29Z" + content=""" +Wishlist implemented ;) + +A full example of doing this would make a nice addition to [[tips]] ... +"""]] diff --git a/doc/todo/wishlist:_an___34__assistant__34___for_web-browsing_--_tracking_the_sources_of_the_downloads/comment_6_285215a4466806baf85b8606f680494a._comment b/doc/todo/wishlist:_an___34__assistant__34___for_web-browsing_--_tracking_the_sources_of_the_downloads/comment_6_285215a4466806baf85b8606f680494a._comment new file mode 100644 index 000000000..d0545153f --- /dev/null +++ b/doc/todo/wishlist:_an___34__assistant__34___for_web-browsing_--_tracking_the_sources_of_the_downloads/comment_6_285215a4466806baf85b8606f680494a._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + nickname="joey" + subject="comment 6" + date="2013-04-11T15:12:00Z" + content=""" +See [[tips/Using_Git-annex_as_a_web_browsing_assistant]]. + +I am leaving this bug open for now, as some javascript that passes the url off to the webapp is still a nice idea to build. + +A solution to the webapp's url always changing could be to have a file:/// url that the webapp creates, and javascript submits to, that then passes the request off to the webapp, probably by using additional javadcript in the html file. Assuming javascript security allows this. +"""]] diff --git a/doc/todo/wishlist:_an___34__assistant__34___for_web-browsing_--_tracking_the_sources_of_the_downloads/comment_7_15bf62e46db4b84ed3156f550f03de42._comment b/doc/todo/wishlist:_an___34__assistant__34___for_web-browsing_--_tracking_the_sources_of_the_downloads/comment_7_15bf62e46db4b84ed3156f550f03de42._comment new file mode 100644 index 000000000..484db1e81 --- /dev/null +++ b/doc/todo/wishlist:_an___34__assistant__34___for_web-browsing_--_tracking_the_sources_of_the_downloads/comment_7_15bf62e46db4b84ed3156f550f03de42._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + nickname="joey" + subject="comment 7" + date="2013-04-11T15:41:16Z" + content=""" +A less round-about method would be to have the webapp listen for links dropped into its page. You could then have the webapp open in a tab, and just drag urls there to add them to the annex. + +I'm not sure if it's possible to intercept drags into a tab. Default browser behavior is certainly to redirect the tab to the url dropped into it. + +If someone finds out how to do this, I will build it. +"""]] diff --git a/doc/todo/wishlist:_annex.largefiles_configuration_in_webapp_and_sync.mdwn b/doc/todo/wishlist:_annex.largefiles_configuration_in_webapp_and_sync.mdwn new file mode 100644 index 000000000..873988eea --- /dev/null +++ b/doc/todo/wishlist:_annex.largefiles_configuration_in_webapp_and_sync.mdwn @@ -0,0 +1 @@ +The `annex.largefiles` feature is very nice to mix annexed files with normal git managed files. I'd like to be able to configure this setting on the webapp and that the configuration directive would be synchronized accross all remotes. diff --git a/doc/todo/wishlist:_annex.largefiles_configuration_in_webapp_and_sync/comment_1_db632de391ce9fce42af51a770ed3aeb._comment b/doc/todo/wishlist:_annex.largefiles_configuration_in_webapp_and_sync/comment_1_db632de391ce9fce42af51a770ed3aeb._comment new file mode 100644 index 000000000..e402d26c3 --- /dev/null +++ b/doc/todo/wishlist:_annex.largefiles_configuration_in_webapp_and_sync/comment_1_db632de391ce9fce42af51a770ed3aeb._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="209.250.56.47" + subject="comment 1" + date="2013-11-05T16:46:28Z" + content=""" +It might make sense to sync this across remotes and have it edited with `git annex vicfg` + +Putting it in the webapp would need some nice interface for constructing the underlying expression. Might be doable for at least simple filtering (ie, files larger than XX or with extensions .A, .B, .C). I tend to think of this as a setting for people comfortable with the command line though. +"""]] diff --git a/doc/todo/wishlist:_annex.largefiles_configuration_in_webapp_and_sync/comment_2_4a0931d9884054d764fde315d4fe4851._comment b/doc/todo/wishlist:_annex.largefiles_configuration_in_webapp_and_sync/comment_2_4a0931d9884054d764fde315d4fe4851._comment new file mode 100644 index 000000000..0e24014d2 --- /dev/null +++ b/doc/todo/wishlist:_annex.largefiles_configuration_in_webapp_and_sync/comment_2_4a0931d9884054d764fde315d4fe4851._comment @@ -0,0 +1,14 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawne_amN4fko4p5cRY_9EYwaYuJKNn7LRio" + nickname="Tobias" + subject="feedback" + date="2013-11-05T21:23:11Z" + content=""" +> It might make sense to sync this across remotes and have it edited with git annex vicfg + +That would be great! + +> Putting it in the webapp would need some nice interface for constructing the underlying expression. Might be doable for at least simple filtering (ie, files larger than XX or with extensions .A, .B, .C). I tend to think of this as a setting for people comfortable with the command line though. + +I could live with a simple filtering interface without too many fancy stuff. The fancy stuff could be done on the command line if needed... +"""]] diff --git a/doc/todo/wishlist:_annex.largefiles_support_for_mimetypes.mdwn b/doc/todo/wishlist:_annex.largefiles_support_for_mimetypes.mdwn new file mode 100644 index 000000000..f38e41dd3 --- /dev/null +++ b/doc/todo/wishlist:_annex.largefiles_support_for_mimetypes.mdwn @@ -0,0 +1 @@ +It would be nice to have mimetype support on the `annex.largefiles` configuration directive. F.e. `git config annex.largefiles "not mimetype=text/plain"` diff --git a/doc/todo/wishlist:_annex.largefiles_support_for_mimetypes/comment_1_304431bb62b5b8a64edc37a11bbaff59._comment b/doc/todo/wishlist:_annex.largefiles_support_for_mimetypes/comment_1_304431bb62b5b8a64edc37a11bbaff59._comment new file mode 100644 index 000000000..b06008475 --- /dev/null +++ b/doc/todo/wishlist:_annex.largefiles_support_for_mimetypes/comment_1_304431bb62b5b8a64edc37a11bbaff59._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawlhE8Xar6m4x3JSILXdm-Y5KhwHTH5qzKQ" + nickname="bruno" + subject="+1" + date="2013-10-14T14:43:11Z" + content=""" +Big +1 for this feature: this would really help configuring annex.largefiles +"""]] diff --git a/doc/todo/wishlist:_archive_from_remote_with_the_least_free_space.mdwn b/doc/todo/wishlist:_archive_from_remote_with_the_least_free_space.mdwn new file mode 100644 index 000000000..acc8b363e --- /dev/null +++ b/doc/todo/wishlist:_archive_from_remote_with_the_least_free_space.mdwn @@ -0,0 +1 @@ +An interesting feature, when an archived file cannot be removed from all clients because of the minimum number of copies required, would be to remove it from the repositories with the smallest amount of free space available. diff --git a/doc/todo/wishlist:_archive_from_remote_with_the_least_free_space/comment_1_6813fdc7ecc98765a5d35d34163a1712._comment b/doc/todo/wishlist:_archive_from_remote_with_the_least_free_space/comment_1_6813fdc7ecc98765a5d35d34163a1712._comment new file mode 100644 index 000000000..89fe4c069 --- /dev/null +++ b/doc/todo/wishlist:_archive_from_remote_with_the_least_free_space/comment_1_6813fdc7ecc98765a5d35d34163a1712._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.153.14.105" + subject="comment 1" + date="2013-09-19T21:01:30Z" + content=""" +That might be useful in some cases. git-annex can only tell how much free space is available on remotes that are mounted to the local filesystem. Did you use case involve removable drives? +"""]] diff --git a/doc/todo/wishlist:_archive_from_remote_with_the_least_free_space/comment_2_21a249cedca1ceb80d10784004735524._comment b/doc/todo/wishlist:_archive_from_remote_with_the_least_free_space/comment_2_21a249cedca1ceb80d10784004735524._comment new file mode 100644 index 000000000..d2b29c239 --- /dev/null +++ b/doc/todo/wishlist:_archive_from_remote_with_the_least_free_space/comment_2_21a249cedca1ceb80d10784004735524._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawmOsy6nbvPyXLd--qqjPMLnVIzxgZwtKlQ" + nickname="Nicolas" + subject="comment 2" + date="2013-09-20T19:03:13Z" + content=""" +I was not thinking of removable drives, but only of \"client\" repositories. Ideally, git-annex would query the remaining space of all connected client repositories to choose on which repositories to drop a copy. +"""]] diff --git a/doc/todo/wishlist:_assistant_autostart_port_and_secret_configuration.mdwn b/doc/todo/wishlist:_assistant_autostart_port_and_secret_configuration.mdwn new file mode 100644 index 000000000..f0d27d0b1 --- /dev/null +++ b/doc/todo/wishlist:_assistant_autostart_port_and_secret_configuration.mdwn @@ -0,0 +1 @@ +When starting the assistant when logging in to the system (`--autostart`) it choses a new port an secret everytime. Having the assistant open in a pinned firefox tab which automatically restores when firefox starts we need to get the url from `.git/annex/url` and copy/paste it into the pinned tab. It would be very nice to have a configuration option which assigns a fixed port and secret so everytime the assistant is autostarted it uses the same settings and firefox is happy to open it automatically on start. diff --git a/doc/todo/wishlist:_assistant_autostart_port_and_secret_configuration/comment_1_be53b8456eed7eadad5d4b8465c8a42b._comment b/doc/todo/wishlist:_assistant_autostart_port_and_secret_configuration/comment_1_be53b8456eed7eadad5d4b8465c8a42b._comment new file mode 100644 index 000000000..651937802 --- /dev/null +++ b/doc/todo/wishlist:_assistant_autostart_port_and_secret_configuration/comment_1_be53b8456eed7eadad5d4b8465c8a42b._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="209.250.56.64" + subject="comment 1" + date="2013-11-22T16:58:51Z" + content=""" +It's impossible to guarantee a program will be able to get a particular port. Any other program could be using that port number. + +This is why `git annex webapp` opens the webapp using whatever url the assistant has picked. I recommend that rather than using a pinned tab, you make an icon on your desktop that runs `git annex webapp`, and use that to open the app when you want it, much as you'd open any other app. If you wanted to, you could make your system run `git annex webapp` on startup instead of `git annex assistant --autostart`, and you'd get the browser tab opened automatically on start. +"""]] diff --git a/doc/todo/wishlist:_command_options_changes.mdwn b/doc/todo/wishlist:_command_options_changes.mdwn new file mode 100644 index 000000000..b15440181 --- /dev/null +++ b/doc/todo/wishlist:_command_options_changes.mdwn @@ -0,0 +1,17 @@ +Some suggestions for changes to command options: + + * --verbose: + * add alternate: -v + + * --from: + * replace with: -s $SOURCE || --source=$SOURCE + + * --to: + * replace with: -d $DESTINATION || --destination=$DESTINATION + + * --force: + * add alternate: -F + * "-f" was removed in v0.20110417 + * since it forces unsafe operations, should be capitalized to reduce chance of accidental usage. + +[[done]], see comments diff --git a/doc/todo/wishlist:_command_options_changes/comment_1_bfba72a696789bf21b2435dea15f967a._comment b/doc/todo/wishlist:_command_options_changes/comment_1_bfba72a696789bf21b2435dea15f967a._comment new file mode 100644 index 000000000..0ab113211 --- /dev/null +++ b/doc/todo/wishlist:_command_options_changes/comment_1_bfba72a696789bf21b2435dea15f967a._comment @@ -0,0 +1,17 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U" + nickname="Richard" + subject="comment 1" + date="2011-04-17T23:46:37Z" + content=""" +--to and --from seem to have different semantics than --source and --destination. Subtle, but still different. + +That being said, I am not sure --from and --to are needed at all. Calling the local repo . and all remotes by their name, they are arguably redundant and removing them would make the syntax a lot prettier; mv and cp don't need them, either. + +I am not sure changing syntax at this point is considered good style though personally, I wouldn't mind adapting and would actually prefer it over using --to and --from. + +-v and -q would be nice. + + +Richard +"""]] diff --git a/doc/todo/wishlist:_command_options_changes/comment_2_f6a637c78c989382e3c22d41b7fb4cc2._comment b/doc/todo/wishlist:_command_options_changes/comment_2_f6a637c78c989382e3c22d41b7fb4cc2._comment new file mode 100644 index 000000000..0072ae1d7 --- /dev/null +++ b/doc/todo/wishlist:_command_options_changes/comment_2_f6a637c78c989382e3c22d41b7fb4cc2._comment @@ -0,0 +1,19 @@ +[[!comment format=mdwn + username="http://joey.kitenet.net/" + nickname="joey" + subject="comment 2" + date="2011-04-19T20:13:10Z" + content=""" +Let's see.. + +* -v is already an alias for --verbose + +* I don't find --source and --destination as easy to type or as clear as --from or --to. + +* -F is fast, so it cannot be used for --force. And I have no desire to make it easy to mistype a short option and enable --force; it can lose data. + +@richard while it would be possible to support some syntax like \"git annex copy . remote\"; what is it supposed to do if there are local files named foo and bar, and a remotes named foo and bar? Does \"git annex copy foo bar\" copy file foo to remote bar, or file bar from remote foo? I chose to use --from/--to to specify remotes independant of files to avoid such +ambiguity, which plain old `cp` doesn't have since it's operating entirely on filesystem objects, not both filesystem objects and abstract remotes. + +Seems like nothing to do here. [[done]] --[[Joey]] +"""]] diff --git a/doc/todo/wishlist:_command_options_changes/comment_3_bf1114533d2895804e531e76eb6b8095._comment b/doc/todo/wishlist:_command_options_changes/comment_3_bf1114533d2895804e531e76eb6b8095._comment new file mode 100644 index 000000000..9fcbae6d2 --- /dev/null +++ b/doc/todo/wishlist:_command_options_changes/comment_3_bf1114533d2895804e531e76eb6b8095._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U" + nickname="Richard" + subject="comment 3" + date="2011-04-20T21:28:06Z" + content=""" +Good point. scp fixes this by using a colon, but as colons aren't needed in git-annex remotes' names... -- RichiH +"""]] diff --git a/doc/todo/wishlist:_define_remotes_that_must_have_all_files.mdwn b/doc/todo/wishlist:_define_remotes_that_must_have_all_files.mdwn new file mode 100644 index 000000000..156cfb009 --- /dev/null +++ b/doc/todo/wishlist:_define_remotes_that_must_have_all_files.mdwn @@ -0,0 +1,18 @@ +I would like to be able to name a few remotes that must retain *all* annexed +files. `git-annex fsck` should warn me if any files are missing from those +remotes, even if `annex.numcopies` has been satisfied by other remotes. + +I imagine this could also be useful for bup remotes, but I haven't actually +looked at those yet. + +Based on existing output, this is what a warning message could look like: + + fsck FILE + 3 of 3 trustworthy copies of FILE exist. + FILE is, however, still missing from these required remotes: + UUID -- Backup Drive 1 + UUID -- Backup Drive 2 + Back it up with git-annex copy. + Warning + +What do you think? diff --git a/doc/todo/wishlist:_define_remotes_that_must_have_all_files/comment_1_cceccc1a1730ac688d712b81a44e31c3._comment b/doc/todo/wishlist:_define_remotes_that_must_have_all_files/comment_1_cceccc1a1730ac688d712b81a44e31c3._comment new file mode 100644 index 000000000..1f65fd982 --- /dev/null +++ b/doc/todo/wishlist:_define_remotes_that_must_have_all_files/comment_1_cceccc1a1730ac688d712b81a44e31c3._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://joey.kitenet.net/" + nickname="joey" + subject="comment 1" + date="2011-04-23T16:27:13Z" + content=""" +Seems to have a scalability problem, what happens when such a repository becomes full? + +Another way to accomplish I think the same thing is to pick the repositories that you would include in such a set, and make all other repositories untrusted. And set numcopies as desired. Then git-annex will never remove files from the set of non-untrusted repositories, and fsck will warn if a file is present on only an untrusted repository. +"""]] diff --git a/doc/todo/wishlist:_define_remotes_that_must_have_all_files/comment_2_eec848fcf3979c03cbff2b7407c75a7a._comment b/doc/todo/wishlist:_define_remotes_that_must_have_all_files/comment_2_eec848fcf3979c03cbff2b7407c75a7a._comment new file mode 100644 index 000000000..1855cdda0 --- /dev/null +++ b/doc/todo/wishlist:_define_remotes_that_must_have_all_files/comment_2_eec848fcf3979c03cbff2b7407c75a7a._comment @@ -0,0 +1,16 @@ +[[!comment format=mdwn + username="gernot" + ip="87.79.209.169" + subject="comment 2" + date="2011-04-24T11:20:05Z" + content=""" +Right, I have thought about untrusting all but a few remotes to achieve +something similar before and I'm sure it would kind of work. It would be more +of an ugly workaround, however, because I would have to untrust remotes that +are, in reality, at least semi-trusted. That's why an extra option/attribute +for that kind of purpose/remote would be nice. + +Obviously I didn't see the scalability problem though. Good Point. Maybe I can +achieve the same thing by writing a log parsing script for myself? + +"""]] diff --git a/doc/todo/wishlist:_detection_of_merge_conflicts.mdwn b/doc/todo/wishlist:_detection_of_merge_conflicts.mdwn new file mode 100644 index 000000000..1b4caeff0 --- /dev/null +++ b/doc/todo/wishlist:_detection_of_merge_conflicts.mdwn @@ -0,0 +1,13 @@ +A conflict during sync or merge is something that requires user intervention, or at least notification. For that reason it would be nice if git annex returned a nonzero exit status when such a conflict happened during a sync or a merge. This is what git does after a conflicting pull, and would make it easier to spot a conflict in automated syncs without having to parse annex output or the logs. + +> Good idea, [[done]]. --[[Joey]] + +Also, it would be nice if your new `git annex status` were able to inform about remaining conflicts in the repo, for instance by reporting files with variant-XXX suffix. + +> Hmm, that would need a separate pass through the whole tree, since +> currently it can use `git ls-files` to find only modified/deleted/new +> files. I would rather not make the new `git annex status` slower for +> this. +> +> It would be possible to add it to `git annex info` (old `status`) +> which already has to look through the entire work tree. diff --git a/doc/todo/wishlist:_disable_automatic_commits.mdwn b/doc/todo/wishlist:_disable_automatic_commits.mdwn new file mode 100644 index 000000000..14c131796 --- /dev/null +++ b/doc/todo/wishlist:_disable_automatic_commits.mdwn @@ -0,0 +1,36 @@ +When using the [[/assistant]] on some of my repositories, I would like to +retain manual control over the granularity and contents of the commit +history. Some motivating reasons: + +* manually specified commit messages makes the history easier to follow +* make a series of minor changes to a file over a period of a few hours would result in a single commit rather than capturing intermediate incomplete edits + +* manual choice of which files to annex (based on predicted usage) could be useful, e.g. a repo might contain a 4MB PDF which you want available in *every* remote even without `git annex get`, and also some 2MB images which are only required in some remotes + +> This particular case is now catered to by the "manual" repository group +> in preferred content settings. --[[Joey]] + +Obviously this needs to be configurable at least per repository, and +ideally perhaps even per remote, since usage habits can vary from machine +to machine (e.g. I could choose to commit manually from my desktop machine +which has a nice comfy keyboard and large screen, but this would be too +much pain to do from my tiny netbook). + +In fact, this is vaguely related to [[design/assistant/partial_content]], +since the usefulness of the commit history depends on the context of the +data being manipulated, which in turn depends on which subdirectories are +being touched. So any mechanism for disabling sync per directory could +potentially be reused for disabling auto-commit per directory. + +According to Joey, it should be easy to arrange for the watcher thread not +to run, but would need some more work for the assistant to notice manual +commits in order to sync them; however the assistant already does some +crazy inotify watching of git refs, in order to detect incoming pushes, so +detecting manual commits wouldn't be a stretch. + +[[!tag design/assistant]] + +> You can do this now by pausing committing via the webapp, +> or setting `annex.autocommit=false`. +> +> The assistant probably doesn't push such commits yet. diff --git a/doc/todo/wishlist:_display_name_of_object_when_addWatcher_gets_a_permission_denied.mdwn b/doc/todo/wishlist:_display_name_of_object_when_addWatcher_gets_a_permission_denied.mdwn new file mode 100644 index 000000000..837f0a587 --- /dev/null +++ b/doc/todo/wishlist:_display_name_of_object_when_addWatcher_gets_a_permission_denied.mdwn @@ -0,0 +1,6 @@ +When addWatcher gets a permission denied, it would be helpful to display the name of the object on which the permission was denied, in the error message which shows in the webapp. + +> I have made the inotify code more robust; now it doesn't crash if it +> cannot read a directory or a file, and only logs a warning, which includes +> the directory name. +> [[done]] --[[Joey]] diff --git a/doc/todo/wishlist:_display_name_of_object_when_addWatcher_gets_a_permission_denied/comment_1_d2665e7347689b520d37561cfddf0aa8._comment b/doc/todo/wishlist:_display_name_of_object_when_addWatcher_gets_a_permission_denied/comment_1_d2665e7347689b520d37561cfddf0aa8._comment new file mode 100644 index 000000000..de0528855 --- /dev/null +++ b/doc/todo/wishlist:_display_name_of_object_when_addWatcher_gets_a_permission_denied/comment_1_d2665e7347689b520d37561cfddf0aa8._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.152.108.220" + subject="comment 1" + date="2013-09-25T18:47:13Z" + content=""" +This is an exception from the inotify library, which is what contains the `addWatch` function. I catch and display the exception. Since `addWatch` is only passed a directory to watch, the most I could do is tack on the name of the directory when displaying the exception. That does not seem likely to be much help? +"""]] diff --git a/doc/todo/wishlist:_display_name_of_object_when_addWatcher_gets_a_permission_denied/comment_2_db153571a32fb072453ed583e3e9ccf4._comment b/doc/todo/wishlist:_display_name_of_object_when_addWatcher_gets_a_permission_denied/comment_2_db153571a32fb072453ed583e3e9ccf4._comment new file mode 100644 index 000000000..e0199a42d --- /dev/null +++ b/doc/todo/wishlist:_display_name_of_object_when_addWatcher_gets_a_permission_denied/comment_2_db153571a32fb072453ed583e3e9ccf4._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawl-g0hYpGY11pBP_42lHh5GWTyFuB4UwH8" + nickname="Nicolas" + subject="comment 2" + date="2013-09-25T23:08:56Z" + content=""" +Well, of course it would not be as helpful as if the inotify exception would contain the name of the exact object on which it got a permission denied (would this be a valid wishlist request for inotify?), but I think that displaying the name of the directory would already be better than nothing. +"""]] diff --git a/doc/todo/wishlist:_display_status_of_remotes_in_the_webapp.mdwn b/doc/todo/wishlist:_display_status_of_remotes_in_the_webapp.mdwn new file mode 100644 index 000000000..741466994 --- /dev/null +++ b/doc/todo/wishlist:_display_status_of_remotes_in_the_webapp.mdwn @@ -0,0 +1 @@ +It would be nice to have an indication of the status of the remotes in the webapp, for example with a field showing "In Sync", "Syncing", or the date of the last successful synchronization for unreachable remotes. diff --git a/doc/todo/wishlist:_do_round_robin_downloading_of_data.mdwn b/doc/todo/wishlist:_do_round_robin_downloading_of_data.mdwn new file mode 100644 index 000000000..6299899e4 --- /dev/null +++ b/doc/todo/wishlist:_do_round_robin_downloading_of_data.mdwn @@ -0,0 +1,5 @@ +Given that git/config will have information on remotes and maybe costs, it might be a good idea to do a simple round robin selection of remotes to download files where the costs are the same. + +This of course assumes that we like the idea of "parallel" launching and running of curl/rsync processes... + +This wish item is probably only useful for the paranoid people who store more than 1 copy of their data. diff --git a/doc/todo/wishlist:_do_round_robin_downloading_of_data/comment_1_460335b0e59ad03871c524f1fe812357._comment b/doc/todo/wishlist:_do_round_robin_downloading_of_data/comment_1_460335b0e59ad03871c524f1fe812357._comment new file mode 100644 index 000000000..6a5fd3d53 --- /dev/null +++ b/doc/todo/wishlist:_do_round_robin_downloading_of_data/comment_1_460335b0e59ad03871c524f1fe812357._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joey.kitenet.net/" + nickname="joey" + subject="comment 1" + date="2011-04-03T16:39:35Z" + content=""" +I dunno about parrallel downloads -- eek! -- but there is at least room for improvement of what \"git annex get\" does when there are multiple remotes that have a file, and the one it decides to use is not available, or very slow, or whatever. +"""]] diff --git a/doc/todo/wishlist:_dropping_git-annex_history.mdwn b/doc/todo/wishlist:_dropping_git-annex_history.mdwn new file mode 100644 index 000000000..8286699c7 --- /dev/null +++ b/doc/todo/wishlist:_dropping_git-annex_history.mdwn @@ -0,0 +1,28 @@ +In real life discussions with git-annex users at DebConf, the idea was proposed to have a way to drop the history of the git-annex branch, and replace it with a new branch with just the current state of the repository. + +The only thing that breaks when this is done, in theory, is `git annex log`, which can't show the location history +of files. + +The crucial thing is that this operation would only need to be done in one repository, and it would then record some information in its (new) git-annex branch, so when it was pushed to other repositories, git-annex there could notice that history had been dropped, and do the same. So, even if you have rarely used offline archive repositories, the history dropping would eventually reach them, without needing to remember to do it. + +There was speculation that it would be enough to record eg, the SHA of the top commit on the old branch. That may not be good enough, because another remote may have not gotten that SHA into its branch yet, or may have commits on top of that SHA. + +Maybe instead we want to record the SHA of the *first* commit to the old git-annex branch. This way, we can tell if the branch that got deleted is the one we're currently using. And if it is, we create a new branch with the current state of *our* branch, and then union merge the other branch into it. + +Hmm, another wrinkle is that, when this indication propigates from remote A to remote B, remote B may also have some git-annex branches available for remotes C and D, which have not transitioned, and E, which has transitioned already. It seems B should first union merge C and D into B, and then flatten B to B', and then union merge A and E into B'. + +I think that'd work! + +--[[Joey]] + +Will also allow dropping dead remotes from history. Just remove all +references to them when rewriting the branch. May or may not be desirable; +I sometimes care about dead remotes that I hope to one day recuscitate. +(OTOH, I can always run git annex fsck in them to get their location +tracking back, if I do manage to get them back.) + +--[[Joey]] + +See also : [[forum/safely_dropping_git-annex_history]] + +> [[done]] --[[Joey]] diff --git a/doc/todo/wishlist:_dropping_git-annex_history/comment_1_a4bee2e26b22a9bdaadc05b7227769ef._comment b/doc/todo/wishlist:_dropping_git-annex_history/comment_1_a4bee2e26b22a9bdaadc05b7227769ef._comment new file mode 100644 index 000000000..043e674ed --- /dev/null +++ b/doc/todo/wishlist:_dropping_git-annex_history/comment_1_a4bee2e26b22a9bdaadc05b7227769ef._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.154.0.63" + subject="comment 1" + date="2013-08-24T19:39:45Z" + content=""" +BTW, a motivation for this is that some of us have old repositories that have been upgrades all the way from annex.version 1 and have a lot of cruft in them because of it. (I have repos that have been upgraded from annex.version 0, but this would not help with that cruft which is on the master branch!) + +Also, people worry that eg, a large copy back and forth bloats history, and having a way to unbloat it if it ever gets actually annoyingly bloated would stop them pestering me. ;) +"""]] diff --git a/doc/todo/wishlist:_dropping_git-annex_history/comment_2_f6d750bfe0c9d8a2aa6bc218ca5c49cc._comment b/doc/todo/wishlist:_dropping_git-annex_history/comment_2_f6d750bfe0c9d8a2aa6bc218ca5c49cc._comment new file mode 100644 index 000000000..a60973b82 --- /dev/null +++ b/doc/todo/wishlist:_dropping_git-annex_history/comment_2_f6d750bfe0c9d8a2aa6bc218ca5c49cc._comment @@ -0,0 +1,14 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U" + nickname="Richard" + subject="comment 2" + date="2013-08-27T20:02:23Z" + content=""" +If starting commit id _and_ commit id from when history is being dropped are documented, you could potentially drop more data. + +* Don't have any commits in common? Full merge. +* Only share the starting ids? Reduce local history as much as possible, and then merge. +* Share both starting id and have the last id somewhere in history? Take history from last id up to current, reduce that, and merge. + +-- RichiH +"""]] diff --git a/doc/todo/wishlist:_encrypted_git_remote_on_hosting_site_from_webapp.mdwn b/doc/todo/wishlist:_encrypted_git_remote_on_hosting_site_from_webapp.mdwn new file mode 100644 index 000000000..b7b1bad0c --- /dev/null +++ b/doc/todo/wishlist:_encrypted_git_remote_on_hosting_site_from_webapp.mdwn @@ -0,0 +1 @@ +It would be great to be able to do **private encrypted git remote on hosting site** and **multiuser encrypted git remote on hosting site** as explained in [[tips/fully encrypted git repositories with gcrypt]] through the webapp. I think it's a pretty common usecase that can be very useful for people not owning a proper server. diff --git a/doc/todo/wishlist:_generic_annex.cost-command.mdwn b/doc/todo/wishlist:_generic_annex.cost-command.mdwn new file mode 100644 index 000000000..6adf1460e --- /dev/null +++ b/doc/todo/wishlist:_generic_annex.cost-command.mdwn @@ -0,0 +1,17 @@ +### Current setup + +ATM git-annex has + +remote.<name>.annex-cost +remote.<name>.annex-cost-command # command is not provided cmdline options by annex + +to set the cost for a given remote. That requires setting up one of those variables per each host, and possibly hardcoding options for the annex-cost-command providing e.g. the remote name. + +### Suggestion + +wouldn't it be more general and thus more flexible to have a repository-wide + +annex.cost-command + +which could take options %remote, %file and assessed accordingly per each file upon '--get' request to allow maximal flexibility: e.g. some files might better be fetched from remotes supporting transfer compression, some from the web, etc. Also it might be worth providing %remote_kind ("special" vs "git") to disambiguate %remote's? + diff --git a/doc/todo/wishlist:_git-annex_replicate.mdwn b/doc/todo/wishlist:_git-annex_replicate.mdwn new file mode 100644 index 000000000..0d926b337 --- /dev/null +++ b/doc/todo/wishlist:_git-annex_replicate.mdwn @@ -0,0 +1,12 @@ +I'd like to be able to do something like the following: + + * Create encrypted git-annex remotes on a couple of semi-trusted machines - ones that have good connectivity, but non-redundant hardware + * set numcopies=3 + * run `git-annex replicate` and have git-annex run the appropriate copy commands to make sure every file is on at least 3 machines + +There would also likely be a `git annex rebalance` command which could be used if remotes were added or removed. If possible, it should copy files between servers directly, rather than proxy through a potentially slow client. + +There might be the need to have a 'replication_priority' option for each remote that configures which machines would be preferred. That way you could set your local server to a high priority to ensure that it is always 1 of the 3 machines used and files are distributed across 2 of the remaining remotes. Other than priority, other options that might help: + + * maxspace - A self imposed quota per remote machine. git-annex replicate should try to replicate files first to machines with more free space. maxspace would change the free space calculation to be `min(actual_free_space, maxspace - space_used_by_git_annex) + * bandwidth - when replication files, copies should be done between machines with the highest available bandwidth. ( I think this option could be useful for git-annex get in general) diff --git a/doc/todo/wishlist:_git-annex_replicate/comment_1_9926132ec6052760cdf28518a24e2358._comment b/doc/todo/wishlist:_git-annex_replicate/comment_1_9926132ec6052760cdf28518a24e2358._comment new file mode 100644 index 000000000..cec971ee3 --- /dev/null +++ b/doc/todo/wishlist:_git-annex_replicate/comment_1_9926132ec6052760cdf28518a24e2358._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U" + nickname="Richard" + subject="comment 1" + date="2011-04-22T18:27:00Z" + content=""" +While having remotes redistribute introduces some obvious security concerns, I might use it. + +As remotes support a cost factor already, you can basically implement bandwidth through that. +"""]] diff --git a/doc/todo/wishlist:_git-annex_replicate/comment_2_c43932f4194aba8fb2470b18e0817599._comment b/doc/todo/wishlist:_git-annex_replicate/comment_2_c43932f4194aba8fb2470b18e0817599._comment new file mode 100644 index 000000000..9d50d1531 --- /dev/null +++ b/doc/todo/wishlist:_git-annex_replicate/comment_2_c43932f4194aba8fb2470b18e0817599._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="http://joey.kitenet.net/" + nickname="joey" + subject="comment 2" + date="2011-04-23T16:22:07Z" + content=""" +Besides the cost values, annex.diskreserve was recently added. (But is not available for special remotes.) + +I have held off on adding high-level management stuff like this to git-annex, as it's hard to make it generic enough to cover use cases. + +A low-level way to accomplish this would be to have a way for `git annex get` and/or `copy` to skip files when `numcopies` is already satisfied. Then cron jobs could be used. +"""]] diff --git a/doc/todo/wishlist:_git-annex_replicate/comment_3_c13f4f9c3d5884fc6255fd04feadc2b1._comment b/doc/todo/wishlist:_git-annex_replicate/comment_3_c13f4f9c3d5884fc6255fd04feadc2b1._comment new file mode 100644 index 000000000..e7eb06b3b --- /dev/null +++ b/doc/todo/wishlist:_git-annex_replicate/comment_3_c13f4f9c3d5884fc6255fd04feadc2b1._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawmBUR4O9mofxVbpb8JV9mEbVfIYv670uJo" + nickname="Justin" + subject="comment 3" + date="2011-04-23T17:54:42Z" + content=""" +Hmm, so it seems there is almost a way to do this already. + +I think the one thing that isn't currently possible is to have 'plain' ssh remotes.. basically something just like the directory remote, but able to take a ssh user@host/path url. something like sshfs could be used to fake this, but for things like fsck you would want to do the sha1 calculations on the remote host. +"""]] diff --git a/doc/todo/wishlist:_git-annex_replicate/comment_4_63f24abf086d644dced8b01e1a9948c9._comment b/doc/todo/wishlist:_git-annex_replicate/comment_4_63f24abf086d644dced8b01e1a9948c9._comment new file mode 100644 index 000000000..3805464a6 --- /dev/null +++ b/doc/todo/wishlist:_git-annex_replicate/comment_4_63f24abf086d644dced8b01e1a9948c9._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joey.kitenet.net/" + nickname="joey" + subject="comment 4" + date="2011-09-19T18:54:46Z" + content=""" +git annex get/copy/drop all now support a --auto flag, which makes them only act on files that have not enough or too many copies. This allows for some crude replication; it doesn't take into account which repositories should be filled up more (beyond honoring annex.diskreserve), nor does it try to optimally use bandwidth (beyond honoring configured annex-cost). You have to run it in every repository that you want to participate in the replication, too. But it's probably a Good Enough solution. See [[walkthrough/automatically_managing_content]]. +"""]] diff --git a/doc/todo/wishlist:_git_annex_diff.mdwn b/doc/todo/wishlist:_git_annex_diff.mdwn new file mode 100644 index 000000000..4acfee255 --- /dev/null +++ b/doc/todo/wishlist:_git_annex_diff.mdwn @@ -0,0 +1,9 @@ +git diff is not very helpful for annexed files. + +How about a git annex diff command that allows to compare two versions of an annexed file? + +Should be relatively simple, only there would have to be a way to deal with the situation where not both versions are present in the repository. Either abort with a message showing the command you need to run to get the missing version(s). Or even interactively volunteer to get it automatically, asking the user for confirmation. + +Of course you wouldn't want to diff two large files, but with git annex assistant, all files are annexed by default (right?), so this would be useful. + +There might already be a way to easily diff two versions of an annexed file which I'm missing -- in that case please point me to it! :) diff --git a/doc/todo/wishlist:_git_annex_put_--_same_as_get__44___but_for_defaults.mdwn b/doc/todo/wishlist:_git_annex_put_--_same_as_get__44___but_for_defaults.mdwn new file mode 100644 index 000000000..9cd56749e --- /dev/null +++ b/doc/todo/wishlist:_git_annex_put_--_same_as_get__44___but_for_defaults.mdwn @@ -0,0 +1,17 @@ +I am running centralized git-annex exclusively. + +Similar to + + git annex get + +I'd like to have a + + git annex put + +which would put all files on the default remote(s). + +My main reason for not wanting to use copy --to is that I need to specify the remote's name in this case which makes writing a wrapper unnecessarily hard. Also, this would allow + + mr push + +to do the right thing all by itself. diff --git a/doc/todo/wishlist:_git_annex_put_--_same_as_get__44___but_for_defaults/comment_1_d5413c8acce308505e4e2bec82fb1261._comment b/doc/todo/wishlist:_git_annex_put_--_same_as_get__44___but_for_defaults/comment_1_d5413c8acce308505e4e2bec82fb1261._comment new file mode 100644 index 000000000..fe1d5520f --- /dev/null +++ b/doc/todo/wishlist:_git_annex_put_--_same_as_get__44___but_for_defaults/comment_1_d5413c8acce308505e4e2bec82fb1261._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://joey.kitenet.net/" + nickname="joey" + subject="comment 1" + date="2011-04-04T18:13:46Z" + content=""" +This begs the question: What is the default remote? It's probably *not* the same repository that git's master branch is tracking (ie, origin/master). It seems there would have to be an annex.defaultremote setting. + +BTW, mr can easily be configured on a per-repo basis so that \"mr push\" copies to somewhere: `push = git push; git annex push wherever` +"""]] diff --git a/doc/todo/wishlist:_git_annex_put_--_same_as_get__44___but_for_defaults/comment_2_0aa227c85d34dfff4e94febca44abea8._comment b/doc/todo/wishlist:_git_annex_put_--_same_as_get__44___but_for_defaults/comment_2_0aa227c85d34dfff4e94febca44abea8._comment new file mode 100644 index 000000000..3090b575b --- /dev/null +++ b/doc/todo/wishlist:_git_annex_put_--_same_as_get__44___but_for_defaults/comment_2_0aa227c85d34dfff4e94febca44abea8._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U" + nickname="Richard" + subject="comment 2" + date="2011-04-04T20:45:30Z" + content=""" +In my case, the remotes are the same, but adding a new option could make sense. + +And while I can tell mr what to do explicitly, I would prefer if it did the right thing all by itself. Having to change configs in two separate places is less than ideal. + +I am not sure what you mean by `git annex push` as that does not exist. Did you mean copy? +"""]] diff --git a/doc/todo/wishlist:_git_annex_put_--_same_as_get__44___but_for_defaults/comment_3_2082f4d708a584a1403cc1d4d005fb56._comment b/doc/todo/wishlist:_git_annex_put_--_same_as_get__44___but_for_defaults/comment_3_2082f4d708a584a1403cc1d4d005fb56._comment new file mode 100644 index 000000000..01dc7813f --- /dev/null +++ b/doc/todo/wishlist:_git_annex_put_--_same_as_get__44___but_for_defaults/comment_3_2082f4d708a584a1403cc1d4d005fb56._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U" + nickname="Richard" + subject="comment 1" + date="2011-04-04T10:28:01Z" + content=""" +Going one step further, a --min-copy could put all files so that numcopies is satisfied. --all could push to all available ones. + +To take everything another step further, if it was possible to group remotes, one could act on the groups. \"all\" would be an obvious choice for a group that always exists, everything else would be set up by the user. +"""]] diff --git a/doc/todo/wishlist:_git_annex_status.mdwn b/doc/todo/wishlist:_git_annex_status.mdwn new file mode 100644 index 000000000..6bb5d71f1 --- /dev/null +++ b/doc/todo/wishlist:_git_annex_status.mdwn @@ -0,0 +1,21 @@ +Ideally, it would look similar to this. And yes, I put "put" in there ;) + + non-annex % git annex status + git annex status: error: not a git annex repository + annex % git annex status + annex object storage version: A + annex backend engine: {WORM,SHA512,...} + Estimated local annex size: B MiB + Estimated total annex size: C MiB + Files without file size information in local annex: D + Files without file size information in total annex: E + Last fsck: datetime + Last git pull: datetime - $annex_name + Last git push: datetime - $annex_name + Last git annex get: datetime - $annex_name + Last git annex put: datetime - $annex_name + annex % + +Datetime could be ISO's YYYY-MM-DDThh:mm:ss or, personal preference, YYYY-MM-DD--hh-mm-ss. I prefer the latter as it's DNS-, tag- and filename-safe which is why I am using it for everything. In a perfect world, ISO would standardize YYYY-MM-DD-T-hh-mm-ss-Z[-SSSSSSSS][--$timezone], but meh. + +[[done]] diff --git a/doc/todo/wishlist:_git_annex_status/comment_1_994bfd12c5d82e08040d6116915c5090._comment b/doc/todo/wishlist:_git_annex_status/comment_1_994bfd12c5d82e08040d6116915c5090._comment new file mode 100644 index 000000000..7b5e7bd44 --- /dev/null +++ b/doc/todo/wishlist:_git_annex_status/comment_1_994bfd12c5d82e08040d6116915c5090._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawkSq2FDpK2n66QRUxtqqdbyDuwgbQmUWus" + nickname="Jimmy" + subject="comment 1" + date="2011-04-08T07:23:08Z" + content=""" ++1 for this feature, I've been longing for something like this other than rolling my own perl/shell scripts to parse the outputs of \"git annex whereis .\" to see how many files are on my machine or not. +"""]] diff --git a/doc/todo/wishlist:_git_annex_status/comment_2_c2b0ce025805b774dc77ce264a222824._comment b/doc/todo/wishlist:_git_annex_status/comment_2_c2b0ce025805b774dc77ce264a222824._comment new file mode 100644 index 000000000..21f9d713c --- /dev/null +++ b/doc/todo/wishlist:_git_annex_status/comment_2_c2b0ce025805b774dc77ce264a222824._comment @@ -0,0 +1,13 @@ +[[!comment format=mdwn + username="http://christian.amsuess.com/chrysn" + nickname="chrysn" + subject="format, respect working directory" + date="2011-04-26T12:31:02Z" + content=""" +we could include the information about the current directory as well, if the command is not issued in the local git root directory. to avoid large numbers of similar lines, that could look like this: + + Estimated annex size: B MiB (of C MiB; [B/C]%) + Estimated annex size in $PWD: B' MiB (of C' MiB; [B'/C']%) + +with the percentages being replaced with \"complete\" if really all files are present (and not just many enough for the value to be rounded to 100%). +"""]] diff --git a/doc/todo/wishlist:_git_annex_status/comment_3_d1fd70c67243971c96d59e1ffb7ef6e7._comment b/doc/todo/wishlist:_git_annex_status/comment_3_d1fd70c67243971c96d59e1ffb7ef6e7._comment new file mode 100644 index 000000000..39986144b --- /dev/null +++ b/doc/todo/wishlist:_git_annex_status/comment_3_d1fd70c67243971c96d59e1ffb7ef6e7._comment @@ -0,0 +1,23 @@ +[[!comment format=mdwn + username="http://joey.kitenet.net/" + nickname="joey" + subject="comment 3" + date="2011-05-17T01:15:10Z" + content=""" +What a good idea! + +150 lines of haskell later, I have this: + +<pre> +# git annex status +supported backends: WORM SHA1 SHA256 SHA512 SHA224 SHA384 SHA1E SHA256E SHA512E SHA224E SHA384E URL +supported remote types: git S3 bup directory rsync hook +local annex keys: 32 +local annex size: 58 megabytes +total annex keys: 38158 +total annex size: 6 terabytes (but 1632 keys have unknown size) +backend usage: + SHA1: 1789 + WORM: 36369 +</pre> +"""]] diff --git a/doc/todo/wishlist:_git_annex_status/comment_4_9aeeb83d202dc8fb33ff364b0705ad94._comment b/doc/todo/wishlist:_git_annex_status/comment_4_9aeeb83d202dc8fb33ff364b0705ad94._comment new file mode 100644 index 000000000..f006f88a0 --- /dev/null +++ b/doc/todo/wishlist:_git_annex_status/comment_4_9aeeb83d202dc8fb33ff364b0705ad94._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://christian.amsuess.com/chrysn" + nickname="chrysn" + subject="status of other remotes?" + date="2011-06-15T08:39:24Z" + content=""" +using the location tracking information, it should be possible to show the status of other remotes as well. what about supporting `--from=...` or `--all`? (thus, among other things, one could determine if a remote has a complete checkout.) +"""]] diff --git a/doc/todo/wishlist:_git_backend_for_git-annex.mdwn b/doc/todo/wishlist:_git_backend_for_git-annex.mdwn new file mode 100644 index 000000000..fd1b40bff --- /dev/null +++ b/doc/todo/wishlist:_git_backend_for_git-annex.mdwn @@ -0,0 +1,9 @@ +Preamble: Obviously, the core feature of git-annex is the ability to keep a subset of files in a local repo. The main trade-off is that you don't get version tracking. + +Use case: On my laptop, I might not have enough disk space to store everything. Not so for my main box nor my backup server. And I would _really_ like to have proper version tracking for many of my files. Thus... + +Wish: ...why not use git as a version backend? That way, I could just push all my stuff to the central instance(s) and have the best of both worlds. Depending on what backend is used in the local repos, it might make sense to define a list of supported client backends with pre-computed keys. + +-- RichiH + +[[done]] (bup) diff --git a/doc/todo/wishlist:_git_backend_for_git-annex/comment_1_04319051fedc583e6c326bb21fcce5a5._comment b/doc/todo/wishlist:_git_backend_for_git-annex/comment_1_04319051fedc583e6c326bb21fcce5a5._comment new file mode 100644 index 000000000..a691393b1 --- /dev/null +++ b/doc/todo/wishlist:_git_backend_for_git-annex/comment_1_04319051fedc583e6c326bb21fcce5a5._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://joey.kitenet.net/" + nickname="joey" + subject="comment 1" + date="2011-03-28T16:01:30Z" + content=""" +Indeed, see [[todo/add_a_git_backend]], where you and I have already discussed this idea. :) + +With the new support for special remotes, which will be used by S3, it would be possible to make such a git repo, using bup, be a special remote. I think it would be pretty easy to implement now. Not a priority for me though. +"""]] diff --git a/doc/todo/wishlist:_git_backend_for_git-annex/comment_2_7f529f19a47e10b571f65ab382e97fd5._comment b/doc/todo/wishlist:_git_backend_for_git-annex/comment_2_7f529f19a47e10b571f65ab382e97fd5._comment new file mode 100644 index 000000000..14798e7a7 --- /dev/null +++ b/doc/todo/wishlist:_git_backend_for_git-annex/comment_2_7f529f19a47e10b571f65ab382e97fd5._comment @@ -0,0 +1,14 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U" + nickname="Richard" + subject="comment 2" + date="2011-03-28T17:47:38Z" + content=""" +On the plus side, the past me wanted exactly what I had in mind. + +On the meh side, I really forgot about this conversation :/ + +When you say this todo is not a priority, does that mean there's no ETA at all and that it will most likely sleep for a long time? Or the almost usual \"what the heck, I will just wizard it up in two lines of haskell\"? + +-- RichiH +"""]] diff --git a/doc/todo/wishlist:_git_backend_for_git-annex/comment_3_a077bbad3e4b07cce019eb55a45330e7._comment b/doc/todo/wishlist:_git_backend_for_git-annex/comment_3_a077bbad3e4b07cce019eb55a45330e7._comment new file mode 100644 index 000000000..8c3286d27 --- /dev/null +++ b/doc/todo/wishlist:_git_backend_for_git-annex/comment_3_a077bbad3e4b07cce019eb55a45330e7._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://joey.kitenet.net/" + nickname="joey" + subject="comment 3" + date="2011-03-28T20:05:13Z" + content=""" +Probably more like 150 lines of haskell. Maybe just 50 lines if the bup repository is required to be on the same computer as the git-annex repository. + +Since I do have some repositories where I'd appreciate this level of assurance that data not be lost, it's mostly a matter of me finding a free day. +"""]] diff --git a/doc/todo/wishlist:_git_backend_for_git-annex/comment_4_ecca429e12d734b509c671166a676c9d._comment b/doc/todo/wishlist:_git_backend_for_git-annex/comment_4_ecca429e12d734b509c671166a676c9d._comment new file mode 100644 index 000000000..cf649a8a2 --- /dev/null +++ b/doc/todo/wishlist:_git_backend_for_git-annex/comment_4_ecca429e12d734b509c671166a676c9d._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U" + nickname="Richard" + subject="comment 4" + date="2011-03-28T20:45:35Z" + content=""" +Personally, I would not mind a requirement to keep a local bup repo. I wouldn't want my data to to unncessarily complex setups, anyway. -- RichiH +"""]] diff --git a/doc/todo/wishlist:_git_backend_for_git-annex/comment_5_3459f0b41d818c23c8fb33edb89df634._comment b/doc/todo/wishlist:_git_backend_for_git-annex/comment_5_3459f0b41d818c23c8fb33edb89df634._comment new file mode 100644 index 000000000..a1300f2e6 --- /dev/null +++ b/doc/todo/wishlist:_git_backend_for_git-annex/comment_5_3459f0b41d818c23c8fb33edb89df634._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joey.kitenet.net/" + nickname="joey" + subject="comment 5" + date="2011-04-08T20:59:37Z" + content=""" +My estimates were pretty close -- the new bup special remote type took 133 lines of code, and 2 hours to write. A testament to the flexibility of the special remote infrastructure. :) +"""]] diff --git a/doc/todo/wishlist:_history_of_operations.mdwn b/doc/todo/wishlist:_history_of_operations.mdwn new file mode 100644 index 000000000..a79f500ff --- /dev/null +++ b/doc/todo/wishlist:_history_of_operations.mdwn @@ -0,0 +1,8 @@ +Hi, + +I would love to have a page of "history" or "events" in the webapp. Similar to how Dropbox or Box show it. +I've been using git-annex for my personal files for a few months now, and I feel like this is the only feature missing to start using it in a production multi-user environment. + +Thanks + +[[!tag design/assistant]] diff --git a/doc/todo/wishlist:_history_of_operations/comment_1_f9a77ce83c6f39b6272d5c577ffbb9f9._comment b/doc/todo/wishlist:_history_of_operations/comment_1_f9a77ce83c6f39b6272d5c577ffbb9f9._comment new file mode 100644 index 000000000..bb872532d --- /dev/null +++ b/doc/todo/wishlist:_history_of_operations/comment_1_f9a77ce83c6f39b6272d5c577ffbb9f9._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawlJEI45rGczFAnuM7gRSj4C6s9AS9yPZDc" + nickname="Kevin" + subject="I second this wishlist item" + date="2013-07-24T16:48:59Z" + content=""" +I too would like to see a recent history of events in the assistant webapp. A simple listing of what changed when and by whom would be sufficient. +"""]] diff --git a/doc/todo/wishlist:_incremental_unannex___40__currently_requires_twice_the_size_of_repo_to_complete__41__.mdwn b/doc/todo/wishlist:_incremental_unannex___40__currently_requires_twice_the_size_of_repo_to_complete__41__.mdwn new file mode 100644 index 000000000..420fb882c --- /dev/null +++ b/doc/todo/wishlist:_incremental_unannex___40__currently_requires_twice_the_size_of_repo_to_complete__41__.mdwn @@ -0,0 +1,10 @@ +I recently tried to use unannex for a large repo and it failed because the repo size was more than half the disk size. Unannex should work incrementally so this isn't a problem. + +Proposed solution: +copy a file, hash it, iff hash check is okay, delete from objects, continue to next file + +> That won't work, because multiple files can point to a key. +> +> I am not happy with unannex's behavior, but I was less happy +> when people were constantly filing bugs about it misbehaving in that +> situttion. If you dislike the copying, use --fast. [[done]] --[[Joey]] diff --git a/doc/todo/wishlist:_incremental_unannex___40__currently_requires_twice_the_size_of_repo_to_complete__41__/comment_1_067b29fc47d26b9da0766f9810684ae8._comment b/doc/todo/wishlist:_incremental_unannex___40__currently_requires_twice_the_size_of_repo_to_complete__41__/comment_1_067b29fc47d26b9da0766f9810684ae8._comment new file mode 100644 index 000000000..372bb621f --- /dev/null +++ b/doc/todo/wishlist:_incremental_unannex___40__currently_requires_twice_the_size_of_repo_to_complete__41__/comment_1_067b29fc47d26b9da0766f9810684ae8._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="GLITTAH" + ip="78.108.63.46" + subject="comment 1" + date="2013-11-19T18:15:15Z" + content=""" +Perhaps when unannex is invoked, it could check to see if there is enough disk space before it begins copying files. Maybe even warn the user of too little disk space and remind them of --fast. + +(Sorry for the dupe!) +"""]] diff --git a/doc/todo/wishlist:_make_git_annex_reinject_work_in_direct_mode.mdwn b/doc/todo/wishlist:_make_git_annex_reinject_work_in_direct_mode.mdwn new file mode 100644 index 000000000..ffdee8840 --- /dev/null +++ b/doc/todo/wishlist:_make_git_annex_reinject_work_in_direct_mode.mdwn @@ -0,0 +1,21 @@ +### Please describe the problem. + +`git annex reinject` refuses to work while in direct mode. + +When in direct mode git annex reinject could simply perform `rm $symlink; mv $file_copy .; git annex add $file`. I prefer having git annex doing that so I am sure I am not messing up (mistakenly adding new files for instance) and everything is properly managed. + +### What version of git-annex are you using? On what operating system? + +git-annex 4.20130516.1 + +~~~~ +$ lsb_release -a +No LSB modules are available. +Distributor ID: Ubuntu +Description: Ubuntu 12.04.2 LTS +Release: 12.04 +Codename: precise +~~~~ + +> [[fixed|done]]. Why did I take so long to do this, it was a trivial 1 +> word change! --[[Joey]] diff --git a/doc/todo/wishlist:_make_partial_files_available_during_transfer.mdwn b/doc/todo/wishlist:_make_partial_files_available_during_transfer.mdwn new file mode 100644 index 000000000..b021c9091 --- /dev/null +++ b/doc/todo/wishlist:_make_partial_files_available_during_transfer.mdwn @@ -0,0 +1,18 @@ +Imagine this situation: +You have a laptop and a NAS. +On your laptop you want to consume a large media file located on the NAS. +So you type: + + git annex get --from nas mediafile + +But now you have to wait for the download to complete, unless either + +* rsync is pointed directly to the file in the object storage ("--inplace") +or +* the symlink temporarily points to the partial file during a transfer + +which would allow you instantaneous consumption of your media. +It might make sense to make this behavior configurable, because not everyone might agree with having partial content (that mismatches its key) around. + + +So what do you say? diff --git a/doc/todo/wishlist:_make_partial_files_available_during_transfer/comment_3_1304a721da6f5133fdfa1dac507f1ecb._comment b/doc/todo/wishlist:_make_partial_files_available_during_transfer/comment_3_1304a721da6f5133fdfa1dac507f1ecb._comment new file mode 100644 index 000000000..8e955fd48 --- /dev/null +++ b/doc/todo/wishlist:_make_partial_files_available_during_transfer/comment_3_1304a721da6f5133fdfa1dac507f1ecb._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.152.108.194" + subject="comment 3" + date="2012-11-04T19:58:28Z" + content=""" +I'm not at all comfortable with either idea. Temporarily repointing the symlink could lead to accidentially git committing a bad symlink. Or the user accidentially doing something with a partially transferred file. Running rsync in place would break lots of things that assume that, once the file is present, it can be assumed to be the full and correct file. (Obviously fsck doesn't assume that, but checks made by `git annex drop` do, for example.) + +However, you can access partially transferred files by key in `.git/annex/tmp`. It would be easy to write some hack that looks at the symlink to get the key, and then spits out the name of the partial file in `.git/annex/tmp.`. +"""]] diff --git a/doc/todo/wishlist:_more_descriptive_commit_messages_in_git-annex_branch.mdwn b/doc/todo/wishlist:_more_descriptive_commit_messages_in_git-annex_branch.mdwn new file mode 100644 index 000000000..3a891fc9b --- /dev/null +++ b/doc/todo/wishlist:_more_descriptive_commit_messages_in_git-annex_branch.mdwn @@ -0,0 +1,39 @@ +as of git-annex version 3.20110719, all git-annex commits only contain the word "update" as a commit message. given that the contents of the commit are pretty non-descriptive (SHA1 hashes for file names, uuids for repository names), i suggest to have more descriptive commit messages, as shown here: + + /mnt/usb_disk/photos/2011$ git annex get + /mnt/usb_disk/photos/2011$ git show git-annex + [...] + usb-disk-photos: get 2011 + + * 10 files retrieved from 2 sources (9 from local-harddisk, 1 from my-server) + * 120 files were already present + * 2 files could not be retrieved + /mnt/usb_disk/photos/2011$ cd ~/photos/2011/07 + ~/photos/2011/07$ git copy --to my-server + ~/photos/2011/07$ git show git-annex + [...] + local-harddisk: copy 2011/07 to my-server + + * 20 files pushed + ~/photos/2011/07$ + +in my opinion, the messages should at least contain + +* what command was used +* in which repository they were executed +* which files or directories they affected (not necessarily all files, but what was given on command line or implicitly from the working directory) + +--[[chrysn]] + +> The implementation of the git-annex branch precludes more descriptive +> commit messages, since a single commit can include changes that were +> previously staged to the branch's index file, or spooled to its journal +> by other git-annex commands (either concurrently running or +> interrupted commands, or even changes needed to automatically merge +> other git-annex branches). +> +> It would be possible to make it *less* verbose, with an empty commit +> message. :) --[[Joey]] + +>> Closing as this is literally impossible to do without making +>> git-annex worse. [[done]] --[[Joey]] diff --git a/doc/todo/wishlist:_more_info_in_the_standard_commit_message_of___96__sync__96__.mdwn b/doc/todo/wishlist:_more_info_in_the_standard_commit_message_of___96__sync__96__.mdwn new file mode 100644 index 000000000..4e5e3a171 --- /dev/null +++ b/doc/todo/wishlist:_more_info_in_the_standard_commit_message_of___96__sync__96__.mdwn @@ -0,0 +1,3 @@ +Could you include the REPO and UUID information in the "automatic sync" commit message? + +This would make troubleshooting easier. (not there was much trouble with git-annex!) diff --git a/doc/todo/wishlist:_move_pending_transfers_for_a_host_to_the_end_of_the_queue_when_one_fails.mdwn b/doc/todo/wishlist:_move_pending_transfers_for_a_host_to_the_end_of_the_queue_when_one_fails.mdwn new file mode 100644 index 000000000..e50ebbde5 --- /dev/null +++ b/doc/todo/wishlist:_move_pending_transfers_for_a_host_to_the_end_of_the_queue_when_one_fails.mdwn @@ -0,0 +1,7 @@ +Right now the assistant can have a huge list of pending transfers for certain hosts if its data is a bit outdated, or a host hasn't been synced lately. When starting up it will then attempt each transfer to said host (which will in turn fail, but at times take time to time out), possibly before doing other stuff like attempting to download new files, or copy files to online hosts. + +I suggest that if a transfer fails for host X, and there are other pending transfers, say to host Y and from Z, then all other pending transfers to/from X gets pushed to the back of the queue, to avoid having to wait a long time for several transfers to time out before doing useful stuff. + +The prime example for me was this morning, when a laptop that was turned off had a huge amount of queued transfers to it, resulting in the assistant attempting a load of transfers to that host before it retrieved a new file that I had created on another machine yesterday. + +[[!tag design/assistant]] diff --git a/doc/todo/wishlist:_move_pending_transfers_for_a_host_to_the_end_of_the_queue_when_one_fails/comment_1_82ee9de610a0ac55cd1c27c211079e5b._comment b/doc/todo/wishlist:_move_pending_transfers_for_a_host_to_the_end_of_the_queue_when_one_fails/comment_1_82ee9de610a0ac55cd1c27c211079e5b._comment new file mode 100644 index 000000000..3b6101ed5 --- /dev/null +++ b/doc/todo/wishlist:_move_pending_transfers_for_a_host_to_the_end_of_the_queue_when_one_fails/comment_1_82ee9de610a0ac55cd1c27c211079e5b._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.154.6.49" + subject="comment 1" + date="2012-12-01T19:22:03Z" + content=""" +There are several difficulties with reordering the queue that way. One is that the failure may be intermittent; another is that the queue is fed by a scanning process, so doesn't always have a well-defined end. + +Another way to deal with this problem, which I think I prefer, is to allow multiple actions from the queue to run at once. Then slow or unreachable remotes don't block it from using other remotes. +"""]] diff --git a/doc/todo/wishlist:_move_pending_transfers_for_a_host_to_the_end_of_the_queue_when_one_fails/comment_2_bea55156bd32cf9e6dd9b946ba1bb8c1._comment b/doc/todo/wishlist:_move_pending_transfers_for_a_host_to_the_end_of_the_queue_when_one_fails/comment_2_bea55156bd32cf9e6dd9b946ba1bb8c1._comment new file mode 100644 index 000000000..62d46bccd --- /dev/null +++ b/doc/todo/wishlist:_move_pending_transfers_for_a_host_to_the_end_of_the_queue_when_one_fails/comment_2_bea55156bd32cf9e6dd9b946ba1bb8c1._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="EskildHustvedt" + ip="84.48.83.221" + subject="comment 2" + date="2012-12-01T19:31:18Z" + content=""" +I agree your method might be preferable, the end result is the same, and would have avoided the issues I had (and, of course, running multiple transfers at once has other benefits as well). + +An alternate way would be to push every transfer NOT from host X to the front of the queue (avoiding most of the \"no defined end\" issue and largely solving the problem), but if multiple actions at once is feasible then that'd still be much nicer. +"""]] diff --git a/doc/todo/wishlist:_option_to_disable_url_checking_with_addurl.mdwn b/doc/todo/wishlist:_option_to_disable_url_checking_with_addurl.mdwn new file mode 100644 index 000000000..d0b847933 --- /dev/null +++ b/doc/todo/wishlist:_option_to_disable_url_checking_with_addurl.mdwn @@ -0,0 +1,9 @@ +I'm testing out an idea of using filter-branch on a git repository to both retroactively annex and AND record a weburl for all relevant files. + +c.f. [http://git-annex.branchable.com/tips/How_to_retroactively_annex_a_file_already_in_a_git_repo/](http://git-annex.branchable.com/tips/How_to_retroactively_annex_a_file_already_in_a_git_repo/) + +The bottleneck I'm hitting here seems to be the fact that `git annex addurl` diligently checks each url to see that it is accessible, which adds up quickly if many files are to be processed. + +It would be great if addurl had an option to disable checking the url, in order to speed up large batch jobs like this. + +> --relaxed added [[done]] --[[Joey]] diff --git a/doc/todo/wishlist:_option_to_disable_url_checking_with_addurl/comment_1_868a380faa1e55faa3c2d314e3258e86._comment b/doc/todo/wishlist:_option_to_disable_url_checking_with_addurl/comment_1_868a380faa1e55faa3c2d314e3258e86._comment new file mode 100644 index 000000000..4bdf4c97b --- /dev/null +++ b/doc/todo/wishlist:_option_to_disable_url_checking_with_addurl/comment_1_868a380faa1e55faa3c2d314e3258e86._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + nickname="joey" + subject="comment 1" + date="2013-03-11T18:19:34Z" + content=""" +It does this to get the size of the url, so it can record this in the key. It would be possible to add a flag to skip that (--fast is already taken of course), but then you tend to get a lot of keys in your repository with no size info attached, which makes `git annex status` complain that it cannot tell you exactly how big your repo is, and is generally not the best. It also defeats annex.diskreserve checking, for example. + +With --fast, the size is the only info available to ensure that the content behind the url has not changed when downloading it later. I suppose for some urls you may not want that checked, and so a --relaxed type option could make sense in that use case as well, although with the above caveat. +"""]] diff --git a/doc/todo/wishlist:_option_to_print_more_info_with___39__unused__39__.mdwn b/doc/todo/wishlist:_option_to_print_more_info_with___39__unused__39__.mdwn new file mode 100644 index 000000000..7a9b81f72 --- /dev/null +++ b/doc/todo/wishlist:_option_to_print_more_info_with___39__unused__39__.mdwn @@ -0,0 +1,37 @@ +It would be nice if the 'unused' command could optionally display info about the actual files behind its cryptic keys. + +I created a (very rough) bash script that simply splices in some info from git log -S'KEY' --numstat into the unused list, like so: + + arand@mas:~/annex(master)$ bash ~/utv/scripts/annex-vunused + unused . (checking for unused data...) (checking master...) (checking synced/master...) (checking origin/HEAD...) (checking seagate/master...) + Some annexed data is no longer used by any files: + NUMBER KEY + 1 SHA256E-s1073741824--49bc20df15e412a64472421e13fe86ff1c5165e18b2afccf160d4dc19fe68a14.img + 8f479a4 Sat Feb 23 16:14:12 2013 +0100 remove bigfile + 0 1 dummy_bigfile.img + 2988d18 Sat Feb 23 16:13:48 2013 +0100 dummy file + 1 0 dummy_bigfile.img + (To see where data was previously used, try: git log --stat -S'KEY')To remove unwanted data: git-annex dropunused NUMBER + ok +The script: + + #!/bin/bash + + pipe="$(mktemp -u)" + mkfifo "$pipe" + + git annex unused >"$pipe" || exit 1 & + + while read -r line + do + key="$(echo "$line" | sed 's/^[^-]*-\([^-]*\)-.*/\1/')" + echo -n "$line" + test -n "$key" && \ + echo && \ + git log --format="%h %cd %s" --numstat -S"$key" | \ + sed '/^$/d;/git-annex automatic sync/,/^ /d;s/^/\t\t/' + + done < "$pipe" + rm "$pipe" + +It would be nice if something like this was available as an option, since it's good way to get a quick overview of what the content is, and if it's safe to drop it. diff --git a/doc/todo/wishlist:_perform_fsck_remotely.mdwn b/doc/todo/wishlist:_perform_fsck_remotely.mdwn new file mode 100644 index 000000000..f2187d912 --- /dev/null +++ b/doc/todo/wishlist:_perform_fsck_remotely.mdwn @@ -0,0 +1,39 @@ +Currently, when `fsck`'ing a remote, files are first downloaded to a temporary +file locally, decrypted if needed, and finally digested; the temporary file is +then either thrown away, or quarantined, depending on the value of that digest. + +Whereas this approach works with any kind of remote, in the particular case +where the user is granted execution rights on the digest command, one could +avoid cluttering the network and digest the file remotely. I propose the +addition of a per-remote git option `annex-remote-fsck` to switch between the +two behaviors. + + +There is an issue with encrypted specialremotes, though. As hinted at +[[here|tips/beware_of_SSD_wear_when_doing_fsck_on_large_special_remotes/#comment-70055f166f7eeca976021d24a736b471]], +since the digest of a ciphertext can't be deduced from that of a plaintext in +general one would needs, before sending an encrypted file to such a remote, to +digest it and store that digest somewhere (together with the cipher's size and +perhaps other meta-information). + +The usual directory structure (`.../.../{backend}-s{size}--{digest}.log`) seems +perfectly suitable to store these informations. Lines there would look like +`{timestamp}s {numcopy} {UUID} {remote digest}`. Of course, it implies that +remote digest commands are trustworthy (are doing the right thing), and that +the digest output are not tampered by others who have access to the git repo. +But that's outside the current threat model, I guess. + +Actually, since git-annex always includes a MDC in the ciphertexts, we could do +something clever and even avoid running a digest algorithm. According to the +[[OpenPGP standard|https://tools.ietf.org/html/rfc4880#section-5.14]] the MDC +is essentially a SHA-1 hash of the plaintext. I'm still investigating if it's +even possible, but in theory it would be enough (with non-chained ciphers at +least) to download a few bytes from the encrypted remote, decrypt those bytes +to retrieve the hash, and compare that hash with the known value. Of course +there is a downside here, namely that files tampered anywhere but on the MDC +packets would not be detected by `fsck` (but gpg will warn when decrypting the +file). + + +My 2 cents :-) Is there something I missed? I suppose there was a reason to +perform `fsck` locally at the first place... diff --git a/doc/todo/wishlist:_perform_fsck_remotely/comment_1_db92311dcdb1ef0ab0413f83e191c70c._comment b/doc/todo/wishlist:_perform_fsck_remotely/comment_1_db92311dcdb1ef0ab0413f83e191c70c._comment new file mode 100644 index 000000000..6bf6af23a --- /dev/null +++ b/doc/todo/wishlist:_perform_fsck_remotely/comment_1_db92311dcdb1ef0ab0413f83e191c70c._comment @@ -0,0 +1,15 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.154.0.63" + subject="comment 1" + date="2013-08-22T15:18:35Z" + content=""" +The only reason fsck is done locally for remotes is ease of implementation and it being a generic operation that supports any kind of special remote. + +Seems that the the only types of remotes where a remote fsck is a possibility are some rsync remotes and git remotes. +git remotes already have git-annex installed, so the fsck could be run locally on the remote system using it. + +I don't know if I see a benefit with the MDC check. Any non-malicious data corruption on the remote is likely to affect the body of the file and not the small portion that holds the MDC. So checking the MDC does not seem much better than the current existence check done by `git annex fsck --fast --from remote`. + +As for storing the remote digest on the git-annex branch, my initial reaction was just that it's potentially a lot of bloat. Thinking about it some more, when using non-shared encryption, there is currently no way, given just a clone of a git repository, to match up files in git with encrypted objects stored on a special remote. So storing the remote digest might be considered to weaken the security. +"""]] diff --git a/doc/todo/wishlist:_perform_fsck_remotely/comment_2_2f0dbaf143d94290bfbebb6869eb7241._comment b/doc/todo/wishlist:_perform_fsck_remotely/comment_2_2f0dbaf143d94290bfbebb6869eb7241._comment new file mode 100644 index 000000000..5418ff991 --- /dev/null +++ b/doc/todo/wishlist:_perform_fsck_remotely/comment_2_2f0dbaf143d94290bfbebb6869eb7241._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="guilhem" + ip="129.16.20.209" + subject="comment 2" + date="2013-08-22T16:56:55Z" + content=""" +Oh yeah, the MDC paragraph was pretty much pointless indeed. Oops :-P + +I agree that this would potentially add some noise to the index, and weaken the +security, but depending on the threat model and people's preferences that's an +option that's worth considering IMHO. +"""]] diff --git a/doc/todo/wishlist:_print_locations_for_files_in_rsync_remote.mdwn b/doc/todo/wishlist:_print_locations_for_files_in_rsync_remote.mdwn new file mode 100644 index 000000000..3876f2197 --- /dev/null +++ b/doc/todo/wishlist:_print_locations_for_files_in_rsync_remote.mdwn @@ -0,0 +1,6 @@ +Based on an irc conversation earlier today: + +19:50 < warp> joeyh: what is the best way to figure out the (remote) filename for a file stored in an rsync remote? + +20:43 < joeyh> warp: re your other question, probably the best thing would be to make the whereis command print out locations for each remote, as it always does for the web special remotes + diff --git a/doc/todo/wishlist:_push_to_cia.vc_from_the_website__39__s_repo__44___not_your_personal_one.mdwn b/doc/todo/wishlist:_push_to_cia.vc_from_the_website__39__s_repo__44___not_your_personal_one.mdwn new file mode 100644 index 000000000..f9b4c8c35 --- /dev/null +++ b/doc/todo/wishlist:_push_to_cia.vc_from_the_website__39__s_repo__44___not_your_personal_one.mdwn @@ -0,0 +1,3 @@ +I just added a CIA bot to #vcs-home and tracking commits immediately would be nice. -- RichiH + +[[done]] diff --git a/doc/todo/wishlist:_push_to_cia.vc_from_the_website__39__s_repo__44___not_your_personal_one/comment_1_3480b0ec629ef29a151408d869186bf8._comment b/doc/todo/wishlist:_push_to_cia.vc_from_the_website__39__s_repo__44___not_your_personal_one/comment_1_3480b0ec629ef29a151408d869186bf8._comment new file mode 100644 index 000000000..5d0edce2e --- /dev/null +++ b/doc/todo/wishlist:_push_to_cia.vc_from_the_website__39__s_repo__44___not_your_personal_one/comment_1_3480b0ec629ef29a151408d869186bf8._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joey.kitenet.net/" + nickname="joey" + subject="comment 1" + date="2011-09-19T18:57:52Z" + content=""" +JFTR, pushing now happens automatically from branchable. +"""]] diff --git a/doc/todo/wishlist:_query_things_like_description__44___trust_level.mdwn b/doc/todo/wishlist:_query_things_like_description__44___trust_level.mdwn new file mode 100644 index 000000000..d158850cd --- /dev/null +++ b/doc/todo/wishlist:_query_things_like_description__44___trust_level.mdwn @@ -0,0 +1,4 @@ +It would be helpful to have a way to query things like a repository's description and trust level, without having to poke in the git-annex branch. For example, "git annex describe ." currently clears the description but could print the current one instead. + +> `git annex status` now breaks down the repository list by type. [[done]] +> --[[Joey]] diff --git a/doc/todo/wishlist:_query_things_like_description__44___trust_level/comment_1_14311384788312b96e550749ab7de9ea._comment b/doc/todo/wishlist:_query_things_like_description__44___trust_level/comment_1_14311384788312b96e550749ab7de9ea._comment new file mode 100644 index 000000000..3ac4ba267 --- /dev/null +++ b/doc/todo/wishlist:_query_things_like_description__44___trust_level/comment_1_14311384788312b96e550749ab7de9ea._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://joey.kitenet.net/" + nickname="joey" + subject="comment 1" + date="2011-10-27T17:09:33Z" + content=""" +`git annex describe` only sets the description to avoid complication. Imagine using it in a script for example. + +`git annex status` shows the description. It does not show the trust level because I have not thought of a visually pleasing and compact way to show it in the repository list there.. suggestions appreciated, since the same list is used by `whereis`, and showing trust levels there would be particularly useful. +"""]] diff --git a/doc/todo/wishlist:_query_things_like_description__44___trust_level/comment_2_342d1ac07573c7ef4e27f003a692e261._comment b/doc/todo/wishlist:_query_things_like_description__44___trust_level/comment_2_342d1ac07573c7ef4e27f003a692e261._comment new file mode 100644 index 000000000..3bb92919f --- /dev/null +++ b/doc/todo/wishlist:_query_things_like_description__44___trust_level/comment_2_342d1ac07573c7ef4e27f003a692e261._comment @@ -0,0 +1,32 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U" + nickname="Richard" + subject="comment 2" + date="2011-10-29T18:28:13Z" + content=""" +Possible solutions: + +This: + + trusted repositories: + UUID -- foo + semi-trusted repositories: + UUID -- bar + untrusted repositories: + UUID -- baz + +or this: + + UUID -- trusted -- foo + UUID -- semi-trusted -- bar + UUID -- untrusted -- baz + +or this: + + known repositories (!/*/X): + UUID -- ! foo + UUID -- * bar + UUID -- X baz + +If you want to reformat this output, putting 'here', 'origin', etc into fixed formatting might make sense, as well. -- Richard +"""]] diff --git a/doc/todo/wishlist:_recursive_directory_remote_setup__47__addurl.mdwn b/doc/todo/wishlist:_recursive_directory_remote_setup__47__addurl.mdwn new file mode 100644 index 000000000..2bfb90b54 --- /dev/null +++ b/doc/todo/wishlist:_recursive_directory_remote_setup__47__addurl.mdwn @@ -0,0 +1,7 @@ +I think it would be interesting to have a way to recursively import a local directory without actually moving files around. And to be able to checksum these files as well (without moving them into the annex). + +This would work somewhat similar to looping over a directory and adding file:// remotes for each file. + +A use case is importing optical media (read-only), whilst keeping that media as a remote, and being able to calculate checksums directly without moving any files around. + +For single files, it would also be interesting if addurl had a "--localchecksum" option that would only work for file:// urls, and make it checksum files directly from their source location?) diff --git a/doc/todo/wishlist:_recursive_directory_remote_setup__47__addurl/comment_1_b79976afc2242791523e63831f30af71._comment b/doc/todo/wishlist:_recursive_directory_remote_setup__47__addurl/comment_1_b79976afc2242791523e63831f30af71._comment new file mode 100644 index 000000000..caefee9a8 --- /dev/null +++ b/doc/todo/wishlist:_recursive_directory_remote_setup__47__addurl/comment_1_b79976afc2242791523e63831f30af71._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="https://launchpad.net/~arand" + nickname="arand" + subject="comment 1" + date="2013-03-10T23:29:55Z" + content=""" +Recursively adding urls is already feasiable with some simple scripting: + +[https://gitorious.org/arand-scripts/arand-scripts/blobs/master/annex-importdir](https://gitorious.org/arand-scripts/arand-scripts/blobs/master/annex-importdir) + +but this is obviously missing the checksumming bit though. +"""]] diff --git a/doc/todo/wishlist:_recursive_directory_remote_setup__47__addurl/comment_2_1741d2392006a9af9cfd1f3b847600b9._comment b/doc/todo/wishlist:_recursive_directory_remote_setup__47__addurl/comment_2_1741d2392006a9af9cfd1f3b847600b9._comment new file mode 100644 index 000000000..64d592479 --- /dev/null +++ b/doc/todo/wishlist:_recursive_directory_remote_setup__47__addurl/comment_2_1741d2392006a9af9cfd1f3b847600b9._comment @@ -0,0 +1,9 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + nickname="joey" + subject="comment 2" + date="2013-03-11T03:04:44Z" + content=""" +See also: [[forum/Managing a large number of files archived on many pieces of read-only medium (E.G. DVDs)]] +particularly [[this comment|forum/Managing a large number of files archived on many pieces of read-only medium (E.G. DVDs)#comment-908dbe02f29e011f030bba4ab5ef73d1]] +"""]] diff --git a/doc/todo/wishlist:_simple_url_for_webapp.mdwn b/doc/todo/wishlist:_simple_url_for_webapp.mdwn new file mode 100644 index 000000000..4549f2e74 --- /dev/null +++ b/doc/todo/wishlist:_simple_url_for_webapp.mdwn @@ -0,0 +1,36 @@ +### Please describe the problem. + +The environment is os/x with chrome as the browser. + +Let's say I close the tab with the webapp running in it. The 'git-annex webapp' process is still running, according to 'ps'. + +So I open a new tab, but then what do I type into the browser url bar to get the app back? What is usually there is a loopback address and an authorisation hash. + +* Should I double-click on the git-annex icon in the dock (or Applications directory)? +* I figured out from observing the startup that if I give the url ://localhost/Users/me/annex/.git/annex/webapp.html I will get redirected to the right place. +Should I set up a bookmark for that? + +### What steps will reproduce the problem? + +see above. + +### What version of git-annex are you using? On what operating system? + +Version: 4.20130723-ge023649 +Build flags: Assistant Webapp Pairing Testsuite S3 WebDAV FsEvents XMPP DNS +os: os/x 10.8.4 + +### Please provide any additional information below. + +I notice that in the webapp ui, all the items at the top of the page highlight when one hovers over them and have useful URLs attached, +with the exception of the 'git-annex' item at the far left.What if that had the entry point url attached to it (so one could bookmark that)? + +> The git-annex assistant is designed to stay running in the background whether you have the web browser open or not. You can open the web display at any time by +> using the git-annex menu item (on linux) or running the git-annex-webapp +> program (which is in the DMG on OSX). +> +> If the file:// url were exposed to users, it would not work if +> the assistant had not already been started. This is why there is a program +> to open the webapp, not an url. +> +> Not a bug; [[done]] --[[Joey]] diff --git a/doc/todo/wishlist:_simple_url_for_webapp/comment_1_552aad504fbb68d1f85abfde8c535e69._comment b/doc/todo/wishlist:_simple_url_for_webapp/comment_1_552aad504fbb68d1f85abfde8c535e69._comment new file mode 100644 index 000000000..1211be9b5 --- /dev/null +++ b/doc/todo/wishlist:_simple_url_for_webapp/comment_1_552aad504fbb68d1f85abfde8c535e69._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawkfHTPsiAcHEEN7Xl7WxiZmYq-vX7azxFY" + nickname="Vincent" + subject="comment 1" + date="2013-07-24T14:46:22Z" + content=""" +typo + +url should be - file://localhost/Users/me/annex/.git/annex/webapp.html +"""]] diff --git a/doc/todo/wishlist:_simpler_gpg_usage.mdwn b/doc/todo/wishlist:_simpler_gpg_usage.mdwn new file mode 100644 index 000000000..1236ee234 --- /dev/null +++ b/doc/todo/wishlist:_simpler_gpg_usage.mdwn @@ -0,0 +1,12 @@ +This is my current understanding on how one must use gpg with git-annex: + + * Generate(or copy around) a gpg key on every machine that needs to access the encrypted remote. + * git annex initremote myremote encryption=KEY for each key that you generated + +What I'm trying to figure out is if I can generate a no-passphrase gpg key and commit it to the repository, and have git-annex use that. That way any new clones of the annex automatically have access to any encrypted remotes, without having to do any key management. + +I think I can generate a no-passphrase key, but then I still have to manually copy it around to each machine. + +I'm not a huge gpg user so part of this is me wanting to avoid having to manage and keeping track of the keys. This would probably be a non-issue if I used gpg on more machines and was more comfortable with it. + +[[done]] diff --git a/doc/todo/wishlist:_simpler_gpg_usage/comment_1_6923fa6ebc0bbe7d93edb1d01d7c46c5._comment b/doc/todo/wishlist:_simpler_gpg_usage/comment_1_6923fa6ebc0bbe7d93edb1d01d7c46c5._comment new file mode 100644 index 000000000..f96f5c377 --- /dev/null +++ b/doc/todo/wishlist:_simpler_gpg_usage/comment_1_6923fa6ebc0bbe7d93edb1d01d7c46c5._comment @@ -0,0 +1,19 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawmBUR4O9mofxVbpb8JV9mEbVfIYv670uJo" + nickname="Justin" + subject="comment 1" + date="2012-04-29T01:41:57Z" + content=""" +Thinking about this more, I think minimally git-annex could support a + + remote.<name>.gpg-options + +or + + remote.<name>.gpg-keyring + +for options to be passed to gpg. I'm not sure how automatically setting it to $ANNEX_ROOT/.gnupg/.. would work. + + +I need to read the encryption code to fully understand it, but I also wonder if there is not also a way to just bypass gpg entirely and store the remote-encryption keys locally in plain text. +"""]] diff --git a/doc/todo/wishlist:_simpler_gpg_usage/comment_2_6fc874b6c391df242bd2592c4a65eae8._comment b/doc/todo/wishlist:_simpler_gpg_usage/comment_2_6fc874b6c391df242bd2592c4a65eae8._comment new file mode 100644 index 000000000..2e12d86d4 --- /dev/null +++ b/doc/todo/wishlist:_simpler_gpg_usage/comment_2_6fc874b6c391df242bd2592c4a65eae8._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://joey.kitenet.net/" + nickname="joey" + subject="comment 2" + date="2012-04-29T02:39:20Z" + content=""" +The encryption uses a symmetric cipher that is stored in the git repository already. It's just stored encrypted to the various gpg keys that have been configured to use it. It would certainly be possible to store the symmetric cipher unencrypted in the git repo. + +I don't see your idea of gpg-options saving any work. It would still require you to do key distribution and run commands in each repo to set it up. +"""]] diff --git a/doc/todo/wishlist:_simpler_gpg_usage/comment_3_012f340c8c572fe598fc860c1046dabd._comment b/doc/todo/wishlist:_simpler_gpg_usage/comment_3_012f340c8c572fe598fc860c1046dabd._comment new file mode 100644 index 000000000..051f17a24 --- /dev/null +++ b/doc/todo/wishlist:_simpler_gpg_usage/comment_3_012f340c8c572fe598fc860c1046dabd._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joey.kitenet.net/" + nickname="joey" + subject="comment 3" + date="2012-04-29T02:41:38Z" + content=""" +BTW re your Tweet.. I was so happy to be able to use 'c i a' in Crypto.hs. :) +"""]] diff --git a/doc/todo/wishlist:_simpler_gpg_usage/comment_4_e0c2a13217b795964f3b630c001661ef._comment b/doc/todo/wishlist:_simpler_gpg_usage/comment_4_e0c2a13217b795964f3b630c001661ef._comment new file mode 100644 index 000000000..c9e337541 --- /dev/null +++ b/doc/todo/wishlist:_simpler_gpg_usage/comment_4_e0c2a13217b795964f3b630c001661ef._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawmBUR4O9mofxVbpb8JV9mEbVfIYv670uJo" + nickname="Justin" + subject="comment 4" + date="2012-04-29T03:09:03Z" + content=""" +I got a good laugh out of it :-) + +Storing the key unencrypted would make things easier.. I think at least for my use-cases I don't require another layer of protection on top of the ssh keys that provide access to the encrypted remotes themselves. +"""]] diff --git a/doc/todo/wishlist:_simpler_gpg_usage/comment_5_9668b58eb71901e1db8da7db38e068ca._comment b/doc/todo/wishlist:_simpler_gpg_usage/comment_5_9668b58eb71901e1db8da7db38e068ca._comment new file mode 100644 index 000000000..60b98bde5 --- /dev/null +++ b/doc/todo/wishlist:_simpler_gpg_usage/comment_5_9668b58eb71901e1db8da7db38e068ca._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joey.kitenet.net/" + nickname="joey" + subject="comment 5" + date="2012-04-29T18:04:13Z" + content=""" +encryption=shared is now supported +"""]] diff --git a/doc/todo/wishlist:_spec.remotes_for_other_peer_network_data_stores___40__gnunet__44___freenet__41__.mdwn b/doc/todo/wishlist:_spec.remotes_for_other_peer_network_data_stores___40__gnunet__44___freenet__41__.mdwn new file mode 100644 index 000000000..545bd861d --- /dev/null +++ b/doc/todo/wishlist:_spec.remotes_for_other_peer_network_data_stores___40__gnunet__44___freenet__41__.mdwn @@ -0,0 +1,26 @@ +Apart from Tahoe-LAFS (covered by [[todo/tahoe lfs for reals]] and [[forum/tips: special_remotes/hook with tahoe-lafs]]), [[special remotes]] (which I understand as real storage backends) for other other [peer network data stores](http://en.wikipedia.org/wiki/Distributed_data_store#Peer_network_node_data_stores_2) would be interesting. + +I mean gnunet, freenet, BitTorrent (also trackerless). + +Before dropping a file locally, the BitTorrent client should check that all parts are still available from the peers. + +Of course, there is no guarantee assumed that the content won't disappear from the peer network in future: they act more like a cache rather than an archive on whose lifespan you decide. (I'm only not sure about gnunet now: whether there is a rule of dropping unused content from it, like in freenet.) + +So, a copy in peer networks shouldn't be counted on by git-annex as much as a copy on a storage you control: probably, by efault, it shouldn't let you delete the local copy if there is a copy in a peer network unless you saved it somewhere else. + +(Think of such a scenario: I could save some of my public large data on external disks/DVDs and keep them at home, and also put them onto peer networks with the same nterface of git-annex which I would be used to; I would also use the git-annex interface to check from time to time that the content is still present, i.e. "cached", on the peer networks. Whenever I'm away from home, and unexpectedly need to show this content to someone, or have a look at it for some reason, I could get it from the peer network "cache".) + +Also networks like namecoin (derived from bitcoin) can be used as a key-value store. Despite being a peer network, a system like namecoin actually could offer the publisher more control over the lifespan of the content: he should be able to offer "financial" reward for others processing his key-value data. (But I'm not sure namecoin is designed reasonably for this reward system to work actually; but there might be appearing other similar systems.) + +## A different view: extend the key-value backends with ways to look for the content in other content-addressable storage systems +We might want to look for the registered files in other [content-addressable storage systems](http://en.wikipedia.org/wiki/Content-addressable_storage#Open-source_implementations) (and also to be able to put the files there for storage). + +For example: + +* [**GNUnet**](http://en.wikipedia.org/wiki/Gnunet) uses its own hash format to address the content. git-annex could extend its own [[backends]] with a one to work with GNUnet, and by default have a built-in [[special remote|special remotes]] that would interact with GNUnet when looking for a content or storing some content. No special setup of the special remote in each repo should be necessary, because GNUnet is "global", so we'd just use the user's already configured GNUnet client. Just turning the builtin GNUnet special remote on or off should be an option (in the repo configuration, and when calling the commands that would query it, like `whereis`). +* **freenet** is similar. +* Similarly, a backend for the hashes used in **BitTorrent** and **magnet links** could be used. If we want a trackerless mode, then probably it's a similar case for a "global"/built-in special remote that needs no local setup in each repo. Using a selected tracker would mean setting up a special remote in our repo. +* **Git** itself can be viwed as place to look for the content. There could be a corresponding backend and a builtin special remote (needing no extra setup) to look for the content among the objects stored in the local Git repo. (What if we have a copy of a file that we've put under the control of git-annex in a previous Git commit? We could get it from the object store of Git.) +* **Venti**, [[**Tahoe-LAFS**|todo/tahoe lfs for reals]] would need a backend for their hashes, and a specially setup special remote in each repo where we'd like to use them--because these are not "global" system, we must setup the path to the instance of the filesystem we'd like to use. +* probably, there must be other interesting cases of this kind... +* (I'm also thinking about using somethng like a **bibliographic information** as a key, but then it wouldn't guarantee identical files: the same paper can be stored in different formats, etc. Cf. [**URNs**](http://en.wikipedia.org/wiki/Uniform_resource_name#Examples), via <http://it.slashdot.org/comments.pl?sid=3032489&cid=40907233>. Also, an URN like bibliographic information can't be computed from the file, it will have to be entered manually or obtained from another directory of URNs.) diff --git a/doc/todo/wishlist:_spec.remotes_for_other_peer_network_data_stores___40__gnunet__44___freenet__41__/comment_1_e2c2047e7401cb95a82ffb686a732859._comment b/doc/todo/wishlist:_spec.remotes_for_other_peer_network_data_stores___40__gnunet__44___freenet__41__/comment_1_e2c2047e7401cb95a82ffb686a732859._comment new file mode 100644 index 000000000..80a245d14 --- /dev/null +++ b/doc/todo/wishlist:_spec.remotes_for_other_peer_network_data_stores___40__gnunet__44___freenet__41__/comment_1_e2c2047e7401cb95a82ffb686a732859._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.153.14.141" + subject="comment 1" + date="2012-09-25T22:57:19Z" + content=""" +The best first step to adding such kinds of data stores to git-annex is probably to use the [[special_remotes/hook]] special remote to access them. +"""]] diff --git a/doc/todo/wishlist:_spec.remotes_for_other_peer_network_data_stores___40__gnunet__44___freenet__41__/comment_2_472b576afdb169b233edd01adcb2123d._comment b/doc/todo/wishlist:_spec.remotes_for_other_peer_network_data_stores___40__gnunet__44___freenet__41__/comment_2_472b576afdb169b233edd01adcb2123d._comment new file mode 100644 index 000000000..c58f97c1c --- /dev/null +++ b/doc/todo/wishlist:_spec.remotes_for_other_peer_network_data_stores___40__gnunet__44___freenet__41__/comment_2_472b576afdb169b233edd01adcb2123d._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://lj.rossia.org/users/imz/" + ip="79.165.56.162" + subject="comment 2" + date="2012-09-25T23:29:49Z" + content=""" +I see. But then, as with Tahoe-LAFS, they also have their own formats for checksums, keys, which could be re-used in git-annex, and that needs special treatment. +"""]] diff --git a/doc/todo/wishlist:_special-case_handling_of_Youtube_URLs_in_Web_special_remote.mdwn b/doc/todo/wishlist:_special-case_handling_of_Youtube_URLs_in_Web_special_remote.mdwn new file mode 100644 index 000000000..229dc258b --- /dev/null +++ b/doc/todo/wishlist:_special-case_handling_of_Youtube_URLs_in_Web_special_remote.mdwn @@ -0,0 +1,22 @@ +The [[Web special remote|special remotes/web]] could possibly be improved by detecting when URLs reference a Youtube video page and using [youtube-dl](http://rg3.github.com/youtube-dl/) instead of wget to download the page. Youtube-dl can also handle several other video sites such as vimeo.com and blip.tv, so if this idea were to be implemented, it might make sense to borrow the regular expressions that youtube-dl uses to identify video URLs. A quick grep through the youtube-dl source for the identifier _VALID_URL should find those regexes (in Python's regex format). + +> This is something I've thought about doing for a while.. +> Two things I have not figured out: +> +> * Seems that this should really be user-configurable or a plugin system, +> to handle more than just this one case. +> * Youtube-dl breaks from time to time, I really trust these urls a lot +> less than regular urls. Perhaps per-url trust levels are called for by +> this. +> +> --[[Joey]] + +> > There's a library for this called [quvi](http://quvi.sourceforge.net/) which supports many +> > different sites and also allows fetching the URL (as opposed to just +> > downloading the file). It seems to me this would be the best tool +> > for this task. One problem to consider here is that a single youtube +> > URL may yield different file contents depending on the quality +> > chosen. Also, it seems that the URLs guessed by quvi may be +> > ephemeral. --[[anarcat]] + +> [[done]]!!! --[[Joey]] diff --git a/doc/todo/wishlist:_special-case_handling_of_Youtube_URLs_in_Web_special_remote/comment_1_1a383c30df4fb1767f13d8c670b0c0b5._comment b/doc/todo/wishlist:_special-case_handling_of_Youtube_URLs_in_Web_special_remote/comment_1_1a383c30df4fb1767f13d8c670b0c0b5._comment new file mode 100644 index 000000000..5569ff94a --- /dev/null +++ b/doc/todo/wishlist:_special-case_handling_of_Youtube_URLs_in_Web_special_remote/comment_1_1a383c30df4fb1767f13d8c670b0c0b5._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="https://rmunn.myopenid.com/" + nickname="rmunn" + subject="comment 1" + date="2012-06-12T15:52:35Z" + content=""" +* One way to handle the configuration might be with regular expressions. If the URL matches regex A, handle it with downloader A' (with option set A''). If the URL matches regex B, handle it with downloader B' and option set B''. And so on. Then if nothing is matched, the default downloader is wget/curl. + +* In my experience, youtube-dl breakages are fixed relatively quickly; a much more serious problem from a trust standpoint is that Youtube videos often disappear. Sometimes due to a legitimate copyright claim, sometimes due to illegitimate copyright claims. (I've seen both happen). Or because the video uploader decided to upload *other* videos that violated copyright, and Youtube closed his/her account, thereby removing *all* his/her videos from the Web. Youtube is definitely an untrustworthy repository as far as \"the file will still be there later on\" is concerned. Perhaps a default trust relationship could go along with the regexes? URLs matching regex A are semitrusted, while URLs matching regex B are untrusted. +"""]] diff --git a/doc/todo/wishlist:_special-case_handling_of_Youtube_URLs_in_Web_special_remote/comment_2_81f7f893ac36c145b31f02db6a682a17._comment b/doc/todo/wishlist:_special-case_handling_of_Youtube_URLs_in_Web_special_remote/comment_2_81f7f893ac36c145b31f02db6a682a17._comment new file mode 100644 index 000000000..a25b3c89d --- /dev/null +++ b/doc/todo/wishlist:_special-case_handling_of_Youtube_URLs_in_Web_special_remote/comment_2_81f7f893ac36c145b31f02db6a682a17._comment @@ -0,0 +1,20 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.154.0.63" + subject="quvi thoughts. excited!" + date="2013-08-22T18:22:51Z" + content=""" +Anarcat's quvi suggestion is interesting, because it seems to simplify the whole thing down to just `addurl`, which git-annex is already good at. + +If quvi manages to find the url that can be used to download the actual video file, without needing to run a special downloader, then all you really need, it seems, is `git annex addurl --relaxed $(quvi youtube-url)` The --relaxed will make git-annex not care if the content or size of the url's content varies in the future. Since --relaxed skips the actual download, you'd want to follow that with `git annex get`, since we don't know how long these urls will last.. + +I suppose git-annex could, if quvi is available, make any attempt to download a web special remote url that +matches the `quvi --support` output run the url through quvi to get the real url and download that instead. The difficulties with this approach: + +* would need to read and parse `quvi --support` every time it gets an url from the web special remote? (I don't think I'd want to link with libquvi, although that would be possible, just because this is an edge thing.) +* what it an url that had been supported stopped being supported -- we'd not want to download the raw url in that case +* putting the quvi support here doesn't allow relaxed mode to be set when `addurl` adds the url. + +Maybe it would be better to keep the support in `addurl`, and record the url generated by quvi. That url would probably be more likely to break in the future, but that kind of breakage is why `git annex untrust web` is wise.. +Keeping the support in `addurl` would also let it use the title that quvi also returns to determine the filename it creates. +"""]] diff --git a/doc/todo/wishlist:_special-case_handling_of_Youtube_URLs_in_Web_special_remote/comment_3_a7e3cd68c5e5f05139151a58f358df95._comment b/doc/todo/wishlist:_special-case_handling_of_Youtube_URLs_in_Web_special_remote/comment_3_a7e3cd68c5e5f05139151a58f358df95._comment new file mode 100644 index 000000000..c4d8cf754 --- /dev/null +++ b/doc/todo/wishlist:_special-case_handling_of_Youtube_URLs_in_Web_special_remote/comment_3_a7e3cd68c5e5f05139151a58f358df95._comment @@ -0,0 +1,13 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.154.0.63" + subject="comment 3" + date="2013-08-22T18:44:15Z" + content=""" +Since the quvi urls are quite likely to break as the CDNs etc change things around, maybe it would be better to somehow have addurl tag an url as needing to be downloaded with quvi. Then `git annex get` could re-run quvi to get an url to download. + +We could expand url syntax for this. `quvi:http://youtube.com/foo` +This also allows for future expansion for other similar things. + +I'd be inclined to still make `addurl` automatically try quvi to see if an url is supported, rather than requiring the user fix up the url themselves. But if trying quvi turns out to be too expensive, manually specifying it in the url would also work. +"""]] diff --git a/doc/todo/wishlist:_special-case_handling_of_Youtube_URLs_in_Web_special_remote/comment_4_a57947ed257b28bbe995a68bfeb5eeaa._comment b/doc/todo/wishlist:_special-case_handling_of_Youtube_URLs_in_Web_special_remote/comment_4_a57947ed257b28bbe995a68bfeb5eeaa._comment new file mode 100644 index 000000000..ee0ab45e7 --- /dev/null +++ b/doc/todo/wishlist:_special-case_handling_of_Youtube_URLs_in_Web_special_remote/comment_4_a57947ed257b28bbe995a68bfeb5eeaa._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://rmunn.myopenid.com/" + nickname="rmunn" + subject="comment 4" + date="2013-08-24T15:31:36Z" + content=""" +Either quvi or youtube-dl might be a good possibility: youtube-dl has the --get-url option (or -g for short) that outputs just the download URL (and nothing else) to stdout. So if for some reason quvi turned out not to be suitable, it wouldn't necessarily mean the idea would have to be abandoned. +"""]] diff --git a/doc/todo/wishlist:_special-case_handling_of_Youtube_URLs_in_Web_special_remote/comment_5_a0612ae05dbda7f7935be648b42b30fc._comment b/doc/todo/wishlist:_special-case_handling_of_Youtube_URLs_in_Web_special_remote/comment_5_a0612ae05dbda7f7935be648b42b30fc._comment new file mode 100644 index 000000000..38ac09511 --- /dev/null +++ b/doc/todo/wishlist:_special-case_handling_of_Youtube_URLs_in_Web_special_remote/comment_5_a0612ae05dbda7f7935be648b42b30fc._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://id.koumbit.net/anarcat" + ip="72.0.72.144" + subject="aaaaawesome!" + date="2013-08-26T04:43:27Z" + content=""" +wow, thanks! i am happy my little suggestion led to an actual implentation, great! +"""]] diff --git a/doc/todo/wishlist:_special_remote_Ubuntu_One.mdwn b/doc/todo/wishlist:_special_remote_Ubuntu_One.mdwn new file mode 100644 index 000000000..b88a038ea --- /dev/null +++ b/doc/todo/wishlist:_special_remote_Ubuntu_One.mdwn @@ -0,0 +1 @@ +Special remote support for [Ubuntu One](http://one.ubuntu.com) would be nice. They're [using propietary but open protocol](https://wiki.ubuntu.com/UbuntuOne/TechnicalDetails#ubuntuone-storageprotocol) based on [Google Protocol Buffers](http://code.google.com/p/protobuf/). There's [protobuf for Haskell](http://code.google.com/p/protobuf-haskell/) so it should be possible to compile [the protocol file](http://bazaar.launchpad.net/~ubuntuone-control-tower/ubuntuone-storage-protocol/trunk/view/head:/ubuntuone/storageprotocol/protocol.proto) to Haskell code and then use that to implement the native Ubuntu special remote. diff --git a/doc/todo/wishlist:_special_remote_for_sftp_or_rsync.mdwn b/doc/todo/wishlist:_special_remote_for_sftp_or_rsync.mdwn new file mode 100644 index 000000000..d3c855655 --- /dev/null +++ b/doc/todo/wishlist:_special_remote_for_sftp_or_rsync.mdwn @@ -0,0 +1,28 @@ +i think it would be useful to have a fourth kind of [[special_remotes]] +that connects to a dumb storage using sftp or rsync. this can be emulated +by using sshfs, but that means lots of round-trips through the system and +is limited to platforms where sshfs is available. + +typical use cases are backups to storate shared between a group of people +where each user only has limited access (sftp or rsync), when using +[[special_remotes/bup]] is not an option. + +an alternative to implementing yet another special remote would be to have +some kind of plugin system by which external programs can provide an +interface to key-value stores (i'd implement the sftp backend myself, but +haven't learned haskell yet). + +> Ask and ye [[shall receive|special_remotes/rsync]]. +> +> Sometimes I almost think that a generic configurable special remote that +> just uses configured shell commands would be useful.. But there's really +> no comparison with sitting down and writing code tuned to work with +> a given transport like rsync, when it comes to reliability and taking +> advantage of its abilities (like resuming). --[[Joey]] + +>> big thanks, and bonus points for identical formats, so converting from +>> directory to rsync is just a matter of changing ``type`` from ``directory`` +>> to ``rsync`` in ``.git-annex/remote.log`` and replacing the directory info +>> with ``annex-rsyncurl = <host>:<dir>`` in ``.git/config``. --[[chrysn]] + +[[done]] diff --git a/doc/todo/wishlist:_special_remote_for_sftp_or_rsync/comment_1_6f07d9cc92cf8b4927b3a7d1820c9140._comment b/doc/todo/wishlist:_special_remote_for_sftp_or_rsync/comment_1_6f07d9cc92cf8b4927b3a7d1820c9140._comment new file mode 100644 index 000000000..c513ed400 --- /dev/null +++ b/doc/todo/wishlist:_special_remote_for_sftp_or_rsync/comment_1_6f07d9cc92cf8b4927b3a7d1820c9140._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawkSq2FDpK2n66QRUxtqqdbyDuwgbQmUWus" + nickname="Jimmy" + subject="comment 1" + date="2011-04-28T07:47:38Z" + content=""" ++1 for a generic user configurable backend that a user can put shell commands in, which has a disclaimer such that if a user hangs themselves with misconfiguration then its their own fault :P + +I would love to be able to quickly plugin an irods/sector set of put/get/delete/stat(get info) commands into git-annex to access my private clouds which aren't s3 compatible. +"""]] diff --git a/doc/todo/wishlist:_special_remote_for_sftp_or_rsync/comment_2_84e4414c88ae91c048564a2cdc2d3250._comment b/doc/todo/wishlist:_special_remote_for_sftp_or_rsync/comment_2_84e4414c88ae91c048564a2cdc2d3250._comment new file mode 100644 index 000000000..6243708f9 --- /dev/null +++ b/doc/todo/wishlist:_special_remote_for_sftp_or_rsync/comment_2_84e4414c88ae91c048564a2cdc2d3250._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joey.kitenet.net/" + nickname="joey" + subject="comment 2" + date="2011-04-28T21:22:03Z" + content=""" +Ask and ye shalle receive with an Abbot on top: [[special_remotes/hook]] +"""]] diff --git a/doc/todo/wishlist:_special_remote_for_sftp_or_rsync/comment_3_79de7ac44e3c0f0f5691a56d3fb88897._comment b/doc/todo/wishlist:_special_remote_for_sftp_or_rsync/comment_3_79de7ac44e3c0f0f5691a56d3fb88897._comment new file mode 100644 index 000000000..dc21ec488 --- /dev/null +++ b/doc/todo/wishlist:_special_remote_for_sftp_or_rsync/comment_3_79de7ac44e3c0f0f5691a56d3fb88897._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawkSq2FDpK2n66QRUxtqqdbyDuwgbQmUWus" + nickname="Jimmy" + subject="comment 3" + date="2011-04-29T10:43:31Z" + content=""" +Cool!, I just tried adding tahoe-lafs as a remote, and it wasn't too hard. +"""]] diff --git a/doc/todo/wishlist:_special_remote_mega.co.nz.mdwn b/doc/todo/wishlist:_special_remote_mega.co.nz.mdwn new file mode 100644 index 000000000..41164084a --- /dev/null +++ b/doc/todo/wishlist:_special_remote_mega.co.nz.mdwn @@ -0,0 +1,3 @@ +mega.co.nz has 50gb for free accounts. They also have an API, so I guess it wouldn't be too hard to use it as a special remote. + +[[done]], see [[tips/megaannex]]. diff --git a/doc/todo/wishlist:_special_remote_mega.co.nz/comment_2_6ca08ef808d4336fc42d0f279d6627b5._comment b/doc/todo/wishlist:_special_remote_mega.co.nz/comment_2_6ca08ef808d4336fc42d0f279d6627b5._comment new file mode 100644 index 000000000..542b92a67 --- /dev/null +++ b/doc/todo/wishlist:_special_remote_mega.co.nz/comment_2_6ca08ef808d4336fc42d0f279d6627b5._comment @@ -0,0 +1,44 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawmLB39PC89rfGaA8SwrsnB6tbumezj-aC0" + nickname="Tobias" + subject="Usage of mega hook" + date="2013-05-21T09:09:28Z" + content=""" +megaannex +========= + +Hook program for gitannex to use mega.co.nz as backend + +# Requirements: + + requests>=0.10 + pycrypto + +Credit for the mega api interface goes to: https://github.com/richardasaurus/mega.py + +## Install +Clone the git repository in your home folder. + + git clone git://github.com/TobiasTheViking/megaannex.git + +This should make a ~/megannex folder + +## Setup +Run the program once to make an empty config file + + cd ~/megaannex; python2 megaannex.py + +Edit the megaannex.conf file. Add your mega.co.nz username and password + +Note: The folder option in the megaannex.conf file isn't yet used. + +## Commands for gitannex: + + git config annex.mega-store-hook '/usr/bin/python2 ~/megaannex/megaannex.py store --subject $ANNEX_KEY --file $ANNEX_FILE' + git config annex.mega-retrieve-hook '/usr/bin/python2 ~/megaannex/megaannex.py getfile --subject $ANNEX_KEY --file $ANNEX_FILE' + git config annex.mega-checkpresent-hook '/usr/bin/python2 ~/megaannex/megaannex.py fileexists --subject $ANNEX_KEY' + git config annex.mega-remove-hook '/usr/bin/python2 ~/megaannex/megaannex.py delete --subject $ANNEX_KEY' + git annex initremote mega type=hook hooktype=mega encryption=shared + git annex describe mega \"the mega.co.nz library\" + +"""]] diff --git a/doc/todo/wishlist:_support_copy_--from__61__x_--to__61__y.mdwn b/doc/todo/wishlist:_support_copy_--from__61__x_--to__61__y.mdwn new file mode 100644 index 000000000..b4f966abb --- /dev/null +++ b/doc/todo/wishlist:_support_copy_--from__61__x_--to__61__y.mdwn @@ -0,0 +1,29 @@ +I'd like to be able to: + + git annex copy --from=x --to=y . + +Use case (true story) follows: + +My desktop hard drive was filling up. I dropped some large files which are also stored (via git-annex) on my backup drive. While these aren't irreplaceable files, I'd prefer to have at least two copies of everything I've decided I care enough about to archive. Later, I get a 2nd external drive, and I: + + git annex copy --to=new-external-drive . + +Fantastic! Now I've got everything that was important/useful enough to keep on my desktop backed up a 2nd time onto my new drive. + +But my new drive doesn't have a copy of any of the files I dropped from my desktop. I would like to be able to: + + git annex copy --from=old-external-drive --to=new-external-drive . + +on my desktop, and then my new drive would have a copy of everything, and my desktop drive would still have plenty of space (ie the files I'd dropped to make space would still not be stored on the desktop). + +The git repos on these external drives are both bare (as in ``git init --bare``) because they are used only for backups. Thus I operate on them only as remotes from my main (desktop) repo. + +> I have now implemented the --all option, and it's the default when +> running `git annex get` inside a bare repo. +> +> So, the solution is to `cd` to the repository on old-external-drive, +> and `git remote add newdrive /path/to/new/drive/repo`. Then run `git +> annex copy --all --to newdrive` and it'll move everything. +> +> Calling this [[done]] unless there are other use cases where the double +> copy method is really needed? --[[Joey]] diff --git a/doc/todo/wishlist:_support_copy_--from__61__x_--to__61__y/comment_1_cf8e0f16b723516374c95a93e4da42fc._comment b/doc/todo/wishlist:_support_copy_--from__61__x_--to__61__y/comment_1_cf8e0f16b723516374c95a93e4da42fc._comment new file mode 100644 index 000000000..cee50a345 --- /dev/null +++ b/doc/todo/wishlist:_support_copy_--from__61__x_--to__61__y/comment_1_cf8e0f16b723516374c95a93e4da42fc._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.154.4.193" + subject="comment 1" + date="2013-06-30T17:43:50Z" + content=""" +A reasonable use case indeed. + +It seems to me that [[add_-all_option]] could also satisfy this use case, as then you could run `git annex get --all` in the new bare remote. + +That would have the benefit of not doing a double copy. +"""]] diff --git a/doc/todo/wishlist:_support_copy_--from__61__x_--to__61__y/comment_2_d35359c9dd4dd4365d9a7caf695ff833._comment b/doc/todo/wishlist:_support_copy_--from__61__x_--to__61__y/comment_2_d35359c9dd4dd4365d9a7caf695ff833._comment new file mode 100644 index 000000000..8305679a3 --- /dev/null +++ b/doc/todo/wishlist:_support_copy_--from__61__x_--to__61__y/comment_2_d35359c9dd4dd4365d9a7caf695ff833._comment @@ -0,0 +1,16 @@ +[[!comment format=mdwn + username="http://jasonwoof.com/" + nickname="JasonWoof" + subject="thanks, good enough for now." + date="2013-07-15T19:27:58Z" + content=""" +the ``--all`` option works for this use case. That takes care of my problem. Thank you! + +I can imagine other use cases where I'd want ``--from`` and ``--to`` at once, such as: + +1. same situation with my two bare external drives, but I want to only copy my audio-book collection to the new drive, and not my movies. + +2. I've got a large online storage (eg rsync.net) and want copy everything from there onto my new external drive. + +I leave it up to your good judgement when/if this is worth doing. +"""]] diff --git a/doc/todo/wishlist:_support_drop__44___find_on_special_remotes.mdwn b/doc/todo/wishlist:_support_drop__44___find_on_special_remotes.mdwn new file mode 100644 index 000000000..24cacbf71 --- /dev/null +++ b/doc/todo/wishlist:_support_drop__44___find_on_special_remotes.mdwn @@ -0,0 +1,18 @@ +Currently there is no way to drop files, or list what files are available, on a special remote. +It would be good if "git annex drop" and "git annex find" supported the --from argument. + +> I agree, drop should support --from. +>> [[done]] --[[Joey]] +> +> To find files *believed* to be present in a given remote, use +> `git annex find --in remote` +> Note that it might show out of date info, since it does not actually go +> check the current contents of the remote. The only reason to support +> `find --from` would be to always check, but I don't think that's needed. +> --[[Joey]] + +For commands that don't support the --from argument, it would also be nice to print an error. +Currently running "git annex drop --from usbdrive" doesn't behave as hoped and instead drops +all content from the local annex. + +> This is done now. --[[Joey]] diff --git a/doc/todo/wishlist:_support_drop__44___find_on_special_remotes/comment_1_f11ed642a83d965076778a162f701e84._comment b/doc/todo/wishlist:_support_drop__44___find_on_special_remotes/comment_1_f11ed642a83d965076778a162f701e84._comment new file mode 100644 index 000000000..6028933b4 --- /dev/null +++ b/doc/todo/wishlist:_support_drop__44___find_on_special_remotes/comment_1_f11ed642a83d965076778a162f701e84._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://joey.kitenet.net/" + nickname="joey" + subject="comment 1" + date="2011-10-27T17:13:43Z" + content=""" +Well, I don't think you mean \"special remotes\", but just any old remote (special or not). +"""]] diff --git a/doc/todo/wishlist:_support_for_more_ssh_urls_.mdwn b/doc/todo/wishlist:_support_for_more_ssh_urls_.mdwn new file mode 100644 index 000000000..55b8120a7 --- /dev/null +++ b/doc/todo/wishlist:_support_for_more_ssh_urls_.mdwn @@ -0,0 +1,22 @@ +git-annex does not seem to support all kinds of urls that git does. + +Specifically, if I have ~/bar set up on host foo: + + [remote "foo"] + ## this one is not recognized as ssh url at all + # url = foo:bar + ## this one makes git-annex try to access '/~/bar' literally + # url = ssh://foo/~/bar + ## this one works + url = ssh://foo/home/tv/bar + +> scp-style is now supported. + +> `~` expansions (for the user's home, or other users) +> are somewhat tricky to support as they require running +> code on the remote to lookup homedirs. If git-annex grows a +> `git annex shell` that is run on the remote side +> (something I am [[considering|todo/git-annex-shell]] for other reasons), it +> could handle the expansions there. --[[Joey]] + +> Update: Now `~` expansions are supported. [[done]] diff --git a/doc/todo/wishlist:_swift_backend.mdwn b/doc/todo/wishlist:_swift_backend.mdwn new file mode 100644 index 000000000..28bd265fa --- /dev/null +++ b/doc/todo/wishlist:_swift_backend.mdwn @@ -0,0 +1,5 @@ +[swift](http://swift.openstack.org/) is the object storage of Openstack. Think S3, but fully open source. As it's backed by rackspace.com, NASA, Dell and several other major players, adoption rates will explode. + +I can provide a test account soonish if need be, else rackspace.com if offering swift storage. Their API gateway lives at https://auth.api.rackspacecloud.com/v1.0 + +Richard diff --git a/doc/todo/wishlist:_swift_backend/comment_1_e6efbb35f61ee521b473a92674036788._comment b/doc/todo/wishlist:_swift_backend/comment_1_e6efbb35f61ee521b473a92674036788._comment new file mode 100644 index 000000000..98a998c1c --- /dev/null +++ b/doc/todo/wishlist:_swift_backend/comment_1_e6efbb35f61ee521b473a92674036788._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawkSq2FDpK2n66QRUxtqqdbyDuwgbQmUWus" + nickname="Jimmy" + subject="comment 1" + date="2011-05-14T10:04:36Z" + content=""" +I don't suppose this SWIFT api is compatible with the eucalytpus walrus api ? +"""]] diff --git a/doc/todo/wishlist:_swift_backend/comment_2_5d8c83b0485112e98367b7abaab3f4e3._comment b/doc/todo/wishlist:_swift_backend/comment_2_5d8c83b0485112e98367b7abaab3f4e3._comment new file mode 100644 index 000000000..97863b095 --- /dev/null +++ b/doc/todo/wishlist:_swift_backend/comment_2_5d8c83b0485112e98367b7abaab3f4e3._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawl9sYlePmv1xK-VvjBdN-5doOa_Xw-jH4U" + nickname="Richard" + subject="comment 2" + date="2011-05-14T15:00:51Z" + content=""" +It does offer a S3 compability layer, but that is de facto non-functioning as of right now. +"""]] diff --git a/doc/todo/wishlist:_swift_backend/comment_3_bf8625b909c3a7321cae40e6f145e874._comment b/doc/todo/wishlist:_swift_backend/comment_3_bf8625b909c3a7321cae40e6f145e874._comment new file mode 100644 index 000000000..90af10c41 --- /dev/null +++ b/doc/todo/wishlist:_swift_backend/comment_3_bf8625b909c3a7321cae40e6f145e874._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://ertai.myopenid.com/" + nickname="npouillard" + subject="+1" + date="2012-09-18T08:52:21Z" + content=""" +OVH (french IT company) is migrating its could storage infrastructure to swift/openstack and the next few weeks. +"""]] diff --git a/doc/todo/wishlist:_swift_backend/comment_4_4d97d12ddd99834788e94648c8eebef9._comment b/doc/todo/wishlist:_swift_backend/comment_4_4d97d12ddd99834788e94648c8eebef9._comment new file mode 100644 index 000000000..21fa52aaf --- /dev/null +++ b/doc/todo/wishlist:_swift_backend/comment_4_4d97d12ddd99834788e94648c8eebef9._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawm5Z8wzidsumLYQnHdrVxpCx8vsd9llSlg" + nickname="Emanuele" + subject="comment 4" + date="2013-11-22T15:52:42Z" + content=""" +OVH is offering free accounts with 25GB of storage on their hubiC service: https://hubic.com/en/ + +The API documentation (based on OAuth2 and OpenStack Swift) is available at https://api.hubic.com/ +"""]] diff --git a/doc/todo/wishlist:_traffic_accounting_for_git-annex.mdwn b/doc/todo/wishlist:_traffic_accounting_for_git-annex.mdwn new file mode 100644 index 000000000..4b661101d --- /dev/null +++ b/doc/todo/wishlist:_traffic_accounting_for_git-annex.mdwn @@ -0,0 +1,3 @@ +As git annex keeps logs about file transfers anyway, it should be relatively easy to add traffic accounting to a repo. That would allow me to monitor how much traffic a given repo generates. As I might end up hosting git-annex repos for a few personal friends, I need/want a way to track the heavy hitters. -- RichiH + +PS: If you ever plan to host git-annex similar branchable, this would probably be of interest to you, as well :) diff --git a/doc/todo/wishlist:_unify_directory_scheme_for_the_store.mdwn b/doc/todo/wishlist:_unify_directory_scheme_for_the_store.mdwn new file mode 100644 index 000000000..83ce53127 --- /dev/null +++ b/doc/todo/wishlist:_unify_directory_scheme_for_the_store.mdwn @@ -0,0 +1,20 @@ +In regular repos, objects are stored in files of the form: .git/annex/objects/xY/z1/SHA1-.../SHA1-.... (scheme 1) + +On (some) special remotes, the corresponding file is stored at: .../abc/def/SHA1-... (scheme 2) + +I'm not sure why the same scheme as in .git/objects isn't used, but it would be useful that the two-directory prefix were the same for all objects stores. + +My use case is: I synchronize a git repo, say containing photos, to a server on which I can't install git-annex. I want the server to store all annexed files. For the photos to be viewed online, the annex store must use the scheme 1 (because the symlinks point to files with scheme 1). So I need to rsync .git/annex/objects manually from my desktop, because a git-annex rsync remote uses scheme 2. On the other hand, the repo on this server is not known by git-annex (like it would if I used a rsync remote). + +At least it would be valuable (to get around above problem) to have a plumbing command giving the 2-directory prefix from a given key, for example: + +$ git annex prefix-dir SHA1-s2--3f786850e387550fdab836ed7e6dc881 + +7w/88 + +f18/122 + + +Even if the 2 schemes were unified, this prefix-dir command would still be useful when hacking around git-annex (for now I need to maintain a dictionary structure). + +Thanks a lot. diff --git a/doc/todo/wishlist:_unify_directory_scheme_for_the_store/comment_1_44da58beaaab359ecaba7fb905ca4ae1._comment b/doc/todo/wishlist:_unify_directory_scheme_for_the_store/comment_1_44da58beaaab359ecaba7fb905ca4ae1._comment new file mode 100644 index 000000000..86c33d434 --- /dev/null +++ b/doc/todo/wishlist:_unify_directory_scheme_for_the_store/comment_1_44da58beaaab359ecaba7fb905ca4ae1._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.152.108.243" + subject="comment 1" + date="2013-10-04T20:50:33Z" + content=""" +Using the mixed case hash directory names is not desirable because people want to use git-annex on a variety of filesystems and operating systems that treat them in a variety of broken ways. However, migrating to the all lower case hash directory names would require changing every git-annex symlink in every git repository, and I do not want to inflict that on my users. + +It seems to me that the best solution to your problem is to install git-annex on your server, which should not be very hard. +"""]] diff --git a/doc/todo/wishlist:_unify_directory_scheme_for_the_store/comment_2_bc698c501ecdb56df57171f4f3bb831a._comment b/doc/todo/wishlist:_unify_directory_scheme_for_the_store/comment_2_bc698c501ecdb56df57171f4f3bb831a._comment new file mode 100644 index 000000000..5c5269f8c --- /dev/null +++ b/doc/todo/wishlist:_unify_directory_scheme_for_the_store/comment_2_bc698c501ecdb56df57171f4f3bb831a._comment @@ -0,0 +1,16 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawnpdM9F8VbtQ_H5PaPMpGSxPe_d5L1eJ6w" + nickname="Rafaël" + subject="comment 2" + date="2013-10-05T10:45:16Z" + content=""" +Thank you for prompt answer. I didn't know there were tarballs, +and indeed I managed to install them easily (although I had to +manually install glibc version 2.9, only 2.7 was installed). + +I found that bare git repos also use lower case hash directory +names... I still would be happy with an optional migration to all +lower case, with a new key-value backend, to avoid this little +complication which happens sometimes (say when converting a repo +from non-bare to bare). +"""]] diff --git a/doc/todo/wishlist:_unify_directory_scheme_for_the_store/comment_3_e555d0dbbaa05528806905c6a940724b._comment b/doc/todo/wishlist:_unify_directory_scheme_for_the_store/comment_3_e555d0dbbaa05528806905c6a940724b._comment new file mode 100644 index 000000000..e6acad3b9 --- /dev/null +++ b/doc/todo/wishlist:_unify_directory_scheme_for_the_store/comment_3_e555d0dbbaa05528806905c6a940724b._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawnpdM9F8VbtQ_H5PaPMpGSxPe_d5L1eJ6w" + nickname="Rafaël" + subject="comment 3" + date="2013-10-07T13:33:40Z" + content=""" +By the way, I just had the case above, i.e. convert a bare repo to non-bare. In order to keep the annex files, I cloned the bare one, and git-annex move'ed all annex content to the new repo, and to my surprise it was slow, as if all files where copied (or maybe they were only checksummed?), instead of being only renamed (old and new repos were on the same partition). So I restate that at least a command line tool giving the prefix dirs would be useful, to allow scripting for this kind of situation. +"""]] diff --git a/doc/todo/wishlist:_use_hardlinks_for_local_clones.mdwn b/doc/todo/wishlist:_use_hardlinks_for_local_clones.mdwn new file mode 100644 index 000000000..4b0694422 --- /dev/null +++ b/doc/todo/wishlist:_use_hardlinks_for_local_clones.mdwn @@ -0,0 +1,9 @@ +as far as I know, if you `git clone` locally a git-annex enabled repository, it will not have all the files available. you would need to use `git annex get` and all files would be copied over, wasting a significant amount of space. + +`git-clone` has this `--local` flags which hardlinks objects in `.git/objects`, but also, maybe more interestingly, has a `--shared` option to simply tell git to look in another repo for objects. it seems to me git-annex could leverage those functionalities to avoid file duplication when using local repositories. + +this would be especially useful for [ikiwiki](http://ikiwiki.info/forum/ikiwiki_and_big_files). + +This is a [[wishlist]], but I would also welcome implementation pointers to do this myself, thanks! --[[anarcat]] + +> [[dup|done]] diff --git a/doc/todo/wishlist:_use_hardlinks_for_local_clones/comment_1_85064fafe472a5bd395d60ce8f7acb56._comment b/doc/todo/wishlist:_use_hardlinks_for_local_clones/comment_1_85064fafe472a5bd395d60ce8f7acb56._comment new file mode 100644 index 000000000..4ef5f8414 --- /dev/null +++ b/doc/todo/wishlist:_use_hardlinks_for_local_clones/comment_1_85064fafe472a5bd395d60ce8f7acb56._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + ip="4.152.108.220" + subject="comment 1" + date="2013-09-25T17:14:28Z" + content=""" +git-annex uses cp --reflink=auto. So on a filesystem supporting COW file copies, like btrfs, `git annex get` will not use any disk space when getting from the same filesystem. + +I do not like the idea of using hardlinks, because changing the file in one repository would change it in the other, which may not be desired. + +[[union_mounting]] seems to cover this item pretty well, so I will close this as a duplicate. +"""]] diff --git a/doc/todo/wishlist:_vicfg_possible_repo_group_names.mdwn b/doc/todo/wishlist:_vicfg_possible_repo_group_names.mdwn new file mode 100644 index 000000000..e30cc619a --- /dev/null +++ b/doc/todo/wishlist:_vicfg_possible_repo_group_names.mdwn @@ -0,0 +1,16 @@ +git annex vicfg should display valid repository group names + +For trust levels the possible values are displayed: + + # Repository trust configuration + # (Valid trust levels: trusted semitrusted untrusted dead) + ... + +The same is not currently done for repository groups + + # Repository groups + # (Separate group names with spaces) + +Thanks. + +> [[done]] --[[Joey]] diff --git a/doc/todo/wishlist:alias_system.mdwn b/doc/todo/wishlist:alias_system.mdwn new file mode 100644 index 000000000..1f5012966 --- /dev/null +++ b/doc/todo/wishlist:alias_system.mdwn @@ -0,0 +1 @@ +To implement things like my custom `git annex-push` without the dash, i.e. `git annex push`, an alias system for git-annex would be nice. diff --git a/doc/todo/wishlist_degraded_files.mdwn b/doc/todo/wishlist_degraded_files.mdwn new file mode 100644 index 000000000..0b265c5eb --- /dev/null +++ b/doc/todo/wishlist_degraded_files.mdwn @@ -0,0 +1,5 @@ +This is an idea to have a small placeholder file that is put into +place when the file's actual content is not available in the local +annex. + +Details being discussed here: <http://bugs.debian.org/728552> |