diff options
Diffstat (limited to 'doc')
11 files changed, 119 insertions, 9 deletions
diff --git a/doc/bugs/CHECKPRESENT_could_check_file_size_as_well/comment_1_7bbba896da745a8a6b69ec62bf71fc89._comment b/doc/bugs/CHECKPRESENT_could_check_file_size_as_well/comment_1_7bbba896da745a8a6b69ec62bf71fc89._comment new file mode 100644 index 000000000..e917ec998 --- /dev/null +++ b/doc/bugs/CHECKPRESENT_could_check_file_size_as_well/comment_1_7bbba896da745a8a6b69ec62bf71fc89._comment @@ -0,0 +1,13 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + subject="comment 1" + date="2015-01-06T17:41:45Z" + content=""" +The encrypted content size is not constant, and not known to git-annex. + +The only git-annex remote that checks the size in its checkpresent implementation is the web special remote, precisely because it's never encrypted. Also because files on the web change content from time to time and so that needs to be detected. + +What would make sense is to extend the reply to `CHECKPRESENT-SUCCESS Key [size]` or perhaps `CHECKPRESENT-SIZE Key size`. git-annex can then compare the value with the key's known size, if any. If the key is encrypted, it would need to skip this check. + +Note that chunk keys currently have their keySize inherited from the parent key, and the keyChunkSize of each chunk key is set to the key size. The last chunk of a key will typically be shorter than its keyChunkSize. That would need to be cleaned up. +"""]] diff --git a/doc/bugs/__34__git-annex:_direct:_1_failed__34___on_Windows/comment_2_dfc398002e2ffbe0b63ce422a1e16d67._comment b/doc/bugs/__34__git-annex:_direct:_1_failed__34___on_Windows/comment_2_dfc398002e2ffbe0b63ce422a1e16d67._comment new file mode 100644 index 000000000..89af53b41 --- /dev/null +++ b/doc/bugs/__34__git-annex:_direct:_1_failed__34___on_Windows/comment_2_dfc398002e2ffbe0b63ce422a1e16d67._comment @@ -0,0 +1,25 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + subject="comment 2" + date="2015-01-06T19:18:26Z" + content=""" +On Linux and OSX, there is a maximum filename size, typically 255 bytes. git-annex always ensures that keys it generates are a maximum of 255 bytes long, no matter the platform. But, in dir/subdir/file, each of the 3 segments of the path is allowed to be that long. The limit on the total path size on Linux is a more reasonable 4096 bytes; OSX has only 1024 bytes. + +I don't know what to do about Windows having such an absurdly small `MAX_PATH` compared to more modern systems. + +The length of just a SHA512 checksum is 128 bytes; that means SHA512 backend cannot be used on windows, at all, since the paths git-annex generates will be at least twice that long, and will easily overflow `PATH_MAX`. I've confirmed this; just adding a file with --backend=SHA512 fails with a \"No such file or directory\" error when it tries to use the path. + +A SHA256 is a more manageable 64 bytes long. So a typical path to such an object will end with eg \".git\annex\objects\566\a33\SHA256E--d728a4c4727febe1c28509482ae1b7b2215798218e544eed7cb7b4dc988f838b\SHA256E--d728a4c4727febe1c28509482ae1b7b2215798218e544eed7cb7b4dc988f838b\" -- 174 bytes long (or a bit longer when there are also extension and size in the key) and leaving only 86 bytes or so for `c:\path\to\repo`. + +Perhaps git-annex should reduce its maximum key size from 255 to 64 bytes, the same as SHA256. Then url keys would work on Windows, except for in deep paths, where git-annex cannot work at all. This would be an easy change. + +git-annex could also avoid using absolute paths, which it currently uses extensively for simplicity (and possiibly robustness against renames of repositories and changes of working directory?), and use relative paths instead. This would probably solve the two examples given in the bug report, and it would make git-annex work better when in a deep path in Windows. It would not make SHA512 work though; with keys that long, the relative path is still too long. (And, it's still possible to get a relative path that has so many '../../' and subdirectories etc that it overflows `PATH_MAX`. It would probably take a really crazy repository directory structure though.) + +The MSDN article has one very interesting bit: + +> The Windows API has many functions that also have Unicode versions to permit an extended-length path for a maximum total path length of 32,767 characters. This type of path is composed of components separated by backslashes, each up to the value returned in the lpMaximumComponentLength parameter of the GetVolumeInformation function (this value is commonly 255 characters). To specify an extended-length path, use the \"\\?\\" prefix. For example, \"\\?\D:\very long path\". + +(It seems that, when using that prefix, `/` is not converted to `\` .. I think git-annex is quite good about getting the slashes the right way round these days.) + +So it might be possible for git-annex to use that prefix and avoid this issue entirely. Haskell's FilePath library does understand that prefix (treats it as part of the drive). Since git-annex always uses the path to the top of the Repo when constructing the problimatic FilePaths, I might be able to just change the Repo constructor to add that prefix, and everything follow from that. I tried doing that, unfortunately this makes *git* fail, with \"fatal: relative path syntax cannot be used outside working tree\" when operating on such a repo. Cause git doesn't understand that prefix. +"""]] diff --git a/doc/bugs/__34__git-annex:_direct:_1_failed__34___on_Windows/comment_3_1d23e9760782a8d6d2ea2dd5a4c6253a._comment b/doc/bugs/__34__git-annex:_direct:_1_failed__34___on_Windows/comment_3_1d23e9760782a8d6d2ea2dd5a4c6253a._comment new file mode 100644 index 000000000..153c48db2 --- /dev/null +++ b/doc/bugs/__34__git-annex:_direct:_1_failed__34___on_Windows/comment_3_1d23e9760782a8d6d2ea2dd5a4c6253a._comment @@ -0,0 +1,9 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + subject="comment 3" + date="2015-01-06T20:51:20Z" + content=""" +I've started a `relativepaths` branch that uses all relative paths to the git repo. After working on it for several hours, there are still 16 test suite failures (update: 10) (update: 1). The potential for uncaught breakage is much higher than I am happy with. (Amoung other problems, git-annex does call setCurrentDirectory in several places, and this utterly breaks the relative paths). + +Using that branch on windows, I am still unable to add files with --backend=SHA512; even relative paths don't make it short enough for such keys. +"""]] diff --git a/doc/bugs/__34__git-annex:_direct:_1_failed__34___on_Windows/comment_4_108f3e4449fc9591bcdeb490b486357f._comment b/doc/bugs/__34__git-annex:_direct:_1_failed__34___on_Windows/comment_4_108f3e4449fc9591bcdeb490b486357f._comment new file mode 100644 index 000000000..3aa56f004 --- /dev/null +++ b/doc/bugs/__34__git-annex:_direct:_1_failed__34___on_Windows/comment_4_108f3e4449fc9591bcdeb490b486357f._comment @@ -0,0 +1,9 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + subject="comment 4" + date="2015-01-06T21:59:15Z" + content=""" +Even with relative paths, Edward's example would use a path of 253 characters, and so a slightly longer url would still break it, even with relative paths. + +So, I think reducing url key length needs to be done anyway, and I've done that. Which hardly closes this bug. +"""]] diff --git a/doc/bugs/__91__Android__93___5.0_needs_PIE_executables___40__git_annex_does_not_work_on_android_5.0__41__/comment_7_c9104ad6d9cfec93f561b9cd514b6f48._comment b/doc/bugs/__91__Android__93___5.0_needs_PIE_executables___40__git_annex_does_not_work_on_android_5.0__41__/comment_7_c9104ad6d9cfec93f561b9cd514b6f48._comment new file mode 100644 index 000000000..3d4f7e230 --- /dev/null +++ b/doc/bugs/__91__Android__93___5.0_needs_PIE_executables___40__git_annex_does_not_work_on_android_5.0__41__/comment_7_c9104ad6d9cfec93f561b9cd514b6f48._comment @@ -0,0 +1,13 @@ +[[!comment format=mdwn + username="http://joeyh.name/" + subject="comment 7" + date="2015-01-06T17:43:45Z" + content=""" +I verified with `eu-readelf --file-header` that the git-annex binary is DYN; ie linked PIE. + +It might be that I also need to tell the C compiler to build it with PIE options. I have now updated the build to include that. Please test the new build. + +It occurs to me that the problem might be not git-annex, but one of the other binaries, like busybox. Does the android app install to the point that there is a working terminal app with a shell? + +It also seems possible that the entire haskell library stack might need to be built with PIE options. If so, that will be a massive pain; I'd need an entire separate autobuilder instance. +"""]] diff --git a/doc/bugs/__91__Android__93___5.0_needs_PIE_executables___40__git_annex_does_not_work_on_android_5.0__41__/comment_8_e628cc5365b1268450f0245b362266cc._comment b/doc/bugs/__91__Android__93___5.0_needs_PIE_executables___40__git_annex_does_not_work_on_android_5.0__41__/comment_8_e628cc5365b1268450f0245b362266cc._comment new file mode 100644 index 000000000..ad25a2694 --- /dev/null +++ b/doc/bugs/__91__Android__93___5.0_needs_PIE_executables___40__git_annex_does_not_work_on_android_5.0__41__/comment_8_e628cc5365b1268450f0245b362266cc._comment @@ -0,0 +1,7 @@ +[[!comment format=mdwn + username="etset" + subject="Still not working" + date="2015-01-06T22:48:18Z" + content=""" +The terminal opens, showing the error message at start and at every new opened tab, without a working shell ever appearing. +"""]] diff --git a/doc/devblog/day_244__relative_paths.mdwn b/doc/devblog/day_244__relative_paths.mdwn new file mode 100644 index 000000000..f631dbc8f --- /dev/null +++ b/doc/devblog/day_244__relative_paths.mdwn @@ -0,0 +1,14 @@ +git-annex internally uses all absolute paths all the time. +For a couple of reasons, I'd like it to use relative paths. +The best reason is, it would let a repository be moved while git-annex was +running, without breaking. A lesser reason is that Windows has some +crazy small limit on the length of a path (260 bytes?!), and using relative +paths would avoid hitting it so often. + +I tried to do this today, in a `relativepaths` branch. I eventually got the +test suite to pass, but I am very unsure about this change. A lot of random +assumptions broke, and the test suite won't catch them all. In a few +places, git-annex commands do change the current directory, and that +will break with relative paths. + +A frustrating day. diff --git a/doc/forum/How_To_Permanently_Delete_a_File__63__/comment_7_1a6fab467a1c4db2dee39ef464f6cc7b._comment b/doc/forum/How_To_Permanently_Delete_a_File__63__/comment_7_1a6fab467a1c4db2dee39ef464f6cc7b._comment new file mode 100644 index 000000000..0b1f6812b --- /dev/null +++ b/doc/forum/How_To_Permanently_Delete_a_File__63__/comment_7_1a6fab467a1c4db2dee39ef464f6cc7b._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawmwjQzWgiD7_I3zw-_91rMRf_6qoThupis" + nickname="Mike" + subject="comment 7" + date="2015-01-06T18:01:40Z" + content=""" +Unfortunately, that is not useful for this at all. We are talking about millions of files here, and the issue is leaving behind old hard links, so that program just won't work at all. + +Furthermore, this questions has all ready been answered in previous comments. +"""]] diff --git a/doc/tips/How_to_retroactively_annex_a_file_already_in_a_git_repo/comment_5_3062c0794ecd7c6237efae66f4d9b62f._comment b/doc/tips/How_to_retroactively_annex_a_file_already_in_a_git_repo/comment_5_3062c0794ecd7c6237efae66f4d9b62f._comment new file mode 100644 index 000000000..e8ef98d99 --- /dev/null +++ b/doc/tips/How_to_retroactively_annex_a_file_already_in_a_git_repo/comment_5_3062c0794ecd7c6237efae66f4d9b62f._comment @@ -0,0 +1,13 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawlcfH7xkyz1kyG_neK4GcFFfFWuIY7l_6A" + nickname="Primiano" + subject="large scale rewrite tips" + date="2015-01-06T22:55:20Z" + content=""" +I recently had the need of re-kind-of-annexing an unusually large repo (one of the largest?). +With some tricks and the right code I managed to get it down to 170000 commits in 19 minutes and extracing ~8GB of blobs. +Attaching the link here as I feel it might be helpful for very large projects (where git-filter-branch can become prohibitively slow) + +[https://www.primianotucci.com/blog/large-scale-git-history-rewrites](https://www.primianotucci.com/blog/large-scale-git-history-rewrites) + +"""]] diff --git a/doc/tips/using_the_web_as_a_special_remote/comment_10_14fb0b1ee50136e0f78ee2b2a6871467._comment b/doc/tips/using_the_web_as_a_special_remote/comment_10_14fb0b1ee50136e0f78ee2b2a6871467._comment deleted file mode 100644 index 0d1790ce9..000000000 --- a/doc/tips/using_the_web_as_a_special_remote/comment_10_14fb0b1ee50136e0f78ee2b2a6871467._comment +++ /dev/null @@ -1,9 +0,0 @@ -[[!comment format=mdwn - username="edward" - subject="URL backend file paths hit the 260 character file path limit on Windows" - date="2014-12-08T19:13:39Z" - content=""" -It isn't possible to checkout a git annex repository on Windows that includes quvi videos because the file path is often greater than 260 characters. - -See [[bugs/\"git-annex: direct: 1 failed\" on Windows]]. -"""]] diff --git a/doc/todo/windows_support.mdwn b/doc/todo/windows_support.mdwn index 3081e425e..42f804195 100644 --- a/doc/todo/windows_support.mdwn +++ b/doc/todo/windows_support.mdwn @@ -3,6 +3,12 @@ usable! ## status +* There can be problems when the git-annex repository is in a deep + or long path. Ie, `C:\loooooooooooooooooongdir\`. + [Details here](http://git-annex.branchable.com/bugs/__34__git-annex:_direct:_1_failed__34___on_Windows) + Workaround: Put your git-annex repo in `C:\annex` or some similar short + path if possible. + * XMPP library not yet built. (See below.) * Local pairing seems to fail, after acking on Linux box, it stalls. |