summaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAge
* leave export logged as incomplete if initial renames failGravatar Joey Hess2017-09-12
| | | | | | | | | | This way, the temp files that might be left due to failure will be cleaned up next time. Also, nub the list of incomplete exports to avoid repeatedly adding the same tree to it when running export repeatedly when it's failing. This commit was supported by the NSF-funded DataLad project.
* export to webdavGravatar Joey Hess2017-09-12
| | | | | | | | | | | This basically works, but there's a bug when renaming a file that leaves a .git-annex-temp-content-key file in the webdav store, that never gets cleaned up. Also, exporting files with spaces to box.com seems to fail; perhaps it does not support it? This commit was supported by the NSF-funded DataLad project.
* update copyright yearGravatar Joey Hess2017-09-12
|
* propigate exception in checkPresentExportS3Gravatar Joey Hess2017-09-12
| | | | checkPresentExport is supposed to throw exceptions
* stop warning about removals from IAGravatar Joey Hess2017-09-12
| | | | | | | In a test, I uploaded a pdf, and several files were derived from it. After removing the pdf, the derived files went away after approximatly half an hour. This window does not seem worth warning about every time. Documented it in the tip.
* avoid showing error when copy failsGravatar Joey Hess2017-09-12
| | | | | | | | | | | | | Since renameExport is allowed to fail for any reason, and its failure is always recovered from by doing a new upload and deleting the old content, this avoids unnecessary noise. Copying a file on the IA failed, apparently something wrong with their emulation of S3: S3Error {s3StatusCode = Status {statusCode = 400, statusMessage = "Bad Request"}, s3ErrorCode = "InvalidArgument", s3ErrorMessage = "Invalid Argument", s3ErrorResource = Just "x-(amz|archive)-copy-source header is bad: 'joeyh-public-test2/foo'", s3ErrorHostId = Nothing, s3ErrorAccessKeyId = Nothing, s3ErrorStringToSign = Nothing, s3ErrorBucket = Nothing, s3ErrorEndpointRaw = Nothing, s3ErrorEndpoint = Nothing} This commit was sponsored by Jake Vosloo on Patreon.
* S3: Allow removing files from IA, but warn about derived versions ↵Gravatar Joey Hess2017-09-12
| | | | | | | | | | | | | potentially still existing there. Removal works, only derives are a potential issue, so allow removing with a warning. This way, unexporting a file works, and behavior is consistent with IA remotes whether or not exporttree=yes. Also tested exporting filenames containing unicode, spaces, underscores. All worked, despite the IA's faq saying it doesn't. This commit was sponsored by Trenton Cronholm on Patreon.
* Added a comment: Issue also affects Samsung devices, git unaffectedGravatar https://christian.amsuess.com/chrysn2017-09-11
|
* Added a comment: The initremote command appears to hang due to low entropyGravatar NathanCollins2017-09-10
|
* (no commit message)Gravatar ilovezfs2017-09-09
|
* closeGravatar Joey Hess2017-09-09
|
* (no commit message)Gravatar ilovezfs2017-09-09
|
* very minor typoGravatar yarikoptic2017-09-08
|
* very delayed response now that feature is addedGravatar Joey Hess2017-09-08
|
* don't show key urls in whereis for S3 with public=yes and exporttree=yesGravatar Joey Hess2017-09-08
|
* consistencyGravatar Joey Hess2017-09-08
|
* devblogGravatar Joey Hess2017-09-08
|
* Merge branch 'master' of ssh://git-annex.branchable.comGravatar Joey Hess2017-09-08
|\
* | S3 export finalizationGravatar Joey Hess2017-09-08
| | | | | | | | Fixed ACL issue, and updated some documentation.
* | S3 export (untested)Gravatar Joey Hess2017-09-08
| | | | | | | | | | | | | | | | | | | | | | It opens a http connection per file exported, but then so does git annex copy --to s3. Decided not to munge exported filenames for IA. Too large a chance of the munging having confusing results. Instead, export of files not supported by IA, eg with spaces in their name, will fail. This commit was supported by the NSF-funded DataLad project.
* | External special remote protocol extended to support export.Gravatar Joey Hess2017-09-08
| | | | | | | | | | | | Also updated example.sh to support export. This commit was supported by the NSF-funded DataLad project.
| * Added a commentGravatar karel-de-macil2017-09-08
|/
* devblogGravatar Joey Hess2017-09-07
|
* mention git-annex exportGravatar Joey Hess2017-09-07
|
* updateGravatar Joey Hess2017-09-07
|
* correctionGravatar Joey Hess2017-09-07
|
* commentGravatar Joey Hess2017-09-07
|
* Merge branch 'master' of ssh://git-annex.branchable.comGravatar Joey Hess2017-09-07
|\
* \ Merge branch 'export'Gravatar Joey Hess2017-09-07
|\ \
| * | interrupted export recovery bugfixesGravatar Joey Hess2017-09-07
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When an export was interrupted, the sqlite database won't have been committed necessarily. Also, the interrupted export might have been run in an entirely different repository. There's not a significant speed benefit in checking getExportLocation in this case anyway, so avoid it. Also, remove the old filename from the export database. Recovery from interrupted exports is now tested working. This commit was supported by the NSF-funded DataLad project.
| * | avoid renaming to temp files before deletingGravatar Joey Hess2017-09-07
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Only rename when actually ncessary. The diff gets buffered in memory. Probably git has to buffer a diff in memory when generating it as well, so this memory usage should not be a problem, even when the diff is very large. I hope. This commit was supported by the NSF-funded DataLad project.
| * | prevent exporttree=yes on remotes that don't support exportsGravatar Joey Hess2017-09-07
| | | | | | | | | | | | | | | | | | | | | | | | | | | Don't allow "exporttree=yes" to be set when the special remote does not support exports. That would be confusing since the user would set up a special remote for exports, but `git annex export` to it would later fail. This commit was supported by the NSF-funded DataLad project.
| * | document new stuff for external special remotesGravatar Joey Hess2017-09-07
| | | | | | | | | | | | | | | | | | | | | Got rid of RENAMEEXPORT-UNSUPPORTED, no reason not to use RENAMEEXPORT-FAILURE for that. This commit was supported by the NSF-funded DataLad project.
| | * (no commit message)Gravatar Horus2017-09-07
| | |
| | * Added a commentGravatar Horus2017-09-07
| | |
| | * Added a commentGravatar anthony@ad39673d230d75cbfd19d2757d754030049c76732017-09-06
| |/ |/|
* | devblogGravatar Joey Hess2017-09-06
| |
| * fix consistency bug reading from export databaseGravatar Joey Hess2017-09-06
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The export database has writes made to it and then expects to read back the same data immediately. But, the way that Database.Handle does writes, in order to support multiple writers, makes that not work, due to caching issues. This resulted in export re-uploading files it had already successfully renamed into place. Fixed by allowing databases to be opened in MultiWriter or SingleWriter mode. The export database only needs to support a single writer; it does not make sense for multiple exports to run at the same time to the same special remote. All other databases still use MultiWriter mode. And by inspection, nothing else in git-annex seems to be relying on being able to immediately query for changes that were just written to the database. This commit was supported by the NSF-funded DataLad project.
| * bugfixGravatar Joey Hess2017-09-06
| |
| * Merge branch 'master' into exportGravatar Joey Hess2017-09-06
| |\ | |/ |/|
| * rename bug fixGravatar Joey Hess2017-09-06
| |
| * todoGravatar Joey Hess2017-09-06
| |
| * export file renamingGravatar Joey Hess2017-09-06
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is seriously super hairy. It has to handle interrupted exports, which may be resumed with the same or a different tree. It also has to recover from export conflicts, which could cause the wrong content to be renamed to a file. I think this works, or is close to working. See the update to the design for how it works. This is definitely not optimal, in that it does more renames than are necessary. It would probably be worth finding the keys that are really renamed and only renaming those. But let's get the "simple" approach to work first.. This commit was supported by the NSF-funded DataLad project.
| * record incomplete exports in export.logGravatar Joey Hess2017-09-06
| | | | | | | | | | | | | | | | | | | | Not yet used, but essential for resuming cleanly. Note that, in normmal operation, only one commit is made to export.log during an export; the incomplete version only gets to the journal and is then overwritten. This commit was supported by the NSF-funded DataLad project.
| * thoughts on handling renames efficientlyGravatar Joey Hess2017-09-06
| | | | | | | | | | | | This gets complicated, but I think this design will work! This commit was supported by the NSF-funded DataLad project.
* | Merge branch 'master' of ssh://git-annex.branchable.comGravatar Joey Hess2017-09-06
|\ \
* | | move line break to fix broken linkGravatar Edward Betts2017-09-06
| | |
* | | updateGravatar Joey Hess2017-09-06
| | |
| * | (no commit message)Gravatar karel-de-macil2017-09-06
|/ /
* | Added datalad "super-dataset".Gravatar yarikoptic2017-09-05
| |
generated by cgit v1.2.3 (git 2.39.1) at 2024-07-01 10:33:57 +0000