| Commit message (Collapse) | Author | Age |
|---|
| |\
| |
| |
| |
| | |
Conflicts:
doc/design/assistant/chunks.mdwn
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
When chunk=0, always try the unchunked key first. This avoids the overhead
of needing to read the git-annex branch to find the chunkcount.
However, if the unchunked key is not present, go on and try the chunks.
Also, when removing a chunked key, update the chunkcounts even when
chunk=0.
|
| | | |
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
No need to process each L.ByteString chunk, instead ask it to split.
Doesn't seem to have really sped things up much, but it also made the code
simpler.
Note that this does (and already did) buffer in memory. It seems that only
the directory special remote could take advantage of streaming chunks to
files w/o buffering, so probably won't add an interface to allow for that.
|
| | |
| |
| |
| |
| |
| |
| | |
This will allow things like WebDAV to opean a single persistent connection
and reuse it for all the chunked data.
The crazy types allow for some nice code reuse.
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| | |
Push it down from needing to be done in every Storer,
to being checked once inside ChunkedEncryptable.
Also, catch exceptions from PrepareStorer and PrepareRetriever,
just in case..
|
| | | |
|
| | | |
|
| | |
| |
| |
| |
| | |
The best thing to do in this case is return Left, so that anything that
tries to access it will fail.
|
| | | |
|
| | | |
|
| | | |
|
| | |\ |
|
| | | | |
|
| | | | |
|
| | | | |
|
| | | | |
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
And clean up legacy chunking code, which is in its own module now.
So much cleaner!
This commit was sponsored by Henrik Ahlgren
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
I'd have liked to keep these two concepts entirely separate,
but that are entagled: Storing a key in an encrypted and chunked remote
need to generate chunk keys, encrypt the keys, chunk the data, encrypt the
chunks, and send them to the remote. Similar for retrieval, etc.
So, here's an implemnetation of all of that.
The total win here is that every remote was implementing encrypted storage
and retrival, and now it can move into this single place. I expect this
to result in several hundred lines of code being removed from git-annex
eventually!
This commit was sponsored by Henrik Ahlgren.
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | | |
Chunk retrieval and reassembly, removal, and checking if all necessary
chunks are present.
This commit was sponsored by Damien Raude-Morvan.
|
| | | | |
|
| | | |
| | |
| | |
| | | |
to the annex
|
| | | | |
|
| | | | |
|
| |\| | |
|
| | |/ |
|
| | |\ |
|
| | | | |
|
| | | |
| | |
| | |
| | |
| | |
| | | |
Encyptable types of remotes that were not actually encrypted still had
the encryptedRemoteCostAdj applied to their configured cost, which was a
bug.
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | | |
Only when storing non-encrypted so far, not retrieving or checking if a key
is present or removing.
This commit was sponsored by Renaud Casenave-Péré.
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
Not yet used by any special remotes, but should not be too hard to add it
to most of them.
storeChunks is the hairy bit! It's loosely based on
Remote.Directory.storeLegacyChunked. The object is read in using a lazy
bytestring, which is streamed though, creating chunks as needed, without
ever buffering more than 1 chunk in memory.
Getting the progress meter update to work right was also fun, since
progress meter values are absolute. Finessed by constructing an offset
meter.
This commit was sponsored by Richard Collins.
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
This avoids a proliferation of hash directories when using new-style
chunking, and should improve performance since chunks are accessed
in sequence and so should have a common locality.
Of course, when a chunked key is encrypted, its hash directories have no
relation to the parent key.
This commit was sponsored by Christian Kellermann.
|
| |/ / |
|
| | | |
|
| | | |
|
| |/ |
|
| | |
|
| |\ |
|
| | | |
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| | |
Slightly tricky as they are not normal UUIDBased logs, but are instead maps
from (uuid, chunksize) to chunkcount.
This commit was sponsored by Frank Thomas.
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Moved old legacy chunking code, and cleaned up the directory and webdav
remotes use of it, so when no chunking is configured, that code is not
used.
The config for new style chunking will be chunk=1M instead of chunksize=1M.
There should be no behavior changes from this commit.
This commit was sponsored by Andreas Laas.
|
| | |
| |
| |
| |
| |
| |
| |
| | |
Added new fields for chunk number, and chunk size. These will not appear
in normal keys ever, but will be used for chunked data stored on special
remotes.
This commit was sponsored by Jouni K Seppanen.
|
| | | |
|
| | | |
|
| | | |
|
Joey Hess
https://www.google.com/accounts/o8/id?id=AItOawmURXBzaYE1gmVc-X9eLAyDat_6rHPl670
https://www.google.com/accounts/o8/id?id=AItOawk9nck8WX8-ADF3Fdh5vFo4Qrw1I_bJcR8
https://www.google.com/accounts/o8/id?id=AItOawlVUq_c3-lrQBculOEUu3yjvdavE7JbvEI
https://www.google.com/accounts/o8/id?id=AItOawnG_weCiH6IleQfVmRPmFJflHQMHdAcDkU