From 2d86f05a7a6aea745453997444d5bfbf26c165c3 Mon Sep 17 00:00:00 2001 From: webanck Date: Tue, 7 Nov 2017 16:03:17 +0000 Subject: --- .../Massive_git_add_produces_sqlite_crashes.mdwn | 57 ++++++++++++++++++++++ 1 file changed, 57 insertions(+) create mode 100644 doc/bugs/Massive_git_add_produces_sqlite_crashes.mdwn diff --git a/doc/bugs/Massive_git_add_produces_sqlite_crashes.mdwn b/doc/bugs/Massive_git_add_produces_sqlite_crashes.mdwn new file mode 100644 index 000000000..dbf46a03b --- /dev/null +++ b/doc/bugs/Massive_git_add_produces_sqlite_crashes.mdwn @@ -0,0 +1,57 @@ +### Please describe the problem. + +When adding plenty of files to my git annex repository, I encounter recurring sqlite errors. + +### What steps will reproduce the problem? +Create a git annex repo, add thousands of annexed binary files, and add thousands of small files tracked only with git. + + +### What version of git-annex are you using? On what operating system? + + >git annex version + git-annex version: 6.20171003 + build flags: Assistant Webapp Pairing Testsuite S3(multipartupload)(storageclasses) WebDAV Inotify DBus DesktopNotify ConcurrentOutput TorrentParser MagicMime Feeds Quvi + dependency versions: aws-0.13.0 bloomfilter-2.0.1.0 cryptonite-0.10 DAV-1.2 feed-0.3.10.4 ghc-7.10.3 http-client-0.4.26.2 persistent-sqlite-2.2 torrent-10000.0.0 uuid-1.3.11 yesod-1.4.2 + key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 SHA1E SHA1 MD5E MD5 WORM URL + remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav tahoe glacier ddar hook external + local repository version: 6 + supported repository versions: 3 5 6 + upgrade supported from repository versions: 0 1 2 3 4 5 + operating system: linux x86_64 + + >lsb_release -d + Description: Ubuntu 16.04.3 LTS + + >uname -r + 4.4.0-43-Microsoft + #(I am using Bash on Windows.) + +### Please provide any additional information below. + + + # If you can, paste a complete transcript of the problem occurring here. + >git ls-files --others | grep txt | wc -l + 1953 + >git add $(git ls-files --others | grep txt) + sqlite worker thread crashed: SQLite3 returned ErrorIO while attempting to perform prepare "SELECT null from content limit 1": disk I/O error + git-annex: sqlite query crashed + error: external filter git-annex smudge --clean %f failed 1 + error: external filter git-annex smudge --clean %f failed + # [...] plenty of errors follow + # End of transcript or log. + + +Triying to solve this problem, I found a part of answer in the form of a similar problem encounter here : + +Deleting git annex databases and running git annex fsck didnt do the trick: + + rm -rf .git/annex/keys/db .git/annex/keys/db-wal + git annex fsck --incremental -J4 + git add $(git ls-files --others | grep txt) + # Again, plenty of sqlite errors :() + +### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) +Oh yeah, I am still discovering this powerfull git annex tool. + +In fact, collegues and I are forming a group during the process to exchange about different use cases, encountered problems and help each other. + -- cgit v1.2.3