From 815fb0c899e80aa2cf72ae4fd6ee0f82974f46ae Mon Sep 17 00:00:00 2001 From: "leavingchicago@c04c893e78d1c4c76cb3e32b5c227cf42bbf7682" Date: Mon, 15 May 2017 04:40:17 +0000 Subject: --- ..._data_loss_but_seems_to_just_lose_tracking.mdwn | 215 +++++++++++++++++++++ 1 file changed, 215 insertions(+) create mode 100644 doc/bugs/sync_claims_data_loss_but_seems_to_just_lose_tracking.mdwn diff --git a/doc/bugs/sync_claims_data_loss_but_seems_to_just_lose_tracking.mdwn b/doc/bugs/sync_claims_data_loss_but_seems_to_just_lose_tracking.mdwn new file mode 100644 index 000000000..3223c61c0 --- /dev/null +++ b/doc/bugs/sync_claims_data_loss_but_seems_to_just_lose_tracking.mdwn @@ -0,0 +1,215 @@ +### Please describe the problem. +Sync / Assistant between a `source` and `transfer` repo results in zero copies for whereis, but data is available. + + +### What steps will reproduce the problem? +See a script below... + +``` sh +#!/bin/sh + +TMP=$(mktemp -d) + + +########################################## +# Set up the repos +########################################## + + +### +# Create working directories +cd ${TMP} +mkdir source transfer + + +### +# Set up the source repo +cd ${TMP}/source +git init +git annex init source +git-annex group here source +git annex wanted here standard + + +### +# Set up the transfer repo +# N.B. It can see the source repo +cd ${TMP}/transfer +git init +git remote add source ${TMP}/source +git annex init transfer +git-annex group here transfer +git annex wanted here \ + 'not (inallgroup=client and copies=client:1) and ((include=* and ((exclude=*/archive/* and exclude=archive/*) or (not (copies=archive:1 or copies=smallarchive:1)))) or approxlackingcopies=1)' + + +########################################## +# Add some files and run the assistant +########################################## + +### +# Make sure it works by adding some files to the source... +cd ${TMP}/source +date > file.1 +date > file.2 +date > file.3 + +# Run the assistant in the source... +(cd ${TMP}/source && \ + git annex assistant && \ + sleep 30 && \ + git annex assistant --stop && \ + git annex whereis file.*) +# PASS: All files are in source. + +# Now run it in the transfer group +(cd ${TMP}/transfer && \ + git annex assistant && \ + sleep 30 && \ + git annex assistant --stop && \ + git annex whereis file.*) +# PASS: All files are in transfer + +# Check the source +(cd ${TMP}/source && \ + git annex whereis file.*) +# FAIL: Only file.1 was updated to be in source. + +# whereis file.1 (1 copy) +# 64b193e0-86f0-4349-b073-70af919ce628 -- transfer +# ok +# whereis file.2 (0 copies) failed +# whereis file.3 (0 copies) failed +# git-annex: whereis: 2 failed + + +########################################## +# Start to scratch head and ask 'Why?' +########################################## + +########## +# What's happening??? + +cd ${TMP}/source && git annex sync +# Source thinks it is updated - working tree clean + +cd ${TMP}/transfer && git annex sync +# Transfer pulls from source, so run again +git annex sync +# OK... working tree clean + + +# Check again +cd ${TMP}/source && git annex whereis file.* +# whereis file.1 (1 copy) +# 64b193e0-86f0-4349-b073-70af919ce628 -- transfer +# ok +# whereis file.2 (0 copies) failed +# whereis file.3 (0 copies) failed +# git-annex: whereis: 2 failed + +cd ${TMP}/transfer && git annex whereis file.* +# whereis file.1 (1 copy) +# 64b193e0-86f0-4349-b073-70af919ce628 -- transfer [here] +# ok +# whereis file.2 (0 copies) failed +# whereis file.3 (0 copies) failed +# git-annex: whereis: 2 failed + + +# HOWEVER... Files are definitely in the transfer repo + +cd ${TMP}/source && cat file.* +# cat: file.1: No such file or directory +# cat: file.2: No such file or directory +# cat: file.3: No such file or directory + + +cd ${TMP}/transfer && cat file.* +# Sun May 14 21:22:02 PDT 2017 +# Sun May 14 21:22:03 PDT 2017 +# Sun May 14 21:22:06 PDT 2017 +``` + + + +### What version of git-annex are you using? On what operating system? + +#### Version: +``` +git-annex version: 6.20170510 +build flags: Assistant Webapp Pairing Testsuite S3(multipartupload)(storageclasses) WebDAV FsEvents ConcurrentOutput TorrentParser MagicMime Feeds Quvi +dependency versions: aws-0.16 bloomfilter-2.0.1.0 cryptonite-0.23 DAV-1.3.1 feed-0.3.12.0 ghc-8.0.2 http-client-0.5.6.1 persistent-sqlite-2.6.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.4.5 +key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 SHA1E SHA1 MD5E MD5 WORM URL +remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav tahoe glacier ddar hook external +local repository version: 5 +supported repository versions: 3 5 6 +upgrade supported from repository versions: 0 1 2 3 4 5 +operating system: darwin x86_64 +``` + +#### OS: +MacOS 10.12.4, but seems to apply to Debian Jessie too. + + +### Please provide any additional information below. + +Annex assistant daemon.log for `source` repo: + +``` sh +[2017-05-15 14:11:24.150202] main: starting assistant version 6.20170510 +[2017-05-15 14:11:24.196597] Cronner: You should enable consistency checking to protect your data. +(scanning...) [2017-05-15 14:11:24.257123] Watcher: Performing startup scan +(started...) +[2017-05-15 14:11:45.505855] Committer: Adding file.1 file.2 file.3 +add file.1 ok +add file.2 ok +add file.3 ok +[2017-05-15 14:11:45.544277] Committer: Committing changes to git +(recording state in git...) +(recording state in git...) +``` + +Annex assistant daemon.log for `transfer` repo: + +``` sh +[2017-05-15 14:12:09.127406] main: starting assistant version 6.20170510 +[2017-05-15 14:12:09.157611] Cronner: You should enable consistency checking to protect your data. +[2017-05-15 14:12:09.177452] TransferScanner: Syncing with source +(scanning...) [2017-05-15 14:12:09.210605] Watcher: Performing startup scan +(started...) +warning: no common commits +From /Users/olaf/tmp/git-annex/play/bug-play/source + * [new branch] git-annex -> source/git-annex + * [new branch] master -> source/master + * [new branch] synced/master -> source/synced/master +(merging source/git-annex into git-annex...) +(recording state in git...) + + +Already up-to-date. +[2017-05-15 14:12:10.256195] Committer: Committing changes to git +(recording state in git...) +(checksum...) [2017-05-15 14:12:10.425494] Transferrer: Downloaded file.1 +[2017-05-15 14:12:11.17664] Pusher: Syncing with source +(recording state in git...) +[2017-05-15 14:12:11.356447] Committer: Committing changes to git +(recording state in git...) +To /Users/olaf/tmp/git-annex/play/bug-play/source + * [new branch] git-annex -> synced/git-annex +(checksum...) [2017-05-15 14:12:11.428676] Transferrer: Downloaded file.2 +(checksum...) [2017-05-15 14:12:11.451457] Transferrer: Downloaded file.3 +drop source file.1 ok +drop source file.2 ok +drop source file.3 ok +[2017-05-15 14:12:13.477636] Pusher: Syncing with source +(recording state in git...) +To /Users/olaf/tmp/git-annex/play/bug-play/source + 825d3dd..60c96cb git-annex -> synced/git-annex +``` + + +### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders) + +Yes. Love it. Donated. Have been using it for years. Recommend it and get(/force) my collaborators to use it. ;-) + -- cgit v1.2.3