1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
|
### Please describe the problem.
I have a Synology DS216+ NAS, which can export shares via SMB2, AFP or NFS. Since I have a lot of "media" data in git-annex, I'd like to be able to put a useful copy of that data (ie, original file names) onto the NAS, and have git-annex track that content as another "local" copy of the content that is available (rather than having to track it by hand).
Because the NFS shares have UID mapping issues (AFAICT NFS v4 id mapping is not supported without Kerberos authentication, due to changes since about Linux 3.4; the Synology is based on a Linux 3.10 kernel), I have been trying to use SMBv2 shares, with symlinks enabled ("Allow symbolic links within shared folders"). I'm aware that `git-annex` on a network file system is less supported, but this is still a use case that would be quite useful to manage larger pools of files.
After updating to the latest release (to [fix git annex init timeouts](https://git-annex.branchable.com/forum/git_annex_init_timeout/)), I am able to complete the `git clone ...`, `git annex init`, and `git sync` steps. (The SMB mount supports symlinks, but not hard links; hence pidlocks being needed.)
But `git annex get` or `git annex copy` to transfer any of the content files fails with:
get README (transfer already in progress, or unable to take transfer lock)
Unable to access these remotes: ashram-data
whether initiated from the SMB mounted `git-annex` or a `git-annex` with the data on a local file system. No content files get transferred.
Of note, on this "SMB share with symlinks enabled" I can manually create symlinks, but `git annex` still detects symlinks as not possible and switches to direct mode. However it leaves the existing symlinks from `git clone ...` in place. It is unclear to me whether this is part of the cause of the "transfer lock" issue, or whether the transfer locks are, eg, not using pidlocks and thus failing because flock() does not work.
### What steps will reproduce the problem?
* Mount a remote file system via SMB v2, with symlinks enabled (eg mounting from a Samba share should be sufficient, as that's what is running on the Synology NAS)
* `git clone` an existing `git-annex` onto a directory on that SMB mounted filesystem
* `git remote ....` configure one or more remotes with the file data
* `git annex init "SMB share"`
* `git annex sync`
* `git annex get ....` of an existing file
or alternatively after the init/sync, go to a source `git-annex` and do:
* `git remote add smb ....`
* `git annex sync`
* `git annex copy --to=smb ...` of an existing file with content in that `git-annex`
### What version of git-annex are you using? On what operating system?
`git-annex` 6.20170320 (downloaded 2017-04-23) on Mac OS X 10.11 (El Capitan):
ewen@ashram:~$ uname -a
Darwin ashram 15.6.0 Darwin Kernel Version 15.6.0: Fri Feb 17 10:21:18 PST 2017; root:xnu-3248.60.11.4.1~1/RELEASE_X86_64 x86_64
ewen@ashram:~$ git annex version
git-annex version: 6.20170320-g41c5d9d
build flags: Assistant Webapp Pairing Testsuite S3(multipartupload)(storageclasses) WebDAV FsEvents ConcurrentOutput TorrentParser MagicMime Feeds Quvi
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 SHA1E SHA1 MD5E MD5 WORM URL
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav tahoe glacier ddar hook external
ewen@ashram:~$
### Please provide any additional information below.
[[!format sh """
# Example session
ewen@ashram:/nas01/annex/purchased$ git clone /bkup/purchased/mediabytes
Cloning into 'mediabytes'...
done.
ewen@ashram:/nas01/annex/purchased$ cd mediabytes/
ewen@ashram:/nas01/annex/purchased/mediabytes$ git remote add ashram-data /data/purchased/mediabytes
ewen@ashram:/nas01/annex/purchased/mediabytes$ git remote add ashram /bkup/purchased/mediabytes
ewen@ashram:/nas01/annex/purchased/mediabytes$ git remote remove origin
ewen@ashram:/nas01/annex/purchased/mediabytes$ git annex init 'nas01 file server'
init nas01 file server
Detected a filesystem without POSIX fcntl lock support.
Enabling annex.pidlock.
Detected a filesystem without fifo support.
Disabling ssh connection caching.
Detected a crippled filesystem.
Disabling core.symlinks.
Enabling direct mode.
ok
(recording state in git...)
ewen@ashram:/nas01/annex/purchased/mediabytes$ git annex sync
commit ok
pull ashram
From /bkup/purchased/mediabytes
* [new branch] git-annex -> ashram/git-annex
* [new branch] master -> ashram/master
* [new branch] synced/git-annex -> ashram/synced/git-annex
* [new branch] synced/master -> ashram/synced/master
ok
pull ashram-data
From /data/purchased/mediabytes
* [new branch] git-annex -> ashram-data/git-annex
* [new branch] master -> ashram-data/master
* [new branch] synced/git-annex -> ashram-data/synced/git-annex
* [new branch] synced/master -> ashram-data/synced/master
ok
(merging ashram-data/git-annex into git-annex...)
(recording state in git...)
push ashram
Counting objects: 8, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (6/6), done.
Writing objects: 100% (8/8), 806 bytes | 0 bytes/s, done.
Total 8 (delta 2), reused 1 (delta 0)
To /bkup/purchased/mediabytes
02b7699..0edf575 git-annex -> synced/git-annex
ok
push ashram-data
Counting objects: 8, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (6/6), done.
Writing objects: 100% (8/8), 806 bytes | 0 bytes/s, done.
Total 8 (delta 2), reused 1 (delta 0)
To /data/purchased/mediabytes
02b7699..0edf575 git-annex -> synced/git-annex
ok
ewen@ashram:/nas01/annex/purchased/mediabytes$ git annex sync
commit ok
pull ashram
ok
pull ashram-data
ok
ewen@ashram:/nas01/annex/purchased/mediabytes$ git annex get .
get README (transfer already in progress, or unable to take transfer lock)
Unable to access these remotes: ashram-data
Try making some of these repositories available:
00420eb2-1c9a-47e8-abfa-675b9ffb0782 -- naos_wd_1 backup drive
0ac1326e-9e03-4c1e-bd51-6cd664988caa -- Naos (colo) fileserver
0e5fd428-ba5b-4817-8489-9780bc62e655 -- naos_wd_2 backup drive
1144edcc-b18f-4cb1-a828-6c272d7788d6 -- naos_wd_3 backup drive
6142cfd4-83ba-410f-bf54-fd4b79e49fdb -- ashram external data drive [ashram-data]
77aeca6a-a4f8-4734-befc-2a0b50687c52 -- tv home fileserver
cc1c45f3-7c96-4103-b9c0-cce0f5f2cc36 -- bethel_data_drive
eb0ca2b6-e8c6-43b0-a805-de17a4d7eeae -- naos_wd_4 backup drive
failed
get alex-lindsay-1.m4a (transfer already in progress, or unable to take transfer lock)
Unable to access these remotes: ashram-data
Try making some of these repositories available:
00420eb2-1c9a-47e8-abfa-675b9ffb0782 -- naos_wd_1 backup drive
0ac1326e-9e03-4c1e-bd51-6cd664988caa -- Naos (colo) fileserver
0e5fd428-ba5b-4817-8489-9780bc62e655 -- naos_wd_2 backup drive
1144edcc-b18f-4cb1-a828-6c272d7788d6 -- naos_wd_3 backup drive
6142cfd4-83ba-410f-bf54-fd4b79e49fdb -- ashram external data drive [ashram-data]
77aeca6a-a4f8-4734-befc-2a0b50687c52 -- tv home fileserver
cc1c45f3-7c96-4103-b9c0-cce0f5f2cc36 -- bethel_data_drive
eb0ca2b6-e8c6-43b0-a805-de17a4d7eeae -- naos_wd_4 backup drive
failed
[...]
[...]
failed
git-annex: get: 59 failed
ewen@ashram:/nas01/annex/purchased/mediabytes$
ewen@ashram:/nas01/annex/purchased/mediabytes$ ls -l | head
total 118
lrwx------ 1 ewen staff 182 23 Apr 18:53 README -> .git/annex/objects/3M/8w/SHA256E-s273--1eb5c07e12d99c99af8c163b5e005162820518020cc9922152852bd1422858ae/SHA256E-s273--1eb5c07e12d99c99af8c163b5e005162820518020cc9922152852bd1422858ae
lrwx------ 1 ewen staff 200 23 Apr 18:53 alex-lindsay-1.m4a -> .git/annex/objects/9M/xJ/SHA256E-s26905448--c8ecda8ca67c3ff8ab1061b34a2974be41eb4b44421e3c39d1dda63f4de6b910.m4a/SHA256E-s26905448--c8ecda8ca67c3ff8ab1061b34a2974be41eb4b44421e3c39d1dda63f4de6b910.m4a
lrwx------ 1 ewen staff 194 23 Apr 18:53 alexlindsay.txt -> .git/annex/objects/P7/7g/SHA256E-s17467--2e7ae57eff6db5a832681443eb3787d1d7f9e57a52e4fedb2b693136a3df19f0.txt/SHA256E-s17467--2e7ae57eff6db5a832681443eb3787d1d7f9e57a52e4fedb2b693136a3df19f0.txt
lrwx------ 1 ewen staff 198 23 Apr 18:53 bourne-1.mp3 -> .git/annex/objects/98/vg/SHA256E-s7560841--a24ce7acabcb0dd46b9a9df6df8aff50145d8b1d5253308a51573a7742652943.mp3/SHA256E-s7560841--a24ce7acabcb0dd46b9a9df6df8aff50145d8b1d5253308a51573a7742652943.mp3
lrwx------ 1 ewen staff 192 23 Apr 18:53 bourne.txt -> .git/annex/objects/QK/2p/SHA256E-s1031--d45a0f0aba935126bf29d6265e9c252faae8b735abdc20edbec69eb3cc9edcd7.txt/SHA256E-s1031--d45a0f0aba935126bf29d6265e9c252faae8b735abdc20edbec69eb3cc9edcd7.txt
lrwx------ 1 ewen staff 200 23 Apr 18:53 chasejarvis-1.m4a -> .git/annex/objects/M1/8z/SHA256E-s80616681--d1b63892a9d68c51ddef9eb5c0a56ced795a449afe13d4884663ca6ecc76cade.m4a/SHA256E-s80616681--d1b63892a9d68c51ddef9eb5c0a56ced795a449afe13d4884663ca6ecc76cade.m4a
lrwx------ 1 ewen staff 194 23 Apr 18:53 chasejarvis.txt -> .git/annex/objects/35/jv/SHA256E-s50443--687b0cf63cc8f5724e3a694e1de6221ca4d1dd3449a6e6ad9b6f34501bfa30ce.txt/SHA256E-s50443--687b0cf63cc8f5724e3a694e1de6221ca4d1dd3449a6e6ad9b6f34501bfa30ce.txt
lrwx------ 1 ewen staff 184 23 Apr 18:53 checksums -> .git/annex/objects/3Q/vv/SHA256E-s1672--31b3cb7ccbcafc96d5061775dc31aa974bb4ce2f16984b2ab0104eb66674d81b/SHA256E-s1672--31b3cb7ccbcafc96d5061775dc31aa974bb4ce2f16984b2ab0104eb66674d81b
lrwx------ 1 ewen staff 200 23 Apr 18:53 chrismarquardt-1.m4a -> .git/annex/objects/k4/f3/SHA256E-s61861112--c14bacb5adfe3eca237547760183a986499808084dd1925a0202598981e4406b.m4a/SHA256E-s61861112--c14bacb5adfe3eca237547760183a986499808084dd1925a0202598981e4406b.m4a
ewen@ashram:/nas01/annex/purchased/mediabytes$
ewen@ashram:/nas01/annex/purchased/mediabytes$ git annex info
repository mode: direct
trusted repositories: 0
semitrusted repositories: 13
00000000-0000-0000-0000-000000000001 -- web
00000000-0000-0000-0000-000000000002 -- bittorrent
00420eb2-1c9a-47e8-abfa-675b9ffb0782 -- naos_wd_1 backup drive
0ac1326e-9e03-4c1e-bd51-6cd664988caa -- Naos (colo) fileserver
0ba93724-85a8-4ac7-855b-e656a6abaf09 -- ashram (laptop) internal drive [ashram]
0e5fd428-ba5b-4817-8489-9780bc62e655 -- naos_wd_2 backup drive
1144edcc-b18f-4cb1-a828-6c272d7788d6 -- naos_wd_3 backup drive
4e5bfc02-1adc-44d2-a60a-885d9120ed5d -- nas01 file server [here]
6142cfd4-83ba-410f-bf54-fd4b79e49fdb -- ashram external data drive [ashram-data]
77aeca6a-a4f8-4734-befc-2a0b50687c52 -- tv home fileserver
bde87f8e-4740-411f-b068-3dbbd9db03d4 -- nas01 file server
cc1c45f3-7c96-4103-b9c0-cce0f5f2cc36 -- bethel_data_drive
eb0ca2b6-e8c6-43b0-a805-de17a4d7eeae -- naos_wd_4 backup drive
untrusted repositories: 0
transfers in progress: none
available local disk space: 3.63 terabytes (+1 megabyte reserved)
local annex keys: 0
local annex size: 0 bytes
annexed files in working tree: 59
size of annexed files in working tree: 1.67 gigabytes
bloom filter size: 32 mebibytes (0% full)
backend usage:
SHA256E: 59
ewen@ashram:/nas01/annex/purchased/mediabytes$
"""]]
For reference Synology DS216+ `/etc/samba/smb.conf`:
ewen@nas01:/volume1$ cat /etc/samba/smb.conf
[global]
printcap name=cups
winbind enum groups=yes
include=/var/tmp/nginx/smb.netbios.aliases.conf
min protocol=SMB2
security=user
local master=no
realm=*
passdb backend=smbpasswd
printing=cups
max protocol=SMB3
winbind enum users=yes
load printers=yes
workgroup=WORKGROUP
ewen@nas01:/volume1$
and the relevant share:
ewen@nas01:/volume1$ cat /etc/samba/smb.share.conf
[annex]
recycle bin admin only=no
ftp disable modify=no
ftp disable download=no
write list=nobody,nobody
browseable=yes
mediaindex=no
hide unreadable=no
win share=yes
enable recycle bin=no
invalid users=nobody,nobody
read list=nobody,nobody
ftp disable list=no
edit synoacl=yes
valid users=nobody,nobody
writeable=yes
guest ok=yes
path=/volume1/annex
skip smb perm=yes
comment="Git Annexes"
[music]
[...]
ewen@nas01:/volume1$
### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
I'm an extremely regular user of git-annex on OS X and Linux, for several years, using it as a podcatcher and to manage most of my "large file" media. It's one of those "couldn't live without" tools. Thanks for writing it.
(Touched, to ask for comments to be emailed to me)
> [[fixed|done]] --[[Joey]]
|
