Added a comment: dtruss OS X 10.11 git annex on SMB mount

1 files changed, 86 insertions, 0 deletions

diff --git a/doc/bugs/SMB__58___git_annex_clone_works__44___get_fails_on_transfer_lock/comment_3_24c6874757783e3c3838db35174870d1._comment b/doc/bugs/SMB__58___git_annex_clone_works__44___get_fails_on_transfer_lock/comment_3_24c6874757783e3c3838db35174870d1._comment
new file mode 100644
index 000000000..c549c1972
--- /dev/null
+++ b/doc/bugs/SMB__58___git_annex_clone_works__44___get_fails_on_transfer_lock/comment_3_24c6874757783e3c3838db35174870d1._comment
@@ -0,0 +1,86 @@
+[[!comment format=mdwn
+ username="ewen"
+ avatar="http://cdn.libravatar.org/avatar/605b2981cb52b4af268455dee7a4f64e"
+ subject="dtruss OS X 10.11 git annex on SMB mount"
+ date="2017-05-14T05:04:29Z"
+ content="""
+A tar file with:
+
+    ewen@ashram:~/Desktop$ tar -tzf dtruss-git-annex-osx-smb-2017-05-14.tar.gz
+    dtruss-git-annex-osx-smb-2017-05-14/
+    dtruss-git-annex-osx-smb-2017-05-14/00-ga-test-smb-osx-symlinks-2017-05-14.script
+    dtruss-git-annex-osx-smb-2017-05-14/01-ga-test-smb-osx-symlinks-dtruss-git-annex-init-2017-05-14.txt
+    dtruss-git-annex-osx-smb-2017-05-14/02-ga-test-smb-osx-symlinks-dtruss-git-annex-sync-2017-05-14.txt
+    dtruss-git-annex-osx-smb-2017-05-14/03-ga-test-smb-osx-symlinks-dtruss-git-annex-get-all-2017-05-14.txt
+    dtruss-git-annex-osx-smb-2017-05-14/04-ga-test-smb-osx-symlinks-dtruss-git-annex-sync-other-way-2017-05-14.txt
+    dtruss-git-annex-osx-smb-2017-05-14/05-ga-test-smb-osx-symlinks-dtruss-git-annex-copy-to-smb-2017-05-14.txt
+    dtruss-git-annex-osx-smb-2017-05-14/dtruss-ga
+    ewen@ashram:~/Desktop$ 
+
+is available at [dtruss-git-annex-osx-smb-2017-05-14.tar.gz](http://ewen.mcneill.gen.nz/temp/dtruss-git-annex-osx-smb-2017-05-14.tar.gz) (at least until I next tidy up the `/temp/` directory :-) ).
+
+The script file is the output of \"`script ....`\" showing all the commands (including typos :-) ) run, except the `dtruss` ones themselves.  The various `0n-ga-tst-smb-osx-symlinks-dtruss...` files contain the output of \"`sudo dtruss -af PID`\" where the `PID` was printed out by the script `dtruss-ga` that is included in the archive.  `dtruss-ga` was used to enable running git-annex as non-root (to ensure that the SMB access wasn't altered by running as a different user), since `dtruss` itself can only run as root.  The script is just:
+
+    ewen@ashram:~/Desktop$ cat dtruss-git-annex-osx-smb-2017-05-14/dtruss-ga 
+    #! /bin/sh
+    # Allow dtruss of non-root git annex
+    #
+    # Based on hints at:
+    # http://sec.cs.ucl.ac.uk/users/smurdoch/blog/2015/08/fixing-slow-start-mc-on-osx.html
+    #---------------------------------------------------------------------------
+
+    echo \"Run: sudo dtruss -af -p $$\"
+    echo \"Then press enter\"
+    read
+    exec git annex \"$@\"
+    ewen@ashram:~/Desktop$ 
+
+so basically \"print PID, wait for go ahead, run git annex command\".
+
+From a quick glance through, it looks like:
+
+    7003/0x16c3126:    121576      14     11 write(0x12, \"PidLock {lockingPid = 7003, lockingHost = \\"ashram\\"}\0\", 0x33)           = 51 0
+    7003/0x16c3126:    121775    3350    191 close(0x12)            = 0 0
+    7003/0x16c3126:    121792      10      6 stat64(\"/Volumes/music/ga-test/.git/annex/locktmp16807282475249\0\", 0x200004940, 0x33)                 = 0 0
+    7003/0x16c3126:    122041    2341    235 link(\"/Volumes/music/ga-test/.git/annex/locktmp16807282475249\0\", \"/Volumes/music/ga-test/.git/annex/locktmp16807282475249.lnk\0\")             = -1 Err#45
+    7003/0x16c3126:    122115     885     49 unlink(\"/Volumes/music/ga-test/.git/annex/locktmp16807282475249.lnk\0\", 0x200004D80, 0x33)             = -1 Err#2
+
+(in `03-...`, the first `get`) probably isn't helping -- that's trying to do a *hardlink* on a SMB file system, and appearing to give up when it fails... (`/Volumes/music` is the SMB mount I was using for testing).
+
+Also since I didn't realise until after the `script` file/`tar` file were made, there *is* an `annex/pidlock` file:
+
+    ewen@ashram:~$ ls -l /Volumes/music/ga-test/.git/annex/pidlock 
+    -rwx------  1 ewen  staff  51 14 May 16:25 /Volumes/music/ga-test/.git/annex/pidlock
+    ewen@ashram:~$ cat /Volumes/music/ga-test/.git/annex/pidlock 
+    PidLock {lockingPid = 7143, lockingHost = \"ashram\"}ewen@ashram:~$ 
+    ewen@ashram:~$ 
+
+created within the SMB `git-annex`, which looks vaguely sensible.  And FTR, there is *not* one in the HFS (local OS X) `git-annex`:
+
+    ewen@ashram:~$ ls -l /var/tmp/ga-test/.git/annex/pidlock
+    ls: /var/tmp/ga-test/.git/annex/pidlock: No such file or directory
+    ewen@ashram:~$ 
+
+so it definitely seems to *try* to do pidlocking on the SMB mount on transfers, but it looks like maybe it's also trying to do something *not* supported.
+
+Looking through the OS X `open(2)` manpage, it appears that:
+
+           O_EXCL          error if O_CREAT and the file exists
+           [...]
+           O_EXLOCK        atomically obtain an exclusive lock
+
+might be plausible non-hardlink locking primitives that'd work better than nothing.  The manpage also says:
+
+    When opening a file, a lock with flock(2) semantics can be obtained by
+    setting O_SHLOCK for a shared lock, or O_EXLOCK for an exclusive lock.
+    If creating a file with O_CREAT, the request for the lock will never fail
+    (provided that the underlying filesystem supports locking).
+
+so maybe that's something to work with?  (SMB as a protocol supports All Kinds Of Locking (tm), and hopefully the OS X SMB file system driver would also translate O_EXLOCK, etc, through to those...)
+
+Hope that's of some help.
+
+Ewen
+
+PS: Yes, I do plan to try to install the stand alone x86-64 git-annex on my Synology when I get a spare few hours to sort out all the details.  It seems like it would be the most robust solution for my specific problem.  But generalised SMB support could also be useful, even if it was \"user needs to be careful about running commands in parallel as there are races\".
+"""]]