1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
|
[[!meta title="arbitrary/configurable backends"]]
(Retitling as this has drifted..)
---
I thought this might be useful, since curl is being used for the URL backend, it might be worth checking for it's existence.
<pre>
diff --git a/configure.hs b/configure.hs
index 772ba54..1a563e0 100644
--- a/configure.hs
+++ b/configure.hs
@@ -13,6 +13,7 @@ tests = [
, TestCase "uuid generator" $ selectCmd "uuid" ["uuid", "uuidgen"]
, TestCase "xargs -0" $ requireCmd "xargs_0" "xargs -0 </dev/null"
, TestCase "rsync" $ requireCmd "rsync" "rsync --version >/dev/null"
+ , TestCase "curl" $ requireCmd "curl" "curl --version >/dev/null"
, TestCase "unicode FilePath support" $ unicodeFilePath
] ++ shaTestCases [1, 256, 512, 224, 384]
</pre>
> Well, curl is an optional extra, so requireCmd is too strong. Changed
> to testCmd and applied, thank you!
>
> I thought about actually *using* the resulting SysConfig.curl
> to disable the URL backend if False.. but probably it's better
> to just let it fail if curl is not available. Although, if we wanted
> to add a check for wget or something and use it when curl was not
> available, that might be worth doing. --[[Joey]]
>> I was thinking that is it worth doing a generic "stat", "delete", "get"
>> and "put" options, I do like the idea of having the possibility of
>> being about to use completely arbitrary storage systems or arbitrary
>> transfer systems. If there was the capability of doing so it would be
>> interesting to see possibilities of using aria2 for using something
>> like bittorrent as backend, or using something like irods or some
>> grid storage system as the storage archive. It's just an idea as
>> I have seen it implemented quite well in irods.
>>> I'm unsure about the idea of having a backend where that is
>>> parameterized. It would mean that one annex's GENERIC-foo key
>>> might be entirely different from another's key with the same backend
>>> and details. And a misconfiguration could get data the wrong
>>> way and get the wrong data, etc.
>>>
>>> I mostly look at the URL backend as an example that can be modified to
>>> make this kind of custom backend. You already probably know enough to
>>> make a TORRENT backend where keys are the urls to torrents to download
>>> with `aria2c --follow-torrent=mem`.
>>>
>>> I am also interested in doing backends that use eg, cloud storage.
>>> A S3 backend that could upload files to S3 in addition to downloading
>>> them, for example, would be handy. --[[Joey]]
>>>> So, rather than use backends to do this, it instead made more sense
>>>> to make them [[special_remotes]]. The URL backend remains a bit
>>>> of a special case, and a bittorrent backend that downloaded a file
>>>> from a bittorrent url would still be a good use of backend, but for
>>>> storing files in external data stores like S3, making it a remote
>>>> makes better sense. I think I can close this bug now, [[done]]
>>>> --[[Joey]]
also in Backend/URL.hs is it worth making a minor change to the way curl is called (I'm not sure if the following is correct or not)
> It's correct, typewise, but I don't see any real reason to bother
> with the change. But I do appreciate patches, which have been rare
> so far, probaby because of Haskell.. :) --[[Joey]]
>> heh agreed
<pre>
diff --git a/Backend/URL.hs b/Backend/URL.hs
index 29dc8fe..4afcf86 100644
--- a/Backend/URL.hs
+++ b/Backend/URL.hs
@@ -50,10 +50,13 @@ dummyFsck _ _ _ = return True
dummyOk :: Key -> Annex Bool
dummyOk _ = return True
+curl :: [CommandParam] -> IO Bool
+curl = boolSystem "curl"
+
downloadUrl :: Key -> FilePath -> Annex Bool
downloadUrl key file = do
showNote "downloading"
showProgress -- make way for curl progress bar
- liftIO $ boolSystem "curl" [Params "-# -o", File file, File url]
+ liftIO $ curl [Params "-# -o", File file, File url]
where
url = join ":" $ drop 1 $ split ":" $ show key
</pre>
|
