summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorGravatar Joey Hess <joeyh@joeyh.name>2017-12-26 12:21:44 -0400
committerGravatar Joey Hess <joeyh@joeyh.name>2017-12-26 12:21:44 -0400
commit0bf7850898db15cca00b1d7ea6056a680aa42930 (patch)
tree7c9644a34b65e029b52af252480920a3fc3b3ebb
parent0358d67fa921c8f21b8c7a4d6d8dfbe188da484c (diff)
fix now-dead gmane links
gmane's disk crashed, I found one thread in another archive, but could not find my whole patch set in any archive (perhaps some of the messages were too long), so pulled it out of my personal mail archives. This commit was supported by the NSF-funded DataLad project.
Diffstat
-rw-r--r--doc/todo/smudge.mdwn5
-rw-r--r--doc/todo/smudge/git-patches1643
2 files changed, 1645 insertions, 3 deletions
diff --git a/doc/todo/smudge.mdwn b/doc/todo/smudge.mdwn
index 266a5d593..3a980b10a 100644
--- a/doc/todo/smudge.mdwn
+++ b/doc/todo/smudge.mdwn
@@ -32,10 +32,9 @@ git-annex should use smudge/clean filters.
read the file itself would let git-annex short-circuit when the file it's
cleaning is one it already knows about. I've proposed extending git with
such an interface:
- <http://news.gmane.org/find-root.php?message_id=20160512182432.GA27427%40kitenet.net>
+ <http://git.661346.n2.nabble.com/proposal-for-extending-smudge-clean-filters-with-raw-file-access-td7656150.html>
- And developed a patch set:
- <http://thread.gmane.org/gmane.comp.version-control.git/297475>
+ And developed a patch set: [[git-patches]]
* Implement git's new `filter.<driver>.process` interface, which will
let only 1 git-annex process be started by git when processing
diff --git a/doc/todo/smudge/git-patches b/doc/todo/smudge/git-patches
new file mode 100644
index 000000000..3736ba2e3
--- /dev/null
+++ b/doc/todo/smudge/git-patches
@@ -0,0 +1,1643 @@
+From mairix@mairix Mon Jan 1 12:34:56 1970
+X-source-folder: /home/joey/mail/.git/annex/objects/VX/gq/SHA256E-s5263957--977af1c7373ecbca2f2df44e782afb143e71a38b0af965f02cd1178ab5b82b4e.gz/SHA256E-s5263957--977af1c7373ecbca2f2df44e782afb143e71a38b0af965f02cd1178ab5b82b4e.gz
+Return-Path: <joey@kitenet.net>
+X-Original-To: joeyh@joeyh.name
+Delivered-To: joey@kitenet.net
+X-Question: 42
+Authentication-Results: kitenet.net;
+ dkim=pass (1024-bit key; unprotected) header.d=joeyh.name header.i=@joeyh.name header.b=ier2ZdRe;
+ dkim-atps=neutral
+DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=joeyh.name; s=mail;
+ t=1468277197; bh=kOlU+hQ5LyTEz/eutsD10qEeMefFCuV9/88MgBjyH3k=;
+ h=From:To:Cc:Subject:Date:From;
+ b=ier2ZdReiz3+8cyLyfBhIuFH0zribYdza324RRso423MVVwX5Z5dXjkqHvfVpRxVA
+ SLJCxSfGmOuiiQikb4CcFpmTi8rt+gcLqcgvQGgQMug+ee8CysZVHW76EpSILa8eIN
+ 1qIOb5q8Q8fHMaRifEWQvbQiw4XTzTdDJZdyPMIE=
+From: Joey Hess <joeyh@joeyh.name>
+To: git@vger.kernel.org
+Cc: Joey Hess <joeyh@joeyh.name>
+Subject: [PATCH v5 0/8] extend smudge/clean filters with direct file access (for pu)
+Date: Mon, 11 Jul 2016 18:45:04 -0400
+Message-Id: <1468277112-9909-1-git-send-email-joeyh@joeyh.name>
+X-Mailer: git-send-email 2.8.1
+X-Spam-Status: No, score=-95.2 required=5.0 tests=DKIM_SIGNED,DKIM_VALID,
+ DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,RCVD_IN_DNSWL_BLOCKED,RCVD_IN_PBL,
+ RCVD_IN_SORBS_DUL,RDNS_DYNAMIC,SPF_SOFTFAIL,URIBL_BLOCKED,USER_IN_WHITELIST
+ autolearn=no autolearn_force=no version=3.4.1
+X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on kite.kitenet.net
+Content-Length: 1441
+Lines: 40
+
+Back from vacation with a reroll of jh/clean-smudge-annex.
+
+Deals with conflicting changes from cc/apply-am in pu.
+
+Since tb/convert-peek-in-index is not currently in pu, this reroll isn't
+based on it, and will conflict if that topic gets added back into pu.
+Not sure what the status of tb/convert-peek-in-index is at this point?
+
+Improvements from Junio's review:
+
+ fix build with DEVELOPER=1
+ style fixes
+ use test_cmp in test cases
+ improve robustness of a test case
+ clean up some confusing code
+ small performance tweak
+
+Joey Hess (8):
+ clarify %f documentation
+ add smudgeToFile and cleanFromFile filter configs
+ use cleanFromFile in git add
+ use smudgeToFile in git checkout etc
+ warn on unusable smudgeToFile/cleanFromFile config
+ better recovery from failure of smudgeToFile filter
+ use smudgeToFile filter in git am
+ use smudgeToFile filter in recursive merge
+
+ Documentation/config.txt | 18 ++++-
+ Documentation/gitattributes.txt | 42 ++++++++++++
+ apply.c | 16 +++++
+ convert.c | 148 ++++++++++++++++++++++++++++++++++++----
+ convert.h | 10 +++
+ entry.c | 59 ++++++++++++----
+ merge-recursive.c | 53 +++++++++++---
+ sha1_file.c | 42 ++++++++++--
+ t/t0021-conversion.sh | 117 +++++++++++++++++++++++++++++++
+ 9 files changed, 459 insertions(+), 46 deletions(-)
+
+--
+2.8.1
+
+From mairix@mairix Mon Jan 1 12:34:56 1970
+X-source-folder: /home/joey/mail/.git/annex/objects/0k/FJ/SHA256E-s162778--6d2a0fc6afd1077eccdca2e92f314c00ef70a06273242429035e0ce52cce2e13.gz/SHA256E-s162778--6d2a0fc6afd1077eccdca2e92f314c00ef70a06273242429035e0ce52cce2e13.gz
+Return-Path: <git-owner@vger.kernel.org>
+X-Original-To: joey@kitenet.net
+Delivered-To: joey@kitenet.net
+Received: from vger.kernel.org (vger.kernel.org [209.132.180.67])
+ by kitenet.net (Postfix) with ESMTP id B1F5B1C677
+ for <joey@kitenet.net>; Mon, 11 Jul 2016 18:46:53 -0400 (EDT)
+Authentication-Results: kitenet.net;
+ dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=joeyh.name header.i=@joeyh.name header.b=hbTM55ad;
+ dkim-atps=neutral
+Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
+ id S1752420AbcGKWqv (ORCPT <rfc822;joey@kitenet.net>);
+ Mon, 11 Jul 2016 18:46:51 -0400
+Received: from kitenet.net ([66.228.36.95]:60934 "EHLO kitenet.net"
+ rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP
+ id S1751854AbcGKWqu (ORCPT <rfc822;git@vger.kernel.org>);
+ Mon, 11 Jul 2016 18:46:50 -0400
+X-Question: 42
+DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=joeyh.name; s=mail;
+ t=1468277197; bh=Q37/p1l59xZISgA1LWyXTUqwJEg7T96CGx2mCEyU9UM=;
+ h=From:To:Cc:Subject:Date:In-Reply-To:References:From;
+ b=hbTM55adTxAE7hwBgj9q7PkR66nfsMxhWSpQn/R9iSUOaJ11fW10rEUxkLzcXKK5x
+ ktVQ6sdgyCC5CbgYfBn6sMJ+EH6sTTJlLS0czzSkl453Izvk85rBtBGYsVSCzgbPai
+ 1rM45vbD/pViJQ2IUyiT/WxLexQFFn0DnVLiWuWA=
+From: Joey Hess <joeyh@joeyh.name>
+To: git@vger.kernel.org
+Cc: Joey Hess <joeyh@joeyh.name>
+Subject: [PATCH v5 1/8] clarify %f documentation
+Date: Mon, 11 Jul 2016 18:45:05 -0400
+Message-Id: <1468277112-9909-2-git-send-email-joeyh@joeyh.name>
+X-Mailer: git-send-email 2.8.1
+In-Reply-To: <1468277112-9909-1-git-send-email-joeyh@joeyh.name>
+References: <1468277112-9909-1-git-send-email-joeyh@joeyh.name>
+X-Spam-Status: No, score=-8.2 required=10.0 tests=BAYES_00,DKIM_SIGNED,
+ HEADER_FROM_DIFFERENT_DOMAINS,RCVD_IN_DNSWL_HI,RP_MATCHES_RCVD,T_DKIM_INVALID,
+ URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.1
+X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on kite.kitenet.net
+Sender: git-owner@vger.kernel.org
+Precedence: bulk
+List-ID: <git.vger.kernel.org>
+X-Mailing-List: git@vger.kernel.org
+Content-Length: 1148
+Lines: 31
+
+It's natural to expect %f to be an actual file on disk; help avoid that
+mistake.
+
+Signed-off-by: Joey Hess <joeyh@joeyh.name>
+---
+ Documentation/gitattributes.txt | 5 +++++
+ 1 file changed, 5 insertions(+)
+
+diff --git a/Documentation/gitattributes.txt b/Documentation/gitattributes.txt
+index f2afdb6..197ece8 100644
+--- a/Documentation/gitattributes.txt
++++ b/Documentation/gitattributes.txt
+@@ -379,6 +379,11 @@ substitution. For example:
+ smudge = git-p4-filter --smudge %f
+ ------------------------
+
++Note that "%f" is the name of the path that is being worked on. Depending
++on the version that is being filtered, the corresponding file on disk may
++not exist, or may have different contents. So, smudge and clean commands
++should not try to access the file on disk, but only act as filters on the
++content provided to them on standard input.
+
+ Interaction between checkin/checkout attributes
+ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+--
+2.8.1
+
+--
+To unsubscribe from this list: send the line "unsubscribe git" in
+the body of a message to majordomo@vger.kernel.org
+More majordomo info at http://vger.kernel.org/majordomo-info.html
+
+From mairix@mairix Mon Jan 1 12:34:56 1970
+X-source-folder: /home/joey/mail/.git/annex/objects/0k/FJ/SHA256E-s162778--6d2a0fc6afd1077eccdca2e92f314c00ef70a06273242429035e0ce52cce2e13.gz/SHA256E-s162778--6d2a0fc6afd1077eccdca2e92f314c00ef70a06273242429035e0ce52cce2e13.gz
+Return-Path: <git-owner@vger.kernel.org>
+X-Original-To: joey@kitenet.net
+Delivered-To: joey@kitenet.net
+Received: from vger.kernel.org (vger.kernel.org [209.132.180.67])
+ by kitenet.net (Postfix) with ESMTP id 04F251C677
+ for <joey@kitenet.net>; Mon, 11 Jul 2016 18:47:07 -0400 (EDT)
+Authentication-Results: kitenet.net;
+ dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=joeyh.name header.i=@joeyh.name header.b=NOnwhtEt;
+ dkim-atps=neutral
+Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
+ id S1752532AbcGKWrD (ORCPT <rfc822;joey@kitenet.net>);
+ Mon, 11 Jul 2016 18:47:03 -0400
+Received: from kitenet.net ([66.228.36.95]:60970 "EHLO kitenet.net"
+ rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP
+ id S1752439AbcGKWrA (ORCPT <rfc822;git@vger.kernel.org>);
+ Mon, 11 Jul 2016 18:47:00 -0400
+X-Question: 42
+DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=joeyh.name; s=mail;
+ t=1468277198; bh=NRbzz9lX+l0UUh9pAD1CQEaHhmhOg2qk05TuAMiBsPo=;
+ h=From:To:Cc:Subject:Date:In-Reply-To:References:From;
+ b=NOnwhtEtJu+wBwiPrABDPIiic0tTHORjOkouxfQbD+rgbTjsY2mKzfmOsW0JdL5c9
+ UyIeOOTmDA8jFAvt/o/9Yj7P39nbgg7fjX/tS8xd5oYSRV15Vg0ePEZ98nSHu+KjOR
+ q06IOF8YyqtTAz4S3lSXXlNFfpqrFw8bQT4Xhq6Y=
+From: Joey Hess <joeyh@joeyh.name>
+To: git@vger.kernel.org
+Cc: Joey Hess <joeyh@joeyh.name>
+Subject: [PATCH v5 2/8] add smudgeToFile and cleanFromFile filter configs
+Date: Mon, 11 Jul 2016 18:45:06 -0400
+Message-Id: <1468277112-9909-3-git-send-email-joeyh@joeyh.name>
+X-Mailer: git-send-email 2.8.1
+In-Reply-To: <1468277112-9909-1-git-send-email-joeyh@joeyh.name>
+References: <1468277112-9909-1-git-send-email-joeyh@joeyh.name>
+X-Spam-Status: No, score=-8.2 required=10.0 tests=BAYES_00,DKIM_SIGNED,
+ HEADER_FROM_DIFFERENT_DOMAINS,RCVD_IN_DNSWL_HI,RP_MATCHES_RCVD,T_DKIM_INVALID,
+ URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.1
+X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on kite.kitenet.net
+Sender: git-owner@vger.kernel.org
+Precedence: bulk
+List-ID: <git.vger.kernel.org>
+X-Mailing-List: git@vger.kernel.org
+Content-Length: 13634
+Lines: 374
+
+This adds new smudgeToFile and cleanFromFile filter commands,
+which are similar to smudge and clean but allow direct access to files on
+disk.
+
+This interface can be much more efficient when operating on large files,
+because the whole file content does not need to be streamed through the
+filter. It even allows for things like cleanFromFile commands that avoid
+reading the whole content of the file, and for smudgeToFile commands that
+populate a work tree file using an efficient Copy On Write operation.
+
+The new filter commands will not be used for all filtering. They are
+efficient to use when git add is adding a file, or when the work tree is
+being updated, but not a good fit when git is internally filtering blob
+objects in memory for eg, a diff.
+
+So, a user who wants to use smudgeToFile should also provide a smudge
+command to be used in cases where smudgeToFile is not used. And ditto
+with cleanFromFile and clean. To avoid foot-shooting configurations, the
+new commands are not used unless the old commands are also configured.
+
+That also ensures that a filter driver configuration that includes these
+new commands will work, although less efficiently, when used with an older
+version of git that does not support them.
+
+Signed-off-by: Joey Hess <joeyh@joeyh.name>
+---
+ Documentation/config.txt | 18 ++++++-
+ Documentation/gitattributes.txt | 37 ++++++++++++++
+ convert.c | 111 +++++++++++++++++++++++++++++++++++-----
+ convert.h | 10 ++++
+ 4 files changed, 160 insertions(+), 16 deletions(-)
+
+diff --git a/Documentation/config.txt b/Documentation/config.txt
+index 19493aa..a55bed8 100644
+--- a/Documentation/config.txt
++++ b/Documentation/config.txt
+@@ -1325,15 +1325,29 @@ format.useAutoBase::
+ format-patch by default.
+
+ filter.<driver>.clean::
+- The command which is used to convert the content of a worktree
++ The command which is used as a filter to convert the content of a worktree
+ file to a blob upon checkin. See linkgit:gitattributes[5] for
+ details.
+
+ filter.<driver>.smudge::
+- The command which is used to convert the content of a blob
++ The command which is used as a filter to convert the content of a blob
+ object to a worktree file upon checkout. See
+ linkgit:gitattributes[5] for details.
+
++filter.<driver>.cleanFromFile::
++ Similar to filter.<driver>.clean but the specified command
++ directly accesses a worktree file on disk, rather than
++ receiving the file content from standard input.
++ Only used when filter.<driver>.clean is also configured.
++ See linkgit:gitattributes[5] for details.
++
++filter.<driver>.smudgeToFile::
++ Similar to filter.<driver>.smudge but the specified command
++ writes the content of a blob directly to a worktree file,
++ rather than to standard output.
++ Only used when filter.<driver>.smudge is also configured.
++ See linkgit:gitattributes[5] for details.
++
+ fsck.<msg-id>::
+ Allows overriding the message type (error, warn or ignore) of a
+ specific message ID such as `missingEmail`.
+diff --git a/Documentation/gitattributes.txt b/Documentation/gitattributes.txt
+index 197ece8..a58aafc 100644
+--- a/Documentation/gitattributes.txt
++++ b/Documentation/gitattributes.txt
+@@ -385,6 +385,43 @@ not exist, or may have different contents. So, smudge and clean commands
+ should not try to access the file on disk, but only act as filters on the
+ content provided to them on standard input.
+
++There are two extra commands "cleanFromFile" and "smudgeToFile", which
++can optionally be set in a filter driver. These are similar to the "clean"
++and "smudge" commands, but avoid needing to pipe the contents of files
++through the filters, and instead read/write files in the filesystem.
++This can be more efficient when using filters with large files that are not
++directly stored in the repository.
++
++Both "cleanFromFile" and "smudgeToFile" are provided a path as an
++added parameter after the configured command line.
++
++The "cleanFromFile" command is provided the path to the file that
++it should clean. Like the "clean" command, it should output the cleaned
++version to standard output.
++
++The "smudgeToFile" command is provided a path to the file that it
++should write to. (This file will already exist, as an empty file that can
++be written to or replaced.) Like the "smudge" command, "smudgeToFile"
++is fed the blob object from its standard input.
++
++Some git operations that need to apply filters cannot use "cleanFromFile"
++and "smudgeToFile", since the files are not present to disk. So, to avoid
++inconsistent behavior, "cleanFromFile" will only be used if "clean" is
++also configured, and "smudgeToFile" will only be used if "smudge" is also
++configured.
++
++An example large file storage filter driver using cleanFromFile and
++smudgeToFile follows:
++
++------------------------
++[filter "bigfiles"]
++ clean = store-bigfile --from-stdin
++ cleanFromFile = store-bigfile --from-file
++ smudge = retrieve-bigfile --to-stdout
++ smudgeToFile = retrieve-bigfile --to-file
++ required
++------------------------
++
+ Interaction between checkin/checkout attributes
+ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+diff --git a/convert.c b/convert.c
+index 214c99f..eb7774f 100644
+--- a/convert.c
++++ b/convert.c
+@@ -358,7 +358,8 @@ struct filter_params {
+ unsigned long size;
+ int fd;
+ const char *cmd;
+- const char *path;
++ const char *path; /* Path within the git repository */
++ const char *fspath; /* Path to file on disk */
+ };
+
+ static int filter_buffer_or_fd(int in, int out, void *data)
+@@ -387,6 +388,15 @@ static int filter_buffer_or_fd(int in, int out, void *data)
+ strbuf_expand(&cmd, params->cmd, strbuf_expand_dict_cb, &dict);
+ strbuf_release(&path);
+
++ /* append fspath to the command if it's set, separated with a space */
++ if (params->fspath) {
++ struct strbuf fspath = STRBUF_INIT;
++ sq_quote_buf(&fspath, params->fspath);
++ strbuf_addstr(&cmd, " ");
++ strbuf_addbuf(&cmd, &fspath);
++ strbuf_release(&fspath);
++ }
++
+ argv[0] = cmd.buf;
+
+ child_process.argv = argv;
+@@ -425,7 +435,8 @@ static int filter_buffer_or_fd(int in, int out, void *data)
+ return (write_err || status);
+ }
+
+-static int apply_filter(const char *path, const char *src, size_t len, int fd,
++static int apply_filter(const char *path, const char *fspath,
++ const char *src, size_t len, int fd,
+ struct strbuf *dst, const char *cmd)
+ {
+ /*
+@@ -454,6 +465,7 @@ static int apply_filter(const char *path, const char *src, size_t len, int fd,
+ params.fd = fd;
+ params.cmd = cmd;
+ params.path = path;
++ params.fspath = fspath;
+
+ fflush(NULL);
+ if (start_async(&async))
+@@ -484,6 +496,8 @@ static struct convert_driver {
+ struct convert_driver *next;
+ const char *smudge;
+ const char *clean;
++ const char *smudge_to_file;
++ const char *clean_from_file;
+ int required;
+ } *user_convert, **user_convert_tail;
+
+@@ -510,8 +524,9 @@ static int read_convert_config(const char *var, const char *value, void *cb)
+ }
+
+ /*
+- * filter.<name>.smudge and filter.<name>.clean specifies
+- * the command line:
++ * filter.<name>.smudge, filter.<name>.clean,
++ * filter.<name>.smudgeToFile, filter.<name>.cleanFromFile
++ * specifies the command line:
+ *
+ * command-line
+ *
+@@ -524,6 +539,12 @@ static int read_convert_config(const char *var, const char *value, void *cb)
+ if (!strcmp("clean", key))
+ return git_config_string(&drv->clean, var, value);
+
++ if (!strcmp("smudgetofile", key))
++ return git_config_string(&drv->smudge_to_file, var, value);
++
++ if (!strcmp("cleanfromfile", key))
++ return git_config_string(&drv->clean_from_file, var, value);
++
+ if (!strcmp("required", key)) {
+ drv->required = git_config_bool(var, value);
+ return 0;
+@@ -821,7 +842,37 @@ int would_convert_to_git_filter_fd(const char *path)
+ if (!ca.drv->required)
+ return 0;
+
+- return apply_filter(path, NULL, 0, -1, NULL, ca.drv->clean);
++ return apply_filter(path, NULL, NULL, 0, -1, NULL, ca.drv->clean);
++}
++
++int can_clean_from_file(const char *path)
++{
++ struct conv_attrs ca;
++
++ convert_attrs(&ca, path);
++ if (!ca.drv)
++ return 0;
++
++ /*
++ * Only use the cleanFromFile filter when the clean filter is also
++ * configured.
++ */
++ return (ca.drv->clean_from_file && ca.drv->clean);
++}
++
++int can_smudge_to_file(const char *path)
++{
++ struct conv_attrs ca;
++
++ convert_attrs(&ca, path);
++ if (!ca.drv)
++ return 0;
++
++ /*
++ * Only use the smudgeToFile filter when the smudge filter is also
++ * configured.
++ */
++ return (ca.drv->smudge_to_file && ca.drv->smudge);
+ }
+
+ const char *get_convert_attr_ascii(const char *path)
+@@ -864,7 +915,7 @@ int convert_to_git(const char *path, const char *src, size_t len,
+ required = ca.drv->required;
+ }
+
+- ret |= apply_filter(path, src, len, -1, dst, filter);
++ ret |= apply_filter(path, NULL, src, len, -1, dst, filter);
+ if (!ret && required)
+ die("%s: clean filter '%s' failed", path, ca.drv->name);
+
+@@ -889,14 +940,34 @@ void convert_to_git_filter_fd(const char *path, int fd, struct strbuf *dst,
+ assert(ca.drv);
+ assert(ca.drv->clean);
+
+- if (!apply_filter(path, NULL, 0, fd, dst, ca.drv->clean))
++ if (!apply_filter(path, NULL, NULL, 0, fd, dst, ca.drv->clean))
+ die("%s: clean filter '%s' failed", path, ca.drv->name);
+
+ crlf_to_git(path, dst->buf, dst->len, dst, ca.crlf_action, checksafe);
+ ident_to_git(path, dst->buf, dst->len, dst, ca.ident);
+ }
+
+-static int convert_to_working_tree_internal(const char *path, const char *src,
++void convert_to_git_filter_from_file(const char *path, struct strbuf *dst,
++ enum safe_crlf checksafe)
++{
++ struct conv_attrs ca;
++ convert_attrs(&ca, path);
++
++ assert(ca.drv);
++ assert(ca.drv->clean);
++ assert(ca.drv->clean_from_file);
++
++ if (!apply_filter(path, path, "", 0, -1, dst, ca.drv->clean_from_file))
++ die("%s: cleanFromFile filter '%s' failed", path, ca.drv->name);
++
++ crlf_to_git(path, dst->buf, dst->len, dst, ca.crlf_action,
++ checksafe);
++ ident_to_git(path, dst->buf, dst->len, dst, ca.ident);
++}
++
++static int convert_to_working_tree_internal(const char *path,
++ const char *destpath,
++ const char *src,
+ size_t len, struct strbuf *dst,
+ int normalizing)
+ {
+@@ -907,7 +978,10 @@ static int convert_to_working_tree_internal(const char *path, const char *src,
+
+ convert_attrs(&ca, path);
+ if (ca.drv) {
+- filter = ca.drv->smudge;
++ if (destpath)
++ filter = ca.drv->smudge_to_file;
++ else
++ filter = ca.drv->smudge;
+ required = ca.drv->required;
+ }
+
+@@ -918,7 +992,7 @@ static int convert_to_working_tree_internal(const char *path, const char *src,
+ }
+ /*
+ * CRLF conversion can be skipped if normalizing, unless there
+- * is a smudge filter. The filter might expect CRLFs.
++ * is a filter. The filter might expect CRLFs.
+ */
+ if (filter || !normalizing) {
+ ret |= crlf_to_worktree(path, src, len, dst, ca.crlf_action);
+@@ -928,21 +1002,30 @@ static int convert_to_working_tree_internal(const char *path, const char *src,
+ }
+ }
+
+- ret_filter = apply_filter(path, src, len, -1, dst, filter);
++ ret_filter = apply_filter(path, destpath, src, len, -1, dst, filter);
+ if (!ret_filter && required)
+- die("%s: smudge filter %s failed", path, ca.drv->name);
++ die("%s: %s filter %s failed", path, destpath ? "smudgeToFile" : "smudge", ca.drv->name);
+
+ return ret | ret_filter;
+ }
+
+ int convert_to_working_tree(const char *path, const char *src, size_t len, struct strbuf *dst)
+ {
+- return convert_to_working_tree_internal(path, src, len, dst, 0);
++ return convert_to_working_tree_internal(path, NULL, src, len, dst, 0);
++}
++
++int convert_to_working_tree_filter_to_file(const char *path, const char *destpath, const char *src, size_t len)
++{
++ struct strbuf output = STRBUF_INIT;
++ int ret = convert_to_working_tree_internal(path, destpath, src, len, &output, 0);
++ /* The smudgeToFile filter stdout is not used. */
++ strbuf_release(&output);
++ return ret;
+ }
+
+ int renormalize_buffer(const char *path, const char *src, size_t len, struct strbuf *dst)
+ {
+- int ret = convert_to_working_tree_internal(path, src, len, dst, 1);
++ int ret = convert_to_working_tree_internal(path, NULL, src, len, dst, 1);
+ if (ret) {
+ src = dst->buf;
+ len = dst->len;
+diff --git a/convert.h b/convert.h
+index 82871a1..6f46d10 100644
+--- a/convert.h
++++ b/convert.h
+@@ -42,6 +42,10 @@ extern int convert_to_git(const char *path, const char *src, size_t len,
+ struct strbuf *dst, enum safe_crlf checksafe);
+ extern int convert_to_working_tree(const char *path, const char *src,
+ size_t len, struct strbuf *dst);
++extern int convert_to_working_tree_filter_to_file(const char *path,
++ const char *destpath,
++ const char *src,
++ size_t len);
+ extern int renormalize_buffer(const char *path, const char *src, size_t len,
+ struct strbuf *dst);
+ static inline int would_convert_to_git(const char *path)
+@@ -53,6 +57,12 @@ extern void convert_to_git_filter_fd(const char *path, int fd,
+ struct strbuf *dst,
+ enum safe_crlf checksafe);
+ extern int would_convert_to_git_filter_fd(const char *path);
++/* Precondition: can_clean_from_file(path) == true */
++extern void convert_to_git_filter_from_file(const char *path,
++ struct strbuf *dst,
++ enum safe_crlf checksafe);
++extern int can_clean_from_file(const char *path);
++extern int can_smudge_to_file(const char *path);
+
+ /*****************************************************************
+ *
+--
+2.8.1
+
+--
+To unsubscribe from this list: send the line "unsubscribe git" in
+the body of a message to majordomo@vger.kernel.org
+More majordomo info at http://vger.kernel.org/majordomo-info.html
+
+From mairix@mairix Mon Jan 1 12:34:56 1970
+X-source-folder: /home/joey/mail/.git/annex/objects/0k/FJ/SHA256E-s162778--6d2a0fc6afd1077eccdca2e92f314c00ef70a06273242429035e0ce52cce2e13.gz/SHA256E-s162778--6d2a0fc6afd1077eccdca2e92f314c00ef70a06273242429035e0ce52cce2e13.gz
+Return-Path: <git-owner@vger.kernel.org>
+X-Original-To: joey@kitenet.net
+Delivered-To: joey@kitenet.net
+Received: from vger.kernel.org (vger.kernel.org [209.132.180.67])
+ by kitenet.net (Postfix) with ESMTP id 92F4B1C677
+ for <joey@kitenet.net>; Mon, 11 Jul 2016 18:47:02 -0400 (EDT)
+Authentication-Results: kitenet.net;
+ dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=joeyh.name header.i=@joeyh.name header.b=O21fZnmu;
+ dkim-atps=neutral
+Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
+ id S1752507AbcGKWq7 (ORCPT <rfc822;joey@kitenet.net>);
+ Mon, 11 Jul 2016 18:46:59 -0400
+Received: from kitenet.net ([66.228.36.95]:60948 "EHLO kitenet.net"
+ rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP
+ id S1752439AbcGKWq6 (ORCPT <rfc822;git@vger.kernel.org>);
+ Mon, 11 Jul 2016 18:46:58 -0400
+X-Question: 42
+DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=joeyh.name; s=mail;
+ t=1468277197; bh=ID7Cvl2BMx/TZGvnWWIJ3y9tfZmZmZ0hcgqOsxLVm9Q=;
+ h=From:To:Cc:Subject:Date:In-Reply-To:References:From;
+ b=O21fZnmuEK0afkVNZdA2KL+bhF+skap1V8rdhqtR0vzDPg7cri/BlGPK8eY5G/xi6
+ 7f7vYgrq3QCN/0Q7cXsHSm0k8sgA2DLnriNR0Ga/gtk6OjqDr2JNHEMDoEt6MMyVto
+ 4wAJLa+dzSUA8Y+8bwZ760DVTucDOLRoEINUvJ0Y=
+From: Joey Hess <joeyh@joeyh.name>
+To: git@vger.kernel.org
+Cc: Joey Hess <joeyh@joeyh.name>
+Subject: [PATCH v5 3/8] use cleanFromFile in git add
+Date: Mon, 11 Jul 2016 18:45:07 -0400
+Message-Id: <1468277112-9909-4-git-send-email-joeyh@joeyh.name>
+X-Mailer: git-send-email 2.8.1
+In-Reply-To: <1468277112-9909-1-git-send-email-joeyh@joeyh.name>
+References: <1468277112-9909-1-git-send-email-joeyh@joeyh.name>
+X-Spam-Status: No, score=-8.2 required=10.0 tests=BAYES_00,DKIM_SIGNED,
+ HEADER_FROM_DIFFERENT_DOMAINS,RCVD_IN_DNSWL_HI,RP_MATCHES_RCVD,T_DKIM_INVALID,
+ URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.1
+X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on kite.kitenet.net
+Sender: git-owner@vger.kernel.org
+Precedence: bulk
+List-ID: <git.vger.kernel.org>
+X-Mailing-List: git@vger.kernel.org
+Content-Length: 3641
+Lines: 127
+
+Includes test cases.
+
+Signed-off-by: Joey Hess <joeyh@joeyh.name>
+---
+ sha1_file.c | 42 ++++++++++++++++++++++++++++++++++++------
+ t/t0021-conversion.sh | 36 ++++++++++++++++++++++++++++++++++++
+ 2 files changed, 72 insertions(+), 6 deletions(-)
+
+diff --git a/sha1_file.c b/sha1_file.c
+index 2fc22b0..549a20f 100644
+--- a/sha1_file.c
++++ b/sha1_file.c
+@@ -3335,6 +3335,29 @@ static int index_stream_convert_blob(unsigned char *sha1, int fd,
+ return ret;
+ }
+
++static int index_from_file_convert_blob(unsigned char *sha1,
++ const char *path, unsigned flags)
++{
++ int ret;
++ const int write_object = flags & HASH_WRITE_OBJECT;
++ struct strbuf sbuf = STRBUF_INIT;
++
++ assert(path);
++ assert(can_clean_from_file(path));
++
++ convert_to_git_filter_from_file(path, &sbuf,
++ write_object ? safe_crlf : SAFE_CRLF_FALSE);
++
++ if (write_object)
++ ret = write_sha1_file(sbuf.buf, sbuf.len, typename(OBJ_BLOB),
++ sha1);
++ else
++ ret = hash_sha1_file(sbuf.buf, sbuf.len, typename(OBJ_BLOB),
++ sha1);
++ strbuf_release(&sbuf);
++ return ret;
++}
++
+ static int index_pipe(unsigned char *sha1, int fd, enum object_type type,
+ const char *path, unsigned flags)
+ {
+@@ -3427,12 +3450,19 @@ int index_path(unsigned char *sha1, const char *path, struct stat *st, unsigned
+
+ switch (st->st_mode & S_IFMT) {
+ case S_IFREG:
+- fd = open(path, O_RDONLY);
+- if (fd < 0)
+- return error_errno("open(\"%s\")", path);
+- if (index_fd(sha1, fd, st, OBJ_BLOB, path, flags) < 0)
+- return error("%s: failed to insert into database",
+- path);
++ if (can_clean_from_file(path)) {
++ if (index_from_file_convert_blob(sha1, path, flags) < 0)
++ return error("%s: failed to insert into database",
++ path);
++ }
++ else {
++ fd = open(path, O_RDONLY);
++ if (fd < 0)
++ return error_errno("open(\"%s\")", path);
++ if (index_fd(sha1, fd, st, OBJ_BLOB, path, flags) < 0)
++ return error("%s: failed to insert into database",
++ path);
++ }
+ break;
+ case S_IFLNK:
+ if (strbuf_readlink(&sb, path, st->st_size))
+diff --git a/t/t0021-conversion.sh b/t/t0021-conversion.sh
+index 7bac2bc..bd84b80 100755
+--- a/t/t0021-conversion.sh
++++ b/t/t0021-conversion.sh
+@@ -12,6 +12,14 @@ tr \
+ EOF
+ chmod +x rot13.sh
+
++cat <<EOF >rot13-from-file.sh
++#!$SHELL_PATH
++fsfile="\$1"
++touch rot13-from-file.ran
++cat "\$fsfile" | ./rot13.sh
++EOF
++chmod +x rot13-from-file.sh
++
+ test_expect_success setup '
+ git config filter.rot13.smudge ./rot13.sh &&
+ git config filter.rot13.clean ./rot13.sh &&
+@@ -268,4 +276,32 @@ test_expect_success 'disable filter with empty override' '
+ test_must_be_empty err
+ '
+
++test_expect_success 'cleanFromFile filter is used when adding a file' '
++ test_config filter.rot13.cleanFromFile ./rot13-from-file.sh &&
++
++ echo "*.t filter=rot13" >.gitattributes &&
++
++ cat test >fstest.t &&
++ git add fstest.t &&
++ test -e rot13-from-file.ran &&
++ rm -f rot13-from-file.ran &&
++
++ rm -f fstest.t &&
++ git checkout -- fstest.t &&
++ test_cmp test fstest.t
++'
++
++test_expect_success 'cleanFromFile filter is not used when clean filter is not configured' '
++ test_config filter.noclean.smudge ./rot13.sh &&
++ test_config filter.noclean.cleanFromFile ./rot13-from-file.sh &&
++
++ echo "*.no filter=noclean" >.gitattributes &&
++
++ cat test >test.no &&
++ git add test.no &&
++ test ! -e rot13-from-file.ran &&
++ git cat-file blob :test.no >actual &&
++ test_cmp test actual
++'
++
+ test_done
+--
+2.8.1
+
+--
+To unsubscribe from this list: send the line "unsubscribe git" in
+the body of a message to majordomo@vger.kernel.org
+More majordomo info at http://vger.kernel.org/majordomo-info.html
+
+From mairix@mairix Mon Jan 1 12:34:56 1970
+X-source-folder: /home/joey/mail/.git/annex/objects/0k/FJ/SHA256E-s162778--6d2a0fc6afd1077eccdca2e92f314c00ef70a06273242429035e0ce52cce2e13.gz/SHA256E-s162778--6d2a0fc6afd1077eccdca2e92f314c00ef70a06273242429035e0ce52cce2e13.gz
+Return-Path: <git-owner@vger.kernel.org>
+X-Original-To: joey@kitenet.net
+Delivered-To: joey@kitenet.net
+Received: from vger.kernel.org (vger.kernel.org [209.132.180.67])
+ by kitenet.net (Postfix) with ESMTP id 363761C677
+ for <joey@kitenet.net>; Mon, 11 Jul 2016 18:47:18 -0400 (EDT)
+Authentication-Results: kitenet.net;
+ dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=joeyh.name header.i=@joeyh.name header.b=P+WpFgpS;
+ dkim-atps=neutral
+Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
+ id S1752612AbcGKWrO (ORCPT <rfc822;joey@kitenet.net>);
+ Mon, 11 Jul 2016 18:47:14 -0400
+Received: from kitenet.net ([66.228.36.95]:32768 "EHLO kitenet.net"
+ rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP
+ id S1752439AbcGKWrN (ORCPT <rfc822;git@vger.kernel.org>);
+ Mon, 11 Jul 2016 18:47:13 -0400
+X-Question: 42
+DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=joeyh.name; s=mail;
+ t=1468277198; bh=tAWpo3wY2ayI3N2P5PIoaKI4XKz6M0EPI7WSh1kSoZc=;
+ h=From:To:Cc:Subject:Date:In-Reply-To:References:From;
+ b=P+WpFgpSST+oHI+yvWOFavctP60LP4dRf0CJA1N75Wk/pQIHue3j4sxVPNsjTAuj6
+ IzUxRBE2Ilz3E0UF3r6iL2lCVMtITzMvvTqYjr9xEXi8mGgInDOWD1bi7bdmt2NY1z
+ ZlAIY9kjZhF+1jmpxhc1YP+tkocCMOsmT/DbiGRI=
+From: Joey Hess <joeyh@joeyh.name>
+To: git@vger.kernel.org
+Cc: Joey Hess <joeyh@joeyh.name>
+Subject: [PATCH v5 4/8] use smudgeToFile in git checkout etc
+Date: Mon, 11 Jul 2016 18:45:08 -0400
+Message-Id: <1468277112-9909-5-git-send-email-joeyh@joeyh.name>
+X-Mailer: git-send-email 2.8.1
+In-Reply-To: <1468277112-9909-1-git-send-email-joeyh@joeyh.name>
+References: <1468277112-9909-1-git-send-email-joeyh@joeyh.name>
+X-Spam-Status: No, score=-8.2 required=10.0 tests=BAYES_00,DKIM_SIGNED,
+ HEADER_FROM_DIFFERENT_DOMAINS,RCVD_IN_DNSWL_HI,RP_MATCHES_RCVD,T_DKIM_INVALID,
+ URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.1
+X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on kite.kitenet.net
+Sender: git-owner@vger.kernel.org
+Precedence: bulk
+List-ID: <git.vger.kernel.org>
+X-Mailing-List: git@vger.kernel.org
+Content-Length: 4367
+Lines: 147
+
+This makes git checkout, git reset, etc use smudgeToFile.
+
+Includes test cases.
+
+(There's a call to convert_to_working_tree in merge-recursive.c
+that could also be made to use smudgeToFile as well.)
+
+Signed-off-by: Joey Hess <joeyh@joeyh.name>
+---
+ entry.c | 40 ++++++++++++++++++++++++++++++++--------
+ t/t0021-conversion.sh | 34 ++++++++++++++++++++++++++++++++--
+ 2 files changed, 64 insertions(+), 10 deletions(-)
+
+diff --git a/entry.c b/entry.c
+index 519e042..81d12a1 100644
+--- a/entry.c
++++ b/entry.c
+@@ -146,6 +146,7 @@ static int write_entry(struct cache_entry *ce,
+ unsigned long size;
+ size_t wrote, newsize = 0;
+ struct stat st;
++ int regular_file, smudge_to_file;
+
+ if (ce_mode_s_ifmt == S_IFREG) {
+ struct stream_filter *filter = get_stream_filter(ce->name, ce->sha1);
+@@ -175,8 +176,13 @@ static int write_entry(struct cache_entry *ce,
+
+ /*
+ * Convert from git internal format to working tree format
++ * unless the smudgeToFile filter can write to the
++ * file directly.
+ */
+- if (ce_mode_s_ifmt == S_IFREG &&
++ regular_file = ce_mode_s_ifmt == S_IFREG;
++ smudge_to_file = regular_file
++ && can_smudge_to_file(ce->name);
++ if (regular_file && !smudge_to_file &&
+ convert_to_working_tree(ce->name, new, size, &buf)) {
+ free(new);
+ new = strbuf_detach(&buf, &newsize);
+@@ -189,13 +195,31 @@ static int write_entry(struct cache_entry *ce,
+ return error_errno("unable to create file %s", path);
+ }
+
+- wrote = write_in_full(fd, new, size);
+- if (!to_tempfile)
+- fstat_done = fstat_output(fd, state, &st);
+- close(fd);
+- free(new);
+- if (wrote != size)
+- return error("unable to write file %s", path);
++ if (!smudge_to_file) {
++ wrote = write_in_full(fd, new, size);
++ if (!to_tempfile)
++ fstat_done = fstat_output(fd, state, &st);
++ close(fd);
++ free(new);
++ if (wrote != size)
++ return error("unable to write file %s", path);
++ }
++ else {
++ close(fd);
++ convert_to_working_tree_filter_to_file(ce->name, path, new, size);
++ free(new);
++ /*
++ * The smudgeToFile filter may have replaced the
++ * file; open it to make sure that the file
++ * exists.
++ */
++ fd = open(path, O_RDONLY);
++ if (fd < 0)
++ return error_errno("unable to create file %s", path);
++ if (!to_tempfile)
++ fstat_done = fstat_output(fd, state, &st);
++ close(fd);
++ }
+ break;
+ case S_IFGITLINK:
+ if (to_tempfile)
+diff --git a/t/t0021-conversion.sh b/t/t0021-conversion.sh
+index bd84b80..ea18b17 100755
+--- a/t/t0021-conversion.sh
++++ b/t/t0021-conversion.sh
+@@ -14,12 +14,20 @@ chmod +x rot13.sh
+
+ cat <<EOF >rot13-from-file.sh
+ #!$SHELL_PATH
+-fsfile="\$1"
++srcfile="\$1"
+ touch rot13-from-file.ran
+-cat "\$fsfile" | ./rot13.sh
++cat "\$srcfile" | ./rot13.sh
+ EOF
+ chmod +x rot13-from-file.sh
+
++cat <<EOF >rot13-to-file.sh
++#!$SHELL_PATH
++destfile="\$1"
++touch rot13-to-file.ran
++./rot13.sh >"\$destfile"
++EOF
++chmod +x rot13-to-file.sh
++
+ test_expect_success setup '
+ git config filter.rot13.smudge ./rot13.sh &&
+ git config filter.rot13.clean ./rot13.sh &&
+@@ -291,6 +299,17 @@ test_expect_success 'cleanFromFile filter is used when adding a file' '
+ test_cmp test fstest.t
+ '
+
++test_expect_success 'smudgeToFile filter is used when checking out a file' '
++ test_config filter.rot13.smudgeToFile ./rot13-to-file.sh &&
++
++ rm -f fstest.t &&
++ git checkout -- fstest.t &&
++ test_cmp test fstest.t &&
++
++ test -e rot13-to-file.ran &&
++ rm -f rot13-to-file.ran
++'
++
+ test_expect_success 'cleanFromFile filter is not used when clean filter is not configured' '
+ test_config filter.noclean.smudge ./rot13.sh &&
+ test_config filter.noclean.cleanFromFile ./rot13-from-file.sh &&
+@@ -304,4 +323,15 @@ test_expect_success 'cleanFromFile filter is not used when clean filter is not c
+ test_cmp test actual
+ '
+
++test_expect_success 'smudgeToFile filter is not used when smudge filter is not configured' '
++ test_config filter.nosmudge.clean ./rot13.sh &&
++ test_config filter.nosmudge.smudgeToFile ./rot13-to-file.sh &&
++
++ echo "*.no filter=nosmudge" >.gitattributes &&
++
++ rm -f fstest.t &&
++ git checkout -- fstest.t &&
++ test ! -e rot13-to-file.ran
++'
++
+ test_done
+--
+2.8.1
+
+--
+To unsubscribe from this list: send the line "unsubscribe git" in
+the body of a message to majordomo@vger.kernel.org
+More majordomo info at http://vger.kernel.org/majordomo-info.html
+
+From mairix@mairix Mon Jan 1 12:34:56 1970
+X-source-folder: /home/joey/mail/.git/annex/objects/0k/FJ/SHA256E-s162778--6d2a0fc6afd1077eccdca2e92f314c00ef70a06273242429035e0ce52cce2e13.gz/SHA256E-s162778--6d2a0fc6afd1077eccdca2e92f314c00ef70a06273242429035e0ce52cce2e13.gz
+Return-Path: <git-owner@vger.kernel.org>
+X-Original-To: joey@kitenet.net
+Delivered-To: joey@kitenet.net
+Received: from vger.kernel.org (vger.kernel.org [209.132.180.67])
+ by kitenet.net (Postfix) with ESMTP id 714E61C67C
+ for <joey@kitenet.net>; Mon, 11 Jul 2016 18:47:12 -0400 (EDT)
+Authentication-Results: kitenet.net;
+ dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=joeyh.name header.i=@joeyh.name header.b=Yy9dgbAc;
+ dkim-atps=neutral
+Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
+ id S1752555AbcGKWrI (ORCPT <rfc822;joey@kitenet.net>);
+ Mon, 11 Jul 2016 18:47:08 -0400
+Received: from kitenet.net ([66.228.36.95]:60984 "EHLO kitenet.net"
+ rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP
+ id S1752439AbcGKWrI (ORCPT <rfc822;git@vger.kernel.org>);
+ Mon, 11 Jul 2016 18:47:08 -0400
+X-Question: 42
+DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=joeyh.name; s=mail;
+ t=1468277198; bh=ZPsSRNaHriT6igc+PUYizSKSXZl5YDOyDcX0bJMWFqk=;
+ h=From:To:Cc:Subject:Date:In-Reply-To:References:From;
+ b=Yy9dgbAcC69LAmxyt4U9d2lqehM4dKl+NuDIHbQ//uTjX1mmOMkj/Pe4n7vrShEUP
+ D4h/CYU5C4oRTJMhV1MF501fgcvPvOujxUMOviTB+g1EH5YfP989HvZT+7xafE3M4m
+ 8+IYWJC1JpDN45UcBwYBROl/ch/87+shPUlms+ds=
+From: Joey Hess <joeyh@joeyh.name>
+To: git@vger.kernel.org
+Cc: Joey Hess <joeyh@joeyh.name>
+Subject: [PATCH v5 5/8] warn on unusable smudgeToFile/cleanFromFile config
+Date: Mon, 11 Jul 2016 18:45:09 -0400
+Message-Id: <1468277112-9909-6-git-send-email-joeyh@joeyh.name>
+X-Mailer: git-send-email 2.8.1
+In-Reply-To: <1468277112-9909-1-git-send-email-joeyh@joeyh.name>
+References: <1468277112-9909-1-git-send-email-joeyh@joeyh.name>
+X-Spam-Status: No, score=-8.2 required=10.0 tests=BAYES_00,DKIM_SIGNED,
+ HEADER_FROM_DIFFERENT_DOMAINS,RCVD_IN_DNSWL_HI,RP_MATCHES_RCVD,T_DKIM_INVALID,
+ URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.1
+X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on kite.kitenet.net
+Sender: git-owner@vger.kernel.org
+Precedence: bulk
+List-ID: <git.vger.kernel.org>
+X-Mailing-List: git@vger.kernel.org
+Content-Length: 2309
+Lines: 84
+
+Let the user know when they have a smudgeToFile/cleanFromFile config
+that cannot be used because the corresponding smudge/clean config
+is missing.
+
+The warning is only displayed a maximum of once per git invocation,
+and only when doing an operation that would use the filter.
+
+Signed-off-by: Joey Hess <joeyh@joeyh.name>
+---
+ convert.c | 36 ++++++++++++++++++++++++++----------
+ 1 file changed, 26 insertions(+), 10 deletions(-)
+
+diff --git a/convert.c b/convert.c
+index eb7774f..e1b0b44 100644
+--- a/convert.c
++++ b/convert.c
+@@ -845,34 +845,50 @@ int would_convert_to_git_filter_fd(const char *path)
+ return apply_filter(path, NULL, NULL, 0, -1, NULL, ca.drv->clean);
+ }
+
++static int can_filter_file(const char *filefilter, const char *filefiltername,
++ const char *stdiofilter, const char *stdiofiltername,
++ const struct conv_attrs *ca,
++ int *warncount)
++{
++ if (!filefilter)
++ return 0;
++
++ if (stdiofilter)
++ return 1;
++
++ if (*warncount == 0)
++ warning("Not running your configured filter.%s.%s command, because filter.%s.%s is not configured",
++ ca->drv->name, filefiltername,
++ ca->drv->name, stdiofiltername);
++ *warncount=*warncount+1;
++
++ return 0;
++}
++
+ int can_clean_from_file(const char *path)
+ {
+ struct conv_attrs ca;
++ static int warncount = 0;
+
+ convert_attrs(&ca, path);
+ if (!ca.drv)
+ return 0;
+
+- /*
+- * Only use the cleanFromFile filter when the clean filter is also
+- * configured.
+- */
+- return (ca.drv->clean_from_file && ca.drv->clean);
++ return can_filter_file(ca.drv->clean_from_file, "cleanFromFile",
++ ca.drv->clean, "clean", &ca, &warncount);
+ }
+
+ int can_smudge_to_file(const char *path)
+ {
+ struct conv_attrs ca;
++ static int warncount = 0;
+
+ convert_attrs(&ca, path);
+ if (!ca.drv)
+ return 0;
+
+- /*
+- * Only use the smudgeToFile filter when the smudge filter is also
+- * configured.
+- */
+- return (ca.drv->smudge_to_file && ca.drv->smudge);
++ return can_filter_file(ca.drv->smudge_to_file, "smudgeToFile",
++ ca.drv->smudge, "smudge", &ca, &warncount);
+ }
+
+ const char *get_convert_attr_ascii(const char *path)
+--
+2.8.1
+
+--
+To unsubscribe from this list: send the line "unsubscribe git" in
+the body of a message to majordomo@vger.kernel.org
+More majordomo info at http://vger.kernel.org/majordomo-info.html
+
+From mairix@mairix Mon Jan 1 12:34:56 1970
+X-source-folder: /home/joey/mail/.git/annex/objects/0k/FJ/SHA256E-s162778--6d2a0fc6afd1077eccdca2e92f314c00ef70a06273242429035e0ce52cce2e13.gz/SHA256E-s162778--6d2a0fc6afd1077eccdca2e92f314c00ef70a06273242429035e0ce52cce2e13.gz
+Return-Path: <git-owner@vger.kernel.org>
+X-Original-To: joey@kitenet.net
+Delivered-To: joey@kitenet.net
+Received: from vger.kernel.org (vger.kernel.org [209.132.180.67])
+ by kitenet.net (Postfix) with ESMTP id 839871C677
+ for <joey@kitenet.net>; Mon, 11 Jul 2016 18:47:27 -0400 (EDT)
+Authentication-Results: kitenet.net;
+ dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=joeyh.name header.i=@joeyh.name header.b=e1tj3LJk;
+ dkim-atps=neutral
+Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
+ id S1752603AbcGKWrW (ORCPT <rfc822;joey@kitenet.net>);
+ Mon, 11 Jul 2016 18:47:22 -0400
+Received: from kitenet.net ([66.228.36.95]:32788 "EHLO kitenet.net"
+ rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP
+ id S1752616AbcGKWrT (ORCPT <rfc822;git@vger.kernel.org>);
+ Mon, 11 Jul 2016 18:47:19 -0400
+X-Question: 42
+DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=joeyh.name; s=mail;
+ t=1468277198; bh=4aNAzyRt9bz6y9MPC2VC5pj9+5O3Gb5fUsDopvg0K64=;
+ h=From:To:Cc:Subject:Date:In-Reply-To:References:From;
+ b=e1tj3LJkIi1ro6kUH77lZhTIqJt1N6KgyzFzcDDEdrFFz4x49MywJnSCKy9xtgY4h
+ gc/cr1oZKUTrgmSXwMT8HD5yRltK8eWXnXOYn816pWnnRSkFlcwB+VWR4hhVcYahcV
+ KM71p6oYWBbs2BULmxjndGZX1lRSx34zt8DePh2Y=
+From: Joey Hess <joeyh@joeyh.name>
+To: git@vger.kernel.org
+Cc: Joey Hess <joeyh@joeyh.name>
+Subject: [PATCH v5 6/8] better recovery from failure of smudgeToFile filter
+Date: Mon, 11 Jul 2016 18:45:10 -0400
+Message-Id: <1468277112-9909-7-git-send-email-joeyh@joeyh.name>
+X-Mailer: git-send-email 2.8.1
+In-Reply-To: <1468277112-9909-1-git-send-email-joeyh@joeyh.name>
+References: <1468277112-9909-1-git-send-email-joeyh@joeyh.name>
+X-Spam-Status: No, score=-8.2 required=10.0 tests=BAYES_00,DKIM_SIGNED,
+ HEADER_FROM_DIFFERENT_DOMAINS,RCVD_IN_DNSWL_HI,RP_MATCHES_RCVD,T_DKIM_INVALID,
+ URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.1
+X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on kite.kitenet.net
+Sender: git-owner@vger.kernel.org
+Precedence: bulk
+List-ID: <git.vger.kernel.org>
+X-Mailing-List: git@vger.kernel.org
+Content-Length: 4558
+Lines: 151
+
+If the smudgeToFile filter fails, it can leave the worktree file with the
+wrong content, or even deleted. Recover from this by falling back to
+running the smudge filter.
+
+Signed-off-by: Joey Hess <joeyh@joeyh.name>
+---
+ entry.c | 66 ++++++++++++++++++++++++++++++++++-----------------
+ t/t0021-conversion.sh | 24 +++++++++++++++++++
+ 2 files changed, 68 insertions(+), 22 deletions(-)
+
+diff --git a/entry.c b/entry.c
+index 81d12a1..7811e31 100644
+--- a/entry.c
++++ b/entry.c
+@@ -182,12 +182,6 @@ static int write_entry(struct cache_entry *ce,
+ regular_file = ce_mode_s_ifmt == S_IFREG;
+ smudge_to_file = regular_file
+ && can_smudge_to_file(ce->name);
+- if (regular_file && !smudge_to_file &&
+- convert_to_working_tree(ce->name, new, size, &buf)) {
+- free(new);
+- new = strbuf_detach(&buf, &newsize);
+- size = newsize;
+- }
+
+ fd = open_output_fd(path, ce, to_tempfile);
+ if (fd < 0) {
+@@ -195,7 +189,51 @@ static int write_entry(struct cache_entry *ce,
+ return error_errno("unable to create file %s", path);
+ }
+
++ if (smudge_to_file) {
++ close(fd);
++ if (convert_to_working_tree_filter_to_file(ce->name, path, new, size)) {
++ free(new);
++ /*
++ * The smudgeToFile filter may have replaced
++ * or deleted the file; reopen it to make
++ * sure that the file exists.
++ */
++ fd = open(path, O_RDONLY);
++ if (fd < 0)
++ return error_errno("unable to create file %s", path);
++ if (!to_tempfile)
++ fstat_done = fstat_output(fd, state, &st);
++ close(fd);
++ }
++ else {
++ /*
++ * The failing smudgeToFile filter may have
++ * deleted or replaced the file; delete
++ * the file and re-open for recovery write.
++ */
++ unlink(path);
++ fd = open_output_fd(path, ce, to_tempfile);
++ if (fd < 0) {
++ free(new);
++ return error_errno("unable to create file %s", path);
++ }
++ /* Fall through to normal write below. */
++ smudge_to_file = 0;
++ }
++ }
++
++ /*
++ * Not an else of above if (smudge_to_file) because the
++ * smudgeToFile filter may fail and in that case this is
++ * run to recover.
++ */
+ if (!smudge_to_file) {
++ if (regular_file &&
++ convert_to_working_tree(ce->name, new, size, &buf)) {
++ free(new);
++ new = strbuf_detach(&buf, &newsize);
++ size = newsize;
++ }
+ wrote = write_in_full(fd, new, size);
+ if (!to_tempfile)
+ fstat_done = fstat_output(fd, state, &st);
+@@ -204,22 +242,6 @@ static int write_entry(struct cache_entry *ce,
+ if (wrote != size)
+ return error("unable to write file %s", path);
+ }
+- else {
+- close(fd);
+- convert_to_working_tree_filter_to_file(ce->name, path, new, size);
+- free(new);
+- /*
+- * The smudgeToFile filter may have replaced the
+- * file; open it to make sure that the file
+- * exists.
+- */
+- fd = open(path, O_RDONLY);
+- if (fd < 0)
+- return error_errno("unable to create file %s", path);
+- if (!to_tempfile)
+- fstat_done = fstat_output(fd, state, &st);
+- close(fd);
+- }
+ break;
+ case S_IFGITLINK:
+ if (to_tempfile)
+diff --git a/t/t0021-conversion.sh b/t/t0021-conversion.sh
+index ea18b17..0efad9b 100755
+--- a/t/t0021-conversion.sh
++++ b/t/t0021-conversion.sh
+@@ -28,6 +28,14 @@ touch rot13-to-file.ran
+ EOF
+ chmod +x rot13-to-file.sh
+
++cat <<EOF >delete-file-and-fail.sh
++#!$SHELL_PATH
++destfile="\$1"
++rm -f "\$destfile"
++exit 1
++EOF
++chmod +x delete-file-and-fail.sh
++
+ test_expect_success setup '
+ git config filter.rot13.smudge ./rot13.sh &&
+ git config filter.rot13.clean ./rot13.sh &&
+@@ -310,6 +318,22 @@ test_expect_success 'smudgeToFile filter is used when checking out a file' '
+ rm -f rot13-to-file.ran
+ '
+
++test_expect_success 'recovery from failure of smudgeToFile filter, using smudge filter' '
++ test_config filter.rot13.smudgeToFile false &&
++
++ rm -f fstest.t &&
++ git checkout -- fstest.t &&
++ test_cmp test fstest.t
++'
++
++test_expect_success 'recovery from failure of smudgeToFile filter that deletes the worktree file' '
++ test_config filter.rot13.smudgeToFile ./delete-file-and-fail.sh &&
++
++ rm -f fstest.t &&
++ git checkout -- fstest.t &&
++ test_cmp test fstest.t
++'
++
+ test_expect_success 'cleanFromFile filter is not used when clean filter is not configured' '
+ test_config filter.noclean.smudge ./rot13.sh &&
+ test_config filter.noclean.cleanFromFile ./rot13-from-file.sh &&
+--
+2.8.1
+
+--
+To unsubscribe from this list: send the line "unsubscribe git" in
+the body of a message to majordomo@vger.kernel.org
+More majordomo info at http://vger.kernel.org/majordomo-info.html
+
+From mairix@mairix Mon Jan 1 12:34:56 1970
+X-source-folder: /home/joey/mail/.git/annex/objects/0k/FJ/SHA256E-s162778--6d2a0fc6afd1077eccdca2e92f314c00ef70a06273242429035e0ce52cce2e13.gz/SHA256E-s162778--6d2a0fc6afd1077eccdca2e92f314c00ef70a06273242429035e0ce52cce2e13.gz
+Return-Path: <git-owner@vger.kernel.org>
+X-Original-To: joey@kitenet.net
+Delivered-To: joey@kitenet.net
+Received: from vger.kernel.org (vger.kernel.org [209.132.180.67])
+ by kitenet.net (Postfix) with ESMTP id 3B2531C677
+ for <joey@kitenet.net>; Mon, 11 Jul 2016 18:47:23 -0400 (EDT)
+Authentication-Results: kitenet.net;
+ dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=joeyh.name header.i=@joeyh.name header.b=P7RXYLHP;
+ dkim-atps=neutral
+Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
+ id S1752629AbcGKWrT (ORCPT <rfc822;joey@kitenet.net>);
+ Mon, 11 Jul 2016 18:47:19 -0400
+Received: from kitenet.net ([66.228.36.95]:32782 "EHLO kitenet.net"
+ rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP
+ id S1752603AbcGKWrS (ORCPT <rfc822;git@vger.kernel.org>);
+ Mon, 11 Jul 2016 18:47:18 -0400
+X-Question: 42
+DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=joeyh.name; s=mail;
+ t=1468277198; bh=R6DA3FWAltlIvvD5mJEp7ZgM7JFZFfGDS/9auV9Z+0Q=;
+ h=From:To:Cc:Subject:Date:In-Reply-To:References:From;
+ b=P7RXYLHPbs0CXSl4ChFu0IJTC7WGua7f0VyocRm/r3ZZK/2HscKr3lMHDy0fxfD3I
+ grgLgEiaffIZ5+vMX0FWQIYDLISJD4NlDMYYt3NoONm10nDHJp/vSm1/YdyoaT8+3p
+ RnZbAzgY6m43Lhx7BxNApQ8/UPzUL5hWD45Jl5YU=
+From: Joey Hess <joeyh@joeyh.name>
+To: git@vger.kernel.org
+Cc: Joey Hess <joeyh@joeyh.name>
+Subject: [PATCH v5 7/8] use smudgeToFile filter in git am
+Date: Mon, 11 Jul 2016 18:45:11 -0400
+Message-Id: <1468277112-9909-8-git-send-email-joeyh@joeyh.name>
+X-Mailer: git-send-email 2.8.1
+In-Reply-To: <1468277112-9909-1-git-send-email-joeyh@joeyh.name>
+References: <1468277112-9909-1-git-send-email-joeyh@joeyh.name>
+X-Spam-Status: No, score=-8.2 required=10.0 tests=BAYES_00,DKIM_SIGNED,
+ HEADER_FROM_DIFFERENT_DOMAINS,RCVD_IN_DNSWL_HI,RP_MATCHES_RCVD,T_DKIM_INVALID,
+ URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.1
+X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on kite.kitenet.net
+Sender: git-owner@vger.kernel.org
+Precedence: bulk
+List-ID: <git.vger.kernel.org>
+X-Mailing-List: git@vger.kernel.org
+Content-Length: 4808
+Lines: 156
+
+git am updates the work tree and so should use the smudgeToFile filter.
+
+This includes some refactoring into convert_to_working_tree_filter_to_file
+to make it check the file after running the smudgeToFile command, and clean
+up from a failing command.
+
+Signed-off-by: Joey Hess <joeyh@joeyh.name>
+---
+ apply.c | 16 ++++++++++++++++
+ convert.c | 25 +++++++++++++++++++++++--
+ entry.c | 21 ++++-----------------
+ t/t0021-conversion.sh | 13 +++++++++++++
+ 4 files changed, 56 insertions(+), 19 deletions(-)
+
+diff --git a/apply.c b/apply.c
+index 4a6b2db..7db8344 100644
+--- a/apply.c
++++ b/apply.c
+@@ -4322,6 +4322,22 @@ static int try_create_file(const char *path, unsigned int mode, const char *buf,
+ if (fd < 0)
+ return 1;
+
++ if (can_smudge_to_file(path)) {
++ close(fd);
++ fd = convert_to_working_tree_filter_to_file(path, path, buf, size);
++ if (fd < 0) {
++ /* smudgeToFile filter failed; continue
++ * with regular file creation instead. */
++ fd = open(path, O_CREAT | O_EXCL | O_WRONLY, (mode & 0100) ? 0777 : 0666);
++ if (fd < 0)
++ return -1;
++ }
++ else {
++ close(fd);
++ return 0;
++ }
++ }
++
+ if (convert_to_working_tree(path, buf, size, &nbuf)) {
+ size = nbuf.len;
+ buf = nbuf.buf;
+diff --git a/convert.c b/convert.c
+index e1b0b44..3746ad5 100644
+--- a/convert.c
++++ b/convert.c
+@@ -1030,13 +1030,34 @@ int convert_to_working_tree(const char *path, const char *src, size_t len, struc
+ return convert_to_working_tree_internal(path, NULL, src, len, dst, 0);
+ }
+
++/*
++ * Returns fd open to read the worktree file on success.
++ * On failure, the worktree file will not exist.
++ */
+ int convert_to_working_tree_filter_to_file(const char *path, const char *destpath, const char *src, size_t len)
+ {
+ struct strbuf output = STRBUF_INIT;
+- int ret = convert_to_working_tree_internal(path, destpath, src, len, &output, 0);
++ int ok = convert_to_working_tree_internal(path, destpath, src, len, &output, 0);
+ /* The smudgeToFile filter stdout is not used. */
+ strbuf_release(&output);
+- return ret;
++ if (ok) {
++ /*
++ * Open the file to make sure that it's present
++ * (and readable) after the command populated it.
++ */
++ int fd = open(path, O_RDONLY);
++ if (fd < 0)
++ unlink(path);
++ return fd;
++ }
++ else {
++ /*
++ * The command could have created the file before failing,
++ * so delete it.
++ */
++ unlink(path);
++ return -1;
++ }
+ }
+
+ int renormalize_buffer(const char *path, const char *src, size_t len, struct strbuf *dst)
+diff --git a/entry.c b/entry.c
+index 7811e31..40662eb 100644
+--- a/entry.c
++++ b/entry.c
+@@ -191,34 +191,21 @@ static int write_entry(struct cache_entry *ce,
+
+ if (smudge_to_file) {
+ close(fd);
+- if (convert_to_working_tree_filter_to_file(ce->name, path, new, size)) {
++ fd = convert_to_working_tree_filter_to_file(ce->name, path, new, size);
++ if (fd >= 0) {
+ free(new);
+- /*
+- * The smudgeToFile filter may have replaced
+- * or deleted the file; reopen it to make
+- * sure that the file exists.
+- */
+- fd = open(path, O_RDONLY);
+- if (fd < 0)
+- return error_errno("unable to create file %s", path);
+ if (!to_tempfile)
+ fstat_done = fstat_output(fd, state, &st);
+ close(fd);
+ }
+ else {
+- /*
+- * The failing smudgeToFile filter may have
+- * deleted or replaced the file; delete
+- * the file and re-open for recovery write.
+- */
+- unlink(path);
++ /* Fall through to normal write below. */
++ smudge_to_file = 0;
+ fd = open_output_fd(path, ce, to_tempfile);
+ if (fd < 0) {
+ free(new);
+ return error_errno("unable to create file %s", path);
+ }
+- /* Fall through to normal write below. */
+- smudge_to_file = 0;
+ }
+ }
+
+diff --git a/t/t0021-conversion.sh b/t/t0021-conversion.sh
+index 0efad9b..42b28aa 100755
+--- a/t/t0021-conversion.sh
++++ b/t/t0021-conversion.sh
+@@ -334,6 +334,19 @@ test_expect_success 'recovery from failure of smudgeToFile filter that deletes t
+ test_cmp test fstest.t
+ '
+
++test_expect_success 'smudgeToFile filter is used by git am' '
++ test_config filter.rot13.smudgeToFile ./rot13-to-file.sh &&
++
++ git commit fstest.t -m "added fstest.t" &&
++ git format-patch HEAD^ --stdout >fstest.patch &&
++ git reset --hard HEAD^ &&
++ git am fstest.patch &&
++
++ test -e rot13-to-file.ran &&
++ rm -f rot13-to-file.ran &&
++ test_cmp test fstest.t
++'
++
+ test_expect_success 'cleanFromFile filter is not used when clean filter is not configured' '
+ test_config filter.noclean.smudge ./rot13.sh &&
+ test_config filter.noclean.cleanFromFile ./rot13-from-file.sh &&
+--
+2.8.1
+
+--
+To unsubscribe from this list: send the line "unsubscribe git" in
+the body of a message to majordomo@vger.kernel.org
+More majordomo info at http://vger.kernel.org/majordomo-info.html
+
+From mairix@mairix Mon Jan 1 12:34:56 1970
+X-source-folder: /home/joey/mail/.git/annex/objects/0k/FJ/SHA256E-s162778--6d2a0fc6afd1077eccdca2e92f314c00ef70a06273242429035e0ce52cce2e13.gz/SHA256E-s162778--6d2a0fc6afd1077eccdca2e92f314c00ef70a06273242429035e0ce52cce2e13.gz
+Return-Path: <git-owner@vger.kernel.org>
+X-Original-To: joey@kitenet.net
+Delivered-To: joey@kitenet.net
+Received: from vger.kernel.org (vger.kernel.org [209.132.180.67])
+ by kitenet.net (Postfix) with ESMTP id C58671C677
+ for <joey@kitenet.net>; Mon, 11 Jul 2016 18:47:18 -0400 (EDT)
+Authentication-Results: kitenet.net;
+ dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=joeyh.name header.i=@joeyh.name header.b=VKChWnWm;
+ dkim-atps=neutral
+Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
+ id S1752615AbcGKWrR (ORCPT <rfc822;joey@kitenet.net>);
+ Mon, 11 Jul 2016 18:47:17 -0400
+Received: from kitenet.net ([66.228.36.95]:32778 "EHLO kitenet.net"
+ rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP
+ id S1752603AbcGKWrO (ORCPT <rfc822;git@vger.kernel.org>);
+ Mon, 11 Jul 2016 18:47:14 -0400
+X-Question: 42
+DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=joeyh.name; s=mail;
+ t=1468277198; bh=e7pd1+fJ7kJE3zbr6Uuf5rQkwlF5ZqMSMu31as4SNuw=;
+ h=From:To:Cc:Subject:Date:In-Reply-To:References:From;
+ b=VKChWnWmsu86vvKjaYknMgW76bozccToHJNzUBKXMXKt57vEc+PiDgxiMNKvA57iJ
+ /S26T8zVRqNUsPdzXsLMoCVA+T2EPLwSv/dLjBIwrTT+yveUiaH2Q4U9jzqDkU/GYy
+ 8iEtbg7ly/pnrWgjApNsnbMY5OogcAINTDod/flY=
+From: Joey Hess <joeyh@joeyh.name>
+To: git@vger.kernel.org
+Cc: Joey Hess <joeyh@joeyh.name>
+Subject: [PATCH v5 8/8] use smudgeToFile filter in recursive merge
+Date: Mon, 11 Jul 2016 18:45:12 -0400
+Message-Id: <1468277112-9909-9-git-send-email-joeyh@joeyh.name>
+X-Mailer: git-send-email 2.8.1
+In-Reply-To: <1468277112-9909-1-git-send-email-joeyh@joeyh.name>
+References: <1468277112-9909-1-git-send-email-joeyh@joeyh.name>
+X-Spam-Status: No, score=-8.2 required=10.0 tests=BAYES_00,DKIM_SIGNED,
+ HEADER_FROM_DIFFERENT_DOMAINS,RCVD_IN_DNSWL_HI,RP_MATCHES_RCVD,T_DKIM_INVALID,
+ URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.1
+X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on kite.kitenet.net
+Sender: git-owner@vger.kernel.org
+Precedence: bulk
+List-ID: <git.vger.kernel.org>
+X-Mailing-List: git@vger.kernel.org
+Content-Length: 3928
+Lines: 134
+
+Recursive merge updates the work tree and so should use the smudgeToFile
+filter.
+
+At this point, smudgeToFile is run by everything that updates work
+tree files.
+
+Signed-off-by: Joey Hess <joeyh@joeyh.name>
+---
+ merge-recursive.c | 53 ++++++++++++++++++++++++++++++++++++++++-----------
+ t/t0021-conversion.sh | 16 +++++++++++++++-
+ 2 files changed, 57 insertions(+), 12 deletions(-)
+
+diff --git a/merge-recursive.c b/merge-recursive.c
+index a4a1195..5fe3f50 100644
+--- a/merge-recursive.c
++++ b/merge-recursive.c
+@@ -758,6 +758,7 @@ static void update_file_flags(struct merge_options *o,
+ enum object_type type;
+ void *buf;
+ unsigned long size;
++ int isreg;
+
+ if (S_ISGITLINK(mode)) {
+ /*
+@@ -774,22 +775,16 @@ static void update_file_flags(struct merge_options *o,
+ die(_("cannot read object %s '%s'"), oid_to_hex(oid), path);
+ if (type != OBJ_BLOB)
+ die(_("blob expected for %s '%s'"), oid_to_hex(oid), path);
+- if (S_ISREG(mode)) {
+- struct strbuf strbuf = STRBUF_INIT;
+- if (convert_to_working_tree(path, buf, size, &strbuf)) {
+- free(buf);
+- size = strbuf.len;
+- buf = strbuf_detach(&strbuf, NULL);
+- }
+- }
+
+ if (make_room_for_path(o, path) < 0) {
+ update_wd = 0;
+ free(buf);
+ goto update_index;
+ }
+- if (S_ISREG(mode) || (!has_symlinks && S_ISLNK(mode))) {
++ isreg = S_ISREG(mode);
++ if (isreg || (!has_symlinks && S_ISLNK(mode))) {
+ int fd;
++ int smudge_to_file;
+ if (mode & 0100)
+ mode = 0777;
+ else
+@@ -797,8 +792,44 @@ static void update_file_flags(struct merge_options *o,
+ fd = open(path, O_WRONLY | O_TRUNC | O_CREAT, mode);
+ if (fd < 0)
+ die_errno(_("failed to open '%s'"), path);
+- write_in_full(fd, buf, size);
+- close(fd);
++
++ smudge_to_file = can_smudge_to_file(path);
++ if (smudge_to_file) {
++ close(fd);
++ fd = convert_to_working_tree_filter_to_file(path, path, buf, size);
++ if (fd < 0) {
++ /*
++ * smudgeToFile filter failed;
++ * continue with regular file
++ * creation.
++ */
++ smudge_to_file = 0;
++ fd = open(path, O_WRONLY | O_TRUNC | O_CREAT, mode);
++ if (fd < 0)
++ die_errno(_("failed to open '%s'"), path);
++ }
++ else {
++ close(fd);
++ }
++ }
++
++ /*
++ * Not an else of above if (smudge_to_file) because
++ * the smudgeToFile filter may fail and in that case
++ * this is run to recover.
++ */
++ if (!smudge_to_file) {
++ if (isreg) {
++ struct strbuf strbuf = STRBUF_INIT;
++ if (convert_to_working_tree(path, buf, size, &strbuf)) {
++ free(buf);
++ size = strbuf.len;
++ buf = strbuf_detach(&strbuf, NULL);
++ }
++ }
++ write_in_full(fd, buf, size);
++ close(fd);
++ }
+ } else if (S_ISLNK(mode)) {
+ char *lnk = xmemdupz(buf, size);
+ safe_create_leading_directories_const(path);
+diff --git a/t/t0021-conversion.sh b/t/t0021-conversion.sh
+index 42b28aa..64b2b8f 100755
+--- a/t/t0021-conversion.sh
++++ b/t/t0021-conversion.sh
+@@ -334,10 +334,24 @@ test_expect_success 'recovery from failure of smudgeToFile filter that deletes t
+ test_cmp test fstest.t
+ '
+
++test_expect_success 'smudgeToFile filter is used in merge' '
++ test_config filter.rot13.smudgeToFile ./rot13-to-file.sh &&
++
++ git commit -m "added fstest.t" fstest.t &&
++ git checkout -b old &&
++ git reset --hard HEAD^ &&
++ git merge master &&
++ git checkout master &&
++
++ test -e rot13-to-file.ran &&
++ rm -f rot13-to-file.ran &&
++
++ test_cmp test fstest.t
++'
++
+ test_expect_success 'smudgeToFile filter is used by git am' '
+ test_config filter.rot13.smudgeToFile ./rot13-to-file.sh &&
+
+- git commit fstest.t -m "added fstest.t" &&
+ git format-patch HEAD^ --stdout >fstest.patch &&
+ git reset --hard HEAD^ &&
+ git am fstest.patch &&
+--
+2.8.1
+
+--
+To unsubscribe from this list: send the line "unsubscribe git" in
+the body of a message to majordomo@vger.kernel.org
+More majordomo info at http://vger.kernel.org/majordomo-info.html
+
generated by cgit v1.2.3 (git 2.39.1) at 2024-12-25 06:28:35 +0000