add tip about metadata driven views (and more flexible view filtering)

While writing this documentation, I realized that there needed to be a way
to stay in a view like tag=* while adding a filter like tag=work that
applies to the same field.

So, there are really two ways a view can be refined. It can have a new
"field=explicitvalue" filter added to it, which does not change the
"shape" of the view, but narrows the files it shows.
Or, it can have a new view added, which adds another level of
subdirectories.

So, added a vfilter command, which takes explicit values to add to the
filter, and rejects changes that would change the shape of the view.

And, made vadd only accept changes that change the shape of the view.

And, changed the View data type slightly; now components that can match
multiple metadata values can be visible, or not visible.

This commit was sponsored by Stelian Iancu.

9 files changed, 201 insertions, 53 deletions

diff --git a/Annex/View.hs b/Annex/View.hs
index abf8f073e..78b4da589 100644
--- a/Annex/View.hs
+++ b/Annex/View.hs
@@ -44,11 +44,7 @@ import Text.Regex.TDFA.String
 import Text.Regex
 #endif
 
-
-data ViewChange = Unchanged | Narrowing | Widening
-	deriving (Ord, Eq, Show)
-
-{- Each multivalued ViewFilter in a view results in another level of
+{- Each visible ViewFilter in a view results in another level of
  - subdirectory nesting. When a file matches multiple ways, it will appear
  - in multiple subdirectories. This means there is a bit of an exponential
  - blowup with a single file appearing in a crazy number of places!
@@ -60,16 +56,38 @@ viewTooLarge :: View -> Bool
 viewTooLarge view = visibleViewSize view > 5
 
 visibleViewSize :: View -> Int
-visibleViewSize = length . filter (multiValue . viewFilter) . viewComponents
+visibleViewSize = length . filter viewVisible . viewComponents
 
-{- Updates a view, adding a new field to filter on (Narrowing), 
- - or allowing a new value in an existing field (Widening). -}
-refineView :: View -> MetaField -> String -> (View, ViewChange)
-refineView view field wanted
+data ViewChange = Unchanged | Narrowing | Widening
+	deriving (Ord, Eq, Show)
+
+{- Updates a view, adding new fields to filter on (Narrowing), 
+ - or allowing new values in an existing field (Widening). -}
+refineView :: View -> [(MetaField, String)] -> (View, ViewChange)
+refineView = go Unchanged
+  where
+  	go c v [] = (v, c)
+	go c v ((f, s):rest) =
+		let (v', c') = refineView' v f s
+		in go (max c c') v' rest
+
+{- Adds an additional filter to a view. This can only result in narrowing
+ - the view. Multivalued filters are added in non-visible form. -}
+filterView :: View -> [(MetaField, String)] -> View
+filterView v vs = v { viewComponents = viewComponents f' ++ viewComponents v}
+  where
+	f = fst $ refineView (v {viewComponents = []}) vs
+	f' = f { viewComponents = map toinvisible (viewComponents f) }
+	toinvisible c = c { viewVisible = False }
+
+refineView' :: View -> MetaField -> String -> (View, ViewChange)
+refineView' view field wanted
 	| field `elem` (map viewField components) = 
 		let (components', viewchanges) = runWriter $ mapM updatefield components
 		in (view { viewComponents = components' }, maximum viewchanges)
-	| otherwise = let view' = view { viewComponents = ViewComponent field viewfilter : components }
+	| otherwise = 
+		let component = ViewComponent field viewfilter (multiValue viewfilter)
+		    view' = view { viewComponents = component : components }
 		in if viewTooLarge view'
 			then error $ "View is too large (" ++ show (visibleViewSize view') ++ " levels of subdirectories)"
 			else (view', Narrowing)
@@ -173,8 +191,8 @@ fileViews view =
 					else map (</> mkfileview file) paths
   where
 	visible = map (fromJust . snd) .
-		filter (multiValue . fst) .
-		zip (map viewFilter (viewComponents view))
+		filter (viewVisible . fst) .
+		zip (viewComponents view)
 
 {- Checks if metadata matches a ViewComponent filter, and if so
  - returns the value, or values that match. Self-memoizing on ViewComponent. -}
@@ -255,7 +273,7 @@ pathProduct (l:ls) = foldl combinel l ls
 fromView :: View -> FileView -> MetaData
 fromView view f = foldr (uncurry updateMetaData) newMetaData (zip fields values)
   where
-	visible = filter (multiValue . viewFilter) (viewComponents view)
+	visible = filter viewVisible (viewComponents view)
 	fields = map viewField visible
 	paths = splitDirectories $ dropFileName f
 	values = map fromViewPath paths
@@ -263,15 +281,15 @@ fromView view f = foldr (uncurry updateMetaData) newMetaData (zip fields values)
 {- Constructing a view that will match arbitrary metadata, and applying
  - it to a file yields a set of FileViews which all contain the same
  - MetaFields that were present in the input metadata
- - (excluding fields that are not multivalued). -}
-prop_view_roundtrips :: FilePath -> MetaData -> Bool
-prop_view_roundtrips f metadata = null f || viewTooLarge view ||
+ - (excluding fields that are not visible). -}
+prop_view_roundtrips :: FilePath -> MetaData -> Bool -> Bool
+prop_view_roundtrips f metadata visible = null f || viewTooLarge view ||
 	all hasfields (fileViews view fileViewFromReference f metadata)
   where
 	view = View (Git.Ref "master") $
-		map (\(mf, mv) -> ViewComponent mf (FilterValues $ S.filter (not . null . fromMetaValue) mv))
+		map (\(mf, mv) -> ViewComponent mf (FilterValues $ S.filter (not . null . fromMetaValue) mv) visible)
 			(fromMetaData metadata)
-	visiblefields = sort (map viewField $ filter (multiValue . viewFilter) (viewComponents view))
+	visiblefields = sort (map viewField $ filter viewVisible (viewComponents view))
 	hasfields fv = sort (map fst (fromMetaData (fromView view fv))) == visiblefields
 
 {- Applies a view to the currently checked out branch, generating a new
@@ -282,7 +300,8 @@ applyView view = applyView' fileViewFromReference view
 
 {- Generates a new branch for a View, which must be a more narrow
  - version of the View originally used to generate the currently
- - checked out branch.
+ - checked out branch. That is, it must match a subset of the files
+ - in view, not any others.
  -}
 narrowView :: View -> Annex Git.Branch
 narrowView = applyView' fileViewReuse
@@ -405,3 +424,6 @@ withIndex :: Annex a -> Annex a
 withIndex a = do
 	f <- fromRepo gitAnnexViewIndex
 	withIndexFile f a
+
+withCurrentView :: (View -> Annex a) -> Annex a
+withCurrentView a = maybe (error "Not in a view.") a =<< currentView
diff --git a/CmdLine/GitAnnex.hs b/CmdLine/GitAnnex.hs
index c8325872d..3604681f9 100644
--- a/CmdLine/GitAnnex.hs
+++ b/CmdLine/GitAnnex.hs
@@ -29,6 +29,7 @@ import qualified Command.ReKey
 import qualified Command.MetaData
 import qualified Command.View
 import qualified Command.VAdd
+import qualified Command.VFilter
 import qualified Command.VPop
 import qualified Command.VCycle
 import qualified Command.Reinject
@@ -142,6 +143,7 @@ cmds = concat
 	, Command.MetaData.def
 	, Command.View.def
 	, Command.VAdd.def
+	, Command.VFilter.def
 	, Command.VPop.def
 	, Command.VCycle.def
 	, Command.Fix.def
diff --git a/Command/VAdd.hs b/Command/VAdd.hs
index d7dee9fb7..3dc1fd4cf 100644
--- a/Command/VAdd.hs
+++ b/Command/VAdd.hs
@@ -10,12 +10,11 @@ module Command.VAdd where
 import Common.Annex
 import Command
 import Annex.View
-import Logs.View
 import Command.View (paramView, parseViewParam, checkoutViewBranch)
 
 def :: [Command]
 def = [notBareRepo $ notDirect $
-	command "vadd" paramView seek SectionMetaData "refine current view"]
+	command "vadd" paramView seek SectionMetaData "add subdirs to current view"]
 
 seek :: CommandSeek
 seek = withWords start
@@ -23,20 +22,15 @@ seek = withWords start
 start :: [String] -> CommandStart
 start params = do
 	showStart "vadd" ""
-	go =<< currentView
-  where
-	go Nothing = error "Not in a view."
-	go (Just view) = do
-		let (view', change) = calc view Unchanged (reverse params)
+	withCurrentView $ \view -> do
+		let (view', change) = refineView view $
+			map parseViewParam $ reverse params
 		case change of
 			Unchanged -> do
 				showNote "unchanged"
 				next $ next $ return True
-			Narrowing -> next $ next $
-				checkoutViewBranch view' narrowView
+			Narrowing -> next $ next $ do
+				if visibleViewSize view' == visibleViewSize view
+					then error "That would not add an additional level of directory structure to the view. To filter the view, use vfilter instead of vadd."
+					else checkoutViewBranch view' narrowView
 			Widening -> error "Widening view to match more files is not currently supported."
-
-	calc v c [] = (v, c)
-	calc v c (p:ps) =
-		let (v', c') = uncurry (refineView v) (parseViewParam p)
-		in calc v' (max c c') ps
diff --git a/Command/View.hs b/Command/View.hs
index 7cc908436..17e136f7b 100644
--- a/Command/View.hs
+++ b/Command/View.hs
@@ -53,12 +53,9 @@ parseViewParam s = case separate (== '=') s of
 mkView :: [String] -> Annex View
 mkView params = do
 	v <- View <$> viewbranch <*> pure []
-	return $ calc v $ reverse params
+	return $ fst $ refineView v $
+		map parseViewParam $ reverse params
   where
-	calc v [] = v
-	calc v (p:ps) =
-		let (v', _) = uncurry (refineView v) (parseViewParam p)
-		in calc v' ps
 	viewbranch = fromMaybe (error "not on any branch!")
 		<$> inRepo Git.Branch.current
 
diff --git a/Logs/View.hs b/Logs/View.hs
index 2a26cfa17..15d472bc6 100644
--- a/Logs/View.hs
+++ b/Logs/View.hs
@@ -71,9 +71,9 @@ branchView view
   where
 	name = intercalate ";" $ map branchcomp (viewComponents view)
 	branchcomp c
-		| multiValue (viewFilter c) = branchcomp' c
+		| viewVisible c = branchcomp' c
 		| otherwise = "(" ++ branchcomp' c ++ ")"
-	branchcomp' (ViewComponent metafield viewfilter)
+	branchcomp' (ViewComponent metafield viewfilter _)
 		| metafield == tagMetaField = branchvals viewfilter
 		| otherwise = concat
 			[ forcelegal (fromMetaField metafield)
diff --git a/Types/View.hs b/Types/View.hs
index 7ef44db11..04b002879 100644
--- a/Types/View.hs
+++ b/Types/View.hs
@@ -28,11 +28,12 @@ instance Arbitrary View where
 data ViewComponent = ViewComponent
 	{ viewField :: MetaField
 	, viewFilter :: ViewFilter
+	, viewVisible :: Bool
 	}
 	deriving (Eq, Read, Show)
 
 instance Arbitrary ViewComponent where
-	arbitrary = ViewComponent <$> arbitrary <*> arbitrary
+	arbitrary = ViewComponent <$> arbitrary <*> arbitrary <*> arbitrary
 
 {- Only files with metadata matching the view are displayed. -}
 type FileView = FilePath
diff --git a/debian/changelog b/debian/changelog
index 36ed0b4d5..c71ef515c 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -7,9 +7,9 @@ git-annex (5.20140211) UNRELEASED; urgency=medium
     to limit them to acting on files that have particular metadata.
   * view: New command that creates and checks out a branch that provides
     a structured view of selected metadata.
-  * vadd, vpop, vcycle: New commands for operating within views.
-  * pre-commit: Update metadata when committing changes to annexed files
-    within a view.
+  * vfilter, vadd, vpop, vcycle: New commands for operating within views.
+  * pre-commit: Update metadata when committing changes to locations
+    of annexed files within a view.
   * Add progress display for transfers to/from external special remotes.
   * Windows webapp: Can set up box.com, Amazon S3, and rsync.net remotes
   * Windows webapp: Can create repos on removable drives.
diff --git a/doc/git-annex.mdwn b/doc/git-annex.mdwn
index 23002cb9a..e7183dc87 100644
--- a/doc/git-annex.mdwn
+++ b/doc/git-annex.mdwn
@@ -722,10 +722,10 @@ subdirectories).
   shown in the view.
   
   Multiple values for a metadata field can be specified, either by using
-  a glob (field="\*") or by listing each wanted value. The resulting view
+  a glob (`field="*"`) or by listing each wanted value. The resulting view
   will put files in subdirectories according to the value of their fields.
-  
-  Once within a view, you can make additional subdirectories, and
+
+  Once within a view, you can make additional directories, and
   copy or move files into them. When you commit, the metadata will
   be updated to correspond to your changes.
 
@@ -736,16 +736,28 @@ subdirectories).
 
   The optional number tells how many views to pop.
 
-* `vadd [field=value ...] [tag ...]`
+* `vfilter [field=value ...] [tag ...]`
+
+  Filters the current view to only the files that have the
+  specified values and tags.
+
+* `vadd [field=glob ...]`
 
-  Refines the currently checked out view branch, adding additional fields
-  or tags.
+  Changes the current view, adding an additional level of directories
+  to categorize the files.
+
+  For example, when the view is by author/tag, `vadd year=*` will 
+  change it to year/author/tag. 
+  
+  So will `vadd year=2014 year=2013`, but limiting the years in view
+  to only those two.
 
 * `vcycle`
 
   When a view involves nested subdirectories, this cycles the order.
-  For example, when the view has date/author/tag, vcycle will switch
-  it to author/tag/date.
+
+  For example, when the view is by year/author/tag, `vcycle` will switch
+  it to author/tag/year.
 
 # UTILITY COMMANDS
 
diff --git a/doc/tips/metadata_driven_views.mdwn b/doc/tips/metadata_driven_views.mdwn
new file mode 100644
index 000000000..85b9d9cbd
--- /dev/null
+++ b/doc/tips/metadata_driven_views.mdwn
@@ -0,0 +1,120 @@
+git-annex now has support for storing arbitrary metadata about annexed
+files. For example, this can be used to tag files, to record the author 
+of a file, etc. The metadata is synced around between repositories with
+the other information git-annex keeps track of.
+
+One nice way to use the metadata is through **views**. You can ask
+git-annex to create a view of files in the currently checked out branch
+that have certian metadata. Once you're in a view, you can move and copy
+files to adjust their metadata further. Rather than the traditional
+hierarchical directory structure, views are dynamic; you can easily
+refine or reorder a view.
+
+Let's get started by setting some tags on files. No views yet, just some
+metadata:
+
+	# git annex metadata --tag todo work/2014/*
+	# git annex metadata --untag todo work/2014/done/*
+	# git annex metadata --tag urgent work/2014/presentation_for_tomorrow.odt
+	# git annex metadata --tag done work/2013/* work/2014/done/*
+	# git annex metadata --tag work work
+	# git annex metadata --tag video videos
+	# git annex metadata --tag work videos/operating_heavy_machinery.mov
+	# git annex metadata --tag done videos/old
+	# git annex metadata --tag new videos/lotsofcats.ogv
+	# git annex metadata --tag sound podcasts
+	# git annex metadata --tag done podcasts/old
+	# git annex metadata --tag new podcasts/recent
+
+So, you had a bunch of different kinds of files sorted into a directory
+structure. But that didn't really reflect how you approach the files.
+Adding some tags lets you categorize the files in different ways.
+
+Ok, metadata is in place, but how to use it? Time to change views!
+
+	# git annex view tag=*
+	view  (searching...)
+
+	Switched to branch 'views/_'
+	ok
+
+This searched for all files with any tag, and created a new git branch
+that sorts the files according to their tags.
+
+	# tree -d
+	work
+	todo
+	urgent
+	done
+	new
+	video
+	sound
+
+Notice that a single file may appear in multiple directories
+depending on its tags. For example, `lotsofcats.ogv` is in
+both `new/` and `video/`.
+
+Ah, but you're at work now, and don't want to be distracted by cat videos.
+Time to filter the view:
+
+	# git annex vfilter tag=work
+	vfilter
+	Switched to branch 'views/(work)/_'
+	ok
+
+Now only the work files are in the view, and they're otherwise categorized
+according to their other tags. So you can check the `urgent/` directory
+to see what's next, and look in `todo/` for other work related files.
+
+Now that you're in a tag based view, you can move files around between the
+directories, and when you commit your changes to git, their tags will be
+updated.
+
+	# git mv urgent/presentation_for_tomorrow_{work;2014}.odt ../done
+	# git commit -m "a good day's work"
+	metadata tag-=urgent
+	metadata tag+=done
+
+You can return to a previous view by running `git annex vpop`. If you pop
+all the way out of all views, you'll be back on the regular git branch you
+originally started from. You can also use `git checkout` to switch between
+views and other branches.
+
+Beyond simple tags, you can add whatever kinds of metadata you like, and
+use that metadata in more elaborate views. For example, let's add a year
+field.
+
+	# git checkout master
+	# git annex metadata --set year=2014 work/2014
+	# git annex metadata --set year=2013 work/2013
+	# git annex view year=* tag=*
+
+Now you're in a view with two levels of directories, first by year and then
+by tag.
+
+	# tree -d
+	2014
+	  |-- work
+          |-- todo
+	  |-- urgent
+          `-- done
+	2013
+	  |-- work
+	  `-- done
+
+Oh, did you want it the other way around? Easy!
+
+	# git annex vcycle
+	# tree -d
+	work
+	  |-- 2014
+	  `-- 2013
+	todo
+	  `-- 2014
+	urgent
+	  `-- 2014
+	done
+	  |-- 2014
+	  `-- 2013
+
+This has probably only scratched the surface of what you can do with views.