documentation overhaul

11 files changed, 203 insertions, 97 deletions

diff --git a/Makefile b/Makefile
index 2d6b8c9..0446f3a 100644
--- a/Makefile
+++ b/Makefile
@@ -21,9 +21,10 @@ install:
 	install -d $(DESTDIR)/usr/share/uzbl/examples
 	install -D -m755 uzbl $(DESTDIR)/usr/bin/uzbl
 	install -D -m755 uzblctrl $(DESTDIR)/usr/bin/uzblctrl
+	cp -ax docs     $(DESTDIR)/usr/share/uzbl/docs
 	cp -ax examples $(DESTDIR)/usr/share/uzbl/
-	install -D -m644 CHECKLIST $(DESTDIR)/usr/share/uzbl/docs
-	install -D -m644 README $(DESTDIR)/usr/share/uzbl/docs
+	install -D -m644 AUTHORS $(DESTDIR)/usr/share/uzbl/docs
+	install -D -m644 README  $(DESTDIR)/usr/share/uzbl/docs
 
 uninstall:
 	rm -rf $(DESTDIR)/usr/bin/uzbl
diff --git a/README b/README
index 60ed089..27e4218 100644
--- a/README
+++ b/README
@@ -1,104 +1,90 @@
 THIS PROJECT IS NOT FOR:
 - people want a browser that does everything
 - people who want a browser with things like a built-in bookmark manager, address bar, forward/back buttons, ...
-
+- people who expect something that works by default.  You'll need to read configs and write/edit scripts
 
 
 TO NEW PEOPLE:
- - please read the README in /usr/share/uzbl/docs
+ - please read the documentation in /usr/share/uzbl/docs
  - invoke uzbl --help
  - to get you started: uzbl --uri 'http://www.archlinux.org' --config /usr/share/uzbl/examples/configs/sampleconfig
  - study the sample config, have a look at all the bindings, and note how you can call the scripts to load new url from history and the bookmarks file
  - note that there is no url bar. all url editing is supposed to happen _outside_ of uzbl.
-   for now, you can use the load_from_* dmenu based scripts to pick a url or type a new one (we should have a dmenu-like that functions as a better editor) or write commands into the fifo (see /usr/share/uzbl/docs/CHECKLIST)
- - If you have questions, you are likely to find answers in the FAQ
-
-CURRENT STATE:
- alpha / prototype
+   for now, you can use the load_from_* dmenu based scripts to pick a url or type a new one or write commands into the fifo (see /usr/share/uzbl/docs/CHECKLIST)
+ - If you have questions, you are likely to find answers in the FAQ or in the other documentation.
 
 
-- Uzbl.
+INTRODUCTION
   In my opinion, any program can only be really useful if it complies to the unix philosophy.
-  Web browsers are frequent violators of this principle.  Time to change that!
-
-Right now uzbl is in a very early state but here are some ideas I would like to (not) implement
-
-- each instance of uzbl renders 1 page (eg it's a small wrapper around webkit), no tabbing, tab previews, or speed dial things. we have window managers for that.
-  -> well actually, there is lots of dicussion about this, i'll probably implement a basic form of tabbing.
-- simple ini config file ("profile") for keyboard, network,.. settings
-- implement some basic keyboard shortcuts for going up, down, refresh etc. preferably vim-like command style.
+  Web browsers are frequent violators of this principle.
+  -> They build in way too much things into the browser, dramatically decreasing the options to do things the way you want. 
+  -> They store things in way too fancy formats (xml, rdf, sqlite, ... ) which are hard to store under version control, reuse in other scripts, ...
+
+Time to change that!
+
+  Here are the general ideas:
+- each instance of uzbl renders 1 page (eg it's a small wrapper around webkit), no tabbing, tab previews, or speed dial things.
+  For "multiple instances management" use your window managers, or scripts. 
+  This way you can get something much more useful than tabbing (see rationale in docs)
+- very simple, plaintext , changeable at runtime configuration
+- various interfaces for (programmatic) interaction with uzbl (see below)
+- customizable keyboard shortcuts in vim or emacs style (whatever user wants)
+- "outsource" logic that is not browsing to external scripts under the users control:
+  - managing bookmarks
+  - loading a url from bookmarks, history,..  Editing the curent url
+  - control cookies
+  - handling of downloads, history logging, etc.
+  - management of cache.
+  - password management
+  Leverage the power of utilities such as grep, awk, dmenu, zenity, wget, gnupg (password file) etc.
 - listen to signals and do useful stuff when triggered.
-- open up a socket file/fifo/.. so we can easily control each instance by writing things like 'uri <foo>' to /tmp/uzbl-windowid
-- MAYBE (if needed): 1 control application called uzblctrl or something. use this to modify the behavior of a uzbl instance (change url, refresh).  use xdotool to get the window with focus.  eg uzblctrl -win <id> -url <http://>.
-  use xbindkeys to bind keys to call uzblctrl.
-- no bookmark management builtin.  make your own solution.  for pulling a bookmark a plaintxt-based program using dmenu would work great here. combine with uzbltcrl and xbindkeys.
-  uzblctrl should support an option to query the current page so you can script something to add to your bookmarks.  use zenity or something to add tags.
-- history: log 'Y-m-d H:M:S <url>' entries to a plaintext file. you can then use dmenu or whatever to select an entry and pipe the output to uzbl's fifo.
-- no ad blocking built in (I think). 
+- no ad blocking built in
   alternatives:
-  ->  /etc/hosts (not very good cause you need root and it affects the whole system)-> uzblctrl would need to support an option to list all images on a page, so you can easily pick the links to ads to add them to your /etc/hosts. (dmenu can again be great here to automate this)
   -> privoxy looks cool and perfectly demonstrates the unix philosphy.
   -> same for http://bfilter.sourceforge.net
-- no download manager. allow user to pick wget/curl/a custom script/... 
-- no build in command interpreters like ubiquity.  uzbl should be accessible and you should use a shell or similar.
-- no "clear cookies/cache/..." menu items. rm ~/$XDG_{DATA,CACHE}_DIR/uzbl/{cache,cookies}/* 
+  -> /etc/hosts (not very good cause you need root and it affects the whole system)-> uzblctrl would need to support an option to list all images on a page, so you can easily pick the links to ads to add them to your /etc/hosts.
 - vimperator/konqueror-like hyperlink following.
 - password management. maybe an encrypted store that unlocks with an ssh key?
-- use the XDG basedir spec for separation of config, data and cache. and state will be a subdir in the config dir (not part of the spec yet) too.
-
-WIDGET ROADMAP:
-* statusbar? (the bar you see in pretty much every gtk program at the bottom. eg firefox)
-  consumes too much space (if always visible) for the little it is used.  (+ you can put only 1 message in it at a time!)
-  -> option 1: no statusbar at all. when hovering over a link (or pressing key to preview the url without changing page) -> show url in tooltip on page.
-  -> option 2: toggle visibility of statusbar on/off when hovering over a link. since it's at the bottom I don't think it will disturb too much.
-* viewing progress/state of pageload?  most programs use statusbar for this.
-  -> option 1: titlebar can show a percentage when it's loading a new page.
-  -> option 2: toggle a statusbar everytime we start loading a new page.
-* uri bar -> yes, even though we can write stuff to the fifo, it can still be convenient to change the url manually and stuff, so a widget in uzbl itself is good.
-* tabs -> yes. you don't have to use them, but you can.
-* back/forward/.. buttons? -> no: use keyboard shortcuts.
-* searching in a page? not sure.. maybe we can abuse the statusbar for that too.
-  eg toggle it on when the user wants to search for something and then do searching in some vim-like fashion.
-  we don't need a gtk text entry widget, just a feedback display of what the current command is.
-* scrollbar? no: use keyboard shortcuts.  we should however have some info about the page length and where we are.
-  -> option 1: put a percentage in the window title
-  -> option 2: everytime you hit a key to change position, temporarily make a statusbar visible and put the percentage in the statusbar.
-  what will we do with pages who are too wide? horizontal scrolling?
-all of the above goes in 1 bar at the top of the program. there should be a key to toggle visibility of it and one to toggle visibilety + focus on the entrybar at once.
-
-input welcome!
-
-
-HISTORY FILE SIZE/PERFORMANCE
-each new pageload -> fopen(history_file, "a"), fwrite one line, fclose.
-I use utf8, so unless you use characters that are "special" (chinese etc)
-each character takes 1 byte.
-So, assume each entry is about 80 chars, you visit 100 pages per day (?), and you wonder when your history file will be 50MB big:
-(50 * 1000 * 1000 ) / ( 80 * 100 ) = 6250 days or 17 years.
-There is code to run a benchmark in the 'extra' dir.  For results & interpretation, see http://dieter.plaetinck.be/poor_mans_dmenu_benchmark
-
-CONTROL:
-- FIFO opened in /tmp/uzbl_pid
-- See config file for commands
-- Press ESC/i to toggle command/insert mode
-
-NOTES:
-- My c skills are very rusty, it will take me a while to get back up to speed
-- For more thoughts & ideas see http://bbs.archlinux.org/viewtopic.php?id=67463
-
-REPO's:
-- http://github.com/Dieterbe/uzbl
-  master -> uzbl stable branch
-  experimental -> bleeding edge stuff that may break. after QA codes gets merged into master
-- various contributors also have their clones on github (http://github.com/dusanx, http://github.com/Barrucadu/uzbl, ...).
-  They may be developing specific features, which get merged into Dieters experimental branch
+- no messing in the users $HOME: no writing of anything unless the users asks for it.  We recommend using XDG basedir spec for separation of config, data and cache. and state should be a subdir in the config dir (not part of the spec yet) too.
+
+
+CONFIGURATION / CONTROL:
+The general idea is that uzbl by default is very bare bones.  you can send it commands to update settings and perform actions, through various interfaces.
+For examples, please see the sample config(s).
+There are several interfaces to interact with uzbl:
+- uzbl --config <filename>: <filename> will be read line by line, and the commands in it will be executed.  useful to configure uzbl at startup.  If you have a file in $XDG\_CONFIG\_HOME/uzbl/config (this expands to ~/.config/uzbl/config on most systems) it will be automatically recognized
+- stdin: you can also write commands into stdin
+- interactive: you can enter commands (and bind them to shortcuts, even at runtime)
+  By default, the behaviour is modal (vi style):
+  command mode: every keystroke is interpreted to run commands
+  insert mode: keystrokes are not interpreted so you can enter text into html forms
+  Press ESC/i to toggle command/insert mode
+  But if you don't like modal interfaces, you can set always_insert_mode and configure a modkey to execute the commands. (emacs style).
+- FIFO & socket file: if enabled by setting their paths through one of the above means, you can have socket and fifo files available, which are very useful to programatically send commands to (coming from scripts etc)
+  The advantage of the fifo is you can write plaintxt commands to it, but it's half duplex only (uzbl cannot send a response to you).
+  The socket is full duplex but you need a socket-compatible wrapper such as netcat to work with it, or uzblctrl of course, an utitly we include with uzbl made especially for writing commnands to the socket (and at some point, it will be able to tell you the response too)
+
+COMMAND SYNTAX
+TODO
+
+VARIABLE REPLACEMENT
+Some of the variables are interpreted.
+- title bar: variable replacement (not yet actually)
+- user agent: variable replacement
+- statusbar: variable replacement + pango markup
+
+This means you can customize how these things appear, what's shown in them and for the statusbar you can even play with the layout.
+For examples, see the example config.
+For a list of possible variables, see uzbl.h
+Forr more info about the markup format see http://library.gnome.org/devel/pango/stable/PangoMarkupFormat.html
 
 
 EXTERNAL SCRIPTS
 You can use external scripts with uzbl the following ways:
 1) let uzbl call them. these scripts are called handlers in the uzbl config. used for handling logging history, handling a new download,.. 
 2) call them yourself from inside uzbl.  you can bind keys for this. examples: add new bookmark, load new url,..
-3) if you want to call scripts that have no option, you can trigger them with something like xbindkeys. example: ? (we try to keep all possibilities inside option 1/2)
+3) You could also use xbindkeys or your WM config to trigger scripts if uzbl does not have focus
+Have a look at the sample configs and scripts!
 
 Scripts that are called by uzbl are passed the following arguments:
 $1 uzbl-config-file
@@ -123,21 +109,11 @@ The script specific arguments are this:
   $10 request address path
   $11 cookie (only with PUT requests)
 
-KNOWN BUGS
+
+BUGS
+known bugs:
 - Segfaults when using zoom commands (happens when max zoom already reached?).
-- Something in the FIFO code causes CPU usage to jump.
 
 Report new issues @ uzbl.org/bugs
 
 
-VALGRIND PROFILING
-add this to Makefile header: CFLAGS=-g
-recompile
-valgrind --tool=callgrind ./uzbl ....
-kcachegrind callgrind.out.foo
-
-CONFIG FILE
-** Variable replacement and markup format
-See http://library.gnome.org/devel/pango/stable/PangoMarkupFormat.html for
-what you can do, markup wise.
-TODO: document possible variables, and what you can do with useragent/title/statusbar
diff --git a/CHECKLIST b/docs/CHECKLIST
index b2c7065..6c4a4f7 100644
--- a/CHECKLIST
+++ b/docs/CHECKLIST
@@ -1,3 +1,6 @@
+THIS FILE IS DEPRECATED.
+TODO: make sure that all this gets merged into the documentation
+
 This file contains all things that are/should be working.
 When you have been working on something, use this list to verify that you did not cause any regressions (things that worked before, but got broken).
 Also, when you finish a new feature, add it to this list so that everyone knows about it, can play with it, and can test it.
diff --git a/docs/CONTRIBUTING b/docs/CONTRIBUTING
new file mode 100644
index 0000000..8aba17e
--- /dev/null
+++ b/docs/CONTRIBUTING
@@ -0,0 +1,24 @@
+Users
+
+Right now, the best way to contribute to Uzbl is to use it, hang around in
+our IRC channel, and tell us when things break. If you're feeling more
+adventerous, you can use one of the development branches and give bug
+reports and suggestions straight to the developer in charge of that, so the
+same problems don't occur when they get merged into the master branch. Have
+a look at the CHECKLIST file to see all the stuff that is supposed to work.
+Play around with the configs and scripts and see if you can improve things.
+
+Developers
+
+If you don't feel like just sending bug reports, by all means dive into the
+code and clone the code to start hacking. (github makes this really easy
+with their "fork" concept).  But it's usually a good thing to tell us first
+what you want to do, to avoid unneeded or duplicate work.
+
+
+
+VALGRIND PROFILING
+add this to Makefile header: CFLAGS=-g
+recompile
+valgrind --tool=callgrind ./uzbl ....
+kcachegrind callgrind.out.foo
diff --git a/FAQ b/docs/FAQ
index e3652f9..e3652f9 100644
--- a/FAQ
+++ b/docs/FAQ
diff --git a/INSTALL b/docs/INSTALL
index c228f5c..270c343 100644
--- a/INSTALL
+++ b/docs/INSTALL
@@ -9,7 +9,7 @@ branch you want.
 From source
 -----------
 	$ git clone git://github.com/Dieterbe/uzbl.git
-	[ $ git checkout master/experimental ] # master == fairly stable.  experimental is more bleeding edge
+	[ $ git checkout master/experimental ] # optional. see below
 	$ cd uzbl
 	$ make
 	$ sudo make install
@@ -30,7 +30,7 @@ Optional/Recommended
 The following tools are quite useful, and some of them are used in the
 sample scripts:
 
-* dmenu
+* dmenu (with vertical patch
 * zenity
 * bash 
 
@@ -50,4 +50,14 @@ You're free to store your personal stuff where you want, but we think the [XDG b
 is very useful for keeping a clean home directory, so we recommend:
 
 * $XDG\_CONFIG\_HOME/uzbl/config* (~/.config/uzbl/config on most systems): config file
-* $XDG\_DATA\_HOME/uzbl (~/.local/share/uzbl on most systems): bookmarks file, history file. and "scripts" directory with scripts
\ No newline at end of file
+* $XDG\_DATA\_HOME/uzbl (~/.local/share/uzbl on most systems): bookmarks file, history file. and "scripts" directory with scripts
+
+Git Repo's & branches
+--------------------
+* Main official repo:
+  http://github.com/Dieterbe/uzbl
+- master -> uzbl stable branch
+- experimental -> bleeding edge stuff that may break. after QA Dieter merges into his master
+
+* Most contributors & developers also have their clones on github (http://github.com/dusanx, http://github.com/Barrucadu/uzbl, ...).
+  They may be developing specific features, which get merged into Dieters experimental branch
diff --git a/TODO b/docs/TODO
index 6e4004d..39ea849 100644
--- a/TODO
+++ b/docs/TODO
@@ -49,13 +49,15 @@
 * configure script
 * a variable that holds the page state: loading, pending, seen. this can be shown in titlebar/statusbar and used for multiple instances management
 * proxy_url is not a good var name. it's not a url.
-
-
+* regex style page searching? so you can do 'or' and 'and' things. flags like case sensitive etc.
+* can we export uri/state/tag/.. as xorg window "properties" or "attributes" or do we need to put them in titlebar and parse that?
 
 * config: check in the default place (XDG_CONFIG_HOME/..) for a config file, and if needed, do file reading (interpret line by line). not ini-based.
 * readd the --config flag to allow entering the same commands either through the file/stdin/fifo/socket
 * check for real command name, not just the first letter.
-
+* let users attach handlers to the most common events/signals in uzbl.
+* write little script to open new urls with the urxvt url thing +document.
+* add Rob/anydots trees on website
 
 SOMEDAY:
 check if we can make the settings loading less hard coded. eg( keep a list of all settings, and for each one, try to load it)
diff --git a/docs/multiple-instances-management b/docs/multiple-instances-management
new file mode 100644
index 0000000..da55002
--- /dev/null
+++ b/docs/multiple-instances-management
@@ -0,0 +1,41 @@
+"multiple instance management" or: how to manage multiple open pages/instances of uzbl.
+
+The way we do MIM in uzbl will be better then what you can do with tabs in other programs.
+Tabs are just one specific gui implementation which aids in some aspects of "multiple instances management" but not all.
+We can get the same and even better features functionality wise, without limiting ourselves to the particular implementation that tabs are.
+We use a "use-case"/tasks driven approach, unlike tabs which is a "lets do this, and then see how we can use it" approach.
+
+The approach we are implementing in uzbl is like this:
+- 1 page per instance.
+- ability to spawn new windows ("open in new window"), keybinds for having the new window focused or not.
+- each instance keeps track of it's page state (loading, loaded but not seen, loaded and seen)
+- ability to "tag" instances (inherited when forking new instances) (eg tag "work" or tag "personal") so you can keep related instances together.
+  The tag can be made visible in the title / statusbar.
+- allow user to make keybinds to focus the next/previous:
+  - window
+  - unseen/seen window
+  - unseen/seen window from the same tag
+  - unseen/seen window from tag <foo> (dmenu)
+- have a program that lists all instances (optionally filtered, or categorized by tag, state, ...) and allow user to select one from it using smart matching techniques which require a minimal amount of keystrokes. (dmenu with vertical patch is great for this)
+
+Here's an overview what do tabs really do, whether we really need it, and if can improve it?
+
+
+* tabs keep multiple open pages together when moving the window other tag/workspace
+   -> in an ideal workspace, you assign windows to a tag/workspace and keep them there.
+   -> in practice, when I was still using Firefox, I noticed I moved my "group of bundled pages" (eg a FF window) sometimes, to aid in copy pasting between browser and another app, or doing something while watching/reading a webpage.
+      This means however I only really need that specific page, not the others. -> With the suggested approach, you can easily temporarily move one window.
+* tabs keep an oversight of which pages you have open
+   -> you should "know" more or less know which pages you have open.  The gui is clutter.  It should however be possible to show the list "on demand".
+   -> a horizontal list of tabs is also not very readable and lacks useful categorisation/grouping.
+   With the suggested approach we will have each url below each other, which is far more readable, and we can use additional text or color themes to denote the state/tag/...
+* tabs aid opening new pages in the background which you will open later and provide shortcuts to go to the previous/next tab.
+   -> our approach is more powerful.
+   -> With tab based browsers you can usually configure if you want new tabs to open "immediately after the current tab", or "at the end of the list". This is much less flexible/usable then what we are building.
+
+Conclusion
+---------
+I think our method is better then tabs.  Bonus advantages:
+* single uzbl instances are simple to implement, no added clutter.
+* one crashing instance does not affect the rest
+* all this stuff is implemented outside of uzbl.  You can even decide to not use it and you won't even know the feature was ever there.  (though i guess most people will want this).
diff --git a/docs/performance b/docs/performance
new file mode 100644
index 0000000..0f9763f
--- /dev/null
+++ b/docs/performance
@@ -0,0 +1,9 @@
+The usage of external scripts causes some slowdowns. (forking new processes, running interpreted scripts,...)
+It can be interesting to keep an eye on this, though I don't expect major problems.
+
+** History file size/performance
+each new pageload -> fopen(history_file, "a"), fwrite one line, fclose.
+I use utf8, so unless you use characters that are "special" (chinese etc) each character takes 1 byte.
+So, assume each entry is about 80 chars, you visit 100 pages per day (?), and you wonder when your history file will be 50MB big:
+(50 * 1000 * 1000 ) / ( 80 * 100 ) = 6250 days or 17 years.
+There is code to run a benchmark in the 'extra' dir.  For results & interpretation, see http://dieter.plaetinck.be/poor_mans_dmenu_benchmark
diff --git a/docs/url-editing b/docs/url-editing
new file mode 100644
index 0000000..fae96c7
--- /dev/null
+++ b/docs/url-editing
@@ -0,0 +1,7 @@
+* All url changing/editing happens outside of uzbl
+
+Use cases:
+* load url from history/bookmarks (dmenu-vertical is great for this)
+* edit current url or one from history/bookmarks: dmenu-vertical is relatively good for this, though it would be better if we had:
+- minimal editing helper key shortcuts (eg to go to beginning/end of input buffer)
+- copy/paste support
diff --git a/docs/widgets b/docs/widgets
new file mode 100644
index 0000000..dc7127a
--- /dev/null
+++ b/docs/widgets
@@ -0,0 +1,33 @@
+
+* statusbar? (the bar you see in pretty much every gtk program at the
+* bottom. eg firefox)
+  consumes too much space (if always visible) for the little it is used.  (+
+* you can put only 1 message in it at a time!)
+  -> option 1: no statusbar at all. when hovering over a link (or pressing
+  -> key to preview the url without changing page) -> show url in tooltip on
+  -> page.
+  -> option 2: toggle visibility of statusbar on/off when hovering over a
+  -> link. since it's at the bottom I don't think it will disturb too much.
+* viewing progress/state of pageload?  most programs use statusbar for this.
+  -> option 1: titlebar can show a percentage when it's loading a new page.
+  -> option 2: toggle a statusbar everytime we start loading a new page.
+* uri bar -> yes, even though we can write stuff to the fifo, it can still
+* be convenient to change the url manually and stuff, so a widget in uzbl
+* itself is good.
+* tabs -> yes. you don't have to use them, but you can.
+* back/forward/.. buttons? -> no: use keyboard shortcuts.
+* searching in a page? not sure.. maybe we can abuse the statusbar for that
+* too.
+  eg toggle it on when the user wants to search for something and then do
+* searching in some vim-like fashion.
+  we don't need a gtk text entry widget, just a feedback display of what the
+* current command is.
+* scrollbar? no: use keyboard shortcuts.  we should however have some info
+* about the page length and where we are.
+  -> option 1: put a percentage in the window title
+  -> option 2: everytime you hit a key to change position, temporarily make
+  -> a statusbar visible and put the percentage in the statusbar.
+  what will we do with pages who are too wide? horizontal scrolling?
+all of the above goes in 1 bar at the top of the program. there should be a
+key to toggle visibility of it and one to toggle visibilety + focus on the
+entrybar at once.