What steps will reproduce the problem?
On my work computer I created a new repository and then added a directory on my home server to backup some of my files. The assistant created a new key pair which then made password login quite hard.
On the commandline I could still login via ssh -o PreferredAuthentications=keyboard-interactive ... but all gui tools (e.g. gigolo) stop after connecting to the restricted shell.
I don't want to allow full passwordless access to my server from this computer.

What is the expected output? What do you see instead?
I was able to fix the problem by moving the keyfiles on my computer at work to a subdirectory. After that I added the following line to the [remote] block in .git/config :
IdentityFile = /home/user/.ssh/annex/id_rsa_annex

I think it should be the default behaviour that git-annex uses its own key pair with restricted access. For portability the keys could be stored somewhere in the repository itself.

What version of git-annex are you using? On what operating system?
git-annex version: 4.20130324
Ubuntu 11.04

Please provide any additional information below.