Source: https://www.opensc-project.org/opensc/wiki/OpenPGP

Gnome's stock GPG agent is a piece of crap that doesn't support smartcards. This is an old known problem (2008 AFAIK), but gnome dev team seems not to give a heck to this issue. The most appropriate solution is to disable gnome's GPG agent and, if necessary, to properly configure your own gpg agent. To do so, run the following program:

gnome-session-properties

uncheck Gnome keyring Agent, then reboot or log out and log back in.

Refer to http://www.jwiltshire.org.uk/content/2010/12/19/the-perfect-gpg-agent-setup/ for a nice GPG agent setup

apt-get install pcscd gnupg2 gpgsm scdaemon

Touch ~/.gnupg/gpg-agent.conf, then put this into ~/.gnupg/scdaemon.conf

reader-port "Gemalto USB Shell Token V2 01 00"
disable-ccid

gpg selects the first card reader it sees by defaut, if the first reader doesn't have a card, it will fail as follow:

13:37 user@hostname ~ % gpg --card-status      
gpg: detected reader `Dell Dell Smart Card Reader Keyboard 00 00'
gpg: detected reader `Gemalto USB Shell Token V2 01 00'
gpg: apdu_send_simple(0) failed: no card

Append –reader-port “reader name” to gpg command for version.