I was having a hard time getting the USB ports to work properly with Ubuntu 10.4 LTS 64bit and VirtualBox 3.1.6 r59338 It turns out that the USB group was missing.

Steps:

open SYSTEM>ADMINISTRATION>USERS AND GROUPS
Click on USER AND GROUPS and then click on MANAGE GROUPS
Then click ADD then enter the group name as usb (lower caps) change the group id to 85
Click the check mark for your group member (your user name) and click OK.

It appears to require a reboot to pick up the permissions. There is probably a simpler way than the reboot but I am not sure what service to restart to see this so the reboot was quick painless.

Now I can add & remove the USB devices on the fly with my virtual machine.

Here is where I found this info:
http://forum.virtualbox.org/viewtopic.php?f=7&t=30906&sid=83b22991a9c34aa69ff180fbc5fb7b20&start=30