First, the gun needs to be equipped. Won't work if you have the darts or whip in your secondary weapon slot. You can change equipment by tapping the weapon icon thing on your screen.
Second, need to target your enemy before it will fire. The game tends to automatically highlight an enemy in front of you but you can also hold down L1 to manual aim. A circle will appear in the middle, so line that up with your target. Will not work on civilians (has been phased out in both this and AC3 for PS3).
Third. Make sure you have bullets, and make sure your gun is loaded. The standard pistol holds only one bullet at a time so you'll need to reload regularly.
Also note that you get a blowpipe not too far in the game. It works almost the same except you need to hold the triangle button and wait for the line to become thinner. Release the button to fire when you have a straight thin line. When not detected use the blowpipe instead as it doesn't make noise.
Fallout 3 isn't Oblivion with guns, it's Fallout 3 with Guns.