Coup de Grace is when your partner is being attacked and you have the option to press "B" to get the attacker off of him/her. If this kills the attacker, it is a coup de grace.
Counter is when enemies are trying to melee you or jump on you, you have a split second(it flashes on screen) to press RT to counter the move.(i found this easiest on Leon's campaign in ch2 in the cemetery, against the zombies with the shovels. Edit: Nevermind, you were inquiring about mercs. the rule stills stands on how to do it though :p)
A Jose Canseco bat? Tell me...you didn't pay money for this. http://tinyurl.com/45pdkzf http://oi48.tinypic.com/ip48x1.jpg http://oi40.tinypic.com/8x7i8o.jpg
Coups are performed from the side of an enemy, when they are in the heavy stun state. Its tricky to perform, I think you either need to shoot 2 stunnable areas (like head + arm), or shoot arm and then slide into them. Can also be done against bigger enemies like bloodshots when they are knocked to their knees.
Counters are exactly what they sound like, hitting the attack button right before an emey hits you (when the prompt appears) which prevents you from taking damage and either hurts them or outright kills them. Best used against armed enemies and bloodshots lunging at you.
A coupe de grace is a special melee finisher. In Mercenaries, you know you got it when you get +7 seconds (instead of +5 for regular melee finisher and +10 for counter finisher).
You can do it by saving a grabbed partner, using a melee on an enemy who has been dropped to their knees, or some characters can pull it off by using a melee while to the side of a stunned enemy (for instance, Jake can do a heel drop in this position, which is his coupe de grace).
Counters are much easier, imo. Press to melee right before an enemy hits you. A melee prompt will flash on screen at the right time, but it is so quick that if you wait until you see the prompt you will probably miss. So time it based on the enemies movement.