I'm almost positive that your first hit basically has to take all of their hit points away. For example;
You can do 300 points of damage with one hit. You (sneak) attack a person or enemy with 100 Hit points and you'll automatically kill them with a animation kill
You attack someone that has 300 Hit points, animation kill.
Attack someone who has more hit points than you can deal out, then your going to have to take more than one strike to kill them.
I've experience this multiple times with a stealth character and can do this at will against an appropriate target. Animation kills work best when your target is standing.
Another example to back up my statements, I was having trouble with a Forsworn Briarheart at a camp. I managed to sneak up behind him and even with a Ebony dagger that does 50 damage I didn't get a instant kill animation (took a good chunk of his health though). Being a perfectionist against a tough opponent, I reloaded my save and equipped a pair of gloves that doubled backstab damage and a potion that increased my one-handed skills and got the throat slitting instant kill animation on my first try.