Each familiar has a certain capture rate. There's a merit award later that increases your chances, but it's a percentage increase on a percentage, I believe. Keep fighting. =)

I've noticed that the familiars are easier to catch if your familiar's sign is compatible with theirs. Could just be coincidence, but it's been working for me. Also that merit badge thundy84 talked about helps a lot.

I've found that beating them with Esther helps.

I'm not 100% sure, and if anyone has any solid evidence to contradict this, by all means, but when doing the "kill florets with frostbite" side quest for the lady in the port town (can't remember name off top of my head, EVERY SINGLE floret that I killed that way went into the "hearts above head" mode. I don't know if it was dumb luck, but i killed four in a row, all of them became capturable. I didn't capture them, because I needed to kill them, but maybe that has something to do with it. Another thought is that this could be a mechanic to keep your allies from killing them so you can finish them, but it's still odd, anyone else have the same experience?

Yes there is, more in to the game you unlock a reward from the stamps. that increases the chance to tame monsters, other than that you just kill them

Just keep on fighting u will get anyone on the game i dont think fighting with esther really matters and the stamp thing is much later in the game and u have to do alot of quests just keep on fighting

and (sixthsage ) even if they have heart above their head u will take the items and exp and gold so dont worry about it

Nothing but killing enemies will tame them. There are no tricks. Fighting with Esther, killing enemies with particular moved or using compatible signs will not help unfortunately. (Nothing was mentioned in the official guide)

As earlier posters have said, a merit award later in the game will improve your chances of taming enemies, but until then taming probability is based on a percentage relevant to each individual monster. Some are easier to tame than others.