I would say Colosseum or XD. ranger games aren't that hard, and the mystery dungeon games aren't really that hard either (excluding a couple of dungeons). I haven't played Conquest, so no opinion on that.
Mystery Dungeon isn't hard because you can level grind and easily abuse type advantages in a lot of dungeons.
The only Ranger game I've played was the first one. I died three times. Once to Slaking early on, another time against Steelix because that thing is annoying, and another time against Suicune because I was unprepared.
Ranger wasn't hard, but a lot of the extra missions were painful to beat. (Mew...)
I'll say the original Pokemon Pinball for GBC was the hardest. If you're trying to complete the Pokedex in that, good luck... I'll do it someday. Pinball Ruby/Sapphire was easy compared to this one.
Official Chespin of the Pokemon X boards. http://www.youtube.com/Decon082/