If magic is broken for you play on Legendary, won't make a dent at all in any enemy, unless you use illusion, which was always is broken.
I started Skyrim and set difficulty to Legendary straightaway...as a mage. Damage has been poor for a long time...been using Vilja to help me survive. But now, at level 25, my magic is getting good enough for when Vilja falls critical- I can drop the enemies coming at me, inclusive of falmer spellcasters. It just takes a long time to be able to grow as a spellcaster.
I'm still able to be killed by falling feathers though :D Even with Stone Skin...will have to work on that. When I play Skyrim...I can play for 3+ hours at a time. If I have less than one hour of free time, I'll consider not starting the game.