I do use all out defense during his ultimate, but he still destroys anyone in the blast. Is my timing off?
That's actually fairly common - it's a STRONG attack. A good strategy is to give each AI-controlled partner a Familiar that has high Defense and Magic Defense. They should switch to that Familiar during All-Out Defense and, hopefully, survive the blast.