I havent played the 3ds version, but I assume the face buttons are the same as the ps2. To do mystic artes you need to do an arcane art, such as Luke's Light Spear Cannon and hold the standard attack. You can tell if the symbol on the arte page is a larger sword with a more filled in box. If artes are B (X on a ps2 controller) then you have to hold A (O on ps2). As for overlimit, maybe press the D pad or there may be a button on the touch screen? On ps2 it was R2, so i'm not sure where they'd place it.
But then again that is all speculation as I have yet to play it on 3DS. I may be completely wrong and have no idea what I'm talking about. Hope I helped!
Gamertag: Jaddmaster Don't trust whatever the poster below says. It's a lie.