What if you use Star Coins to buy too much at the start of the game, in where you possibly can't get 90 coins to open Star World? I spent quite a bit of star coins at first, and, I added up all the coins I could get to open star world. Though it was close. And I noticed I could have spent at least 25 more coins to open other passages, which would have left me with not enough Star Coins to open Star World. Or, is there another way to get Star Coins that I don't know about?
Hm... now that I think about it, yeah, you probably can overspend yourself. It costs 90 to get into the Star World, but you only get 24 back, meaning if you open most exits, you could find yourself stuck.
Someone should test this, because it'd be a huge design oversight.
"It looks great, it plays great, it sounds great... I hate it." --JonTron on the New Super Mario Bros. series
I got every single star coin in all 8 worlds (1-6, mushroom, flower) and paid to get access to every single Toad house and secret level before heading to star world. When I got there, I had 99 Star Coins left, with 9 for the Toad house and 90 for the levels.
So to answer your question, no you can't actually overspend star coins, there is enough for every "sign" in the game.
"I have FURY!" - Fawful from Mario and Luigi Superstar Saga