Actually, you can have less than 4 people on your team.
When selecting your team members, you need to push start once you've got your team of three or less set. If you really wanted to, you could have someone solo an area, which helps cut load times for battles.
Though, I only recommend doing so if you've got someone powerful enough to deal with the dungeon monsters and such. This also helps if you're trying to farm mana for any purpose.