I haven't noticed any EXP penalties while partying. I always bring the maximum two townsfolk even when I'm just farming bosses in order to level them. At higher levels, the townsfolks' AI improves and they are much more aggressive.
There really isn't any downside to partying. Unlike monsters, human party members are smart enough to not attack an enemy if their attacks will heal them. You can also tell them to stand on top of switches for later dungeons. A very minor thing is that they can stand on top of dropped items and make it a tiny bit harder to pick up, but you can always tell them to stand back.
I've noticed that some of them actually fight like their personalities. For example, bado seems to do very little, xiao pai seems to trip often, and i think clorica fell asleep once or twice even when i wasn't fighting monsters that can cause sleep --- 3DS: 3496-9744-9813 friend safari pokemon: magneton, metang, and excadrill. Pm if you want to add me.
I've noticed that some of them actually fight like their personalities. For example, bado seems to do very little, xiao pai seems to trip often, and i think clorica fell asleep once or twice even when i wasn't fighting monsters that can cause sleep
That's actually true.
That's why taking Vishnal with you could be very helpful (better work together on his fighting skill first). Also Forte for sure. They're both loyal.
If someone is on high LP level would they fight harder for you? Cuz that's what I think.
A party is always better than solo in so many ways.
I played my first playthrough almost entirely with no party and I'm sure I made it harder on myself.
Clorica throws food occasionally and I read that if it misses you then you can pick it up and save it. Someone got a super expensive dish that way.
If you do take people with you make sure you give them better equipment. When I take people for the first time I outfit them from head to toe, sometimes skipping the headgear, but increasing their attack and defense so that I don't have to keep healing them.