16GB should be enough unless you're a PS+ member and/or plan to buy many games digitally. There are digital Vita games almost 4GB on their own so space gets eaten. --- I own a Vita and a Wii U, bite me.
You just can't straight swap if you have info on the card. You'll need the memory manager on your PC, or a PS3 to transfer info from card to card. 32GB is the way to go, as I don't have any full retail games on my Vita, but I have tons of PSN titles, and they can add up over time. 16GB may be enough for you, it just depends on your needs.