What starting cave? Do you mean the very first area? The first cave of the area after you leave the inquisition?
Anyway. It's simple.
1. You need the map. If you do not have the map of the area you can NOT fast travel.
2. Once you have the map, you can fast travel when you have visited the place by opening the map, "M", and clicking on the place you want to go to, if there is a "circle" there on the map.
3. If there is no fast travel to where you want to go on the map and you already went there. Then there is no fast travel there. Fast travel is available by visiting and owning a map, nothing else.
4. If by "first cave" you somehow mean one of the previous area you were in (like right at the begining, which isn't a cave btw), and you do not mean "fast travel", just "traveling between island", then you will need a boat, which will be available half way through the game.