You have to first have picked which people you want at the totems. Then do the quest for captain hermosa, to save the wandering shipmate. You will find a book, take it to the windy area with the 3 candles to the east and read it. Follow that questline and you'll eventually open the way.
Gametag: Jawless Dan Here at Sofa King, our prices are Sofa King low that you will call us Sofa King crazy
When you go all the way up the tower with the ladder (on the outside), take the right path down, not left up to the skeleton blacksmith DOWN! At the end of that path climb down 1 ladder, jump down and enter the tower to Lovis. And if your stuck in the room whit the blacksmith, you need to jump in the hole with the fire get the stone, jump back quickly and activate the stone at the glowing wall. Hope that helps