"I uncovered the whole map, but the story wasn't starting.
I did find the problem! I was fast travelling direct to lair of infinite agony, which wasn't triggering the story event.
I fast traveled to Hatred's Shadow, went though the whole thing and used the travel point at the end to go to the Lair of Infinite agony. This dropped me into the wailer drop and started the story bit properly (follow the spirit etc).