It's really hard to only do main story unless 1) you have zero interest in this game and aren't enjoying it or 2) rushing it for the sake of trying to get to the ending. You will always spend a lot of time in these games. it's the way the game is designed. Sometimes doing side missions complements the main story. If your'e only doing mainline story the games pacing can feel a bit off.
If you don't know what I mean basically i'm saying that sometimes you naturally do some sidequests because you might want to take a break from the story. It's similar to GTA to a lesser extent.....I mean you can try to do the mainline missions only but there are too many distractions and you'll want to do other stuff to get the best out of the game.
I've been playing literally for like 15 hours and am not that far into the story. It'd be pretty hard just to just do the main missions. The game has an expansive amount of content which, I think, a lot of people will miss out on. Which is unfortunate because some of it is epic. Have an explore around the Vault as Des and you'll get a hint of what I mean.