It really depends on which skills you want the most. If you plan out your build and find that most of the skills you want are in the earth mastery, then go with earth first. And vice versa. Also what kind of damage do you want to do? Do you want to do fire, ice or lightning? Because that could give you a clue to which one to pick first.
I would go with storm first because I would want to do either lightning or ice damage. I would want to do ice or lightning damage because 1) The wisp majorly helps with ice/lightning damage, and 2) Rare staves with either be fire, ice or lightning based damage. Now if you have storm based damage then you will have 2 catagories to choose from. If you go with earth based damage means you are stuck with doing fire damage and fire damage only.
For the earth mastery you can just use that for the skills such as the major butt-kicking core dweller, Eruption, or the fireball one.