One of the nodes on the Western side of the map allows you to build a crystal mine. There is only one that allows it.
Once you have that and a mage tower, you will need to let the town cycle (let it go for 6 minutes.) At this point, one of the shops in your town will have crystals in it. Run around and find it. If you don't like the crystals for sale, quit, restart Village and run back to the same shop.
In my village, the Crystal Mine is on the eastern side of the map. I have two Mage Towers up, and there is only one guy selling crystals. He starts out with five, and if I buy one and quit the game, he only has four. Every time you buy one and restart, he has one less. When he gets down to two, I loot all the chests and that seems to reset him to having five crystals. I guess you could just wait the six minutes for that to happen, but I like to make money and maybe get a level 7 crystal while I wait.
Forgot to mention that you also need to make sure that there's enough bread and meat being produced, or the vendors won't have squat. I believe your village also has to be turning a profit - if the upkeep costs are not being paid, I think that affects the shops too. For me, this means that I specialize in crystal production. After I'm done with willpower crystals, I'm going to change my focus to advanced weapons and armors.