I don't know the exact requirements, but I think you can get these seeds later on in the game after finishing the first arc. I believe the quest is given by Forte, and it mentions growing dungeon seeds (yes you can "grow dungeons" in your fields).
Here is the site that I found describing where exactly it's located. Read at your own descretion - there are slight story spoilers on the site. http://kaze-no-travelers.tumblr.com/post/62446741834/non-spoiler-pic-free-version-theres-this-area-in