When you reset it, you get back any perk points in that skill. Also, after you reset it, you can level it back up again and you gain even more perk points, as well as more to add to you health/stamina/magicka pools.
A quick walkthrough of the process, just to further illustrate (numbers are completely made up):
1) Start a new character.
2) Level "Smithing" from 15 to 100, bumping your character's overall level to 14.
3) Switch "Smithing" to Legendary. Your character is still level 14 with all of your accumulated perk-points.
4) Level "Smithing" from 15 to 100 again, bumping your character's overall level to, say, 22.
5) Now you're level 22 with all of your perk points (which, in this example, is 8 more than you had before going Legendary) and you've only leveled the "Smithing" skill. Repeat as long as you want.
Basically, the Legendary system just lets you keep re-leveling your skills and boosting your character level (gaining another perk point per level as you normally would) while doing so. Before, you only got one opportunity to reach 100 with each skill and then your progress stopped.
It also helps with maxing the skills that you use. Once you max all the skills you use as a thief then you can't level anymore. If you want to "perk" out your pickpocketing you would have to level other skills to do so. But, with the legendary system, you no longer have to use other skills like heavy armor to be a rogue. You can just reset a skill.