If you want rare weapons then take over the outposts and radio towers early on in the game. Basically as soon as you have free reign. Also collect as many relics as you can. I am not even half way through the game and I have already upgraded almost all of my pouches and I have 4 special limited edition guns.
You can find hard-drives with small messages or WWII letters from lost soldiers. Either case, the messages arent very relevant with the story and are for entertainment. Like mrbeef523 said, you mainly collect them to unlock "signature" weapons.
Ive spotted several caves with no purpose than aesthetics. No chests, no letters, just cave. They're neat to find, if thats what you're looking for.
To be honest, you may find better kicks in user maps dedicated to adventuring and "dungeon diving". Map editor can be played in a multiplayer and single player aspect, depending on how their fitted.
"...and when there was no meat, we ate fowl and when there was no fowl, we ate crawdad and when there was no crawdad to be found, we ate sand. "
Pretty much only for the sake of exploration itself and for the chances of random encounters. like getting stuck in the middle of a tiger's hunting session or to see a bear charge out of the woods to be hit by a jeep and have the guys in the jeep jump out screaming and shoot it only to be mauled to death.
But on the exploration end itself, its a beautifully rendered world, but it does seem very very empty and boring once you clear the enemy outposts. I also was puzzled by their refusal to add traps. Some of the areas are just screaming for it.
There's really no point to exploring other than maybe finding chests to loot for money or exploring a specific area to hunt an animal type. Everything else gets marked on the map and it's faster to just pinpoint whatever you're looking for and go straight to it. Later on though, you'll put on a disguise and be able to take outposts unopposed, but you have to worry about the warriors not knowing your disguise and attacking you. It usually works out because they follow you to a camp and while the guards fight the warriors, you can pick them off. As soon as you take a camp though, don't hang around outside with the warriors there.