The harpoon is used on fish such as mollids, jellyfish, sharks, and whatnot. It's the only way to kill them in such a way that they can be carved.
For example, you swim up to a shark and start hacking away forever. Eventually you'll do enough damage that the shark will swim away and disappear. However, if you weakened the shark till it is "limping" in the water, and then use a harpoon on it, it'll kill it, and it's remains will then be carvable.
It's performed in the same manner you mine or catch bugs with a net.
there are smaller fish that stay near the coasts which you can get with a fishing pole, and there are bigger ones that you have to go underwater to get, and the harpoon is the only way of gathering stuff out of them I think.
Also depending on the fish you will have to weaken it first with your weapon, then when it shows signs of being hurt you use the harpoon and gather stuff from them, otherwise they'll just vanish, like the shark, and there are other fish that you just simply use the harpoon without the need of weakening them beforehand like that big round fish and the tuna like one.