You're still entirely missing the point, but you're also just wrong about something, and that is saying the stories aren't open for interpretation. They are, hence why these questions are being asked in the first place. I already explained how the magic thing works and no, you would not have to ignore parts of the story but instead take them as a way other than literally. Seeing as how most of it could easily be explained as "this part was just his imagination", it's not entirely difficult to accomplish.
You can interpret things however you like, but there are correct theories and there are incorrect theories. A popular incorrect theory is that Lord of the Rings is an allegorical take on the events of World War II. Another pop culture item that suffers from this exact same question is Pan's Labyrinth where viewers argue that none of the fantastical events of the movie took place even though they were intended to be real.
There's no point in explaining things again as you are free to hold whatever irrational belief that you have. "Taking it in a way other than literally" is the same thing as, well, ignoring it. It's twisting it to your own image of what it should be. It's, as you say, fanfiction. If you can't assume any baseline for the "real" events in the story, then there is no story and there is no argument for either side.