I just missed a tonne of crystals when you fall down to the centre of the earth, so it would be nice to give it another shot.
Sorry, but IIRC that is the one part -- I think that might be what people refer to as the pillar -- that you CANNOT go back to (you'll need to use an NPC to go back to that lower city, just as you do to revisit the Rift areas).
Don't worry, however, there are lots of gems, I upgraded Kat almost 100% (just short on some of the special attacks) just returning to areas I've been and got both gem trophies (the 2nd after already having finished the game). To get the second gem trophy you'll need to do some work to complete challenges for more jewels.
Yes, just as you can go back to the rift planes, you can go back to the city of the past (or whatever it's called). The little girl and the older guy wait by Kat's sewer pipe home and you just walk up to them and ask them to go to either the city or the rift planes.
You have to be past certain points of the game for this to happen.