Haven't played it yet myself but I've heard great things about Outlast. Keeps the jump scares to a minimum yet creates a panicked atmosphere with its invulnerable monsters.
Silent hill 1 on ps1 was the scariest game I played though. The terrible graphics and amazing sound design really puts it above most. Chances are its a bit of a case of nostalgia glasses but, your first scary game as a young one tends to stick.
edit: Yeah I did say that terrible graphics was a plus. Horror for me requires you to let your imagination do most of the work. Catching only a glimpse or seeing something and not understanding exactly what it is you saw is much scarier than just seeing the monster. Monsters tend to be the least scary part of a game once you 'understand' them both mechanically and visually. Jump scares no longer really work on me in 'horror' games because you are always in a state of alertness. A good horror game offers atmosphere over and above anything and tries to make you feel helpless.
The old Realms of the Haunting game was a very scary game at the time. Enemies that appeared out of nowhere (specially in places that where previously empty before), the lack of a soundtrack in most places, and the constant storm outside the mansion made me pee in my pants when I was a kid! :p