I seen't him twice. The first time he was chilling in the graveyard and not moving anywhere. The second time he was riding past Falkreath and I followed him all the way to the graveyard.
Both times I tried to get him off his horse, but nothing worked. I tried attacking him, telling my follower to attack him, casting frenzy, using animal allegiance. Nothing got his attention. Apparently he can sometimes spawn next to his horse instead of riding it, and that seems to be the only time you can steal the horse.