I stumbled on a quest for Ser Darious although I think I made the wrong choice.. But I really think that quest will reward you with the key to that fortress if you made the right choice on that quest. --- Its "Betty" you son of a pig! The name is "Betty" -Master Pain-
If you're talking about that small castle way to the north then that's part of the Duchess questline and to get in you need a special set of armour that her servant gives to you. You get the quest just before fighting the dragon when you talk to her servant. You must also have tried to sleep with her and seen the whole choking love scene.
Ser Darious' quest is in Windbluff tower. When he asks you to get him supplies, you have the choice of talking to a guard or to Julien. Talk to Jules, tell him you need a week's supply, and give Darious the 5 pieces of meat. Once you do that, he will give you the key to open the room with 2 chests.