it depends the gun. For smgs, you dont even need any attachments to do good, so no perks is def worse. For ARs, not having stock hurts, but having no perks is still worse (especially since toughness is most times needed for longer range battles).
The thing is, everybody else having attachments like stock and quickdraw and you don't. Other than toughness, perks really don't help you gunfights. That's why I think no attachments is a little worse. Not having perks gets you killed by things other than bullets.