This is an exploit that isn't obvious. Multiple sources state that if a quest is red, traits and abilities have no effect. When you add in a follower that is red, the icon does not display next to his picture indicating the ability is used. The only way it is seen is after adding said follower it is added to the quest bonuses.
Once you accidentally do this, it becomes minor/common knowledge pretty quick. But if you never accidentally did this, it is not obvious the game works this way as it is not intended.