I, personally, don't find any singular class frustrating to play against but rather I find particular situations frustrating.
Each class has effective counters and if you're patient enough you can eventually force a class change eg. when a deadringer-spy grows tired of dying over and over because they're being chased down after their feign. If there is an enemy team-combo that is difficult to break ie. Vac-sniper-with-DDS, there are always counter-measures that can shift them out of their cosy-corner. One suggestion: kritzkrieg-DH-Solly.
Playing as a team is the BEST counter against any potentially frustrating situation. Any time the 'team' system breaks down is when those frustrating situations surface and overwhelm.