There's probably a degree of sense in that.
There are many reasons why a manager might fail. Having a stable platform and good supporting processes (esp. recruiting) does a lot to limit what can go wrong.
So when it does go wrong you can probably safely say the manager wasn't up to it...or wasn't a good fit (...which is rare, as they recruit managers more intelligently than most clubs)
Plus, more than most clubs they will give you the benefit of the doubt for longer if your results are lagging your underlying performances.
But also this. Managing in the PL is hard, no matter what club you are at.
But maybe we should weigh managerial success at Brentford or Brighton (...or maybe even Liverpool?) a little less than at some other clubs?