Well this was the inevitable problem with switching the emphasis from whether offence was intended to whether it was taken. It turns the accusation into a possible weapon. And as you say, it's now used all the time - Danny Baker and Alastair Stewart lost their jobs for the wrong tweet to or about the wrong person, Laurence Fox gets labelled racist and death threats for daring to point out that anyone can be racist to anyone, and now even the Conservatives defend Priti Patel by saying opponents would cry racism rather than assess the claims (although that Guardian cartoon... ). 17.4m were instantly branded racist 4 years ago for how they voted in a democratic referendum, regardless of their reasons. PC went mad a long, long time ago, why's nobody done anything to sort it out?
As for Trevor Phillips specifically, I saw the letter Labour had sent him, and it's like something out of 1984. I heard a few commentators defend Labour by saying they had to follow a process if there'd been a complaint. But it's clear from the letter a judgement has already been passed - it actually asks if he regrets his comments and if he intends to share similar content in future. Bizarrely, they suggest he can contact the Samaritans for support, but not, say, a lawyer... Beyond that, the evidence against him is that he made some references to muslims and quoted Enoch Powell one time. If that's the bar and context is not important, it only emphasises the anti-Semitism problem in the party.