They want Jeremy Corbyn to step in and take the whip away from Mr Williamson – meaning he would no longer be a Labour MP and probably couldn’t stand again for the party in the event of a snap general election.
And the prospect of that election is fuelling – at the very least – further scepticism towards the leadership among Labour MPs.
The parliamentary selection process has just been opened up and some critics of Mr Corbyn are concerned about possible deselection.
One of the 160 signatories of a original document criticising the Williamson decision – co-ordinated by deputy leader Tom Watson – told me he felt the leadership was trying to use the threat of deselection to keep people in line, but it seemed to have had the opposite effect.
So while the outrage over anti-Semitism is real, the motivation to speak out on the party leadership’s response has been even greater.
And now, there has been a further initiative. Seventy-one Labour politicians have written to the chair of the Parliamentary Labour Party (PLP), John Cryer, asking him to begin a process that could lead to the whip being withdrawn from Mr Williamson if Mr Corbyn doesn’t act.
Hull MP Diana Johnson argues in the letter that the PLP can take matters in to its own hands – its standing orders – or rules – would allow “in exceptional circumstances” for the withdrawal of the whip from a member whose conduct has been investigated.
And the signatories to the letter, which includes “soft-left” MPs such as Lisa Nandy and Alex Sobel, say Mr Williamson should be suspended for a year. If they are successful, that would mean that he would be ineligible to stand as a Labour candidate if there is an election before July 2020.
What is also significant is that it isn’t just the usual Corbyn critics that have weighed in to the Williamson row. There has also been pressure from some -wingers who were once seen as his allies.
For example, Jon Lansman, chairman of the Labour grassroots group Momentum, has said Mr Williamson “has to go”.
Interestingly, sources close to the Labour leadership say that relations with Mr Williamson broke down some time ago and they genuinely believed that the panel who made the decision to readmit him would in fact have been tougher.
It was Keith Vaz – a former minister who is not seen as politically close to Mr Williamson – who had the pivotal vote.
He is now calling the process into question and suggesting the whole thing could be re-run – an action replay that could give a different result and defuse a potentially explosive internal row.
Mr Vaz was brought into the process at the last minute and the theory being advanced by some in leadership circles is that he initially and mistakenly did what he thought Mr Corbyn wanted, the better to avoid a deselection threat.
When he saw the strength of the backlash, he suggested effectively re-running the process.
But Mr Corbyn is in a difficult position – one his internal opponents relish.
Earlier this year his office was criticised for intervening in disciplinary cases. His spokespeople said this had been a temporary measure and had stopped.
So Mr Corbyn says he can’t get involved now in this case, but his critics say this displays a lack of leadership at a time when he is also being portrayed as dithering on Brexit.
The party’s processes has given the leader’s opponents a stick with which to beat him, as the whole Williamson affair turns in to something of a sticky mess.