I personally don't think anyone would be that arsed if England withdrew, no matter how noble the gesture. It happened with Brexit.
"We'll show you, we'll boycott that, and the whole thing will fall apart".
Look how that went. The EU was largely unaffected, just like the World Cup would be.