Just looked up the name of the book: The Unforgiven - it looks like a good read.
A rough time is half of it - by all accounts most of the players were social outcasts and have had a wretched life since.
There's a good book out about the West Indies tour of South Africa. I haven't read it but it has some good reviews. Despite the promised financial security, it didn't end well for many of the particpants
Fry was actually one of few non right wing cricketers - he was a Liberal parliamentary candidate three times. He did, however, have some weird ideas - such as fostering strong links between Boy Scouts and Hitler Youth but that seems to be more down to naivete rather than ideology
There's a lot about this in Peter Oborne's excellent biography of Dolly.
Oborne's take on this is that the chairman (who is elected yearly) didn't have as much cloud as the secretary and treasurer of MCC, both permanent positions: Gubby Allen and Billy Griffith (also ex Sussex) respectively...