They're a great read.
Tory diaries seem to be more interesting: Duff Cooper's and Chips Cannon's give a great insight into the political scene of the 30s. Labour diarists seem a bit more boring - although I'd make an exception for Robin Cook, his are worth reading too.
The thing is that most MPs are perfectly pleasant when engaging in small talk, it's what they do all the time. I've met Portillo a couple of times, for example, and he's never tried to egg me on to play Kick the Beggar or something. The real question is whether you could have a few pints down...