Can I just ask if you are making an allegation of that?
In what way can "it wouldn't surprise me" be construed as an allegation. By it's very definition the phrase shows ambiguity as to the authority or otherwise of the suggestion subsequently made.
Er...no, I'm not.