Like you, I can't remember using 'fieldsman' instead of 'fielder', although it was in use at one time - Denis Compton used it in his autobiography, but that was in 1948 - slightly before even my time!
I remember cricketers being interviewed a few years ago started talking about batters rather...