My lad is 15 now and as a baby he was lactose intolerant and was put on soya milk (Wysoy IIRC) on the doctor's recommendation. I think we even got it on prescription.
Personally I'd take the view that a doctor trumps a midwife and health visitor; the health visitor said "never mix breast and...