I think it's the haste element that catches people out. I seem to remember Harry Wilson's tackle's son getting scammed a while ago with a very clever, very tightly timed scam. I think it was only after HWT posted it on here and a few people had contributed that we actually worked out exactly...
I'm guessing because a lot of organisations like charities, are allowed (by the banks) to set up DDs without a signature, so I can see how someone who had your account details could be a pain, but these would be refunded the moment it was raised as an issue by companies who are allowed to set up...