You wouldn't believe.... Really.
You've two options, in my experience. I've tried both.
1) Allow crawlers, jet up the search rankings, then become a prime target. Sort your firewall out (although even the best ones can be beaten) and cross your fingers.
2) Block crawlers, drift down the...