Searches for “more information about text formats”

I have a site using Drupal as CMS, and got “infected” badly with spammer bots.

Moreover, recently I signed up for Google’s Webmaster Tools, and I see that one of the top search queries that result in my site is “more information about text formats
And I know that this block of text appears in drupal sites.

So I’m guessing that this is the bots’ way to find spammable drupal sites. So is there a precaution for this? I already made robots.txt to Disallow: /drupal-subfolder/ but apparently this does not affect malicious bots.

One Response to “Searches for “more information about text formats””

  1. Sumoanand says:

    You can prepare a list of bots which you don’t want to allow on your website & mention them in robots.txt like

    User-agent: <bot_identifier>
    Disallow: /

    Here is a exhaustive bad bots list.

    But still many bad bots will not respect it & crawl your site. For that you need to install application like Fail2ban to restrict the crawlers which are crawling at a higher rate.

Leave a Reply

Question and Answer is proudly powered by WordPress.
Theme "The Fundamentals of Graphic Design" by Arjuna
Icons by FamFamFam