Robots TXT

dania · September 09, 2008, 04:15:58 AM

Wow,

Today I was wondering if Google had a way to NOT crawl a site of mine as I uninstalled and people still somehow after all this time looking for it. I found a way in Google with the robots.txt

Does anyone know what robot.txt will get Google crawling the new one - a code maybe. Would it just be like this:
User-agent: Googlebot
Disallow: /

But allow. I have put the new site in Google but Google still not once listed it.

Thanks

Steve · September 09, 2008, 08:04:42 AM

Your code is correct.

http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=40364

Just give to the bot some time

dania · October 07, 2008, 03:49:57 PM

Google is visiting a new site thanks Steve.

How do we stop bots like picsearch. A robots txt in every image folder with that disallow?

Steve · October 07, 2008, 04:10:28 PM

User-agent: *
Disallow: /
Crawl-delay: 10

dania · February 24, 2009, 03:18:42 PM

Steve,

I am late..again..

What does that code do?
Thanks!

Steve · February 25, 2009, 01:34:30 AM

Disallows ALL the robots.

dania · February 28, 2009, 11:50:14 AM

But don't you want some good guys Steve?

Steve · March 01, 2009, 12:13:28 PM

only google adsense for my forum, noone else.