Author Topic: Finding robots.txt  (Read 3519 times)

DrStir

  • Newbie
  • *
  • Posts: 2
Finding robots.txt
« on: October 02, 2008, 08:45:01 AM »
Hi
Google has said it has a problem with crawling my site as it had found - but was unable to download - the robots.txt file, and suggested making it accessible or removing it all together. I've looked on the site using Yummy and can't find this file.
I've tried toggling to view the hidden files but it doesn't show up. When I do a search using Yummy, it looks like the file is somehow in .htaccess, which is in a _private directory.
I can't download .htaccess - it just vanishes from the desktop. But anyway, it's obviously not robots.txt.
Can anyone offer any advice?
Thanks
David

paulc

  • Sr. Member
  • ****
  • Posts: 189
Re: Finding robots.txt
« Reply #1 on: October 02, 2008, 09:01:12 AM »
I'd talk to the sysadmin of the server your site is hosted on... the existence of a .htaccess file implies you are hosted on somebody else's server.

JD

  • Administrator
  • FTP Guru
  • *****
  • Posts: 2157
Re: Finding robots.txt
« Reply #2 on: October 02, 2008, 09:10:53 AM »
As paulc said, you need to speak with your hosting provider to see what's going on

Just for the record, when you download the .htaccess file it is actually successful, but by default all files (and folders) with names that start with a period (full point) are considered 'invisible' on Mac OS X and that's why you don't see it. This is where OS X shows its Unix roots. If you instead downloaded it to the Mac browser in Yummy you would see it, because Yummy lets you see invisible files like those.

DrStir

  • Newbie
  • *
  • Posts: 2
Re: Finding robots.txt
« Reply #3 on: October 02, 2008, 02:48:25 PM »
Thanks for the responses. It's much appreciated. I've created a new robots.txt file - so I'll see if Google has any more luck finding and downloading this one. David

paulc

  • Sr. Member
  • ****
  • Posts: 189
Re: Finding robots.txt
« Reply #4 on: October 03, 2008, 09:14:25 AM »
Doesn't matter what you put in your web space, if web crawlers are halted at the server level. Besides, "robots.txt" is meant to STOP crawling, not allow for it (last I knew...).