A

Robots.txt and sitemap questions

  • Thread starter Thread starter Alvin63
  • Start date Start date
A

Alvin63

Guest

I've never actually used a robots.txt (didn't get round to it and it hadn't been an issue generally) and have been reading around on here to find various examples and still find it a bit confusing (due to there being different examples) and also not quite understanding some of it. I thought it was just to block robots from scanning the site, but it seems people are adding forum sections to it as well.

Last time I looked into this I thought you needed to add "Allow" at the end to allow...

Read more

Continue reading...
 
Similar threads Most view View more
Back
Top Bottom