Google reveals details of its Robots Exclusion Protocol
Google has released its robots.txt web parser, the Robots Exclusion Protocol that instructs Google's web crawlers to ignore website content. The robots.txt Robots Exclusion Protocol enables webs...
To continue reading this article...
Join Computing
- Unlimited access to real-time news, analysis and opinion from the technology industry
- Receive important and breaking news in our daily newsletter
- Be the first to hear about our events and awards programmes
- Join live member only interviews with IT leaders at the ‘IT Lounge’; your chance to ask your burning tech questions and have them answered
- Access to the Computing Delta hub providing market intelligence and research
- Receive our members-only newsletter with exclusive opinion pieces from senior IT Leaders