Google open sources robots.txt web parser

clock • 3 min read

Google reveals details of its Robots Exclusion Protocol

Google has released its robots.txt web parser, the Robots Exclusion Protocol that instructs Google's web crawlers to ignore website content. The robots.txt Robots Exclusion Protocol enables webs...

To continue reading this article...

Join Computing

  • Unlimited access to real-time news, analysis and opinion from the technology industry
  • Receive important and breaking news in our daily newsletter
  • Be the first to hear about our events and awards programmes
  • Join live member only interviews with IT leaders at the ‘IT Lounge’; your chance to ask your burning tech questions and have them answered
  • Access to the Computing Delta hub providing market intelligence and research
  • Receive our members-only newsletter with exclusive opinion pieces from senior IT Leaders

Join now

 

Already a Computing member?

Login

You may also like
IT Essentials: Time to take the deepfake fight to Big Tech

Government

Speak softly and carry a big stick

clock 04 March 2024 • 3 min read
Critics furious about Microsoft-Mistral AI partnership

Legislation and Regulation

Flies in the face of the AI Act

clock 28 February 2024 • 2 min read
Google to use Reddit posts for training AI models

Big Data and Analytics

Reddit will get access to Vertex AI as part of the deal as it heads for IPO

clock 23 February 2024 • 2 min read

More on Cloud and Infrastructure

'A lot of confusion': Unpicking SAP's recent changes

'A lot of confusion': Unpicking SAP's recent changes

SAP and Lemongrass talk S/4HANA migration deadlines, RISE and AI

John Leonard
clock 27 February 2024 • 6 min read
Datacentre demand 'a self-fulfilling cycle': National Grid CIO Sarah Milton-Hunt

Datacentre demand 'a self-fulfilling cycle': National Grid CIO Sarah Milton-Hunt

Build better

Tom Allen
clock 21 February 2024 • 2 min read
Vinted's stylish security: Navigating fashion-tech fusion

Vinted's stylish security: Navigating fashion-tech fusion

Turning to open source to address containers and microservices

Tom Allen
clock 07 December 2023 • 4 min read