Meta removes language that may have allowed it to scrape news sites

Meta removed language from its commercial terms that said it could use content from domains, web pages, or URLs to provide, develop, improve, research, train, and maintain its products. The wording drew attention because it appeared broader than ordinary advertising terms.

Critics, including Neiman Lab and Cornell professor James Grimmelmann, interpreted the passage as permission to collect website content for AI training. They also thought it might override instructions in a site’s robots.txt file, which websites use to signal crawler preferences.

Meta says the update does not create new rights or change its web-crawler practices. Spokesperson Daniel Roberts said the terms applied only to ads and business products, and that Meta clarified this point while removing the controversial language.

A web crawler is an automated program that visits web pages and follows links to discover more pages. Web scraping is the process of copying or extracting information from those pages. A crawler may simply index a page, while a scraper can collect its text, headlines, images, prices, or other structured data.

For example, a crawler could start at a news site’s homepage, follow links to articles, and send each page’s contents to a company’s systems. Software can organize the results into a searchable database or dataset. Repeated visits can gather information from many sites and millions of pages.

The article focuses on Meta’s terms and says Meta uses web crawlers, but it does not describe their technical operation in detail. Meta said there was no change to that use, and that website owners can use robots.txt to signal preferences.

The public web contains an enormous and constantly changing amount of material. An AI company with large crawling systems could potentially collect information from billions of pages, including articles, books posted online, forums, images, and other data. The article does not provide a precise number for Meta’s possible collection.

The mechanism is automation. Crawlers visit pages, follow links, and send information back to company systems. Scraping tools can then extract and organize the material for search, analysis, or AI development. Repeating this process across many domains allows coverage far beyond what people could gather manually.

Scale does not automatically settle whether collection is permitted. Copyright rules, contracts, access controls, and technical signals can affect particular uses. In this case, critics focused on whether Meta’s wording could support broad collection, while Meta said it created no new rights and did not change crawler behavior.

The concern came from the breadth of Meta’s wording. It said Meta might use content from domains, web pages, or URLs to provide, develop, improve, research, train, and maintain its products. Critics read that language as covering website material collected for AI training, not merely content supplied through an advertising service.

Neiman Lab and James Grimmelmann, a professor of digital and information law at Cornell, interpreted the passage as an attempt to claim free use of scraped site content. They also thought the wording could override a publisher’s robots.txt instructions. That interpretation raised particular concern for news organizations, whose articles could be valuable training material.

Meta rejected the interpretation. Andy Stone and Daniel Roberts said there was no change to how Meta uses web crawlers. In its update, Meta said the terms applied only to ads and business products and did not give it new rights over site owners’ preferences.

Robots.txt is a small file placed on a website that communicates instructions to automated crawlers. A site can use it to signal that certain pages, directories, or the whole site should not be crawled by particular bots. It is a technical preference, not a physical barrier that prevents every visit.

The worry arose because critics read Meta’s terms as giving the company permission to use content from websites for training and other purposes. Neiman Lab and James Grimmelmann thought the wording might supersede robots.txt instructions. That would have weakened a tool publishers use to express access preferences.

Meta disputes that reading. Andy Stone and Daniel Roberts said there was no change to web-crawler use. Roberts later said content owners can still use robots.txt to signal their preferences, and that the updated terms do not give Meta rights that override those preferences.

For news organizations and other website owners, the main change is clarity. Meta removed the controversial language and stated that its consolidated crawling terms applied only to ads and business products. That may reduce concern that the commercial terms themselves authorize broad scraping for AI training.

The practical mechanism is robots.txt. Website owners can use the file to signal whether crawlers should access particular areas or content. Critics had feared Meta’s earlier wording could supersede those signals. Meta’s follow-up statement says it does not, preserving the stated role of robots.txt in communicating preferences.

The clarification does not promise that every dispute disappears. The article says Meta’s web-crawler practices are unchanged, but it does not discuss enforcement, copyright outcomes, or publisher payments. Its immediate effect is to remove the disputed wording and clarify Meta’s position about access preferences.

Copyright law generally concerns who owns protected expression and when copying or reuse may be allowed. Website terms can add contractual conditions for using a service or its content. Robots.txt is different: it is a technical signal asking crawlers to follow stated access preferences. These systems can overlap, but one does not automatically replace the others.

For example, a publisher might hold copyright in an article, place usage rules in its website terms, and use robots.txt to tell crawlers not to access it. An AI company would need to consider all three signals, along with applicable law and any permission or license. The exact result can depend on the use and jurisdiction.

The article does not analyze copyright law or contract enforceability. It reports that Meta removed disputed language and said robots.txt preferences are not overridden. That clarifies Meta’s position, but it does not decide every legal question about training AI on online content.