Meta revises terms amid concerns over AI training on news content

Meta changed its commercial terms after publishers and legal experts questioned whether website content could be collected for artificial intelligence training. The earlier proposal said collected material could be used to “train” Meta products. The revised wording removes that reference and narrows the stated purpose to developing, improving, researching and maintaining advertising and business products.

The terms still permit automated access through web scrapers, bots and crawlers. These tools may collect content from websites linked to businesses using Meta’s platforms. For example, a crawler could retrieve product information from a business website so Meta can support advertising or content recommendations.

The revised terms are scheduled to take effect on Oct. 30. Meta says the change does not alter its existing crawling practices. It also says website owners can continue using robots.txt instructions to express access preferences, while Andy Stone rejected claims that the update was designed to enable scraping news websites.

Web crawlers, bots and scrapers are computer programs that automatically visit websites instead of relying on a person to browse each page. They request webpages, read the information returned, and may copy or organize selected content. The terms are often used broadly, although a scraper usually emphasizes collecting data.

A crawler might follow links from one page to another, while a bot can repeatedly check a site for changes. A scraper may extract details such as product names, prices or descriptions. In Meta’s case, the article says automated tools can access and collect content from websites associated with businesses using its platforms.

The article does not describe Meta’s technical process in detail. It says the revised terms permit these tools for developing, improving, researching and maintaining advertising and business products. Website owners can use robots.txt instructions to indicate which automated visitors or areas they prefer to allow or restrict, though that file is an instruction rather than a physical barrier.

Automated crawlers can potentially reach a large portion of publicly available webpages, but they cannot automatically access all internet content. Some pages require logins, payments or special permissions. Others may be blocked by technical systems, unavailable to automated requests, or excluded by website instructions.

A crawler typically starts with an accessible page and follows links or uses known addresses. The website can influence access through robots.txt, account controls, rate limits and other technical measures. The crawler’s operator also decides which sites, pages and data fields to target. In this article, Meta’s stated target is content from websites associated with businesses using its platforms.

The article gives no percentage or total for how much of the internet Meta can access. It says only that Meta may use web scrapers, bots and crawlers to collect website content. Therefore, the practical scope depends on each site’s design, restrictions and stated preferences, not a single universal figure.

Meta says it needs website content to make information in advertising and business products more accurate. A business’s own website may contain product details, descriptions or other information that helps Meta understand what the business offers. That information can support services connected with advertising and recommendations.

For example, a crawler could visit a business website and retrieve current product information. Meta could then use that information when building an advertisement or making a content recommendation. The article does not specify the exact systems, ranking methods or data fields involved, but Andy Stone described the purpose as retrieving accurate product information.

Meta says the revised provisions cover developing, improving, researching and maintaining advertising and business products. Stone rejected suggestions that the update was intended to enable scraping news websites. The article therefore presents the stated purpose as business and service functionality, while publishers’ earlier concerns focused on possible artificial intelligence training.

Robots.txt is a plain-text file placed at a website’s main address. It gives automated crawlers instructions about which parts of the site they may visit. Owners can name particular crawlers or use a general rule, then identify paths that should not be accessed. They can also leave paths available by not disallowing them.

For example, a site could tell a particular bot not to visit a folder containing articles, while allowing access to product pages. A crawler that follows the instructions should avoid the disallowed path. Robots.txt usually communicates a preference or rule for cooperative crawlers; it does not itself require a crawler to obey or prevent direct access.

The article does not provide specific robots.txt syntax. It says Meta’s spokesperson Daniel Roberts believes website owners can continue using robots.txt to indicate their access preferences. That means the revised commercial terms do not remove the stated mechanism for communicating a site’s choices.

If copyrighted news articles are collected and used to train an AI model without permission, publishers could argue that their protected work was reused without authorization. They might seek payment, licensing arrangements or legal remedies. The article does not state that this happened, but it explains why publishers and legal experts were concerned about the earlier wording.

The concern centered on a proposed reference to using collected material to “train” Meta products. News content gathered by automated tools could potentially become part of a model-development process rather than simply support a particular advertising or recommendation feature. That difference matters because training may involve using material to develop a system’s capabilities.

The revised terms remove the training reference and clarify that crawling provisions apply to advertising and business products. Meta says this does not change existing crawling practices. The article leaves open how future disputes over permission, copyright and AI development will be resolved, but it shows that wording in commercial terms can trigger major questions.

Using web content to power a service means applying information to an immediate function. For example, product details might help create an advertisement or support a content recommendation. The content helps the service provide a result, such as showing relevant information to users or businesses.

Training an AI model is a different purpose. Collected material is used during model development so the system can learn patterns or improve its capabilities. The model may then generate outputs across many later tasks. In the article, the earlier terms raised concern because they referred to using collected material to “train” Meta products.

The revised terms say automated crawling is intended to develop, improve, research and maintain advertising and business products. Meta removed the training reference and says the change does not alter existing crawling practices. The article does not explain the technical boundary between product improvement and model training, so that distinction may remain important in future debates.