Huawei, Pramana AI training linked to pirated books by Korean authors - Korea JoongAng Daily

Investigators found that Chinese AI companies had used, or may have used, datasets containing unauthorized copies of books by Korean authors. The connection matters because these datasets can carry copyright risks into AI training, even when companies do not publicly identify every book they selected.

Huawei’s Pangu Sigma used The Pile as an English-language training source. The Pile includes Books3, a collection of pirated e-books containing translations of at least 19 works by 10 Korean authors. Pramana reported using e-books from Anna’s Archive, another site offering pirated books, but did not reveal which titles it selected.

The evidence is therefore different for each company. Huawei’s source dataset contained the books, though its exact selections remain unknown. Pramana’s selection is also uncertain, but Pramana reproduced 19 consecutive words from Han Kang’s The Vegetarian after fine-tuning. Researchers called that indirect evidence of pretraining on the novel.

Books3 is a collection of pirated e-books. The Pile is a much larger dataset containing books, academic papers, web documents and other material, totaling approximately 800 gigabytes. Anna’s Archive is a website that provides access to pirated books. All three matter because they connect unauthorized book copies with AI training.

The Pile included Books3, according to the dataset’s research paper. A GitHub list showed at least 19 works by 10 Korean authors, including Han Kang’s The Vegetarian and Human Acts. Huawei identified The Pile as an English-language source for training Pangu Sigma. Pramana reported obtaining 860,000 English-language and 180,000 Chinese-language e-books from Anna’s Archive.

These sources do not prove that every listed book entered a model. Huawei and Pramana selected training materials, and the article says their exact choices remain unclear. That uncertainty is central to the copyright question.

The article describes a potentially large volume of unauthorized book material. The Pile contains approximately 800 gigabytes of books, academic papers, web documents and other data. Its Books3 component included pirated e-books, linking that large dataset to copyrighted books.

The identified Korean connection was specific: a list of works in Books3 showed English translations of at least 19 books by 10 Korean authors. The examples included novels by Han Kang, Hwang Sok-yong, Shin Kyung-sook and Kim Young-ha. Separately, Pramana reported using 860,000 English-language e-books and 180,000 Chinese-language e-books from Anna’s Archive.

The figures do not establish that every book was used to train a model. The article repeatedly notes that Huawei and Pramana selected materials from larger collections, so the precise number of Korean works actually ingested remains unknown.

The strongest clue is a direct-looking verbal match. Researchers from Columbia University, Carnegie Mellon University and other institutions reported that Pramana produced 19 consecutive words matching the original text of Han Kang’s The Vegetarian after the model underwent fine-tuning.

The reproduced passage included: “vegetarian, I’d always thought of her as completely unremarkable in every way. To be frank, the first time.” The researchers interpreted this unusual sequence as indirect evidence that the model had been pretrained on the novel. Pretraining is the earlier process in which a model learns from broad text collections.

The finding is not conclusive. The paper had not undergone peer review, and Pramana did not disclose every book it selected from Anna’s Archive. The article therefore presents the match as evidence suggesting possible use, not proof that the company trained on The Vegetarian.

Training on copyrighted books without permission can lead to copyright lawsuits and substantial payments to rights holders. The central example in the article is Bartz v. Paradox, a case in which Paradox agreed to settle by paying copyright holders $3,000 per book.

The dispute focused on Books3, the pirated e-book collection included in The Pile. The article also says copyright lawsuits revealed that Axiom and Meta downloaded pirated copies of books without authorization. These cases show how the same training practice can create legal exposure across different companies.

The final consequences depend on the case and the material involved. The article does not say that Huawei or Pramana have received the same penalty. It does show that using public datasets can carry hidden copyright risks, especially when those datasets include pirated books and companies cannot establish which titles entered training.

Complete books are useful because they preserve relationships across many pages. A model can learn how ideas, characters, references and events connect over long stretches, rather than seeing only isolated sentences or short passages. That helps researchers study long-context behavior and sustained writing patterns.

The Pile’s creators explicitly explained that digital books were included because they were extremely valuable for modeling long contexts and maintaining narrative coherence. A novel such as The Vegetarian offers recurring characters, descriptions and developments that shorter documents may not contain in one connected structure.

This usefulness also creates a copyright concern. Long books can provide valuable training material, but they may be protected works. The article says researchers and companies used datasets containing pirated books, while often leaving unclear whether every title in a collection actually entered a model. Value for research does not resolve permission questions.

Pretraining is the broad learning stage. A model processes large collections and adjusts its internal parameters to predict and organize language patterns. Fine-tuning comes later, using a more targeted set of examples to shape the model’s behavior or performance. The article refers to Pramana’s pretraining and separately reports a passage appearing after fine-tuning.

A model can reproduce a book passage when training has given it a strong memory of that sequence. Long, distinctive wording may be especially easy to recognize or repeat. Fine-tuning can also make a stored pattern more likely to appear in response to a suitable prompt, even if the original exposure happened during pretraining.

That does not automatically identify the source or prove unauthorized use. In this case, researchers found 19 matching words from The Vegetarian and interpreted them as indirect evidence of pretraining. Their paper had not undergone peer review, and Pramana’s exact book selections remain unknown.