Internet Archive Under Attack: Why Big Publishers Want to Block the Wayback Machine
The preservation of the web's historical memory is facing an unprecedented crisis. The Internet Archive, through its renowned Wayback Machine, finds itself in the paradoxical position of being both a tool for investigative journalism and, at the same time, the target of systematic blocks by the very publishers who could potentially benefit from it. According to data collected by the startup Originality AI, cited by Wired, there are currently 23 major news outlets that have decided to inhibit access to ia_archiverbot, the web crawler responsible for indexing and saving historical versions of web pages. Recently, the social platform Reddit has also joined this list.
The most emblematic case involves USA Today: the outlet recently published a high-profile report on the impact of ICE detention policies (the American federal agency responsible for immigration control and the deportation of undocumented immigrants). To carry out this analysis, journalists extensively used the Wayback Machine, utilizing it to track changes in statistics and communications from the government agency over the years.
Despite this deep and instrumental use, the publishing group actively prevents the Internet Archive from archiving its content. Mark Graham, director of the Wayback Machine, pointed out how this dichotomy is a logical contradiction: newsrooms exploit the archive for research but deny it sustenance by blocking their servers.
Publishing Raises Walls Against ia_archiverbot, Between Scraping and Copyright Defense
The official justification provided by USA Today lies in the need to counter indiscriminate scraping. The publisher states that it does not want to specifically target the Internet Archive but has implemented restrictive measures against all automated bots to protect its infrastructure and digital assets. However, this position ignores the technical distinction between crawlers used for training generative artificial intelligence and the historical preservation activities carried out by the San Francisco foundation.
As emphasized in an open letter promoted by organizations such as the Electronic Frontier Foundation and Public Knowledge, available in full here, the Internet Archive adopts a respectful approach that does not include bypassing paywalls or the wild scraping typical of AI companies. The fear among publishers is linked to the fact that responses provided by AI agents could cannibalize traffic to source sites, reducing advertising revenue. However, the Wayback Machine operates on a different level, serving as a digital library and ensuring the persistence of over 5 million bibliographic citations on Wikipedia that would otherwise be lost due to the gradual deterioration of hyperlinks.
The reaction from the professional base has been swift. Over 100 signatures from prominent journalists have been collected at the bottom of a document that defines the Internet Archive as the only heir to the old physical archives of local newspapers. With the systematic closure of many local outlets and the absence of clear protocols for the preservation of "digital-only" publications by public libraries, the responsibility for safeguarding the chronicle largely falls on this nonprofit infrastructure.
The concrete risk is that large segments of recent history may become inaccessible, erased by short-term business logic or simple technical restructurings. If crawlers are blocked, the ability to verify governments' past statements or monitor the evolution of news diminishes, weakening one of the pillars of democratic transparency. The request of the signatories is simple: distinguish between those who exploit content for technological profit and those, like the Internet Archive, who ensure its availability for researchers and future citizens.