Skip to main content

Internet Archive Web Crawls

The Internet Archive discovers and captures web pages through many different web crawls.



rss RSS

1,708,311
RESULTS


Show sorted alphabetically

Show sorted alphabetically

SHOW DETAILS
up-solid down-solid
eye
Title
Date Archived
Creator
Worldwide Web Crawls
Worldwide Web Crawls
collection
634,931
ITEMS
19.9B
VIEWS
collection

eye 19.9B

Wide crawls of the Internet conducted by Internet Archive. Please visit the Wayback Machine to explore archived web sites. Since September 10th, 2010, the Internet Archive has been running Worldwide Web Crawls of the global web, capturing web elements, pages, sites and parts of sites. Each Worldwide Web Crawl was initiated from one or more lists of URLs that are known as "Seed Lists". Descriptions of the Seed Lists associated with each crawl may be provided as part of the metadata for...
Survey Crawls
Survey Crawls
collection
100,903
ITEMS
12B
VIEWS
collection

eye 12B

Survey crawls are run about twice a year, on average, and attempt to capture the content of the front page of every web host ever seen by the Internet Archive since 1996.
Topic: survey crawls
Live Web Proxy Crawls
Live Web Proxy Crawls
collection
108,965
ITEMS
11.9B
VIEWS
collection

eye 11.9B

Content crawled via the Wayback Machine Live Proxy mostly by the Save Page Now feature on web.archive.org. Liveweb proxy is a component of Internet Archive’s wayback machine project. The liveweb proxy captures the content of a web page in real time, archives it into a ARC or WARC file and returns the ARC/WARC record back to the wayback machine to process. The recorded ARC/WARC file becomes part of the wayback machine in due course of time.
Wide Crawl Number 17: Started August 3rd, 2018
Wide Crawl Number 17: Started August 3rd, 2018
collection
69,709
ITEMS
2.3B
VIEWS
collection

eye 2.3B

Wide17 was seeded with the "Total Domains" list of 256,796,456 URLs provided by  Domains Index   on June 26th, and crawled with max-hops set to "3" and de-duplication set "on".   
collection

eye 2.7B

The seed for Wide00014 was: - Slash pages from every domain on the web: -- a list of domains using Survey crawl seeds -- a list of domains using Wide00012 web graph -- a list of domains using Wide00013 web graph - Top ranked pages (up to a max of 100) from every linked-to domain using the Wide00012 inter-domain navigational link graph -- a ranking of all URLs that have more than one incoming inter-domain link (rank was determined by number of incoming links using Wide00012 inter domain links)...
collection

eye 2.2B

The seed for this crawl was a list of every host in the Wayback Machine This crawl was run at a level 1 (URLs including their embeds, plus the URLs of all outbound links including their embeds) The WARC files associated with this crawl are not currently available to the general public.
collection

eye 1.6B

Web wide crawl.
Survey Crawl Number 8
collection
8,758
ITEMS
1.4B
VIEWS
collection

eye 1.4B

collection

eye 1.4B

Web wide crawl number 16 The seed list for Wide00016 was made from the join of the top 1 million domains from CISCO and the top 1 million domains from Alexa.
GDELT
GDELT
collection
57,657
ITEMS
1.3B
VIEWS
collection

eye 1.3B

A daily crawl of more than 200,000 home pages of news sites, including the pages linked from those home pages. Site list provided by The GDELT Project
Topics: GDELT, News
collection

eye 1.4B

The seeds for this crawl came from: 251 million Domains that had at least one link from a different domain in the Wayback Machine, across all time ~ 300 million Domains that we had in the Wayback, across all time 55,945,067 Domains from https://archive.org/details/wide00016 This crawl was run with a Heritrix setting of "maxHops=0" (URLs including their embeds) The WARC files associated with this crawl are not currently available to the general public.
collection

eye 1.5B

The seed for this crawl was a list of every host in the Wayback Machine This crawl was run at a level 1 (URLs including their embeds, plus the URLs of all outbound links including their embeds) The WARC files associated with this crawl are not currently available to the general public.
Wide Crawl Number 12 - started March, 14th 2015
Wide Crawl Number 12 - started March, 14th 2015
collection
49,621
ITEMS
1.5B
VIEWS
collection

eye 1.5B

Web wide crawl with initial seedlist and crawler configuration from January 2015.
Wide Crawl started June 2014
Wide Crawl started June 2014
collection
45,341
ITEMS
1.4B
VIEWS
collection

eye 1.4B

Web wide crawl with initial seedlist and crawler configuration from June 2014.
Wide Crawl started April 2013
Wide Crawl started April 2013
collection
25,035
ITEMS
1.5B
VIEWS
collection

eye 1.5B

Web wide crawl with initial seedlist and crawler configuration from April 2013.
collection

eye 1.2B

The seed for this crawl was a list of every host in the Wayback Machine This crawl was run at a level 1 (URLs including their embeds, plus the URLs of all outbound links including their embeds) The WARC files associated with this crawl are not currently available to the general public.
Wikipedia Outlinks March 2016
Wikipedia Outlinks March 2016
collection
27,594
ITEMS
1B
VIEWS
collection

eye 1B

Crawl of outlinks from wikipedia.org started March, 2016. These files are currently not publicly accessible. Properties of this collection. It has been several years since the last time we did this. For this collection, several things were done: 1. Turned off duplicate detection. This collection will be complete, as there is a good chance we will share the data, and sharing data with pointers to random other collections, is a complex problem. 2. For the first time, did all the different wikis....
Survey Crawl Number 7
Survey Crawl Number 7
collection
6,605
ITEMS
996.8M
VIEWS
collection

eye 996.8M

This "Survey" crawl was started on Feb. 24, 2018. This crawl was run with a Heritrix setting of "maxHops=0" (URLs including their embeds) Survey 7 is based on a seed list of 339,249,218 URLs which is all the URLs in the Wayback Machine that we saw a 200 response code from in 2017 based on a query we ran on Feb. 1st, 2018.   The WARC files associated with this crawl are not currently available to the general public.
Wide Crawl Number 13
Wide Crawl Number 13
collection
46,050
ITEMS
1.1B
VIEWS
collection

eye 1.1B

Web Wide Crawl Number 13
Wayback Indexes
Wayback Indexes
collection
554
ITEMS
1.2B
VIEWS
collection

eye 1.2B

Wayback indexes. This data is currently not publicly accessible.
Wide Crawl started August 2013
Wide Crawl started August 2013
collection
21,932
ITEMS
984.3M
VIEWS
collection

eye 984.3M

Web wide crawl with initial seedlist and crawler configuration from August 2013.
collection

eye 856.8M

The seed for this crawl was a list of every host in the Wayback Machine This crawl was run at a level 1 (URLs including their embeds, plus the URLs of all outbound links including their embeds) The WARC files associated with this crawl are not currently available to the general public.
.com survey started January 2011
.com survey started January 2011
collection
2,535
ITEMS
570.4M
VIEWS
collection

eye 570.4M

Survey crawl of .com domains started January 2011.
Topic: webcrawl
collection

eye 737.3M

The seed for this crawl was a list of every host in the Wayback Machine This crawl was run at a level 1 (URLs including their embeds, plus the URLs of all outbound links including their embeds) The WARC files associated with this crawl are not currently available to the general public.
Wide Crawl started January 2012
Wide Crawl started January 2012
collection
30,373
ITEMS
848.8M
VIEWS
collection

eye 848.8M

Web wide crawl with initial seedlist and crawler configuration from January 2012 using HQ software.
Wide Crawl started April 2012
Wide Crawl started April 2012
collection
39,279
ITEMS
744.5M
VIEWS
collection

eye 744.5M

Web wide crawl with initial seedlist and crawler configuration from April 2012.
Wide Crawl started February 2014
Wide Crawl started February 2014
collection
9,806
ITEMS
636.2M
VIEWS
collection

eye 636.2M

Web wide crawl with initial seedlist and crawler configuration from February 2014.
Wide Crawl started September 2012
Wide Crawl started September 2012
collection
22,423
ITEMS
553.2M
VIEWS
collection

eye 553.2M

Web wide crawl with initial seedlist and crawler configuration from September 2012.
Wide Crawl started October 2010
Wide Crawl started October 2010
collection
15,839
ITEMS
571M
VIEWS
collection

eye 571M

Web wide crawl with initial seedlist and crawler configuration from October 2010
Wide Crawl Started January 2013
Wide Crawl Started January 2013
collection
15,157
ITEMS
557.2M
VIEWS
collection

eye 557.2M

Wide crawls of the Internet conducted by Internet Archive. Access to content is restricted. Please visit the Wayback Machine to explore archived web sites.
collection

eye 427.7M

The seed for this crawl was a list of every host in the Wayback Machine This crawl was run at a level 1 (URLs including their embeds, plus the URLs of all outbound links including their embeds) The WARC files associated with this crawl are not currently available to the general public.
Wide Crawl started October 2011
Wide Crawl started October 2011
collection
12,648
ITEMS
530.9M
VIEWS
collection

eye 530.9M

Web wide crawl with initial seedlist and crawler configuration from March 2011 using HQ software.
Host Screen Captures
Host Screen Captures
collection
17,458
ITEMS
239.8M
VIEWS
collection

eye 239.8M

Screen captures of hosts discovered during wide crawls. This data is currently not publicly accessible.
Survey Crawl Number 9
collection
561
ITEMS
246.5M
VIEWS
collection

eye 246.5M

Wide Crawl started March 2011
Wide Crawl started March 2011
collection
8,528
ITEMS
486.6M
VIEWS
collection

eye 486.6M

Web wide crawl with initial seedlist and crawler configuration from March 2011. This uses the new HQ software for distributed crawling by Kenji Nagahashi. What’s in the data set: Crawl start date: 09 March, 2011 Crawl end date: 23 December, 2011 Number of captures: 2,713,676,341 Number of unique URLs: 2,273,840,159 Number of hosts: 29,032,069 The seed list for this crawl was a list of Alexa’s top 1 million web sites, retrieved close to the crawl start date. We used Heritrix (3.1.1-SNAPSHOT)...
International News Crawls
International News Crawls
collection
13,331
ITEMS
298.9M
VIEWS
collection

eye 298.9M

Crawls of International News Sites
Around The World Crawl
Around The World Crawl
collection
2,150
ITEMS
443.5M
VIEWS
collection

eye 443.5M

Data crawled by Sloan Foundation on behalf of Internet Archive
Wikipedia Outlinks February 2012
Wikipedia Outlinks February 2012
collection
2,951
ITEMS
387.5M
VIEWS
collection

eye 387.5M

Crawl of outlinks from wikipedia.org started February, 2012. These files are currently not publicly accessible.
Collections News Crawls v3
Collections News Crawls v3
collection
11,215
ITEMS
173.6M
VIEWS
by ximm@archive.org
collection

eye 173.6M

Miscellaneous high-value news sites 
Topics: World news, US news, news
Live Web Proxy Crawls
web

eye 152.5M

favorite 5

comment 2

favoritefavoritefavoritefavoritefavorite ( 2 reviews )
Wide Crawl Number 18
Wide Crawl Number 18
collection
7,442
ITEMS
62M
VIEWS
collection

eye 62M

Hackernews crawl number 0
collection
6,164
ITEMS
192.3M
VIEWS
collection

eye 192.3M

Hacker News Crawl of their links.
End of Term Web Crawls
End of Term Web Crawls
collection
31,917
ITEMS
138.8M
VIEWS
by IA
collection

eye 138.8M

This collection includes web crawls of the Federal Executive, Legislative, and Judicial branches of government performed at the end of US presidential terms of office.
Topics: web, end of term, US, federal government
Wikipedia Outlinks May 2011
Wikipedia Outlinks May 2011
collection
1,638
ITEMS
192.2M
VIEWS
collection

eye 192.2M

Crawl of outlinks from wikipedia.org started May, 2011. These files are currently not publicly accessible.
Shallow Crawls
Shallow Crawls
collection
1,042
ITEMS
187.2M
VIEWS
collection

eye 187.2M

Shallow crawls that collect content 1 level deep including embeds. This data is currently not publicly accessible.
Youtube Videos
Youtube Videos
collection
746,390
ITEMS
109.9M
VIEWS
collection

eye 109.9M

Captures of pages from YouTube. Currently these are discovered by searching for YouTube links on Twitter.
Topics: YouTube, Twitter, Video
Geocities Closing Crawl
Geocities Closing Crawl
collection
149
ITEMS
110.6M
VIEWS
collection

eye 110.6M

Geocities crawl performed by Internet Archive. This data is currently not publicly accessible. from Wikipedia : Yahoo! GeoCities is a Web hosting service. GeoCities was originally founded by David Bohnett and John Rezner in late 1994 as Beverly Hills Internet (BHI), and by 1999 GeoCities was the third-most visited Web site on the World Wide Web. In its original form, site users selected a "city" in which to place their Web pages. The "cities" were metonymously named after...
Wikipedia Outlinks July 2011
Wikipedia Outlinks July 2011
collection
1,011
ITEMS
133.2M
VIEWS
collection

eye 133.2M

Crawl of outlinks from wikipedia.org started July, 2011. These files are currently not publicly accessible.
Wayback CDX Shards
Wayback CDX Shards
collection
1,214
ITEMS
159.2M
VIEWS
collection

eye 159.2M

CDX Index shards for the Wayback Machine. The Wayback Machine works by looking for historic URL's based on a query. This is done by searching an index of all the web objects (pages, images, etc) that have been archived over the years. This collection holds the index used for this purpose, which is broken up into 300 pieces so they fit into items more naturally and distribute the lookup load. Each of these 300 pieces is stored in at least 2 items, and then those are also stored on the backup...
Live Web Proxy Crawls
web

eye 39.7M

favorite 0

comment 1

favoritefavoritefavoritefavoritefavorite ( 1 reviews )
End Of Term 2020 Web Crawls
End Of Term 2020 Web Crawls
collection
29,131
ITEMS
23.8M
VIEWS
collection

eye 23.8M

COM Survey Crawl 2009-2010
COM Survey Crawl 2009-2010
collection
729
ITEMS
86.9M
VIEWS
collection

eye 86.9M

COM survey crawl data collected by Internet Archive in 2009-2010. This data is currently not publicly accessible.
survey_net00000
survey_net00000
collection
300
ITEMS
70.9M
VIEWS
collection

eye 70.9M

Survey crawl of .net domains started December 2010.
Topic: webcrawl
Shallow Crawl Started 2013
Shallow Crawl Started 2013
collection
544
ITEMS
75.7M
VIEWS
collection

eye 75.7M

Shallow crawl started 2013 that collects content 1 level deep, including embeds. Access to content is restricted. Please visit the Wayback Machine to explore archived web sites.
Shallow Crawl Started 2013
Shallow Crawl Started 2013
collection
252
ITEMS
76M
VIEWS
collection

eye 76M

Shallow crawl started 2013 that collects content 1 level deep, including embeds. Access to content is restricted. Please visit the Wayback Machine to explore archived web sites.
collection

eye 24.1M

This collection contains web crawls performed on the US Federal Executive, Legislative & Judicial branches of government in 2020-2021.   Information about this project can be found here: https://end-of-term.github.io/eotarchive/   You can submit URLs to be archived here: https://digital2.library.unt.edu/nomination/eth2020/add/
End of Term 2016 Web Crawls
End of Term 2016 Web Crawls
collection
19,083
ITEMS
49.9M
VIEWS
collection

eye 49.9M

This collection contains web crawls performed as part of the End of Term Web Archive, a collaborative project that aims to preserve the U.S. federal government web presence at each change of administration. Content includes publicly-accessible government websites hosted on .gov, .mil, and relevant non-.gov domains, as well as government social media materials. The web archiving was performed in the Fall and Winter of 2016 and Spring of 2017. For more information, see...
Topics: end of term, federal government, 2016, president, congress, government data
nsdlweb
collection
91
ITEMS
53.4M
VIEWS
collection

eye 53.4M

this data is currently not publicly accessible.
ORG Survey Crawls
ORG Survey Crawls
collection
191
ITEMS
37.8M
VIEWS
collection

eye 37.8M

Survey of .org domains. This data is currently not publicly accessible.
Collections news crawls v2
collection
820
ITEMS
44.3M
VIEWS
by ximm@archive.org
collection

eye 44.3M

Live Web Proxy Crawls
web

eye 7M

favorite 0

comment 0

International News Crawl started in September 2010
International News Crawl started in September 2010
collection
1,083
ITEMS
41.8M
VIEWS
collection

eye 41.8M

Crawl of International News Sites with initial seedlist and crawler configuration from Sep 1, 2010.
Internet Archive crawldata from Survey Webwide Crawl, captured by crawl800.us.archive.org:survey from Fri Jan 29 19:24:19 PST 2016 to Fri Jan 29 15:37:03 PST 2016.
Topic: crawldata
Live Web Proxy Crawls
web

eye 6.8M

favorite 0

comment 0

End of Term 2012 Web Crawls
End of Term 2012 Web Crawls
collection
2,383
ITEMS
37M
VIEWS
collection

eye 37M

This collection contains web crawls performed on the US Federal Executive, Legislative & Judicial branches of government in 2012-2013.
Topics: end of term, US, Federal government, 2012, Obama
Internet Archive crawldata from Webwide Crawl, captured by crawl427.us.archive.org:wide from Sun Oct 16 02:25:38 PDT 2016 to Sat Oct 15 21:19:02 PDT 2016.
Topic: crawldata
survey_net00001
collection
170
ITEMS
25.6M
VIEWS
collection

eye 25.6M

Survey crawl of .net domains started October 2011.
Topics: webwidecrawl, net
End Of Term 2016 Pre-Inauguration Crawls
End Of Term 2016 Pre-Inauguration Crawls
collection
4,693
ITEMS
18.6M
VIEWS
collection

eye 18.6M

This collection contains web crawls performed as the pre-inauguration crawl for part of the End of Term Web Archive, a collaborative project that aims to preserve the U.S. federal government web presence at each change of administration. Content includes publicly-accessible government websites hosted on .gov, .mil, and relevant non-.gov domains, as well as government social media materials. The web archiving was performed in the Fall and Winter of 2016 to capture websites prior to the January...
Topics: end of term, federal government, 2016, president, congress
Edu & Gov Crawl, June 2010
Edu & Gov Crawl, June 2010
collection
704
ITEMS
23.1M
VIEWS
collection

eye 23.1M

TEST COLLECTION: Crawl of .edu and .gov sites started in June 2010.
Topic: crawldata
End Of Term 2020 Post Inauguration Crawls
End Of Term 2020 Post Inauguration Crawls
collection
19,102
ITEMS
8.2M
VIEWS
collection

eye 8.2M

End Of Term 2016 UNT Crawls
End Of Term 2016 UNT Crawls
collection
1,275
ITEMS
14.6M
VIEWS
collection

eye 14.6M

End of Term 2016 Web Archive government web crawls by project partner the University of North Texas.
Topics: end of term, federal government, 2016, president, congress, university of north texas
Survey Crawl Number 8
web

eye 3M

favorite 0

comment 0

Internet Archive crawldata from Survey Webwide Crawl, captured by crawl836.us.archive.org:survey from Fri Feb 22 02:17:40 PST 2019 to Thu Feb 21 23:46:50 PST 2019.
Topic: crawldata
Shallow Crawl Started February 2013
Shallow Crawl Started February 2013
collection
86
ITEMS
18.8M
VIEWS
collection

eye 18.8M

Shallow crawl started February 2013 that collects content 1 level deep, including embeds. Access to content is restricted. Please visit the Wayback Machine to explore archived web sites.
Crawl Data
Crawl Data
collection
32,956
ITEMS
22M
VIEWS
collection

eye 22M

Crawl Data. This data is currently not publicly accessible.
crawl_UNK
crawl_UNK
collection
32,949
ITEMS
22M
VIEWS
collection

eye 22M

Crawl data. This data is currently not publicly accessible.