bnewbold

Archivist

UPLOADS

POSTS

REVIEWS

COLLECTIONS

LOANS

WEB ARCHIVES WEB

SHOW DETAILS

SORT BY

VIEWS

TITLE

DATE ARCHIVED

CREATOR

4.2M 4.2M

MAG-PDF-CRAWL-2020-03

489

ITEMS

4.2M

VIEWS

Mar 5, 2020 03/20

by Internet Archive Web Group

eye 4.2M

1.8M 1.8M

UNPAYWALL-PDF-CRAWL-2020-05

282

ITEMS

1.8M

VIEWS

May 4, 2020 05/20

by Internet Archive Web Group

eye 1.8M

3.2M 3.2M

Wide Web Targeted PDF Crawling (2017)

922

ITEMS

3.2M

VIEWS

Sep 21, 2017 09/17

by Internet Archive Web Group

eye 3.2M

1.6M 1.6M

OA-DOI-CRAWL-2020-12

191

ITEMS

1.6M

VIEWS

Dec 9, 2020 12/20

by Internet Archive Web Group

eye 1.6M

22 22

Scholarly TDM Corpora

ITEMS

VIEWS

Jul 14, 2021 07/21

by Internet Archive Web Group

eye 22

Access-restricted text and data-mining corpora. If you are interested in getting access to work with this content, contact info@archive.org

10.7M 11M

OA-JOURNAL-CRAWL-2020-07

1,923

ITEMS

10.7M

VIEWS

Jul 6, 2020 07/20

by Internet Archive Web Group

eye 10.7M

2.9M 2.9M

OA-JOURNAL-CRAWL-2019-08

201

ITEMS

2.9M

VIEWS

Aug 1, 2019 08/19

by Internet Archive Web Group

eye 2.9M

456,372 456K

PubMed Central Crawl (2019-10)

216

ITEMS

456,372

VIEWS

Oct 12, 2019 10/19

by Internet Archive Web Group

eye 456,372

522,698 523K

PLATFORM-CRAWL-2020

649

ITEMS

522,698

VIEWS

Aug 27, 2020 08/20

by Internet Archive Web Group

eye 522,698

4M 4.0M

DATACITE-DOI-CRAWL-2020-01

1,417

ITEMS

VIEWS

Jan 24, 2020 01/20

by Internet Archive Web Group

eye 4M

15.8M 16M

UNPAYWALL-PDF-CRAWL-2018-07

1,241

ITEMS

15.8M

VIEWS

Jul 17, 2018 07/18

by Internet Archive Web Group

eye 15.8M

Web archive data from a crawl of open access PDF URLs provided by Unpaywall.

17 17

Web PDF GROBID Corpus (July 2019)

ITEMS

VIEWS

Jul 30, 2019 07/19

by Internet Archive Web Group

eye 17

973,161 973K

UNPAYWALL-PDF-CRAWL-2021-05

123

ITEMS

973,161

VIEWS

Apr 27, 2021 04/21

by Internet Archive Web Group

eye 973,161

1.8M 1.8M

UNPAYWALL-PDF-CRAWL-2020-11

199

ITEMS

1.8M

VIEWS

Nov 2, 2020 11/20

by Internet Archive Web Group

eye 1.8M

6,644 6.6K

Crossref DOI Resolution Test Crawl (May 2018)

ITEMS

6,644

VIEWS

May 7, 2018 05/18

by Internet Archive Web Group

eye 6,644

This collection contains web crawl data for a random selection of 500k (0.5 million) Crossref DOI redirects, including the doi.org redirect requests. The intent of this crawl is to gather loose statistics on the number of failing redirects, number of host websites that block automated crawling, and a corpus of HTML landing pages for metadata extraction (eg, "signposting" HTTP headers, linked data HTML metadata, semantic markup). Total size of (uncompressed) WARC data is 50 GB,...

1.8M 1.8M

MAG-PDF-CRAWL-2020-07

196

ITEMS

1.8M

VIEWS

Jul 9, 2020 07/20

by Internet Archive Web Group

eye 1.8M

204,694 205K

SCIELO-CRAWL-2020-07

ITEMS

204,694

VIEWS

Jul 6, 2020 07/20

by Internet Archive Web Group

eye 204,694

14,146 14K

OMICS-DOI-LANDING-CRAWL-2019-04

ITEMS

14,146

VIEWS

Apr 26, 2019 04/19

by Internet Archive Web Group

eye 14,146

This crawl started in April 2019, as an informal collaboration with Crossref. Crawling a smallish number (100k) DOI redirects and landing pages (plus PDF outlinks, and maybe a couple other hops) for a single large publisher (OMICS, which has multiple subsidiaries). Intent is to get reasonably good capture that can be used as canonical preservation copies of the landing pages. Secondary goal is to get decent fulltext capture coverage.

11.5M 12M

Open Access Journal Test Crawl (2018)

794

ITEMS

11.5M

VIEWS

Apr 9, 2018 04/18

by Internet Archive Web Group

eye 11.5M

178 178

Web PDF Training Sets

ITEMS

178

VIEWS

Jun 12, 2019 06/19

by Internet Archive Web Group

eye 178

79,989 80K

arXiv Content Crawl (2019-10)

ITEMS

79,989

VIEWS

Oct 12, 2019 10/19

by Internet Archive Web Group

eye 79,989

1.6M 1.6M

SEMSCHOLAR-DIRECT-PDF-CRAWL-2020-02

1,011

ITEMS

1.6M

VIEWS

Feb 6, 2020 02/20

by Internet Archive Web Group

eye 1.6M

959,382 959K

DOAJ-CRAWL-2020-11

102

ITEMS

959,382

VIEWS

Nov 24, 2020 11/20

by Internet Archive Web Group

eye 959,382

5.9M 5.9M

UNPAYWALL-PDF-CRAWL-2019-04

641

ITEMS

5.9M

VIEWS

Apr 26, 2019 04/19

by Internet Archive Web Group

eye 5.9M

2M 2.0M

UNPAYWALL-PDF-CRAWL-2020-03

344

ITEMS

VIEWS

Mar 5, 2020 03/20

by Internet Archive Web Group

eye 2M

13M 13M

MSAG-PDF-CRAWL-2017

1,855

ITEMS

13M

VIEWS

Aug 4, 2017 08/17

by Internet Archive Web Group

eye 13M

Microsoft Academic Graph public corpus (Feb 2016) PDF URLs, filtered to remove large sites (pubmed, citeseerx, arxiv) and already-crawled URLs.
Topics: papers, journals

50 50

Web PDF GROBID Corpus (June 2019)

ITEMS

VIEWS

Jun 7, 2019 06/19

by Internet Archive Web Group

eye 50

262,794 263K

PUBMEDCENTRAL-CRAWL-2020-02

108

ITEMS

262,794

VIEWS

Feb 5, 2020 02/20

by Internet Archive Web Group

eye 262,794

5.6M 5.6M

DIRECT-OA-CRAWL-2019

2,566

ITEMS

5.6M

VIEWS

Feb 15, 2019 02/19

by Internet Archive Web Group

eye 5.6M

1.9M 1.9M

CORE-UPSTREAM-CRAWL-2018-11

741

ITEMS

1.9M

VIEWS

Oct 31, 2018 10/18

by Internet Archive Web Group

eye 1.9M

Crawl of "upstream" URLs from CORE (core.ac.uk) metadata dump. Only a partial seedlist of files crawled.

3.4M 3.4M

DOI-LANDING-CRAWL-2018-06

279

ITEMS

3.4M

VIEWS

Jun 1, 2018 06/18

by Internet Archive Web Group

eye 3.4M

3.5M 3.5M

OA-DOI-CRAWL-2020-02

278

ITEMS

3.5M

VIEWS

Feb 5, 2020 02/20

by Internet Archive Web Group

eye 3.5M

5.9M 5.9M

OAI-PMH-CRAWL-2020-06

2,946

ITEMS

5.9M

VIEWS

May 28, 2020 05/20

by Internet Archive Web Group

eye 5.9M

114,355 114K

ARXIV-PUBMEDCENTRAL-CRAWL-2020-04

ITEMS

114,355

VIEWS

Apr 24, 2020 04/20

by Internet Archive Web Group

eye 114,355

112.5M 113M

Internet Archive Research Publication Crawls

21,177

ITEMS

112.5M

VIEWS

Dec 19, 2017 12/17

by Internet Archive Web Group

eye 112.5M

A series of open web crawls targeting journal articles, technical memos, essays, datasets, and other research publications. This collection contains WARC and CDX files that end up in Wayback ( https://web.archive.org ). See also bibliographic metadata corpuses at  https://archive.org/details/ia_biblio_metadata

MORE RESULTS
Fetching more results

Subject	Poster	Forum	Replies	Date

View more forum posts

bnewbold

Archivist

Filters (2) 35 UPLOADS

Media TypeMedia Type

YearYear

Topics & SubjectsTopics & Subjects

CollectionCollection

CreatorCreator

eye 4.2M

eye 1.8M

eye 3.2M

eye 1.6M

eye 22

eye 10.7M

eye 2.9M

eye 456,372

eye 522,698

eye 4M

eye 15.8M

eye 17

eye 973,161

eye 1.8M

eye 6,644

eye 1.8M

eye 204,694

eye 14,146

eye 11.5M

eye 178

eye 79,989

eye 1.6M

eye 959,382

eye 5.9M

eye 2M

eye 13M

eye 50

eye 262,794

eye 5.6M

eye 1.9M

eye 3.4M

eye 3.5M

eye 5.9M

eye 114,355

eye 112.5M

Posts by bnewbold

35
UPLOADS

Media Type

Year

Topics & Subjects

Collection

Creator