bnewbold

Archivist

UPLOADS

POSTS

REVIEWS

COLLECTIONS

LOANS

WEB ARCHIVES WEB

SHOW DETAILS

SORT BY

VIEWS

TITLE

DATE ARCHIVED

CREATOR

22 22

Scholarly TDM Corpora

ITEMS

VIEWS

Jul 14, 2021 07/21

by Internet Archive Web Group

eye 22

Access-restricted text and data-mining corpora. If you are interested in getting access to work with this content, contact info@archive.org

386,390 386K

PLATFORM-CRAWL-2020

649

ITEMS

386,390

VIEWS

Aug 27, 2020 08/20

by Internet Archive Web Group

eye 386,390

9.5M 9.5M

OA-JOURNAL-CRAWL-2020-07

1,923

ITEMS

9.5M

VIEWS

Jul 6, 2020 07/20

by Internet Archive Web Group

eye 9.5M

1.6M 1.6M

UNPAYWALL-PDF-CRAWL-2020-11

199

ITEMS

1.6M

VIEWS

Nov 2, 2020 11/20

by Internet Archive Web Group

eye 1.6M

6,476 6.5K

Crossref DOI Resolution Test Crawl (May 2018)

ITEMS

6,476

VIEWS

May 7, 2018 05/18

by Internet Archive Web Group

eye 6,476

This collection contains web crawl data for a random selection of 500k (0.5 million) Crossref DOI redirects, including the doi.org redirect requests. The intent of this crawl is to gather loose statistics on the number of failing redirects, number of host websites that block automated crawling, and a corpus of HTML landing pages for metadata extraction (eg, "signposting" HTTP headers, linked data HTML metadata, semantic markup). Total size of (uncompressed) WARC data is 50 GB,...

100.1M 100M

Internet Archive Research Publication Crawls

21,054

ITEMS

100.1M

VIEWS

Dec 19, 2017 12/17

by Internet Archive Web Group

eye 100.1M

A series of open web crawls targeting journal articles, technical memos, essays, datasets, and other research publications. This collection contains WARC and CDX files that end up in Wayback ( https://web.archive.org ). See also bibliographic metadata corpuses at  https://archive.org/details/ia_biblio_metadata

3M 3.0M

Wide Web Targeted PDF Crawling (2017)

922

ITEMS

VIEWS

Sep 21, 2017 09/17

by Internet Archive Web Group

eye 3M

1.4M 1.4M

OA-DOI-CRAWL-2020-12

191

ITEMS

1.4M

VIEWS

Dec 9, 2020 12/20

by Internet Archive Web Group

eye 1.4M

3.6M 3.6M

MAG-PDF-CRAWL-2020-03

489

ITEMS

3.6M

VIEWS

Mar 5, 2020 03/20

by Internet Archive Web Group

eye 3.6M

1.6M 1.6M

UNPAYWALL-PDF-CRAWL-2020-05

282

ITEMS

1.6M

VIEWS

May 4, 2020 05/20

by Internet Archive Web Group

eye 1.6M

847,305 847K

UNPAYWALL-PDF-CRAWL-2021-05

123

ITEMS

847,305

VIEWS

Apr 27, 2021 04/21

by Internet Archive Web Group

eye 847,305

11.6M 12M

MSAG-PDF-CRAWL-2017

1,855

ITEMS

11.6M

VIEWS

Aug 4, 2017 08/17

by Internet Archive Web Group

eye 11.6M

Microsoft Academic Graph public corpus (Feb 2016) PDF URLs, filtered to remove large sites (pubmed, citeseerx, arxiv) and already-crawled URLs.
Topics: papers, journals

1.6M 1.6M

MAG-PDF-CRAWL-2020-07

196

ITEMS

1.6M

VIEWS

Jul 9, 2020 07/20

by Internet Archive Web Group

eye 1.6M

188,303 188K

SCIELO-CRAWL-2020-07

ITEMS

188,303

VIEWS

Jul 6, 2020 07/20

by Internet Archive Web Group

eye 188,303

13,827 14K

OMICS-DOI-LANDING-CRAWL-2019-04

ITEMS

13,827

VIEWS

Apr 26, 2019 04/19

by Internet Archive Web Group

eye 13,827

This crawl started in April 2019, as an informal collaboration with Crossref. Crawling a smallish number (100k) DOI redirects and landing pages (plus PDF outlinks, and maybe a couple other hops) for a single large publisher (OMICS, which has multiple subsidiaries). Intent is to get reasonably good capture that can be used as canonical preservation copies of the landing pages. Secondary goal is to get decent fulltext capture coverage.

10.6M 11M

Open Access Journal Test Crawl (2018)

794

ITEMS

10.6M

VIEWS

Apr 9, 2018 04/18

by Internet Archive Web Group

eye 10.6M

17,205 17K

Bulk Bibliographic Metadata

223

ITEMS

17,205

VIEWS

Dec 14, 2017 12/17

by Internet Archive Web Group

eye 17,205

This collection contains both external ("upstream") metadata dumps and Internet Archive generated databases and reports on our holdings of papers, books, and other documents.

17 17

Web PDF GROBID Corpus (July 2019)

ITEMS

VIEWS

Jul 30, 2019 07/19

by Internet Archive Web Group

Crawl of "upstream" URLs from CORE (core.ac.uk) metadata dump. Only a partial seedlist of files crawled.

858,478 858K

DOAJ-CRAWL-2020-11

102

ITEMS

858,478

VIEWS

Nov 24, 2020 11/20

by Internet Archive Web Group

eye 858,478

1.4M 1.4M

SEMSCHOLAR-DIRECT-PDF-CRAWL-2020-02

1,011

ITEMS

1.4M

VIEWS

Feb 6, 2020 02/20

by Internet Archive Web Group

eye 1.4M

5.2M 5.2M

UNPAYWALL-PDF-CRAWL-2019-04

641

ITEMS

5.2M

VIEWS

Apr 26, 2019 04/19

by Internet Archive Web Group

eye 5.2M

1.8M 1.8M

UNPAYWALL-PDF-CRAWL-2020-03

344

ITEMS

1.8M

VIEWS

Mar 5, 2020 03/20

101,993 102K

ARXIV-PUBMEDCENTRAL-CRAWL-2020-04

ITEMS

101,993

VIEWS

Apr 24, 2020 04/20

by Internet Archive Web Group

Web archive data from a crawl of open access PDF URLs provided by Unpaywall.

403,156 403K

PubMed Central Crawl (2019-10)

216

ITEMS

403,156

VIEWS

Oct 12, 2019 10/19

by Internet Archive Web Group

eye 403,156

2.7M 2.7M

OA-JOURNAL-CRAWL-2019-08

201

ITEMS

2.7M

VIEWS

Aug 1, 2019 08/19

by Internet Archive Web Group

eye 2.7M

3.6M 3.6M

DATACITE-DOI-CRAWL-2020-01

1,417

ITEMS

3.6M

VIEWS

Jan 24, 2020 01/20

by Internet Archive Web Group

eye 3.6M

MORE RESULTS
Fetching more results

Subject	Poster	Forum	Replies	Date

View more forum posts

bnewbold

Archivist

Filters (2) 37 UPLOADS

Media TypeMedia Type

YearYear

Topics & SubjectsTopics & Subjects

CollectionCollection

CreatorCreator

eye 22

eye 386,390

eye 9.5M

eye 1.6M

eye 6,476

eye 100.1M

eye 3M

eye 1.4M

eye 3.6M

eye 1.6M

eye 847,305

eye 11.6M

eye 1.6M

eye 188,303

eye 13,827

eye 10.6M

eye 17,205

eye 17

eye 64,880

eye 3.2M

eye 1.5M

eye 858,478

eye 1.4M

eye 5.2M

eye 1.8M

eye 49

eye 4.8M

eye 3.2M

eye 101,993

eye 1,358

eye 178

eye 5.1M

eye 234,015

eye 14.4M

eye 403,156

eye 2.7M

eye 3.6M

Posts by bnewbold

37
UPLOADS

Media Type

Year

Topics & Subjects

Collection

Creator