A growing list of the public sources for datasets relevant to the Internet History Initiative.

These sections are a collection of minimally-documented deep links into public repositories maintained by their originating institutions. Their inclusion in this index does not imply endorsement of this project by any of these institutions.

This rough index is supplied purely as a convenience for researchers, in preparation for constructing a more carefully curated index (and potential mirrored storage) to follow.

Please consult the original sources for complete documentation, and abide by their original acceptable use policies and licenses in your use of these datasets. Finally, please support these organizations and let them know that you appreciate their continuing stewardship and research over the years!

Public sources of historical BGP data, generally RIBs and Updates.

Some sources of BGP RIBs and updates are known to have existed, but no known copy exists. If you have a copy of any of these, please share!

“Research Packet Clearing House investigates technological, economic, and policy issues in areas related to Internet traffic exchange.

“PCH’s longest-running research project is the Internet Routing Topology Archive, a database of Internet topology measurements begun in 1997. This archive of routing data from all major and many minor Internet provider networks is available to academic and commercial researchers and the operations community, to aid in the understanding of the dynamic nature and topology of the Internet.

“Other topics of ongoing research include the economic impact of local traffic exchange in developing nations, inter-provider notifications and communication, and database schemas for topological data.

“Packet Clearing House facilitates research, instruments the Internet, collects, archives, and disseminates information and data, and creates a climate conducive to analytical examination of all aspects of Internet topology, operational practice, and economics. Although we have collected and maintain the world’s largest database of Internet routing information, we prefer to put researchers in academia together with the data that they need, rather than analyze all the data we collect ourselves - a task beyond the capacity of any one organization. We believe that, by facilitating partnerships between industry and academia and enabling communication between the two communities, we can achieve a more enduring beneficial effect.”

• Source
• License
• Raw data

“The RIPE Routing Information Service (RIS) is a RIPE NCC service. With the help of network operators all over the world, RIS employs a globally distributed set of Remote Route Collectors (RRCs), typically located at Internet Exchange Points, to collect and store Internet routing data. Volunteers peer with the RRCs using the BGP protocol and RIS stores the update and withdraw messages. RIS data can be accessed via:

• RIPEstat, the “one-stop shop” for all available information about Internet number resources. RIPEstat uses individual widgets to display routing and other information;
• RIS Live, a real time BGP streaming API allowing server-side filtering of BGP messages by prefix or autonomous system;
• RIS Raw Data, available for each route collector, with state dumps and batches of updates made available periodically;
• RISwhois, that searches the latest RIS data for details of an IP address using a plaintext “whois”-style interface. It is useful when querying RIS data using scripts.”

Terms of Service

“The University’s RouteViews project was initially conceived as a tool for Internet operators to obtain real-time information about the global routing system from the perspectives of several different backbones and locations around the Internet. Although other tools handle related tasks, such as the various Looking Glass Collections (see e.g., TRACEROUTE.ORG), they typically either provide only a constrained view of the routing system (e.g., either a single provider or the route server) or they do not provide real-time access to routing data.

“While the RouteViews project was initially motivated by interest on the part of operators in determining how the global routing system viewed their prefixes and/or AS space, there have been many other interesting uses of this RouteViews data. For example, NLANR has used RouteViews data for AS path visualization and to study IPv4 address space utilization (archive). Others have used RouteViews data to map IP addresses to origin AS for various topological studies. CAIDA has used it in conjunction with the NetGeo database in generating geographic locations for hosts, functionality that both CoralReef and the Skitter project support.”

• FAQ
• License

Public sources of historical DNS data of various sorts. Formats may vary considerably.

“This public dataset contains all IPv4 measurements from Archipelago (Ark) that are older than approximately one year, and all IPv6 measurements (upto the present).”

…

“The IPv4 Routed /24 DNS Names Dataset provides fully-qualified domain names for IP addresses seen in the traces of the IPv4 Routed /24 Topology Dataset.

“DNS names are useful for obtaining additional information about routers and hosts making up the Internet topology. For example, DNS names of routers often encode the link type (backbone vs. access), link capacity, Point of Presence (PoP), and geographic location. We have DNS Names data starting March 2008.”

• Source
• The CAIDA UCSD IPv4 Routed /24 DNS Names Dataset
• The CAIDA UCSD IPv6 DNS Names Dataset
• Data URL, IPv4
• Data URL, IPv6
• Acceptable Use Agreement

“This dataset consists of a daily snapshot of the reverse zones (in-addr.arpa and ip6.arpa) as published by the RIRs on their public FTP servers in the /pub/zones/ directory. The RIRs use this data operationally. The zones under in-addr.arpa and ip6.arpa are assembled from these files.”

• Source
• Data URL
• Terms of Service

Public sources of historical data about peering and IXPs. Formats may vary considerably.

“This dataset is a repository of daily snapshots of historic PeeringDB data. The repository consists of two parts, version 1 and version 2.

“The old v1 format is available from July 29, 2010 through March 13, 2016 as sql and sqlite files. The new v2 format is available from May 27, 2016 to March 10, 2018 as sqlite files, and from March 11, 2016 onwards as json files.”

• Source
• Data URL
• Acceptable Use Agreement

“The IXP History Collection is an ongoing project which seeks to record and document global histories of computer networking and internet exchange points (IXPs), due to valid concerns that these histories will be lost from the global record unless attempts are made to start preserving them now. In particular, the project is concerned with the fragility of electronic information and born digital documents, records, and multimedia, otherwise known as born digital heritage.”

“As a starting point, the project utilizes the Packet Clearing House Internet Exchange Directory (https://www.pch.net/ixp/dir) to track global IXP development. The project then focuses on documenting IXP histories through as many public online sources as possible (e.g., web sites/pages, reports, journals, magazine/newspaper articles, old emails on public mail lists). In addition, the project relies on the use of web archives as a research tool for tracing IXP histories, but also as a preservation tool using the Save Page functions in the Wayback Machine and Arquivo.pt.”

• Source

Mirrored public sources of historical traceroute data. Formats may vary considerably.

“This public dataset contains all IPv4 measurements from Archipelago (Ark) that are older than approximately one year, and all IPv6 measurements (upto the present).”

• Source
• The CAIDA UCSD IPv4 Routed /24 Topology Dataset
• The CAIDA UCSD IPv6 Topology Dataset
• Data URL, IPv4
• Data URL, IPv4
• Data URL, IPv6
• Acceptable Use Agreement

“RIPE Atlas measurement data is always available via the API, but many distinct measurements run every single day and aggregating the results manually can be a pain. We collect a rich set of measurement data and, as good citizens, we don’t want to generate more measurement traffic than is necessary. It’s important that we make this data as easy to retrieve as possible.

And you all want a ton of data, right? Right.

Last year, we prototyped a service to expose this data for around one month from the day of collection. The feedback has all been positive, so we’ve worked on making it more permanent.”

• Source
• Data URL
• Terms of Service
• Note: postings disappear after 30 days