Datasets

We provide large graphs compressed using LLP + WebGraph. Please read the tutorial to learn how to download and use them.

The graphs can also be loaded and analyzed in JGraphT using suitable adapters. Moreover, for graphs with less than 2³¹ arcs we provide a serialized succinct JGraphT representation.

Acknowledging data

If you publish results based on these graphs, please acknowledge the usage of WebGraph and LLP by quoting the following papers:

@inproceedings{BoVWFI,
  author = "Paolo Boldi and Sebastiano Vigna",
  title = "The {W}eb{G}raph Framework {I}: {C}ompression Techniques",
  year = 2004,
  booktitle = "Proc. of the Thirteenth International World Wide Web Conference (WWW 2004)",
  address = "Manhattan, USA",
  pages = "595--601",
  publisher = "ACM Press"
}

@inproceedings{BRSLLP,
  author = "Paolo Boldi and Marco Rosa and Massimo Santini and Sebastiano Vigna",
  title = "Layered Label Propagation: A MultiResolution Coordinate-Free Ordering for Compressing Social Networks",
  booktitle = "Proceedings of the 20th international conference on World Wide Web",
  editor = "Sadagopan Srinivasan and Krithi Ramamritham and Arun Kumar and M. P. Ravindra and Elisa Bertino and Ravi Kumar",
  publisher = "ACM Press",
  year = 2011,
  pages = "587--596"
}

Acknowledging UbiCrawer

If the graphs you are using were gathered by UbiCrawler, please acknowledge the usage of UbiCrawler by quoting the following paper:

@article{BCSU3,
  author = "Paolo Boldi and Bruno Codenotti and Massimo Santini and Sebastiano Vigna",
  title = "UbiCrawler: A Scalable Fully Distributed Web Crawler",
  journal = "Software: Practice \& Experience",
  year = 2004,
  volume = 34,
  number = 8,
  pages = "711--726"
}

Acknowledging BUbiNG

If the graphs you are using were gathered by BUbiNG, please acknowledge the usage of BUbiNG by quoting the following paper:

@inproceedings{BMSB,
  author = "Boldi, Paolo and Marino, Andrea and Santini, Massimo and Vigna, Sebastiano",
  title = "{BUbiNG}: Massive Crawling for the Masses",
  booktitle = "Proceedings of the Companion Publication of the 23rd International Conference on World Wide Web",
  year = 2014,
  pages = "227--228",
  publisher = "International World Wide Web Conferences Steering Committee"
}

The graphs

We have gathered our datasets in reasonably homogenous groups. Besides a brief description, for each group there is a table providing basic information about the available graphs, such as the crawl date, the number of nodes and arcs, and the number of bits per link of the highly compressed version (the version we provide for general usage has faster random access, but worse compression ratio). The percentage after the number of bits per link gives the compression ratio with respect to the information-theoretical lower bound (that is, the logarithm of “n² choose m” divided by m). We report, when available, the maximum depth per host and the maximum number of URLs per host. The last column shows the institution that provided support for the crawl.

For each graph, we provide a wealth of statistical data and graphs, ranging from plots of the distribution of indegrees and outdegrees to the size of the giant component and a drawing of the largest components. In particular, thanks to HyperBall we can provide the average and median distance, the harmonic diameter and the spid (shortest-paths index of dispersion) of the graph. The statistic are obtained by applying the jackknife method to 100 statistically independent approximations of the neighbourhood function.

Small graphs

The graphs in this section are very small web graphs that were crawled for testing purposes, or to have a look at some weird domain. We provide them as testing and debugging tools, as their size makes them easily manageable, but clearly they cannot be used as the only testbed for compression.

Larger crawls

The graphs in this section are larger graphs from different domain. With the exclusion of the series of .uk datasets, these are the largest graphs available from us.

Social networks

The graphs in this section are not web graph, but rather (directed and undirected) networks generated by human activities, such as a collaboration network, a co-starring network, LiveJournal, etc.

Facebook graphs

These graphs have been studied in the “Four degrees of separation” paper by Lars Backstrom, Paolo Boldi, Marco Rosa, Johan Ugander and Sebastiano Vigna. The series sections the Facebook graph geographically and temporally: please see the paper for the details. Note that years up to 2011 are timed at the first of January, but the “current” graphs are timed at May 2011. In particular, the latest whole Facebook graph prompted the title of the paper.

We cannot distribute the graphs for obvious reasons, but you can download aggregated data (WebGraph properties, HyperANF runs and some degree distributions). If you publish results based on our data, please acknowledge our work by quoting the “Four degrees of separation” paper.

The .uk DELIS series

From 2006 to 2007, we gathered twelve monthly snapshots of about 100 millions pages each for the DELIS project. If you want a large recent graph we suggest to use uk-2007-05. A special effort has been made in compacting into the dataset uk-union all the snapshots by labelling each node and arc, thus obtaining a large time-aware graph.

Other datasets

In 2001, we downloaded from the Stanford Webbase project site a graph that we distribute in compact form. We also provide some statistical data about the Altavista dataset distributed by Yahoo! within the Webscope program (AltaVista webpage connectivity dataset, version 1.0). The “nd” (no-dangling) version was obtained by pruning (one level of) dangling nodes.