Downloads

In addition to the HTTP Archive source code being open source, all of the data gathered is available as downloadable data dumps. These dumps are also available in Google BigQuery thanks to Ilya Grigorik. For more information see Ilya's blog post HTTP Archive + BigQuery = Web Performance Answers and video. And checkout all the custom queries shared on discuss.httparchive.org.

Data Dumps

The results from each crawl are saved as dump files in CSV format. Dumps are made for both the desktop and mobile crawls. The dump files do not contain the commands to create the database and tables. To restore a dump:

  1. Import the schema to create the tables.
    cat httparchive_schema.sql | mysql -u MYSQL_USERNAME -p -h MYSQL_HOSTNAME MYSQL_DB
  2. Import the desired dump file.
    For CSV-formatted dumps, gunzip the file and rename it to either pages.csv or requests.csv, then run this command:
    mysqlimport --local --fields-optionally-enclosed-by="\"" --fields-terminated-by=, --user=MYSQL_USERNAME -p MYSQL_DB MYSQLDUMP_FILE.csv
  3. If you want to run a private instance of the source code, you need to also import the stats and crawls dumps.

Files

These files define the schema and the meta-level tables:

There's a download file for each type of crawl: