Repositories list

• web-languages

  Public
  Crowd-sourced lists of urls to help Common Crawl crawl under-resourced languages. See https://github.com/commoncrawl/web-languages-code/ for the code

  crawlinglanguage-detectiondataset

  32•28•0•1•Updated Jan 18, 2025Jan 18, 2025

• uap-core

  Public
  The regex file necessary to build language ports of Browserscope's user agent parser.

  JavaScript

  •

  Other

  •452•0•0•0•Updated Jan 17, 2025Jan 17, 2025

• cc-citations

  Public
  Scientific articles using or citing Common Crawl data

  bibtexbibliographyopendata

  Jupyter Notebook

  •3•12•0•0•Updated Jan 15, 2025Jan 15, 2025

• cc-downloader

  Public
  A polite and user-friendly downloader for Common Crawl data

  rustdownloadercommoncrawl

  Rust

  •

  Apache License 2.0

  •1•10•0•0•Updated Jan 13, 2025Jan 13, 2025

• web-languages-code

  Public
  The code used to generate templates for the web-languages repo https://github.com/commoncrawl/web-languages

  Python

  •

  Apache License 2.0

  •0•1•0•0•Updated Jan 11, 2025Jan 11, 2025

• cc-crawl-statistics

  Public
  Statistics of Common Crawl monthly archives mined from URL index files

  commoncrawlcommon-crawlstatistics

  Python

  •

  Apache License 2.0

  •11•167•0•0•Updated Jan 9, 2025Jan 9, 2025

• nutch

  Public
  Common Crawl fork of Apache Nutch

  big-datahadoopweb-crawler+ 2commoncrawljava

  Java

  •

  Apache License 2.0

  •1.3k•29•8•0•Updated Jan 8, 2025Jan 8, 2025

• ia-hadoop-tools

  Public
  Web archiving tools on Hadoop

  Java

  •27•3•2•1•Updated Jan 8, 2025Jan 8, 2025

• ia-web-commons

  Public
  Web archiving utility library

  cdx-fileswarc-fileswat-files

  Java

  •

  Apache License 2.0

  •72•9•4•2•Updated Jan 8, 2025Jan 8, 2025

• cc-webgraph-statistics

  Public
  Statistics of Common Crawl monthly Web Graphs

  Python

  •

  Apache License 2.0

  •0•2•0•0•Updated Jan 2, 2025Jan 2, 2025

• whirlwind-python

  Public
  A whirlwind tour of Common Crawl's data using Python

  pythontutorialarchive+ 1warc

  Python

  •

  Apache License 2.0

  •5•15•0•1•Updated Dec 26, 2024Dec 26, 2024

• webarchive-indexing

  Public
  Tools for bulk indexing of WARC/ARC files on Hadoop, EMR or local file system.

  cc-index

  Python

  •

  MIT License

  •10•5•0•2•Updated Dec 20, 2024Dec 20, 2024

• cc-pyspark

  Public
  Process Common Crawl data with Python and Spark

  pysparksparksqlwet+ 5commoncrawlcommon-crawlsparkwarc-fileswat-files

  Python

  •

  MIT License

  •88•411•4•2•Updated Dec 20, 2024Dec 20, 2024

• cc-webgraph

  Public
  Tools to construct and process webgraphs from Common Crawl data

  pagerankwebgraphcommoncrawl+ 3common-crawlcentrality-measureswebgraph-framework

  Java

  •

  Apache License 2.0

  •5•84•2•0•Updated Dec 19, 2024Dec 19, 2024

• cc-warc-examples

  Public
  CommonCrawl WARC/WET/WAT examples and processing code for Java + Hadoop

  javahadoopmapreduce+ 1commoncrawl

  Java

  •

  MIT License

  •46•37•0•0•Updated Dec 17, 2024Dec 17, 2024

• crawler-commons

  Public
  A set of reusable Java components that implement functionality common to any web crawler

  Java

  •

  Apache License 2.0

  •77•0•0•0•Updated Nov 26, 2024Nov 26, 2024

• open-data-registry

  Public
  A registry of publicly available datasets on AWS

  Python

  •

  Apache License 2.0

  •941•1•0•0•Updated Nov 24, 2024Nov 24, 2024

• cc-index-table

  Public
  Index Common Crawl archives in tabular format

  columnar-storageaws-athenaapache-parquet+ 3commoncrawlsqlspark

  Java

  •

  Apache License 2.0

  •9•110•5•3•Updated Nov 19, 2024Nov 19, 2024

• language-detection-cld2

  Public
  Natural language detection, Java bindings for CLD2

  natural-languagelanguage-detectionlanguage-identification

  Java

  •

  Apache License 2.0

  •2•14•2•0•Updated Nov 9, 2024Nov 9, 2024

• eotarchive

  Public
  Website for End of Term project, eotarchive.org.

  HTML

  •5•0•0•0•Updated Oct 20, 2024Oct 20, 2024

• ccf-git-github-filesystem-unicode-test

  Public
  Test files to diagnose git and filesystem problems with unicode normalization

  Python

  •0•0•0•0•Updated Oct 18, 2024Oct 18, 2024

• ccf-eot-analysis-2024

  Public
  Analysis code for the End of Term 2024 crawl

  Python

  •0•0•0•0•Updated Oct 14, 2024Oct 14, 2024

• ccf-eot-seeds-2024

  Public
  Common Crawl's contribution of seeds to the End of Term Archive 2024

  Makefile

  •

  Apache License 2.0

  •0•1•0•0•Updated Oct 7, 2024Oct 7, 2024

• ai.robots.txt

  Public
  A list of AI agents and robots to block.

  Python

  •

  MIT License

  •56•0•0•0•Updated Sep 26, 2024Sep 26, 2024

• eot2024

  Public
  End of Term Web Archive 2024

  Apache License 2.0

  •8•0•0•0•Updated Sep 16, 2024Sep 16, 2024

• warcio

  Public
  Streaming WARC/ARC library for fast web archive IO

  Python

  •

  Apache License 2.0

  •58•1•0•0•Updated Aug 23, 2024Aug 23, 2024

• cc-monitoring

  Public
  Code that monitors Common Crawl infrastructure

  Python

  •0•2•0•0•Updated May 27, 2024May 27, 2024

• cc-legal

  Public
  Repository for legal documentation at the Common Crawl Foundation

  0•1•0•0•Updated Mar 24, 2024Mar 24, 2024

• ml-opt-out-experiments

  Public
  A series of experiments into ML opt–out protocols

  Python

  •0•5•0•0•Updated Feb 22, 2024Feb 22, 2024

• commoncrawl_notebooks

  Public
  Jupyter Notebook

  •2•0•0•0•Updated Feb 12, 2024Feb 12, 2024