[WARC writer / protocol-okhttp] WARC-Truncated header issues and improvements

[sebastian-nagel]

There are some oddities how truncated captures are recorded in WARC files. See also Henry Thompson's report and the discussion in the Common Crawl user group.

• (protocol-okhttp) reliably annotate content truncated by length limit:
  • this happens mostly (100 out of 110) for pages with Content-Encoding: gzip
  • no truncation flag is added if the loop to read content chunk by chunk is exited reaching the content limit exactly: verify and open issue to fix this in upstream Nutch (NUTCH-2729)
• analyze truncations flagged as "disconnect" due to an IOException
  • in 3 analyzed WARC files all records flagged by "disconnect" have either "gzip" content encoding or "chunked" transfer encoding (or even both) - the reason could be also a broken encoding not a "network disconnect". Note: would need also to clarify how to annotate truncations due to protocol-level errors.
  • see More detailed marking of truncated records due to "network disconnect" #13 for more details
• always add a Content-Length header to HTTP headers in WARC file, even if there wasn't one in the original HTTP response (eg. for chunked transfer encoding). Implemented in 3663f35.
• (in the course) upgrade to latest okhttp library (NUTCH-2728)
• port fixes to StormCrawler: okhttp protocol: reliably mark trimmed content because of content limit apache/incubator-stormcrawler#757

[sebastian-nagel]

For analysis and verification see

• cc-pyspark#9 to mine (presumably) truncated records from WARC files
• Python Notebook to aggregate metrics from the list of records

[sebastian-nagel]

Implemented and fixed for August 2019 crawl (CC-MAIN-2019-35). Solution verified on 100 randomly selected WARC files.

[tfmorris]

Implemented and fixed for August 2019 crawl (CC-MAIN-2019-35). Solution verified on 100 randomly selected WARC files.

The above notebook is now here