Add semi-automated test system

Basically, a set of pcaps and baseline hars in a new tests directory. run_tests.sh tests har files from the pcaps against the existing har files and reports any errors. For more details see the readme.
sboisson · May 14, 2011 · 1f1559e · 1f1559e
1 parent 71d8f99
commit 1f1559e
Show file tree

Hide file tree

Showing 19 changed files with 48,335 additions and 0 deletions.
diff --git a/tests/README.txt b/tests/README.txt
@@ -0,0 +1,37 @@
+What we have in this director is a set of pcaps to be used as tests for
+pcap2har. Many of them exhibited specific previous bugs in pcap2har.
+Also included are HAR's that are expected from the pcaps. They thus 
+serve as a primitive regression test suite.
+
+To run the tests, just run the bash script run_tests.sh. This iterates
+through all the pcaps in the directory, runs pcap2har on them, and diffs
+the output with saved hars to check for errors. If either pcap2har or the diff
+fails, the log is saved and the script continues to check files.
+
+Here is a list of pcaps, their properties, and where they came from.
+
+http.pcap
+Previously http.cap. A simple http pageload. It has some awkward feature
+I forgot.
+
+fhs.pcap
+A complete pageload of andrewfleenor.users.sourceforge.net/fhs/fhs.xml.
+Streams are gzip compressed.
+
+fhs_ncomp.pcap
+Above, but not compressed.
+
+empty.pcap
+Empty file. dpkt doesn't like it, but we have to handle it.
+
+out-of-order.pcap
+From from Dekel Amrani. A big pcap, with out-of-order starting packet on
+incoming stream of tcp port 59743.
+
+github.pcap
+A pageload of github.
+
+pcapr.net.pcap
+A pageload of pcapr.net, an online pcap repository. Includes a redirect
+from pcapr.net to pcapr.net/home
+
diff --git a/tests/empty.pcap b/tests/empty.pcap
diff --git a/tests/empty.pcap.har b/tests/empty.pcap.har
@@ -0,0 +1,15 @@
+{
+  "log": {
+    "browser": {
+      "name": null, 
+      "version": "mumble"
+    }, 
+    "creator": {
+      "name": "pcap2har", 
+      "version": "0.1"
+    }, 
+    "entries": [], 
+    "pages": [], 
+    "version": "1.1"
+  }
+}
diff --git a/fhs.pcap → tests/fhs.pcap b/fhs.pcap → tests/fhs.pcap
diff --git a/tests/fhs.pcap.har b/tests/fhs.pcap.har
diff --git a/fhs_ncomp.pcap → tests/fhs_ncomp.pcap b/fhs_ncomp.pcap → tests/fhs_ncomp.pcap
diff --git a/tests/fhs_ncomp.pcap.har b/tests/fhs_ncomp.pcap.har
diff --git a/tests/github.pcap b/tests/github.pcap
diff --git a/tests/github.pcap.har b/tests/github.pcap.har
@@ -0,0 +1,296 @@
+{
+  "log": {
+    "browser": {
+      "name": "Mozilla/5.0 (X11; U; Linux x86_64; en-US) AppleWebKit/534.10 (KHTML, like Gecko) Chrome/8.0.552.224 Safari/534.10", 
+      "version": "mumble"
+    }, 
+    "creator": {
+      "name": "pcap2har", 
+      "version": "0.1"
+    }, 
+    "entries": [
+      {
+        "cache": {}, 
+        "pageref": "page_0", 
+        "request": {
+          "bodySize": 0, 
+          "cookies": [], 
+          "headers": [
+            {
+              "name": "accept-language", 
+              "value": "en-US,en;q=0.8"
+            }, 
+            {
+              "name": "accept-encoding", 
+              "value": "gzip,deflate,sdch"
+            }, 
+            {
+              "name": "connection", 
+              "value": "keep-alive"
+            }, 
+            {
+              "name": "accept", 
+              "value": "application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5"
+            }, 
+            {
+              "name": "user-agent", 
+              "value": "Mozilla/5.0 (X11; U; Linux x86_64; en-US) AppleWebKit/534.10 (KHTML, like Gecko) Chrome/8.0.552.224 Safari/534.10"
+            }, 
+            {
+              "name": "accept-charset", 
+              "value": "ISO-8859-1,utf-8;q=0.7,*;q=0.3"
+            }, 
+            {
+              "name": "host", 
+              "value": "www.google.com"
+            }, 
+            {
+              "name": "referer", 
+              "value": "http://www.google.com/"
+            }, 
+            {
+              "name": "cookie", 
+              "value": "PREF=ID=7492fece205ea3d4:U=317eaeca3484f597:FF=0:TM=1294255593:LM=1294259267:GM=1:S=kqaPbRCfx_L6k5cA; rememberme=true; HSID=ArpR8Rl1rxNDri76e; NID=43=CWDH2mgaolrF4VAgexh4gekl-txAfml233t4KwghgCARsbawFGI2xB5H4S47JKgE1dpWG3BB0lsLwaI2rJ1sdB7wHr5m09fBcDlylsWimRZJPSeOGvPB1fZI9e_Icu6J; SID=DQAAAKYAAAD1zusm-LtWNAa5SwBgXnyvA_kdbIPw8nsu3kaQ_ZK7oy8-PH_UsfbbuUSXdCk1YT6j7jQrcKPsDZvEnGxoXiZM_cR007HdiSspv7gMCPvXmgfa1Uj-1S1abvl2uqz88lcMx4qf_Q3IwrfNdSX5pQlV83QvReP_7iHw4mGd8PVwA3UEXIF5KV-WHn9v65czIoU2lPLa8BxI1rTMzgLO_dZ1eNkw1ARAixh3oQ_5Lwf_sw"
+            }
+          ], 
+          "headersSize": -1, 
+          "httpVersion": "1.1", 
+          "method": "GET", 
+          "queryString": [
+            {
+              "name": "usg", 
+              "value": "AFQjCNGtdj1QdkZlT74_PKB0_ne3b9h2Vw"
+            }, 
+            {
+              "name": "ei", 
+              "value": "skdHTYe3HpHSsAOI0NTiAQ"
+            }, 
+            {
+              "name": "url", 
+              "value": "https://github.com/andrewf/pcap2har"
+            }, 
+            {
+              "name": "rct", 
+              "value": "j"
+            }, 
+            {
+              "name": "q", 
+              "value": "pcap2har"
+            }, 
+            {
+              "name": "source", 
+              "value": "web"
+            }, 
+            {
+              "name": "ved", 
+              "value": "0CBMQFjAA"
+            }, 
+            {
+              "name": "sig2", 
+              "value": "SJ0322g854RsyqtEDxQC2w"
+            }, 
+            {
+              "name": "sa", 
+              "value": "t"
+            }, 
+            {
+              "name": "cd", 
+              "value": "1"
+            }
+          ], 
+          "url": "http://www.google.com/url?sa=t&source=web&cd=1&ved=0CBMQFjAA&url=https%3A%2F%2Fgithub.com%2Fandrewf%2Fpcap2har&rct=j&q=pcap2har&ei=skdHTYe3HpHSsAOI0NTiAQ&usg=AFQjCNGtdj1QdkZlT74_PKB0_ne3b9h2Vw&sig2=SJ0322g854RsyqtEDxQC2w"
+        }, 
+        "response": {
+          "bodySize": 219, 
+          "content": {
+            "compression": 108, 
+            "mimeType": "text/html", 
+            "size": 327, 
+            "text": "<script>var a=parent,b=parent.google,c=location;if(a!=window&&b){if(b.r){b.r=0;a.location.href=\"https://github.com/andrewf/pcap2har\";c.replace(\"about:blank\");}}else{c.replace(\"https://github.com/andrewf/pcap2har\");};</script><noscript><META http-equiv=\"refresh\" content=\"0;URL='https://github.com/andrewf/pcap2har'\"></noscript>"
+          }, 
+          "cookies": [], 
+          "headers": [
+            {
+              "name": "content-length", 
+              "value": "219"
+            }, 
+            {
+              "name": "x-xss-protection", 
+              "value": "1; mode=block"
+            }, 
+            {
+              "name": "content-encoding", 
+              "value": "gzip"
+            }, 
+            {
+              "name": "expires", 
+              "value": "Fri, 01 Jan 1990 00:00:00 GMT"
+            }, 
+            {
+              "name": "server", 
+              "value": "gws"
+            }, 
+            {
+              "name": "pragma", 
+              "value": "no-cache"
+            }, 
+            {
+              "name": "cache-control", 
+              "value": "no-cache, must-revalidate"
+            }, 
+            {
+              "name": "date", 
+              "value": "Mon, 31 Jan 2011 23:41:47 GMT"
+            }, 
+            {
+              "name": "content-type", 
+              "value": "text/html; charset=UTF-8"
+            }
+          ], 
+          "headersSize": -1, 
+          "httpVersion": "1.1", 
+          "redirectURL": "", 
+          "status": 200, 
+          "statusText": "OK"
+        }, 
+        "startedDateTime": "2011-01-31T15:41:47.204937Z", 
+        "time": 158, 
+        "timings": {
+          "blocked": -1, 
+          "connect": 47, 
+          "dns": 44, 
+          "receive": 0, 
+          "send": 0, 
+          "wait": 110
+        }
+      }, 
+      {
+        "cache": {}, 
+        "pageref": "page_1", 
+        "request": {
+          "bodySize": 383, 
+          "cookies": [], 
+          "headers": [
+            {
+              "name": "content-length", 
+              "value": "383"
+            }, 
+            {
+              "name": "accept-language", 
+              "value": "en-US,en;q=0.8"
+            }, 
+            {
+              "name": "accept-encoding", 
+              "value": "gzip,deflate,sdch"
+            }, 
+            {
+              "name": "connection", 
+              "value": "keep-alive"
+            }, 
+            {
+              "name": "user-agent", 
+              "value": "Mozilla/5.0 (X11; U; Linux x86_64; en-US) AppleWebKit/534.10 (KHTML, like Gecko) Chrome/8.0.552.224 Safari/534.10"
+            }, 
+            {
+              "name": "accept-charset", 
+              "value": "ISO-8859-1,utf-8;q=0.7,*;q=0.3"
+            }, 
+            {
+              "name": "host", 
+              "value": "toolbarqueries.clients.google.com"
+            }, 
+            {
+              "name": "cookie", 
+              "value": "PREF=ID=7492fece205ea3d4:U=317eaeca3484f597:FF=0:TM=1294255593:LM=1294259267:GM=1:S=kqaPbRCfx_L6k5cA; rememberme=true; HSID=ArpR8Rl1rxNDri76e; NID=43=CWDH2mgaolrF4VAgexh4gekl-txAfml233t4KwghgCARsbawFGI2xB5H4S47JKgE1dpWG3BB0lsLwaI2rJ1sdB7wHr5m09fBcDlylsWimRZJPSeOGvPB1fZI9e_Icu6J; SID=DQAAAKYAAAD1zusm-LtWNAa5SwBgXnyvA_kdbIPw8nsu3kaQ_ZK7oy8-PH_UsfbbuUSXdCk1YT6j7jQrcKPsDZvEnGxoXiZM_cR007HdiSspv7gMCPvXmgfa1Uj-1S1abvl2uqz88lcMx4qf_Q3IwrfNdSX5pQlV83QvReP_7iHw4mGd8PVwA3UEXIF5KV-WHn9v65czIoU2lPLa8BxI1rTMzgLO_dZ1eNkw1ARAixh3oQ_5Lwf_sw"
+            }, 
+            {
+              "name": "content-type", 
+              "value": "text/plain"
+            }
+          ], 
+          "headersSize": -1, 
+          "httpVersion": "1.1", 
+          "method": "POST", 
+          "queryString": [], 
+          "url": "http://toolbarqueries.clients.google.com/tbproxy/af/query"
+        }, 
+        "response": {
+          "bodySize": 68, 
+          "content": {
+            "compression": 135, 
+            "mimeType": "text/xml", 
+            "size": 203, 
+            "text": "<autofillqueryresponse><field autofilltype=\"0\" /><field autofilltype=\"0\" /><field autofilltype=\"0\" /><field autofilltype=\"0\" /><field autofilltype=\"0\" /><field autofilltype=\"0\" /></autofillqueryresponse>"
+          }, 
+          "cookies": [], 
+          "headers": [
+            {
+              "name": "content-length", 
+              "value": "68"
+            }, 
+            {
+              "name": "x-xss-protection", 
+              "value": "1; mode=block"
+            }, 
+            {
+              "name": "content-encoding", 
+              "value": "gzip"
+            }, 
+            {
+              "name": "server", 
+              "value": "AutoFill Server"
+            }, 
+            {
+              "name": "cache-control", 
+              "value": "private, x-gzip-ok=\"\""
+            }, 
+            {
+              "name": "date", 
+              "value": "Mon, 31 Jan 2011 23:41:48 GMT"
+            }, 
+            {
+              "name": "content-type", 
+              "value": "text/xml; charset=utf-8"
+            }
+          ], 
+          "headersSize": -1, 
+          "httpVersion": "1.1", 
+          "redirectURL": "", 
+          "status": 200, 
+          "statusText": "OK"
+        }, 
+        "startedDateTime": "2011-01-31T15:41:48.579586Z", 
+        "time": 197, 
+        "timings": {
+          "blocked": -1, 
+          "connect": 83, 
+          "dns": 45, 
+          "receive": 0, 
+          "send": 0, 
+          "wait": 114
+        }
+      }
+    ], 
+    "pages": [
+      {
+        "id": "page_0", 
+        "pageTimings": {
+          "onContentLoad": -1, 
+          "onLoad": -1
+        }, 
+        "startedDateTime": "2011-01-31T15:41:47.204937Z", 
+        "title": "http://www.google.com/url?sa=t&source=web&cd=1&ved=0CBMQFjAA&url=https%3A%2F%2Fgithub.com%2Fandrewf%2Fpcap2har&rct=j&q=pcap2har&ei=skdHTYe3HpHSsAOI0NTiAQ&usg=AFQjCNGtdj1QdkZlT74_PKB0_ne3b9h2Vw&sig2=SJ0322g854RsyqtEDxQC2w"
+      }, 
+      {
+        "id": "page_1", 
+        "pageTimings": {
+          "onContentLoad": -1, 
+          "onLoad": -1
+        }, 
+        "startedDateTime": "2011-01-31T15:41:48.579586Z", 
+        "title": "http://toolbarqueries.clients.google.com/tbproxy/af/query"
+      }
+    ], 
+    "version": "1.1"
+  }
+}
diff --git a/http.cap → tests/http.pcap b/http.cap → tests/http.pcap