Johnetordoff · NyanHelsing · Mar 7, 2018 · Mar 8, 2018 · Jan 19, 2018 · Mar 1, 2018
diff --git a/CHANGELOG b/CHANGELOG
@@ -2,6 +2,93 @@
 ChangeLog
 *********
 
+0.40.1 (2018-07-25)
+===================
+- Feature: Add `X-CSRFToken` to list of acceptable CORS headers.
+- Feature: Tell Keen analytics to strip ip on upload.
+- Code: Remove never-implemented anonymous geolocation code.
+
+0.40.0 (2018-06-22)
+===================
+- Feature: Listen for MFR-originating metadata requests and relay the nature of the request to
+the OSF. This will allow the OSF to tally better metrics on file views.
+- Feature: Add new "sizeInt" file metadata field. Some providers return file size as a string.
+The "size" field in WaterButler's file metadata follows this behavior.  The "sizeInt" field will
+always be an integer (if file size is available) or `null` (if not).
+- Code: Expand tests for WaterButler's APIv1 server.
+
+0.39.2 (2018-06-05)
+===================
+- Fix: Brown-paper-bag release: actually change version in the code.
+
+0.39.1 (2018-06-05)
+===================
+- Code: Pin base docker image to python:3.5-slim-jessie.  3.5-slim was recently updated to Debian
+stretch, and WaterButler has not yet been verified to work on it.
+
+0.39.0 (2018-05-08)
+===================
+- Feature: WaterButler now lets osfstorage know who the requesting user is when listing folder
+contents, so that it can return metadata about whether the user has seen the most recent version
+of the file.  (thanks, @erinspace!)
+- Feature: WaterButler includes metadata about the request in the logging callback to help the
+logger tally more accurate download counts.  (thanks, @johnetordoff!)
+- Feature: Stop logging revisions metadata requests to logging callback.  (thanks, @johnetordoff!)
+- Fix: Don't try to decode the body of HEAD requests that error.  HEAD requests don't have bodies!
+- Code: Clean up existing mypy annotations and add new ones.  WaterButler still isn't 100% clean,
+but it's getting there!
+- Code: Allow passing JSON-encoded objects as config settings via environment variables.
+
+0.38.6 (2018-04-25)
+===================
+- Fix: `CELERY_RESULT_PERSISTENT` should default to True, now that `amqp` is the default result
+backend.
+
+0.38.5 (2018-04-24)
+===================
+- Fix: Don't overwrite evokable url with generated url inside the make_request retry loop.  Third
+time's the charm, right?
+
+0.38.4 (2018-04-24)
+===================
+- Fix: Evoke url-generating functions *inside* the make_request retry loop to make sure signed
+urls are as fresh as possible.
+
+0.38.3 (2018-04-24)
+===================
+- Fix: Remove temporary files after osfstorage provider has completed its archiving and
+parity-generation tasks.
+
+0.38.2 (2018-04-23)
+===================
+- Fix: Delay construction of Google Cloud signed urls until right before issuing them.  The provider
+was building them outside of the retry loop, resulting in slow-to-fail requests having their
+signatures expire before all subsequent retries could be issued.
+
+0.38.1 (2018-04-10)
+===================
+- Fix: Don't log 206 Partial requests to the download callback.  Most of these are for streaming
+video or resumable downloads and shouldn't be tallied as a download.
+
+0.38.0 (2018-03-28)
+===================
+- Fix: WaterButler now handles Range header requests correctly!  There was an off-by-one error where
+WB was returning one byte more than requested.  This manifested in Safari being unable to load
+videos rendered by MFR.  Safari requests the first two bytes of a video before issuing a full
+request.  When WB returns three bytes, Safari refuses to try to render the file.  Safari can now
+render videos from most providers that support Range headers.  Unfortunately, it still struggles
+with osfstorage for reasons that are still being investigated.  (thanks, @abought and @johetordoff!)
+- Feature: WaterButler now logs download and download-as-zip requests to the callback given by the
+auth provider.  This is intended to allow the OSF (the default auth provider) to better log download
+counts for files.  If WB detects that a request originates from MFR, it will relay that on to the
+callback, so the auth provider can account for that when compiling statistics.
+- Feature: Add a new limited provider for Google Cloud Storage!  A limited provider is one that can
+be used as a data-storage backend provider for osfstorage.  It does NOT have a corresponding OSF
+addon interface, nor does it support the full range of WB actions.  It has been tested locally, but
+has yet to be tested in a staging or production environment, so EARLY ADOPT AT YOUR OWN RISK!
+- Fix: Stop installing aiohttpretty in editable mode to avoid containers hanging while awaiting user
+input.
+
 0.37.1 (2018-02-06)
 ===================
 - Feature: Increase default move/copy task timeout to 20s.  WaterButler will now wait 20 seconds for

diff --git a/Dockerfile b/Dockerfile
@@ -1,4 +1,4 @@
-FROM python:3.5-slim
+FROM python:3.5-slim-jessie
 
 RUN usermod -d /home www-data && chown www-data:www-data /home
 
@@ -25,7 +25,16 @@ ENV GOSU_VERSION 1.4
 RUN apt-get update \
     && apt-get install -y \
         curl \
-    && gpg --keyserver pool.sks-keyservers.net --recv-keys B42F6819007F00F88E364FD4036A9C25BF357DD4 \
+        gnupg2 \
+    && for key in \
+      # GOSU
+      B42F6819007F00F88E364FD4036A9C25BF357DD4 \
+    ; do \
+      gpg --keyserver hkp://ipv4.pool.sks-keyservers.net:80 --recv-keys "$key" || \
+      gpg --keyserver hkp://ha.pool.sks-keyservers.net:80 --recv-keys "$key" || \
+      gpg --keyserver hkp://pgp.mit.edu:80 --recv-keys "$key" || \
+      gpg --keyserver hkp://keyserver.pgp.com:80 --recv-keys "$key" \
+    ; done \
     && curl -o /usr/local/bin/gosu -SL "https://github.com/tianon/gosu/releases/download/$GOSU_VERSION/gosu-$(dpkg --print-architecture)" \
   	&& curl -o /usr/local/bin/gosu.asc -SL "https://github.com/tianon/gosu/releases/download/$GOSU_VERSION/gosu-$(dpkg --print-architecture).asc" \
   	&& gpg --verify /usr/local/bin/gosu.asc \

diff --git a/dev-requirements.txt b/dev-requirements.txt
@@ -5,7 +5,7 @@ colorlog==2.5.0
 flake8==3.0.4
 ipdb
 ipython
-mypy==0.560
+mypy==0.580
 pydevd==0.0.6
 pyflakes
 pytest==2.8.2

diff --git a/docs/code.rst b/docs/code.rst
@@ -3,13 +3,12 @@ Code
 
 Important code links
 
-
 .. toctree::
     :maxdepth: 2
 
+    waterbutler
     waterbutler.core
+    waterbutler.core.streams
     waterbutler.providers
     waterbutler.server
-    streams
     waterbutler.tasks
-
diff --git a/docs/modules.rst b/docs/modules.rst
diff --git a/docs/provider.googlecloud.metadata.rst b/docs/provider.googlecloud.metadata.rst
@@ -0,0 +1,7 @@
+Metadata
+--------
+
+.. automodule:: waterbutler.providers.googlecloud.metadata
+    :members:
+    :undoc-members:
+    :show-inheritance:
diff --git a/docs/provider.googlecloud.rst b/docs/provider.googlecloud.rst
@@ -0,0 +1,13 @@
+Google Cloud Provider
+=====================
+
+.. toctree::
+
+    provider.googlecloud.metadata
+    provider.googlecloud.utils
+
+.. autoclass:: waterbutler.providers.googlecloud.provider.GoogleCloudProvider
+    :members:
+    :private-members:
+    :show-inheritance:
+
diff --git a/docs/provider.googlecloud.utils.rst b/docs/provider.googlecloud.utils.rst
@@ -0,0 +1,7 @@
+Utils
+-----
+
+.. automodule:: waterbutler.providers.googlecloud.utils
+    :members:
+    :undoc-members:
+    :show-inheritance:
diff --git a/docs/provider.osfstorage.metadata.rst b/docs/provider.osfstorage.metadata.rst
@@ -0,0 +1,7 @@
+Metadata
+--------
+
+.. automodule:: waterbutler.providers.osfstorage.metadata
+    :members:
+    :undoc-members:
+    :show-inheritance:
diff --git a/docs/provider.osfstorage.rst b/docs/provider.osfstorage.rst
@@ -1,6 +1,10 @@
-OSF Storage Provider
+OSFStorage Provider
 ====================
 
+.. toctree::
+
+    provider.osfstorage.metadata
+
 .. autoclass:: waterbutler.providers.osfstorage.provider.OSFStorageProvider
     :members:
     :undoc-members:

diff --git a/docs/providers.rst b/docs/providers.rst
@@ -12,6 +12,7 @@ Providers
    provider.filesystem
    provider.github
    provider.gitlab
+   provider.googlecloud
    provider.googledrive
    provider.onedrive
    provider.osfstorage

diff --git a/docs/waterbutler.core.rst b/docs/waterbutler.core.rst
@@ -2,7 +2,7 @@ waterbutler.core package
 ========================
 
 waterbutler.core.auth module
-----------------------------------
+----------------------------
 
 .. automodule:: waterbutler.core.auth
     :members:
@@ -17,6 +17,14 @@ waterbutler.core.exceptions module
     :undoc-members:
     :show-inheritance:
 
+waterbutler.core.log_payload module
+-----------------------------------
+
+.. automodule:: waterbutler.core.logging
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
 waterbutler.core.logging module
 -------------------------------
 
@@ -33,14 +41,39 @@ waterbutler.core.metadata module
     :undoc-members:
     :show-inheritance:
 
+waterbutler.core.metrics module
+-------------------------------
+
+.. automodule:: waterbutler.core.metrics
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
 waterbutler.core.path module
---------------------------------
+----------------------------
 
 .. automodule:: waterbutler.core.path
     :members:
     :undoc-members:
     :show-inheritance:
 
+waterbutler.core.provider module
+--------------------------------
+
+.. automodule:: waterbutler.core.provider
+    :members:
+    :undoc-members:
+    :show-inheritance:
+    :noindex:
+
+waterbutler.core.remote_logging module
+--------------------------------------
+
+.. automodule:: waterbutler.core.remote_logging
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
 waterbutler.core.signing module
 -------------------------------
 
@@ -49,7 +82,6 @@ waterbutler.core.signing module
     :undoc-members:
     :show-inheritance:
 
-
 waterbutler.core.utils module
 -----------------------------
 
@@ -58,7 +90,6 @@ waterbutler.core.utils module
     :undoc-members:
     :show-inheritance:
 
-
 Module contents
 ---------------
 

diff --git a/docs/streams.rst → docs/waterbutler.core.streams.rst b/docs/streams.rst → docs/waterbutler.core.streams.rst
@@ -1,14 +1,12 @@
-Streams
-=======
+waterbutler.core.streams package
+================================
 
 BaseStream
 ----------
 
 .. autoclass:: waterbutler.core.streams.BaseStream
     :members:
     :undoc-members:
-    :show-inheritance:
-    :inherited-members:
 
 ResponseStreamReader
 --------------------
@@ -72,3 +70,10 @@ FormDataStream
     :undoc-members:
     :show-inheritance:
     :inherited-members:
+
+CutoffStream
+------------
+
+.. autoclass:: waterbutler.core.streams.CutoffStream
+    :members:
+    :undoc-members:
diff --git a/docs/waterbutler.rst b/docs/waterbutler.rst
@@ -1,18 +1,13 @@
 waterbutler package
 ===================
 
-Subpackages
------------
+waterbutler.constants module
+----------------------------
 
-.. toctree::
-
-    waterbutler.core
-    waterbutler.providers
-    waterbutler.server
-    waterbutler.tasks
-
-Submodules
-----------
+.. automodule:: waterbutler.constants
+    :members:
+    :undoc-members:
+    :show-inheritance:
 
 waterbutler.settings module
 ---------------------------
@@ -22,6 +17,21 @@ waterbutler.settings module
     :undoc-members:
     :show-inheritance:
 
+waterbutler.sizes module
+------------------------
+
+.. automodule:: waterbutler.sizes
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+waterbutler.version module
+--------------------------
+
+.. automodule:: waterbutler.version
+    :members:
+    :undoc-members:
+    :show-inheritance:
 
 Module contents
 ---------------
@@ -30,3 +40,4 @@ Module contents
     :members:
     :undoc-members:
     :show-inheritance:
+
diff --git a/docs/waterbutler.server.rst b/docs/waterbutler.server.rst
@@ -1,9 +1,6 @@
 waterbutler.server package
 ==========================
 
-Submodules
-----------
-
 waterbutler.server.app module
 -----------------------------
 

diff --git a/docs/waterbutler.tasks.rst b/docs/waterbutler.tasks.rst
@@ -1,9 +1,6 @@
 waterbutler.tasks package
 =========================
 
-Submodules
-----------
-
 waterbutler.tasks.app module
 ----------------------------