Log datasette access IPs #3669

jdangerx · 2024-06-12T20:35:28Z

Overview

Relates to catalyst-cooperative/pudl-usage-metrics#128

Right now, our datasette instance only logs out the IP of the load balancer it sits behind. Useless!

If we shove datasette behind nginx, we can then use nginx's "real IP" module to grab the IPs of the clients that the load balancer is forwarding.

This PR does that, and:

reorganizes the deploy logic slightly so there is less duplication between the various deploy targets: "metadata" is an early exit, but everything else should share all the logic except for "how do we take this Dockerfile and deploy it?"
make --local target actually use Docker - this makes the deploy slightly slower but also much closer to what is actually getting deployed on fly.io

Testing

How did you make sure this worked?

I did a fast deploy to our staging server:

$ python publish.py --staging --only ferc2_xbrl.sqlite --only ferc6_xbrl.sqlite

Then I went to the staging url: https://catalyst-coop-pudl-staging.fly.dev/

And I saw my IP show up in the fly.io live logs:

https://fly.io/apps/catalyst-coop-pudl-staging/monitoring

Which I verified with a quick curl icanhazip.com / asking other people to do the same.

To-do list

Give feedback

Review the PR yourself and call out any questions or issues you have
Options

jdangerx · 2024-06-12T20:36:14Z

devtools/datasette/fly/50-mod-http-realip.conf

@@ -0,0 +1 @@
+load_module modules/ngx_http_realip_module.so;


This could technically also live in nginx.conf but this is more idiomatic.

jdangerx · 2024-06-12T20:37:07Z

devtools/datasette/fly/run.sh

@@ -8,4 +8,5 @@ ls
 mv all_dbs.tar.zst /data
 zstd -f -d /data/all_dbs.tar.zst -o /data/all_dbs.tar
 tar -xf /data/all_dbs.tar --directory /data
-datasette serve --host 0.0.0.0 ${DATABASES} --cors --inspect-file inspect-data.json --metadata metadata.yml --setting sql_time_limit_ms 5000 --port $PORT
+datasette serve --host 0.0.0.0 ${DATABASES} --cors --inspect-file inspect-data.json --metadata metadata.yml --setting sql_time_limit_ms 5000 --port $DATASETTE_PORT > /dev/null &


We want to suppress the datasette serve logs since we get the same information in the nginx access logs.

jdangerx · 2024-06-12T20:37:48Z

devtools/datasette/publish.py

+    && ln -sf /dev/stdout /var/log/nginx/access.log \
+    && ln -sf /dev/stderr /var/log/nginx/error.log
+
+COPY . /app


Moved this down to the bottom since all the other setup changes much less frequently than the contents of the Docker context - this helps with build caching.

jdangerx · 2024-06-13T14:21:16Z

src/pudl/metadata/classes.py

@@ -2216,7 +2216,7 @@ def from_data_source_ids(
            xbrl_resources=xbrl_resources,
        )

-    def to_yaml(self) -> None:
+    def to_yaml(self) -> str:


mypy was complaining about this.

jdangerx · 2024-06-13T14:22:50Z

devtools/datasette/fly/nginx.conf

+}
+http {
+  server {
+    listen 8080;


In theory we could turn this (and the Dockerfile) into a jinja template, which might be cleaner, but it felt like gilding the lily to me.

bendnorman

This all mostly makes sense to me! On the staging server I get a 502 but when I reload the page it works.

Also, how do we transfer the logs into a bucket?

bendnorman · 2024-06-14T22:29:33Z

devtools/datasette/publish.py

+    check_tables_have_metadata(metadata_yml, databases)
+
+    if deploy == "metadata":
+        logging.info("Only writing metadata. Aborting now.")


You maybe forgot to abort the script here?

Sure did, fixing! Good catch :) Fortunately we only actually deploy if we actively choose a valid deployment target 😅

jdangerx · 2024-06-17T21:54:16Z

Yeah, the 502 is happening because nginx will start up and become available before datasette is fully initialized. Since the staging server shuts down with inactivity, you'll see a 502 first every time the VM comes out of inactivity.

In terms of how to transfer logs to a bucket: the official solution is fly-log-shipper: https://github.com/superfly/fly-log-shipper.

bendnorman · 2024-06-18T18:37:35Z

We can't declare victory on tracking logs until we set up the sink right? Any idea how the fly-log-shipper actually runs? Also, looks like there isn't a GCS sink :(

jdangerx · 2024-06-18T20:14:56Z

We'd have to set up the sink to declare victory, yes. The quick-start in their docs seems reasonable.

The confusing thing about the GCS sink is that it's clearly supported by Vector, the library that fly-log-shipper uses... but the fly-log-shipper docs seem to omit GCS from the documentation. So we could try configuring Vector for GCS, or we can shove the logs into S3 and then pull them into GCS separately.

bendnorman

Sounds good! Looks like those next steps are tracked in the issue 👍

* use nginx real IP module * get nginx logs to show in stdout/err and suppress datasette/gunicorn logs

jdangerx commented Jun 12, 2024

View reviewed changes

jdangerx requested review from a team and zaneselvans and removed request for a team June 12, 2024 20:46

Base automatically changed from 3664-fly-staging-environment to main June 12, 2024 22:14

jdangerx force-pushed the 3664-get-user-ips branch from 8b37c9a to 635384a Compare June 13, 2024 14:20

jdangerx commented Jun 13, 2024

View reviewed changes

jdangerx self-assigned this Jun 13, 2024

zaneselvans requested review from bendnorman and removed request for zaneselvans June 13, 2024 16:37

zaneselvans added cloud Stuff that has to do with adapting PUDL to work in cloud computing context. datasette Issues related the accessing PUDL data via Datasette. labels Jun 13, 2024

bendnorman requested changes Jun 14, 2024

View reviewed changes

jdangerx force-pushed the 3664-get-user-ips branch from 635384a to f2e3983 Compare June 17, 2024 21:44

jdangerx requested a review from bendnorman June 17, 2024 21:54

bendnorman approved these changes Jun 19, 2024

View reviewed changes

jdangerx added 4 commits July 2, 2024 10:17

Add nginx

8d0e822

Get real IPs in fly.io logs

497e30f

* use nginx real IP module * get nginx logs to show in stdout/err and suppress datasette/gunicorn logs

Update docs / clean up

6239ce6

Actually abort!

317bea4

jdangerx force-pushed the 3664-get-user-ips branch from f2e3983 to 317bea4 Compare July 2, 2024 14:19

jdangerx enabled auto-merge July 2, 2024 14:31

Move noqa comments up.

56216e5

jdangerx added this pull request to the merge queue Jul 2, 2024

Merged via the queue into main with commit a605d67 Jul 2, 2024
12 checks passed

jdangerx deleted the 3664-get-user-ips branch July 2, 2024 17:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Log datasette access IPs #3669

Log datasette access IPs #3669

jdangerx commented Jun 12, 2024 •

edited

Loading

To-do list

jdangerx Jun 12, 2024

jdangerx Jun 12, 2024

jdangerx Jun 12, 2024

jdangerx Jun 13, 2024

jdangerx Jun 13, 2024

bendnorman left a comment

bendnorman Jun 14, 2024

jdangerx Jun 17, 2024 •

edited

Loading

jdangerx commented Jun 17, 2024

bendnorman commented Jun 18, 2024

jdangerx commented Jun 18, 2024

bendnorman left a comment

Log datasette access IPs #3669

Log datasette access IPs #3669

Conversation

jdangerx commented Jun 12, 2024 • edited Loading

Overview

Testing

To-do list

jdangerx Jun 12, 2024

Choose a reason for hiding this comment

jdangerx Jun 12, 2024

Choose a reason for hiding this comment

jdangerx Jun 12, 2024

Choose a reason for hiding this comment

jdangerx Jun 13, 2024

Choose a reason for hiding this comment

jdangerx Jun 13, 2024

Choose a reason for hiding this comment

bendnorman left a comment

Choose a reason for hiding this comment

bendnorman Jun 14, 2024

Choose a reason for hiding this comment

jdangerx Jun 17, 2024 • edited Loading

Choose a reason for hiding this comment

jdangerx commented Jun 17, 2024

bendnorman commented Jun 18, 2024

jdangerx commented Jun 18, 2024

bendnorman left a comment

Choose a reason for hiding this comment

jdangerx commented Jun 12, 2024 •

edited

Loading

jdangerx Jun 17, 2024 •

edited

Loading