Big Cleanup

[notjosh]

Heya 👋

This is more of a meta-PR than a single PR. It's quite large, so we can split it up if there are changes you're interested in.

I wanted to extend some of the parsing for my own personal needs, and I took it probably a couple of steps further I needed to...but some of the changes might be helpful to you too.

At a high level, here's what's different:

• add linting/formatting + precommit hook to encourage it (builds on top of Cleanup #25)
• add BOT_EMAIL and BOT_ID env vars, to send as HTTP headers, as semi-officially documented in nicbou/burgeramt-appointments-websockets
  • related: extended the sleep to 3mins as documented above...but we still send requests for more services, so we're not really adhering to their requiements
• added script to generate URLs for city-wide appointments, but also bezirk/location specific URLs (as is necessary for citizenship - fixing url for naturalization #23 incorrectly only queries one of many locations)
  • note: this isn't exposed to the Telegram interface, so it's still hardcoded in the actual code
• added some basic tests for the calendar parsing
• added pagination to calendar page requests
  • note: different services will want different rules for this to be useful. i.e. it's "normal" to schedule a citizenship appointment 6 months into the future, but anmeldung should be, say, max 14 days into the future

At this point I was starting to think about the best way to move this project forward overall:

On one hand, having a centralised bot to handle all requests is a lot of a requests from a single IP but can easily scale to serve 1000s of people. Officially (based on notes from nicbou/burgeramt-appointments-websockets) that is too many requests, so an alternative might be necessary.

Therefore on the other hand, maybe making this tool easy to spin up as a single-use thing might make sense. i.e. having a "DigialOcean 1-click app", or "deploy to AWS" so someone could manage their own instance until they get an appointment, then shut it down.

Combining the two options could be a viable step forward: having each service as an independent worker, which all notify back to a central Telegram bot server, will mean the bot should be able to scale more easily. In this case, I imagine a "service" to mean a unique dienstleisterlist and anliegen pair.

Some TODOs I'd like to get to at some point:

• simplify subscriptions: add a command like /add https://service.berlin.de/dienstleistung/120702/ or /add [berlin-wide appointment search URL] is (sometimes) easier than selecting it from the menu. Especially if you want an appointment from a specific location
• add location-specific appointments from the menu: /add 121616 sonnenallee will only add the Sonnenallee amt to the dienstleisterlist parameter
• simplify deployment i.e. Dockerise it
  • simplify non-local deployment (or proxy) during development to prevent getting IP-banned at home :[
• decouple Telegram bot from parsing, so different notification options become easier (i.e. Discord)
• update Telegram bot to the new (pre-release) API

That's probably enough for now :) If there's anything that sounds interesting to you, I can carve out separate PRs. Let me know!

[sonac]

wow, that's a lot. i'll need some time to carve the time for review of all the things.
thanks a lot for the PR!
even though my initial desire was to keep this project as small and simple as possible - it might be a good idea to make it more "production ready" and your PR is a major step there.

[sonac]

ok, so after taking long look and thought - i think this PR goes a bit against my initial thought with this project (short and simple hosted service that i've shared with others)
your refactoring makes it way bigger (and better, that goes without saying), but it also makes it completely different from my initial design and i'd be hesitant to maintain it (and tbh I don't have any spare time now =)
current license allows you to copy code and use it in whatever way you like. i can add link in the readme to your repo, but i'd prefer to keep this project as simple as possible