Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[wip] Use schemas for regions #70

Open
gislars opened this issue Mar 18, 2023 · 0 comments
Open

[wip] Use schemas for regions #70

gislars opened this issue Mar 18, 2023 · 0 comments
Assignees

Comments

@gislars
Copy link
Collaborator

gislars commented Mar 18, 2023

currently we import all regions into one schema (import) and all the processing happens in a separate schema (processing). This enables us to continuously update the OSM data and run the processing independently from the import.

When importing a new region we only want to provide a new pbf and region boundary. But there is a chance (a very little one) that the updating goes wrong when we mix data with different timestamps and try to update them.
One solution would be to only import all regions at once and make sure that all have the same timestamp. One could do this with the configuration provided in extracts/*.

This approach has some disadvantages:

  • create and import all extracts
  • downtime while updating
  • with each new region a whole import is needed

One idea to solve this is to use separate db schemas for each region.
Advantages:

  • regions can be updated, inserted, deleted independently from each other
  • no mixup of timestamps, each region can start with an individual timestamp (not too old!) and osm2pgsql will update it
  • for each region a individual settings could be used
    • update interval
    • projection

Disadvantages:

  • for each region a separate update job is running
  • multiple import and middle schemas use more storage space

This issue is about to keep track of this topic, collect experiences and progress.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant