Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dockerized osm import and added osm update. #52

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

Dockerized osm import and added osm update. #52

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
6705a9b
Reorganize the repo contents. Created import scripts for osm/imposm. …
gregtzar Mar 31, 2019
52560bc
Updates to readme
gregtzar Mar 31, 2019
9a556c9
Renamed some files. Updated readme. New scripts for ne import and osm…
gregtzar Apr 3, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
59 changes: 59 additions & 0 deletions Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
##
# tegola/osm
#
# This creates an Ubuntu derived base image and installs the necessary
# dependencies for running osm import and updates for tegola.

FROM ubuntu:bionic

# Set the session to noninteractive. Only applies for life of dockerfile.
ARG DEBIAN_FRONTEND=noninteractive

# Install basic dependencies
RUN apt-get update -y && \
apt-get install -y \
apt-utils \
software-properties-common \
unzip \
curl \
wget

# Install Java runtime
RUN apt-get update -y && \
apt-get install -y default-jre

# Install gdal/ogr v2.1.3 from the ubuntugis ppa
RUN add-apt-repository ppa:ubuntugis/ubuntugis-unstable && \
apt-get update -y && \
apt-get install -y \
gdal-bin

# Install psql (postgres client) and postgis (contains shp2pgsql) from the postgresql repo
RUN wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | apt-key add - && \
add-apt-repository "deb http://apt.postgresql.org/pub/repos/apt/ $(lsb_release -sc)-pgdg main" && \
apt-get update -y && \
apt-get install -y \
postgresql-client-11 \
postgis

# Install osmosis
RUN mkdir -p /usr/local/bin/osmosis-src && \
wget --quiet -O - https://bretth.dev.openstreetmap.org/osmosis-build/osmosis-latest.tgz | tar -xz -C /usr/local/bin/osmosis-src && \
ln -s /usr/local/bin/osmosis-src/bin/osmosis /usr/local/bin/osmosis

# Install imposm
RUN mkdir -p /usr/local/bin/imposm-src && \
wget --quiet -O - https://github.com/omniscale/imposm3/releases/download/v0.6.0-alpha.4/imposm-0.6.0-alpha.4-linux-x86-64.tar.gz | tar -xz -C /usr/local/bin/imposm-src && \
ln -s /usr/local/bin/imposm-src/imposm-0.6.0-alpha.4-linux-x86-64/imposm /usr/local/bin/imposm && \
ln -s /usr/local/bin/imposm-src/imposm-0.6.0-alpha.4-linux-x86-64/lib/* /usr/lib/

# Install additional packages
RUN apt-get update -y && \
apt-get install -y \
jq

# Install the local scripts
COPY scripts/osm_imposm_import.sh /usr/local/bin/osm_imposm_import.sh
COPY scripts/osm_imposm_update.sh /usr/local/bin/osm_imposm_update.sh
COPY scripts/osm_land_import.sh /usr/local/bin/osm_land_import.sh
COPY scripts/ne_import.sh /usr/local/bin/ne_import.sh
212 changes: 176 additions & 36 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,60 +1,200 @@
# Tegola OSM

This repo houses instructions and configuration files to aid with standing up an OpenStreetMap export and Natural Earth dataset into a PostGIS enabled database that uses [tegola](https://github.com/terranodo/tegola) for creating and serving vector tiles.
This repo houses instructions, configuration files, and a docker container to aid with standing up an OpenStreetMap export and Natural Earth dataset into a PostGIS enabled database that uses [tegola](https://github.com/terranodo/tegola) for creating and serving vector tiles.

## Repo config files
## Dependencies

- imposm3.json - an [imposm3](https://github.com/omniscale/imposm3) mapping file for the OSM PBF file.
- tegola.toml - a [tegola](https://github.com/terranodo/tegola) configuration file for the OSM import produced by imposm3.
* [Docker](https://www.docker.com)
* Postgres server with [PostGIS](http://www.postgis.net) enabled
* tegola ([download](https://github.com/terranodo/tegola/releases))

## Dependencies
Additional dependencies if not using docker:

* imposm ([download](https://imposm.org/static/rel/) - linux only)
* [gdal](http://www.gdal.org/) - required for Natural Earth import

## Example Config

* `config/osm-imposm-mapping.json`: An [imposm](https://github.com/omniscale/imposm3) mapping file for mapping the osm pbf data to postgis.
* `config/osm-imposm-config.json`: An [imposm](https://github.com/omniscale/imposm3) config file for replication and srid settings. Note that imposm config properties such as the db connection string, cache and diff directories are not specified via the config but are configured as env vars to docker instead.
* `tegola-osm.toml` - A [tegola](https://github.com/terranodo/tegola) configuration file for the OSM import produced by imposm and osm land.
* `tegola-natural-earth.toml` - A [tegola](https://github.com/terranodo/tegola) configuration file for the natural earth data import.

- Postgres server with [PostGIS](http://www.postgis.net) enabled.
- imposm3 ([download](https://imposm.org/static/rel/) - linux only)
- tegola ([download](https://github.com/terranodo/tegola/releases))
- [gdal](http://www.gdal.org/) - required for Natural Earth import
## Docker Setup

## Download the OSM planet database in PBF format
To build the container run the following command from the root directory of this repo:

```bash
curl -O https://planet.openstreetmap.org/pbf/planet-latest.osm.pbf
docker build -t tegola-osm .
```

## Import the OSM export into PostGIS using imposm3
The container can also be pulled down from dockerhub.

## Available Scripts / Docker Commands

Each of the bash scripts is installed into the docker container and available as a global command. The scripts could also be run outside of a docker container if you meet the dependencies. The scripts all expect certain env vars to be set.

### scripts/osm_imposm_import.sh

Imports an OSM PBF file into a PostgreSQL/PostGIS instance using imposm `import` command. You are required to provide an imposm-cache and imposm-diff folder to which imposm will persist data. Those same directories must then be used for the osm update script. If you run an import against an existing db and cache/diff folders the existing data will be squashed.

#### Required Env Vars

* `DB_HOST`: PostgreSQL database host name
* `DB_PORT`: PostgreSQL database port
* `DB_NAME`: PostgreSQL database name
* `DB_USER`: PostgreSQL database username
* `DB_PW`: PostgreSQL database password
* `OSM_SOURCE_PBF`: Full path to the osm pbf file that we will import. Eg: `/osm_data/data/north-america-latest.osm.pbf`
* `IMPOSM_CONFIG`: Full path to the imposm config file we want to use. Eg: `/osm_data/config/osm-imposm-config.json`
* `IMPOSM_MAPPING`: Full path to the imposm mapping file we want to use. Eg: `/osm_data/config/osm-imposm-mapping.json`
* `IMPOSM_CACHE_DIR`: Full path to an empty writeable directory where imposm will persist a cache of the import. Eg: `/osm_data/cache`
* `IMPOSM_DIFF_DIR`: Full path to an empty writeable directory where imposm will persist diffs for the import. Eg: `/osm_data/cache`

#### Example Docker Command

All of our data, config files, and directories remain on the host and are exposed to the docker container through bind mounts. We then set the appropriate env vars on the container and reference the internal mount point of each object. This way imposm can not only read data and config dynamically from the host, but it can also persist cache/diff data back.

```bash
./imposm3 import -connection postgis://username:password@host/database-name -mapping imposm3.json -read /path/to/osm/planet-latest.osm.pbf -write
./imposm3 import -connection postgis://username:password@host/database-name -mapping imposm3.json -deployproduction
docker run -i --rm \
-u "${UID}" \
--mount "type=bind,source=/osm_data/data/north-america-latest.osm.pbf,target=/osm/data/north-america-latest.osm.pbf" \
--mount "type=bind,source=/osm_data/config/osm-imposm-config.json,target=/osm/config/imposm-config.json" \
--mount "type=bind,source=/osm_data/config/osm-imposm-mapping.json,target=/osm/config/imposm-mapping.json" \
--mount "type=bind,source=/osm_data/cache,target=/osm/cache" \
--mount "type=bind,source=/osm_data/diff,target=/osm/diff" \
-e "DB_HOST=localhost" \
-e "DB_PORT=5432" \
-e "DB_NAME=osm_data" \
-e "DB_USER=postgres" \
-e "DB_PW=postgres" \
-e "OSM_SOURCE_PBF=/osm/data/north-america-latest.osm.pbf" \
-e "IMPOSM_CONFIG=/osm/config/imposm-config.json" \
-e "IMPOSM_MAPPING=/osm/config/imposm-mapping.json" \
-e "IMPOSM_CACHE_DIR=/osm/cache" \
-e "IMPOSM_DIFF_DIR=/osm/diff" \
"tegola-osm:latest" \
bash -c "osm_import.sh"
```

## Import the OSM Land and Natural Earth dataset (requires gdal, Natural Earth can be skipped if you're only interested in OSM)
### scripts/osm_imposm_update.sh

### Option 1: Embed Credentials
Update the database credentials inside of `natural_earth.sh` and `osm_land.sh`, then run each file: `./natural_earth.sh && ./osm_land.sh`. This will download the natural earth and osm land datasets and insert it into PostGIS under a database named `natural_earth` and `osm` respectively.
Updates an existing PostgreSQL/PostGIS instance using imposm `diff` command. In this case we also use osmosis to generate a single changes list which we pass to imposm `diff`. This method is robust and supports updates from different data sources and data subsets across any replication interval. You are required to provide the imposm-cache and imposm-diff folder which was populated during the initial import and prior updates. Additionally, updates require a working directory for osmosis which may or may not be persisted.

#### Required Env Vars

* `DB_HOST`: PostgreSQL database host name
* `DB_PORT`: PostgreSQL database port
* `DB_NAME`: PostgreSQL database name
* `DB_USER`: PostgreSQL database username
* `DB_PW`: PostgreSQL database password
* `OSMOSIS_DIR`: Full path to a working directory for osmosis to create changes lists. Eg: `/osm_data/osmosis`
* `IMPOSM_CONFIG`: Full path to the imposm config file we want to use. Eg: `/osm_data/config/osm-imposm-config.json`
* `IMPOSM_MAPPING`: Full path to the imposm mapping file we want to use. Eg: `/osm_data/config/osm-imposm-mapping.json`
* `IMPOSM_CACHE_DIR`: Full path to an empty writeable directory where imposm will persist a cache of the import. Eg: `/osm_data/cache`
* `IMPOSM_DIFF_DIR`: Full path to an empty writeable directory where imposm will persist diffs for the import. Eg: `/osm_data/cache`

#### Example Docker Command

### Option 2: Create a dbcredentials.sh file
Create a `dbcredentials.sh` file which will be shared with the `osm_land` script. This option is ideal for when the `natural_earth` and `osm` databases will reside on the same database server, and will use the same credentials. Ensure that the following variables are defined in your file:
```bash
DB_HOST="mydbhost"
DB_PORT="myport"
DB_USER="myuser"
DB_PW="mypassword"
docker run -i --rm \
-u "${UID}" \
--mount "type=bind,source=/osm_data/osmosis,target=/osm/osmosis" \
--mount "type=bind,source=/osm_data/config/osm-imposm-config.json,target=/osm/config/imposm-config.json" \
--mount "type=bind,source=/osm_data/config/osm-imposm-mapping.json,target=/osm/config/imposm-mapping.json" \
--mount "type=bind,source=/osm_data/cache,target=/osm/cache" \
--mount "type=bind,source=/osm_data/diff,target=/osm/diff" \
-e "DB_HOST=localhost" \
-e "DB_PORT=5432" \
-e "DB_NAME=osm_data" \
-e "DB_USER=postgres" \
-e "DB_PW=postgres" \
-e "OSMOSIS_DIR=/osm/osmosis" \
-e "IMPOSM_CONFIG=/osm/config/imposm-config.json" \
-e "IMPOSM_MAPPING=/osm/config/imposm-mapping.json" \
-e "IMPOSM_CACHE_DIR=/osm/cache" \
-e "IMPOSM_DIFF_DIR=/osm/diff" \
"tegola-osm:latest" \
bash -c "osm_import.sh"
```
Once you have configured the `dbcredentials.sh` file, run the scripts as above: `./natural_earth.sh && ./osm_land.sh`

### Option 3:
Create separate configuration files in the same pattern as the above `dbcredentials.sh` file and pass the path to the config file using the `-c` option. This is ideal if you have two different servers for the databases. Ensure the file you create follows this format:
### scripts/ne_import.sh

Imports the Natural Earth data into the specified database. The import process will drop and recreate existing tables. Data is reprojected to 3857 on import but can be changed.

**Note:** If you are looking for the old `natural_earth.sh` script it can be found in `scripts/deprecated/`.

#### Command Args

* `-d` Setting this flag will force the entire database to be dropped and recreated.
* `-s SRID` Setting this will reproject that data to the specified EPSG:SRID during import.
* `-v` Verbose mode

#### Required Env Vars

* `DB_HOST`: PostgreSQL database host name
* `DB_PORT`: PostgreSQL database port
* `DB_NAME`: PostgreSQL database name
* `DB_USER`: PostgreSQL database username
* `DB_PW`: PostgreSQL database password
* `NE_MAPPING`: Full path to the json mapping file we want to use. Eg: `/ne/config/ne-mapping.json`
* `TEMP_DATA_DIR`: Full path to the directory where this script will temporarily store data. Script user must have permissions to read/write/delete from this directory. Eg: `/tmp`

#### Example Docker Command

```bash
DB_NAME="mydb"
DB_HOST="mydbhost"
DB_PORT="myport"
DB_USER="myuser"
DB_PW="mypassword"
docker run -i --rm \
-u "${UID}" \
--mount "type=bind,source=/ne_data/config/ne-mapping.json,target=/ne/mapping.json" \
--mount "type=bind,source=/tmp,target=/ne/temp" \
-e "DB_HOST=localhost" \
-e "DB_PORT=5432" \
-e "DB_NAME=ne_data" \
-e "DB_USER=postgres" \
-e "DB_PW=postgres" \
-e "NE_MAPPING=/ne/mapping.json" \
-e "TEMP_DATA_DIR=/ne/temp" \
"tegola-osm:latest" \
bash -c "ne_import.sh"
```
Once you have configured the files, run the scripts with the `-c` flag and provide the path to the credentials file, ie: `./natural_earth.sh -c natural_earth_creds.sh && ./osm_land.sh -c osm_creds.sh`

### Usage:
Both scripts support a `-v` flag for debugging. `natural_earth.sh` also supports a `-d` flag, which will drop the existing natural earth database prior to import if set. Since the `osm_land.sh` imports into a database shared with other data, it lacks this functionality. Instead, only the relevent tables are dropped.
### scripts/osm_land_import.sh

Imports the OSM land polygons data into the specified database. The import process will drop and recreate existing tables. Also, you will want to write to the same DB as the main OSM data if you are using the sample `tegola-osm.toml` config.

**Note:** If you are looking for the old `osm_land.sh` script it can be found in `scripts/deprecated/`.

#### Command Args

* `-v` Verbose mode

#### Required Env Vars

* `DB_HOST`: PostgreSQL database host name
* `DB_PORT`: PostgreSQL database port
* `DB_NAME`: PostgreSQL database name
* `DB_USER`: PostgreSQL database username
* `DB_PW`: PostgreSQL database password
* `OSM_LAND_MAPPING`: Full path to the json mapping file we want to use. Eg: `/osm/config/osm-land-mapping.json`
* `TEMP_DATA_DIR`: Full path to the directory where this script will temporarily store data. Script user must have permissions to read/write/delete from this directory. Eg: `/tmp`

#### Example Docker Command

```bash
docker run -i --rm \
-u "${UID}" \
--mount "type=bind,source=/osm_data/config/osm-land-mapping.json,target=/osm/config/osm-land-mapping.json" \
--mount "type=bind,source=/tmp,target=/osm/temp" \
-e "DB_HOST=localhost" \
-e "DB_PORT=5432" \
-e "DB_NAME=ne_data" \
-e "DB_USER=postgres" \
-e "DB_PW=postgres" \
-e "OSM_LAND_MAPPING=/ne/mapping.json" \
-e "TEMP_DATA_DIR=/ne/temp" \
"tegola-osm:latest" \
bash -c "osm_land_import.sh"
```

## Install SQL helper functions
Execute `postgis_helpers.sql` against your OSM database. Currently this contains a single utility function for converting building heights from strings to numbers which is important if you want to extrude buildings for the 3d effect.
Expand All @@ -72,13 +212,13 @@ psql -U tegola -d database-name -a -f postgis_index.sql
## Launch tegola

```bash
./tegola -config=tegola.toml
./tegola -config=tegola-osm.toml
```

Open your browser to localhost and the port you configured tegola to run on (i.e. localhost:8080) to see the built in viewer.

## Data Layers
To view these data layers in a map and query the features for a better understanding of each data layer, use the [Tegola-OSM Inspector](https://osm.tegola.io). The data layers described here are in the "Tegola-OSM" database as laid out in the tegola.toml (i.e., not the Natural Earth database that is specified in tegola-natural-earth.toml).
To view these data layers in a map and query the features for a better understanding of each data layer, use the [Tegola-OSM Inspector](https://osm.tegola.io). The data layers described here are in the "Tegola-OSM" database as laid out in the tegola-osm.toml (i.e., not the Natural Earth database that is specified in tegola-natural-earth.toml).

| source | Description |
|--------|-------------|
Expand Down
Loading