Data Updates

Data Updates

To enable regular graph updates (other data providers often make no sense), you'll have to set up cron jobs for:

• daily updating the OSM PBF files
• updating the graphs of the registered jobs according to the desired intervals, one/more of ["daily", "weekly", "monthly"]

Note, if you go the docker-compose route, this will all be set up for you by the ./docker-entrypoint.sh script.

Update OSM PBF files

To update the OSM PBF files in regular intervals, you can schedule a cron job using the ./cron/routing_packager_update_osm.sh script:

routing_packager_update_osm.sh --interval -i [minutely|hourly|*daily*|weekly] --dir-pbf -d /app/data/osm"

The arguments are:

• --interval: determines the granularity of the PBF file updates. One of ["minutely", "hourly", "daily", "weekly"]. Default "daily".
• --dir-pbf: the directory path of the OSM files path. Default /app/data/osm.

To install the script in your crontab to run every night at 3 am, you could do

(crontab -l || true; echo "0 3 * * * /app/cron/routing_packager_update_osm.sh -i daily -d /app/data/osm > /var/logs/osm_updater.log 2>&1") | crontab -

Note, the update process uses osmupdate which lacks one important feature: complete_ways. Which means that during update processes the edge ways might not get updates. We recommend to pull fresh OSM extracts every now and then.

This step is not necessary when running the stack with docker-compose.

Update registered graphs

The app is capable of running scheduled updates on registered graph generation jobs. The update frequency is determined by the job entry's interval field. Valid values for interval are ["once", "daily", "weekly", "monthly", "yearly"], while "once" is never updated.

The app provides a command line interface (flask update) to automate the updates:

$PWD/.venv/bin/flask update --help
Usage: flask update [OPTIONS] INTERVAL

  Update routing packages according to INTERVALs, one of ['once', 'daily',
  'weekly', 'monthly', 'yearly'].

Options:
  -c, --config [development|production|testing]
                                  Internal option
  --help                          Show this message and exit.

The script will pull the job entries from Postgres with matching INTERVAL and let the worker spin up the update procedures one-by-one, starting with the biggest bbox.

You can find the appropriate scripts in the ./cron directory. Inside the scripts change the location of the flask executable according to your setup and copy them to the respective cron folders:

sudo cp ./cron/routing_packager_daily.sh /etc/cron.daily
sudo cp ./cron/routing_packager_weekly.sh /etc/cron.weekly
sudo cp ./cron/routing_packager_monthly.sh /etc/cron.monthly

Of course you can also use crontab to manage the jobs with more scheduling granularity.

This step is not necessary when running the stack with docker-compose.