Stop allowing bulk downloading of tile.openstreetmap.org tiles

[Firefishy]

I am part of the OpenStreetMap.org sysadmin team.
tile.openstreetmap.org traffic exceeds 1.6TB/hour and 41600 req/second during our daily peak times. The service is run by us, we are all volunteers.

We have a simple usage policy and don't require API keys or anything like that. We intentionally try keep the bar to using our tiles low.

BUT! Our usage policy specifically disallows bulk downloading of tiles because it puts extreme pressure on our infrastructure and strains our limited on-demand rendering capacity. Bulk downloading of tiles disrupts our service and severely degrades our ability to service tiles.

1. We ask that QTiles STOP allowing bulk downloading of tile.openstreetmap.org tiles.
2. We ask that QTiles use a dedicated HTTP user-agent (eg: QTiles/1.0.0) to allow us to differentiate between QGIS (built-in TMS support) and QTiles requests.

It would be a massive loss to the QGIS community if we were forced to block QGIS entirely because of QTiles allowing bulk downloading of OSM tiles.

[simgislab]

1. QTiles for QGIS 3 that is currently available though official QGIS plugins repo is not our development. It was an unsupported (by us) port (by someone) and we don't control it. I.e .you might be at the wrong address.

2. I don't understand the solution yet. QTiles does NOT separately pull OSM tiles. The common use case is: (a) Add OSM to QGIS (b) Render a tileset from project. It is QGIS that renders and pulls data (QgsMapRendererCustomPainterJob).

We can do something on our side (our version of QTiles that we indeed control).

[tomhughes]

The problem is that we on the server side can't tell the difference between somebody using QGIS interactively and somebody "rendering a tileset" as you put it.

I don't know enough about QGIS to know exactly what you mean by that - is that essentially asking it to render and export an image for a given area using it's configured layers? If so then I guess we need to talk to QGIS about using a modified user agent in that case.

[simgislab]

is that essentially asking it to render and export an image for a given area using it's configured layers

yes

need to talk to QGIS

yes

[Zverik]

@Firefishy are there some numbers linking QTiles or QGIS to bulk downloading tiles? I assumed this was too complex for a regular person. Maybe some other tool like SASPlanet is spoofing the user agent?

[Firefishy]

Hmmm ok, so this is likely the bulk tile downloader we are after then: https://bitbucket.org/we-do-it/qgis-latlongo-stage-1/src/master/

[Firefishy]

@Firefishy are there some numbers linking QTiles or QGIS to bulk downloading tiles? I assumed this was too complex for a regular person. Maybe some other tool like SASPlanet is spoofing the user agent?

We primarily track QGIS by its User-Agent. Most QGIS plugins use the QGIS user-agent. QGIS is the single biggest tile consuming app. @pnorman can provide stats breakdown.

Yes, spoofing the User-Agent is possible. Classic tragedy of the commons.

[simgislab]

reopen if necessary

[Firefishy]

@simgislab Please be aware we are going to escalate this issue now. Your tool is abusing OpenStreetMap services. You are degrading our services used by others. I will contact you offline with more details.

[Firefishy]

PS: I cannot re-open this issue.

[simgislab]

are there any numbers showing that this is specifically QTiles problem and not general QGIS or other plugins?

[Zverik]

@Firefishy please be reasonable. NextGIS is not abusing OSM services: they just made the plugin. QTiles is not abusing OSM services: it renders a QGIS project into tiles, not downloads OSM tiles specifically. There cannot be an easy solution, and threats won't help resolve it faster. Discussion is better than ultimatums. We are open-source developers, just like you.

[Zverik]

By me, the better solution would be throttling OSM tiles downloads in a plugin that supplies OSM background layer to QGIS.

[simgislab]

throttling OSM tiles downloads

this sounds like something for core QGIS level not plugins

[Firefishy]

@Firefishy please be reasonable. NextGIS is not abusing OSM services: they just made the plugin. QTiles is not abusing OSM services: it renders a QGIS project into tiles, not downloads OSM tiles specifically. There cannot be an easy solution, and threats won't help resolve it faster. Discussion is better than ultimatums. We are open-source developers, just like you.

QTiles plugin is specifically a plugin for mass downloading tiles. The plugin traffic is superficially indistinguishable (user-agent, referer) from using OpenStreetMap as a background layer in QGIS. We regularly have users mass downloading tiles for 20+ hours as fast as possible. No human browsers a map for 20+ hours continuously constantly zooming-in and them moving through the zoom layers tile by tile.

The current status quo is OpenStreetMap.org will run out of capacity to serve traffic to QGIS (currently 50% of our tile traffic) later this year.

[Firefishy]

The traffic is as far as I can tell caused specifically by this QGIS plugin and not standard QGIS.

[Zverik]

QTiles plugin is specifically a plugin for mass downloading tiles

I frankly have no idea how you came to this conclusion, given there's no mention of OSM tiles in the code, and the readme tells a different story.

How did you even find this plugin?

Why don't you blame the traffic on exporting QGIS projects to pdf/printing, for example? It's not a very popular plugin.

[simgislab]

how about this? no plugin needed at all, core QGIS
https://gis.stackexchange.com/questions/415775/is-it-possible-to-extract-tiles-layer-as-raster-in-qgis/415781#415781

Please. We don't mind fixing QTiles somehow, but I'd like some certainty it's really QTiles behind the problems you mention before jumping to work on some solution.

[Zverik]

I'd suggest moving this discussion to QGIS repo, and coming with solutions there. From my experience, QGIS shows tiles of higher zoom than expected (e.g. on z13.5 it shows z14, while z13 would be perfectly suitable), has issues with caching, and doesn't throttle or otherwise limit tile layers.

[Firefishy]

how about this? no plugin needed at all, core QGIS https://gis.stackexchange.com/questions/415775/is-it-possible-to-extract-tiles-layer-as-raster-in-qgis/415781#415781

Please. We don't mind fixing QTiles somehow, but I'd like some certainty it's really QTiles behind the problems you mention before jumping to work on some solution.

That downloads a single zoom level. We are seeing multiple zoom levels mass scrapped.

[Firefishy]

QTiles plugin is specifically a plugin for mass downloading tiles

I frankly have no idea how you came to this conclusion, given there's no mention of OSM tiles in the code, and the readme tells a different story.

How did you even find this plugin?

Why don't you blame the traffic on exporting QGIS projects to pdf/printing, for example? It's not a very popular plugin.

QGIS native export tools do not systematically scrape multiple zoom levels.
Your ill-informed comments are not helpful. OSM is the default background layer in QGIS and this plugin facilitates the scraping of that laying in violation of our usage terms. QGIS core team are aware of the issue, I will raise it again with them.

I or other OSM operations team member will provide some anonymised logs to demonstrate the scraping in the next day or 2.

[simgislab]

@Firefishy QTiles can be used to generate a raster pyramid using QGIS project as an input. QGIS is doing all the requests rendering needed tiles. OSM basemap may be one of the layers. If this is the case and user request several zoom levels, QGIS will render all layers consequently requesting OSM tiles.

However, I'm still doubtful that QTiles became so popular that lots of people would start using this feature so much that it may compare with orders of magnitude larger crowd using OSM as a basemap in a common way: browsing, printing maps. etc.

as fast as possible

If I remember correctly, QTiles is single thread which further limits its abilities for 'mass download'. It was never optimized for this.

All that said, again, we're ready to work on fixing this issue together, just need bit more info to come up with the best approach.

[pnorman]

If I remember correctly, QTiles is single thread which further limits its abilities for 'mass download'. It was never optimized for this.

What order is it downloading them in? I could check this.

[Firefishy]

The best approach would be to limit max zoom to z12 or z13, beyond z12 we are rendering on demand, we have very limited render capacity which can be overwhelmed by mass downloading tiles which are rendered-on-demand.

[eduard-kazakov]

QGIS native export tools do not systematically scrape multiple zoom levels.

@Firefishy It is worth saying that QGIS now has a built-in processing tool to create a multiscale tile cache from the project.
https://docs.qgis.org/3.28/en/docs/user_manual/processing_algs/qgis/rastertools.html#generate-xyz-tiles-directory
This tool downloads all tiles from all configured zoom levels, as QTiles does. But as long as it is a standard built-in tool, I guess it is much more popular, it could be integrated into model builder models, run from PyQGIS code easily, etc.

[Firefishy]

QGIS native export tools do not systematically scrape multiple zoom levels.

@Firefishy It is worth saying that QGIS now has a built-in processing tool to create a multiscale tile cache from the project. https://docs.qgis.org/3.28/en/docs/user_manual/processing_algs/qgis/rastertools.html#generate-xyz-tiles-directory This tool downloads all tiles from all configured zoom levels, as QTiles does. But as long as it is a standard built-in tool, I guess it is much more popular, it could be integrated into model builder models, run from PyQGIS code easily, etc.

Sigh. Destroying the commons bit by bit.

[simgislab]

hopefully not so dramatic, if you finally 'talk to QGIS' :)

[simgislab]

Our dev is back from a vacation and we've checked this again.

QTiles doesn't have it's own mechanism to download tiles and uses the same mechanism as QGIS: QgsWmsTiledImageDownloadHandler (uses QgsWmsProvider::QgsWmsProvider)
https://github.com/qgis/QGIS/blob/master/src/providers/wms/qgswmsprovider.cpp

So more general solution would be: add an exception to these methods in QGIS so that if sees *.osm.org it does something to lessen the load. When it's done, QTiles will automatically be 'fixed'.

We've created another issue to discuss interim solution for QTiles specifically.
#125

[Firefishy]

Our dev is back from a vacation and we've checked this again.

QTiles doesn't have it's own mechanism to download tiles and uses the same mechanism as QGIS: QgsWmsTiledImageDownloadHandler (uses QgsWmsProvider::QgsWmsProvider) https://github.com/qgis/QGIS/blob/master/src/providers/wms/qgswmsprovider.cpp

So more general solution would be: add an exception to these methods in QGIS so that if sees *.osm.org it does something to lessen the load. When it's done, QTiles will automatically be 'fixed'.

We've created another issue to discuss interim solution for QTiles specifically. #125

Thanks. We don't mind a small amount of downloaded tiles, but 10,000+ are way too many.

[simgislab]

more general discussion continued under OSM operations openstreetmap/operations#1019

more specific discussion on QTiles itself should be continued here #125