Prepare request to QGIS to better manage OSM tile demand

[Firefishy]

QGIS's tile.openstreetmap.org tile usage continues to grow rapidly primarily due to a very small minority of users mass downloading tiles. The mass downloading of tiles is a strain on OpenStreetMap's limited resources.

At the moment QGIS's user-agent is requesting ~1300 tiles per second (24h average) with a very high portion (520 tiles per second) of these request are uncached. OpenStreetMap.org for reference is ~1630 tiles per second of which 485 tiles per second are uncached.

Many of the requests seem to be in violation of our usage policy

1. Extremely heavy usage by a very small % of users who appear to be mass scraping tiles in violation of our usage policy.
2. QGIS appears to insufficiently cache tiles. QGIS appears to only cache 100MB of tiles which is insufficient and likely in violation of our caching policy.
3. While strictly not in violation of our policy, we are unable to identify if the tile request traffic is from QGIS proper or a plugin or a processing provider.

We should investigate the usage and formulate a request the QGIS to better manage the usage. QGIS are our friends, and we want to continue to allow OpenStreetMap tiles to be used in QGIS.

[Firefishy]

There are plugins which specifically facilitate mass tile downloading like nextgis/qgis_qtiles#111

[Firefishy]

QGIS now also has the ability to mass scrape tiles via the processing tool which is a direct violation of our usage policy.

[craigallan]

I read this with concern. QGIS are our friends so we approach it gently. Is this not a matter where the Board should contact QGIS management and politely say that we have a problem and want to set up a channel of co-operation to fix it.

[Firefishy]

I read this with concern. QGIS are our friends so we approach it gently. Is this not a matter where the Board should contact QGIS management and politely say that we have a problem and want to set up a channel of co-operation to fix it.

Absolutely agree. QGIS are our friends. This ticket is to collect the details of the problem and clearly define what is the issue.

99% of the QGIS users are not the problem, it is the 1% of users who are mass downloading map tiles that are unlikely to ever be viewed.

[craigallan]

That was my thought. Someone is downloading crazy amounts of tiles they don't need. Nobody could ever view so many tiles. Is it an denial of service attack? But whatever it is the QGIS Board should not allow plug-ins that request crazy amounts of tiles. Anyway, once you have a plan, I'm happy to do formal emails - let me know.

[Firefishy]

I have clarified the opening paragraph of this ticket.

[Firefishy]

Is it an denial of service attack?

It is not a denial of service attack. It is a small minority of users mass downloading tiles likely for offline usage. There are much better ways to use OpenStreetMap offline (eg: raster maps like OrganicMaps).

An example might be a QGIS user attempting to download all tile.openstreetmap.org map tiles for Germany. Up to zoom level 15 it would 8GB across 1.3 million request, upto zoom 16 is 23GB 5.4 million request, upto zoom 17 is 61GB across 21 million requests, upto zoom 18 is 217GB across 86 million requests. Our tile.openstreetmap.org service also has zoom 19. We generate the higher zoom levels on-demand and which is taxing on server resources. Normally only a fraction of tiles are ever viewed.

[craigallan]

I'm thinking this through...

Can you clarify "very small % of users" even if it is a guesstimate.
These few users are consuming 45% of our tile serving capacity and costing us a lot of money.

You're guessing that users are doing this to build a local store of all tiles at all zoom levels that reflects a QGIS project.
Only if OSM data is a layer of the project will OSM tiles be added to the new local store of tiles.

And maybe they'll refresh that local store by doing the same again in a few days time.
Quite possibly they are not aware of the problem that this causes to OSM.

We certainly cannot allow this to persist.

To control it from our end:

• Technically can we throttle (or flatly deny) any one user that requests more than x MB of tiles in a minute?

To control it from QGIS:

• we need to find the exact guilty plug-in, which will probably require help from QGIS to reproduce the problem -- the likely suspect is QTiles made by NextGis https://gis-lab.info/qa/qtiles-eng.html Contact: Maxim Dubinin

• QGIS will ask the Developers to always identify the plug-in as a unique user agent, and

• require the plug-in to self-throttle itself or QGIS will de-register it and OSM will block it.

is that reasonable?

[Firefishy]

Great!

[craigallan]

Firefishy: Guillaume advises that you have it all under control and the Board does not need to assist. I stand down.

[pnorman]

assigning myself as I need to gather some data and examples

[simgislab]

we're ready to help on QTiles side
custom user-agent is a great suggestion that will finally allow everyone to understand if QTiles is the correct suspect

[matkoniecz]

QGIS now also has the ability to mass scrape tiles via the processing tool which is a direct violation of our usage policy.

would it be helpful for me to file feature request that would add domain checking and fail on using this functionality with OSMF tile servers? As this by definition violates tile usage policy?

[Firefishy]

Upstream QGIS have made improvements to improve caching. The improved features will be adopted as new versions are rolled out.

[pnorman]

The linked issues talk about prefetching and other improvements to live map views. Have they also done something that will stop or allow us to identify users who are trying to bulk-download tiles?