200k+ requests per hour from GoogleAssociationService to /.well-known/assetlinks.json on off2

[stephanegigandet]

In the last hour, we received more than 200k queries from the GoogleAssociationService bot on off2, that's 55k queries per second on average.

root@proxy:/var/log/nginx# tail -n 400000 openpetfoodfacts.org.log | grep 2023:16: | grep -c GoogleAssociationService
155288
root@proxy:/var/log/nginx# tail -n 400000 openbeautyfacts.org.log | grep 2023:16: | grep -c GoogleAssociationService
24577
root@proxy:/var/log/nginx# tail -n 400000 openproductsfacts.org.log | grep 2023:16: | grep -c GoogleAssociationService
24480

Sample queries:

66.102.9.206 - - [10/Oct/2023:16:00:00 +0000] "GET /.well-known/assetlinks.json HTTP/1.1" 404 136
 "-" "GoogleAssociationService" "-"
66.102.9.75 - - [10/Oct/2023:16:00:00 +0000] "GET /.well-known/assetlinks.json HTTP/1.1" 404 136 
"-" "GoogleAssociationService" "-"
66.102.9.100 - - [10/Oct/2023:16:00:00 +0000] "GET /.well-known/assetlinks.json HTTP/1.1" 404 136
 "-" "GoogleAssociationService" "-"
66.249.93.96 - - [10/Oct/2023:16:00:00 +0000] "GET /.well-known/assetlinks.json HTTP/1.1" 404 136
 "-" "GoogleAssociationService" "-"
66.249.93.96 - - [10/Oct/2023:16:00:00 +0000] "GET /.well-known/assetlinks.json HTTP/1.1" 404 136
 "-" "GoogleAssociationService" "-"
66.102.9.130 - - [10/Oct/2023:16:00:00 +0000] "GET /.well-known/assetlinks.json HTTP/1.1" 404 136
 "-" "GoogleAssociationService" "-"
66.102.9.135 - - [10/Oct/2023:16:00:00 +0000] "GET /.well-known/assetlinks.json HTTP/1.1" 404 136
 "-" "GoogleAssociationService" "-"
66.102.9.104 - - [10/Oct/2023:16:00:00 +0000] "GET /.well-known/assetlinks.json HTTP/1.1" 404 136
 "-" "GoogleAssociationService" "-"
66.249.93.101 - - [10/Oct/2023:16:00:00 +0000] "GET /.well-known/assetlinks.json HTTP/1.1" 404 13
6 "-" "GoogleAssociationService" "-"
66.249.93.169 - - [10/Oct/2023:16:00:00 +0000] "GET /.well-known/assetlinks.json HTTP/1.1" 404 13
6 "-" "GoogleAssociationService" "-"
66.102.9.167 - - [10/Oct/2023:16:00:00 +0000] "GET /.well-known/assetlinks.json HTTP/1.1" 404 136
 "-" "GoogleAssociationService" "-"
66.102.9.7 - - [10/Oct/2023:16:00:00 +0000] "GET /.well-known/assetlinks.json HTTP/1.1" 404 136 "
-" "GoogleAssociationService" "-"
66.249.93.65 - - [10/Oct/2023:16:00:00 +0000] "GET /.well-known/assetlinks.json HTTP/1.1" 404 136
 "-" "GoogleAssociationService" "-"
66.102.9.202 - - [10/Oct/2023:16:00:00 +0000] "GET /.well-known/assetlinks.json HTTP/1.1" 404 136
 "-" "GoogleAssociationService" "-"
66.102.9.4 - - [10/Oct/2023:16:00:00 +0000] "GET /.well-known/assetlinks.json HTTP/1.1" 404 136 "
-" "GoogleAssociationService" "-"
66.102.9.68 - - [10/Oct/2023:16:00:00 +0000] "GET /.well-known/assetlinks.json HTTP/1.1" 404 136 
"-" "GoogleAssociationService" "-"
142.250.32.4 - - [10/Oct/2023:16:00:00 +0000] "GET /.well-known/assetlinks.json HTTP/1.1" 404 136
 "-" "GoogleAssociationService" "-"
66.249.93.233 - - [10/Oct/2023:16:00:00 +0000] "GET /.well-known/assetlinks.json HTTP/1.1" 404 13
6 "-" "GoogleAssociationService" "-"
66.249.93.3 - - [10/Oct/2023:16:00:00 +0000] "GET /.well-known/assetlinks.json HTTP/1.1" 404 136 
"-" "GoogleAssociationService" "-"
66.102.9.133 - - [10/Oct/2023:16:00:00 +0000] "GET /.well-known/assetlinks.json HTTP/1.1" 404 136
 "-" "GoogleAssociationService" "-"
66.102.9.227 - - [10/Oct/2023:16:00:00 +0000] "GET /.well-known/assetlinks.json HTTP/1.1" 404 136
 "-" "GoogleAssociationService" "-"
66.102.9.37 - - [10/Oct/2023:16:00:00 +0000] "GET /.well-known/assetlinks.json HTTP/1.1" 404 136 
"-" "GoogleAssociationService" "-"
66.102.9.3 - - [10/Oct/2023:16:00:00 +0000] "GET /.well-known/assetlinks.json HTTP/1.1" 404 136 "
-" "GoogleAssociationService" "-"
66.102.9.134 - - [10/Oct/2023:16:00:00 +0000] "GET /.well-known/assetlinks.json HTTP/1.1" 404 136
 "-" "GoogleAssociationService" "-"
66.102.9.39 - - [10/Oct/2023:16:00:00 +0000] "GET /.well-known/assetlinks.json HTTP/1.1" 404 136 
"-" "GoogleAssociationService" "-"
142.250.32.34 - - [10/Oct/2023:16:00:00 +0000] "GET /.well-known/assetlinks.json HTTP/1.1" 404 13
6 "-" "GoogleAssociationService" "-"
66.102.9.1 - - [10/Oct/2023:16:00:00 +0000] "GET /.well-known/assetlinks.json HTTP/1.1" 404 136 "
-" "GoogleAssociationService" "-"
192.178.8.33 - - [10/Oct/2023:16:00:00 +0000] "GET /.well-known/assetlinks.json HTTP/1.1" 404 136
 "-" "GoogleAssociationService" "-"
66.102.9.168 - - [10/Oct/2023:16:00:00 +0000] "GET /.well-known/assetlinks.json HTTP/1.1" 404 136
 "-" "GoogleAssociationService" "-"

For some reason this behavior seems to happen with obf, opf and opff but not off.

[stephanegigandet]

On opf the file is missing and Google gets a 404, but on off and obf we serve a file.

[alexgarel]

Could it be that we should add a Cache-Control header ?

[stephanegigandet]

I tried it on obf, but for some reason Google requests the file and gets a 301, while I get a 200...

66.249.93.65 - - [08/Nov/2023:15:10:36 +0000] "GET /.well-known/assetlinks.json HTTP/1.1" 301 169 "-" "GoogleAssociationService" "-"
66.249.93.64 - - [08/Nov/2023:15:10:36 +0000] "GET /.well-known/assetlinks.json HTTP/1.1" 301 169 "-" "GoogleAssociationService" "-"

This is what I added in the nginx config file on the obf container:

        location = /.well-known/assetlinks.json {
                include snippets/off.cors-headers.include;
                expires 1d;
                add_header Cache-Control "public, no-transform";
                try_files $uri $uri/ =404;
        }