openaddress import elasticsearch duplicate values search for pip-service.

[imran-5]

Hey team!

I was geocoding with the manual build Openaddress data import to Elasticsearch.
I have noticed that there is a duplicate search request for pip-service.
Latitude and longitude are the same for duplicate requests.

logs are following.

2019-11-15T17:19:10.332Z - debug: [pip] pip: http://0.0.0.0:3102/-45.554319/-20.289574?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.332Z - debug: [pip] pip: http://0.0.0.0:3102/-45.554319/-20.289574?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.332Z - debug: [pip] pip: http://0.0.0.0:3102/-45.554319/-20.289574?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.332Z - debug: [pip] pip: http://0.0.0.0:3102/-45.554319/-20.289574?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.332Z - debug: [pip] pip: http://0.0.0.0:3102/-45.554319/-20.289574?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.356Z - debug: [pip] pip: http://0.0.0.0:3102/-45.554529/-20.288676?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.356Z - debug: [pip] pip: http://0.0.0.0:3102/-45.554529/-20.288676?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.357Z - debug: [pip] pip: http://0.0.0.0:3102/-45.554529/-20.288676?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.357Z - debug: [pip] pip: http://0.0.0.0:3102/-45.554529/-20.288676?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.358Z - debug: [pip] pip: http://0.0.0.0:3102/-45.553866/-20.288582?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.358Z - debug: [pip] pip: http://0.0.0.0:3102/-45.553866/-20.288582?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.358Z - debug: [pip] pip: http://0.0.0.0:3102/-45.553866/-20.288582?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.359Z - debug: [pip] pip: http://0.0.0.0:3102/-45.553866/-20.288582?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.359Z - debug: [pip] pip: http://0.0.0.0:3102/-45.553866/-20.288582?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.359Z - debug: [pip] pip: http://0.0.0.0:3102/-45.553866/-20.288582?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.360Z - debug: [pip] pip: http://0.0.0.0:3102/-45.553342/-20.288973?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.361Z - debug: [pip] pip: http://0.0.0.0:3102/-45.553342/-20.288973?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.361Z - debug: [pip] pip: http://0.0.0.0:3102/-45.553342/-20.288973?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.361Z - debug: [pip] pip: http://0.0.0.0:3102/-45.554006/-20.289067?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.362Z - debug: [pip] pip: http://0.0.0.0:3102/-45.554006/-20.289067?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.362Z - debug: [pip] pip: http://0.0.0.0:3102/-45.554006/-20.289067?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.362Z - debug: [pip] pip: http://0.0.0.0:3102/-45.554006/-20.289067?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.363Z - debug: [pip] pip: http://0.0.0.0:3102/-45.554006/-20.289067?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.363Z - debug: [pip] pip: http://0.0.0.0:3102/-45.554006/-20.289067?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.364Z - debug: [pip] pip: http://0.0.0.0:3102/-45.554006/-20.289067?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.364Z - debug: [pip] pip: http://0.0.0.0:3102/-45.554366/-20.288117?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.364Z - debug: [pip] pip: http://0.0.0.0:3102/-45.554366/-20.288117?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.365Z - debug: [pip] pip: http://0.0.0.0:3102/-45.554366/-20.288117?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.365Z - debug: [pip] pip: http://0.0.0.0:3102/-45.554366/-20.288117?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.365Z - debug: [pip] pip: http://0.0.0.0:3102/-45.554366/-20.288117?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.372Z - debug: [pip] pip: http://0.0.0.0:3102/-45.553693/-20.287994?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.372Z - debug: [pip] pip: http://0.0.0.0:3102/-45.553693/-20.287994?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.373Z - debug: [pip] pip: http://0.0.0.0:3102/-45.553693/-20.287994?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.373Z - debug: [pip] pip: http://0.0.0.0:3102/-45.553693/-20.287994?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.373Z - debug: [pip] pip: http://0.0.0.0:3102/-45.553179/-20.288414?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.374Z - debug: [pip] pip: http://0.0.0.0:3102/-45.553179/-20.288414?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.374Z - debug: [pip] pip: http://0.0.0.0:3102/-45.553179/-20.288414?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.375Z - debug: [pip] pip: http://0.0.0.0:3102/-45.553179/-20.288414?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.375Z - debug: [pip] pip: http://0.0.0.0:3102/-45.553179/-20.288414?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.375Z - debug: [pip] pip: http://0.0.0.0:3102/-45.553853/-20.288537?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.376Z - debug: [pip] pip: http://0.0.0.0:3102/-45.553853/-20.288537?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.376Z - debug: [pip] pip: http://0.0.0.0:3102/-45.553853/-20.288537?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.376Z - debug: [pip] pip: http://0.0.0.0:3102/-45.553853/-20.288537?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.377Z - debug: [pip] pip: http://0.0.0.0:3102/-45.553853/-20.288537?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.377Z - debug: [pip] pip: http://0.0.0.0:3102/-45.553853/-20.288537?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.377Z - debug: [pip] pip: http://0.0.0.0:3102/-45.553853/-20.288537?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.378Z - debug: [pip] pip: http://0.0.0.0:3102/-45.553882/-20.286808?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.378Z - debug: [pip] pip: http://0.0.0.0:3102/-45.553882/-20.286808?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.379Z - debug: [pip] pip: http://0.0.0.0:3102/-45.553882/-20.286808?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.379Z - debug: [pip] pip: http://0.0.0.0:3102/-45.553137/-20.286131?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.379Z - debug: [pip] pip: http://0.0.0.0:3102/-45.553137/-20.286131?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.380Z - debug: [pip] pip: http://0.0.0.0:3102/-45.553253/-20.287123?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.380Z - debug: [pip] pip: http://0.0.0.0:3102/-45.553253/-20.287123?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.380Z - debug: [pip] pip: http://0.0.0.0:3102/-45.553961/-20.287879?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.381Z - debug: [pip] pip: http://0.0.0.0:3102/-45.553961/-20.287879?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.381Z - debug: [pip] pip: http://0.0.0.0:3102/-45.55321/-20.287138?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.381Z - debug: [pip] pip: http://0.0.0.0:3102/-45.552698/-20.288198?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.382Z - debug: [pip] pip: http://0.0.0.0:3102/-45.549691/-20.291573?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.382Z - debug: [pip] pip: http://0.0.0.0:3102/-45.547237/-20.293432?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.383Z - debug: [pip] pip: http://0.0.0.0:3102/-45.547237/-20.293432?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.383Z - debug: [pip] pip: http://0.0.0.0:3102/-45.547237/-20.293432?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.383Z - debug: [pip] pip: http://0.0.0.0:3102/-45.547237/-20.293432?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.384Z - debug: [pip] pip: http://0.0.0.0:3102/-45.547237/-20.293432?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.384Z - debug: [pip] pip: http://0.0.0.0:3102/-45.547237/-20.293432?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.384Z - debug: [pip] pip: http://0.0.0.0:3102/-45.547237/-20.293432?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.385Z - debug: [pip] pip: http://0.0.0.0:3102/-45.547237/-20.293432?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.385Z - debug: [pip] pip: http://0.0.0.0:3102/-45.547237/-20.293432?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.386Z - debug: [pip] pip: http://0.0.0.0:3102/-45.547237/-20.293432?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.386Z - debug: [pip] pip: http://0.0.0.0:3102/-45.547237/-20.293432?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.386Z - debug: [pip] pip: http://0.0.0.0:3102/-45.547237/-20.293432?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.387Z - debug: [pip] pip: http://0.0.0.0:3102/-45.547237/-20.293432?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.387Z - debug: [pip] pip: http://0.0.0.0:3102/-45.547237/-20.293432?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.387Z - debug: [pip] pip: http://0.0.0.0:3102/-45.547237/-20.293432?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.388Z - debug: [pip] pip: http://0.0.0.0:3102/-45.546644/-20.295129?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.388Z - debug: [pip] pip: http://0.0.0.0:3102/-45.546644/-20.295129?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.389Z - debug: [pip] pip: http://0.0.0.0:3102/-45.546644/-20.295129?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.389Z - debug: [pip] pip: http://0.0.0.0:3102/-45.546644/-20.295129?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.389Z - debug: [pip] pip: http://0.0.0.0:3102/-45.546644/-20.295129?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.390Z - debug: [pip] pip: http://0.0.0.0:3102/-45.553255/-20.289628?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.390Z - debug: [pip] pip: http://0.0.0.0:3102/-45.552727/-20.290016?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.390Z - debug: [pip] pip: http://0.0.0.0:3102/-45.555402/-20.291661?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.391Z - debug: [pip] pip: http://0.0.0.0:3102/-45.553247/-20.291234?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.391Z - debug: [pip] pip: http://0.0.0.0:3102/-45.554006/-20.289067?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.391Z - debug: [pip] pip: http://0.0.0.0:3102/-46.144255/-21.356963?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.392Z - debug: [pip] pip: http://0.0.0.0:3102/-46.144255/-21.356963?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.392Z - debug: [pip] pip: http://0.0.0.0:3102/-46.144255/-21.356963?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.393Z - debug: [pip] pip: http://0.0.0.0:3102/-46.144255/-21.356963?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.393Z - debug: [pip] pip: http://0.0.0.0:3102/-46.144255/-21.356963?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.393Z - debug: [pip] pip: http://0.0.0.0:3102/-46.144255/-21.356963?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.394Z - debug: [pip] pip: http://0.0.0.0:3102/-46.144255/-21.356963?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.394Z - debug: [pip] pip: http://0.0.0.0:3102/-46.144255/-21.356963?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.395Z - debug: [pip] pip: http://0.0.0.0:3102/-46.145334/-21.356781?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.395Z - debug: [pip] pip: http://0.0.0.0:3102/-46.145334/-21.356781?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.395Z - debug: [pip] pip: http://0.0.0.0:3102/-46.145334/-21.356781?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.396Z - debug: [pip] pip: http://0.0.0.0:3102/-46.145334/-21.356781?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.396Z - debug: [pip] pip: http://0.0.0.0:3102/-46.145334/-21.356781?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country
2019-11-15T17:19:10.396Z - debug: [pip] pip: http://0.0.0.0:3102/-46.145334/-21.356781?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country`

[imran-5]

Repost for the first 5 URL.

1. http://0.0.0.0:3102/-45.554319/-20.289574?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country

2. http://0.0.0.0:3102/-45.554319/-20.289574?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country

3. http://0.0.0.0:3102/-45.554319/-20.289574?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country

4. http://0.0.0.0:3102/-45.554319/-20.289574?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,country

5. http://0.0.0.0:3102/-45.554319/-20.289574?layers=neighbourhood,borough,locality,localadmin,county,macrocounty,region,macroregion,dependency,

[missinglink]

That does seem redundant, I suspect what you're seeing is multiple addresses within the same building which share the same lat/long in the source CSV.

It would be a nice optimization of we could prevent these duplicate (or even better very near duplicate) lat/lon queries.

Are you able to confirm that the duplication is in the source data and not an application error?

[imran-5]

Yes. There are multiple values in the CSV file.

---

[missinglink]

I'm not sure how easy it would be to apply some sort of query caching for these queries...

If there was no parallelism it would be simple, it's still totally possible, maybe a rolling buffer which queues those addresses multiple times, queries once and then calls the callback for each matching item in the queue?

Anyway I think for now it's fine how it is, it's not causing any issues and so it may be preferable to leave it as-is (with a minor performance hit) rather than add complex code which may be prone to bugs?

I could be wrong, depends on the implementation..

[missinglink]

If someone is interested in looking at this, I think it could be a nice addition to https://github.com/pelias/microservice-wrapper/blob/master/service.js

I suspect that either there is an existing superagent plugin which does this or there is a way to do it using the plugin API.

Of course the implementation would need to be able to handle parallelism, although this is probably easy in nodejs because it's single-threaded.

[missinglink]

see: jpodwys/superagent-cache#41

[jpodwys]

Hi, I'm the author of superagent-cache. I just stumbled upon this issue and thought I'd link to the docs for the preventDuplicateCalls feature.