pcp2elasticsearch: support dynamic mapping #2117
Conversation
Elasticsearch 7.X removed the concept of mapping types (initially this was
depreciated in Elasticsearch 6.X).
Change over to support dynamic mapping by making use of the updated
'POST {server}/{index}/_doc' index API with auto-generated ids by default.
Users can still use the the search type '-p pcp-metric' command line argument
on launch to make use of the original pcp-metric mapping name for environments
with Elasticsearch 5.X and 6.X, ('POST {server}/{index}/{search_type}' API).
Note: These older versions are considered EOL by Elastic.
This fix addresses github issue performancecopilot#1998.
|
Thanks! I haven't followed Elasticsearch recently but based on your explanation this certainly makes sense. However, looking at the CI runs were some of the related QA test cases skipped? Have you checked locally is there any QA impact with this change? |
Update QA test 1130 to handle the dynamic mapping changes to the exporter along with providing an updated out file. Also commented out socat connection reset check in test due to finding it was being constantly flagged, even for pcp2elasticsearch version prior to the dynamic mapping patches.
|
Hi Marko, Apologies for missing the QA test update, have provided in a further commit. I was originally running into an issue with QA/1130 skipping, i'm assuming that CI was skipping this test also? It seems that 'socat' was giving errors for connection resets during the test runs ("socat on this platform is not behaving as expected"). This behaviour was also being triggered for me with the current upstream version of the exporter also. I have commented out the 'socat' connection reset notrun check in the test for now. Cheers |
|
Thank you! LGTM now but given that Ken spent a lot of time getting the QA running across different platform let's check with him if there would be any additional feedback - @kmcdonell ? Otherwise I think we could merge this. Thanks. |
|
I don't have anything to add on the code change, this is Python after all. |
Elasticsearch 7.X removed the concept of mapping types (initially this was depreciated in Elasticsearch 6.X).
Change over to support dynamic mapping by making use of the updated 'POST {server}/{index}/_doc' index API with auto-generated ids by default.
Users can still use the the search type '-p pcp-metric' command line argument on launch to make use of the original pcp-metric mapping name for environments with Elasticsearch 5.X and 6.X, ('POST {server}/{index}/{search_type}' API).
Note: These older versions are considered EOL by Elastic.
This fix addresses github issue #1998.