[Task] Use apache arrow to store metrics data

[tomsun28]

Description

use apache arrow to store metrics data instead of protobuf Field ValueRow

    // monitoring collect metric field
    repeated Field fields = 9;
    // monitoring collect metric data, mapping with the fields
    repeated ValueRow values = 10;

Task List

No response

[Calvin979]

Is it a task now? I'm interested in it

[tomsun28]

yes welcome. 👍

[tomsun28]

maybe we can use

   // collect response data
    bytes data = 9;

to instead of

    // monitoring collect metric field
    repeated Field fields = 9;
    // monitoring collect metric data, mapping with the fields
    repeated ValueRow values = 10;

in collect_rep.proto

and then use the apache arrow to construct the metrics data bytes data = 9

[Calvin979]

maybe we can use

   // collect response data
    bytes data = 9;

to instead of

    // monitoring collect metric field
    repeated Field fields = 9;
    // monitoring collect metric data, mapping with the fields
    repeated ValueRow values = 10;

in collect_rep.proto

and then use the apache arrow to construct the metrics data bytes data = 9

Thanks for your suggestion, I need to learn Apache Arrow first.

[Calvin979]

Todo List

• Complete comment on new method and class.
• Delete Field and ValueRow in collect_rep.proto.
• Modify collected data in package hertzbeat-collector
• Add JDK options in scripts. See also: https://arrow.apache.org/docs/java/install.html#java-compatibility
• Change unit test case in package hertzbeat-collector
• org.apache.hertzbeat.warehouse.store.history.HistoryDataWriter#saveData
• org.apache.hertzbeat.warehouse.store.realtime.RealTimeDataWriter#saveData
• Complete related doc, such as CONTRIBUTING