MINIFICPP-2501 Add processorStatuses C2 metric node to FlowInformation

apache · Dec 14, 2024 · efa4f0c · efa4f0c
1 parent 25882c1
commit efa4f0c
Show file tree

Hide file tree

Showing 20 changed files with 727 additions and 80 deletions.
diff --git a/C2.md b/C2.md
@@ -26,10 +26,23 @@ options defined are located in minifi.properties.
   - [Description](#description)
   - [Configuration](#configuration)
     - [Base Options](#base-options)
+      - [Flow Id and URL](#flow-id-and-url)
+      - [Agent Identifier Fallback](#agent-identifier-fallback)
     - [Metrics](#metrics)
     - [UpdatePolicies](#updatepolicies)
     - [Triggers](#triggers)
       - [C2 File triggers](#c2-file-triggers)
+  - [C2 Response Nodes](#c2-response-nodes)
+    - [AgentInformation](#agentinformation)
+    - [AgentStatus](#agentstatus)
+    - [AssetInformation](#assetinformation)
+    - [BuildInformation](#buildinformation)
+    - [ConfigurationChecksums](#configurationchecksums)
+    - [DeviceInfoNode](#deviceinfonode)
+    - [FlowInformation](#flowinformation)
+    - [QueueMetrics](#queuemetrics)
+    - [RepositoryMetrics](#repositorymetrics)
+    - [Processor Metric Response Nodes](#processor-metric-response-nodes)
 
 ## Description
 
@@ -271,3 +284,352 @@ in minifi.properties to activate the file update trigger specify
     # specifying a trigger
     nifi.c2.agent.trigger.classes=FileUpdateTrigger
     nifi.c2.file.watch=<full path of file to monitor>
+
+## C2 Response Nodes
+
+The following is a list of nodes that can be defined in the minifi.properties file for the C2 heartbeat response as part of the C2 root nodes defined in the `nifi.c2.root.classes` property or in the metrics nodes defined in the tree under `nifi.c2.root.class.definitions` as stated above.
+
+### AgentInformation
+
+Contains information about the agent's build, extensions, supported C2 operations and status of its components.
+
+```
+"agentInfo": {
+    "agentManifest": {
+        "buildInfo": {
+            "compiler": "/usr/lib/ccache/g++",
+            "flags": " -std=c++20;-Wall;-Wextra;-Werror;-Wno-error=restrict;SODIUM_STATIC=1",
+            "revision": "cc9aaac37a9a6b7efeb3c4394a97522a600a1758",
+            "timestamp": 1734001238,
+            "version": "0.99.1"
+        },
+        "bundles": [
+            {
+                "componentManifest": {
+                    "processors": [
+                    ...
+                    ]
+                },
+                "artifact": "minifi-civet-extensions",
+                "group": "org.apache.nifi.minifi",
+                "version": "0.99.1"
+            }
+        ],
+        "schedulingDefaults": {
+            "defaultMaxConcurrentTasks": 1,
+            "defaultRunDurationNanos": 0,
+            "defaultSchedulingPeriodMillis": 1000,
+            "defaultSchedulingStrategy": "TIMER_DRIVEN",
+            "penalizationPeriodMillis": 30000,
+            "yieldDurationMillis": 1000
+        },
+        "supportedOperations": [
+            {
+                "type": "acknowledge"
+            }
+            ...
+        ],
+        "agentType": "cpp",
+        "identifier": "bH77vXakM0Lkgt8VcDOGZVW3"
+    },
+    "status": {
+        "repositories": {
+            "content_repo": {
+                "entryCount": 0,
+                "full": false,
+                "maxSize": 0,
+                "running": true,
+                "size": 0
+            },
+            "flow_file_repo": {
+                "entryCount": 0,
+                "full": false,
+                "maxSize": 0,
+                "running": true,
+                "size": 0
+            },
+            "org::apache::nifi::minifi::core::repository::VolatileContentRepository": {
+                "entryCount": 4,
+                "full": false,
+                "maxSize": 7864320,
+                "running": true,
+                "size": 40
+            }
+        },
+        "components": {
+            "LogAttribute": {
+                "running": true,
+                "uuid": "5128e3c8-015a-1000-79ca-83af40ec1990"
+            },
+            "GenerateFlowFile": {
+                "running": true,
+                "uuid": "4fe2d51d-076a-49b0-88de-5cf5adf52b8f"
+            },
+            "FlowController": {
+                "running": true,
+                "uuid": "2438e3c8-015a-1000-79ca-83af40ec1990"
+            }
+        },
+        "resourceConsumption": {
+            "cpuUtilization": 0.05,
+            "memoryUsage": 97955840
+        },
+        "uptime": 1025
+    },
+    "agentClass": "test",
+    "agentManifestHash": "9FFC8326121A816E5B2FD674CE9A34321F89CC690AD0D1FD79DFB5969B3B523D6570520382E82C68CFA347FBD9897FC027E518E98CFA229C18617B062E1C9E77",
+    "identifier": "9628acfe-b9fe-11ef-a0c0-10f60a596f64"
+}
+```
+
+### AgentStatus
+
+Contains information about the agent's status, including the status of its components, repositories, and resource consumption.
+
+```
+"AgentStatus": {
+    "repositories": {
+        "repo_name": {
+            "entryCount": 0,
+            "full": false,
+            "maxSize": 0,
+            "running": true,
+            "size": 0
+        },
+        "ff": {
+            "entryCount": 0,
+            "full": false,
+            "maxSize": 0,
+            "running": true,
+            "size": 0
+        },
+        "org::apache::nifi::minifi::core::repository::VolatileContentRepository": {
+            "entryCount": 4,
+            "full": false,
+            "maxSize": 7864320,
+            "running": true,
+            "size": 40
+        }
+    },
+    "components": {
+        "LogAttribute": {
+            "running": true,
+            "uuid": "5128e3c8-015a-1000-79ca-83af40ec1990"
+        },
+        "GenerateFlowFile": {
+            "running": true,
+            "uuid": "4fe2d51d-076a-49b0-88de-5cf5adf52b8f"
+        },
+        "FlowController": {
+            "running": true,
+            "uuid": "2438e3c8-015a-1000-79ca-83af40ec1990"
+        }
+    },
+    "resourceConsumption": {
+        "cpuUtilization": 0.0028846153846153849,
+        "memoryUsage": 97955840
+    },
+    "uptime": 995
+}
+```
+
+### AssetInformation
+
+Contains the calculated hash of the assets.
+
+```
+"resourceInfo": {
+    "hash": "null"
+}
+```
+
+### BuildInformation
+
+Contains information about the agent's build.
+
+```
+"BuildInformation": {
+    "compiler": {
+        "compiler_command": "/usr/lib/ccache/g++",
+        "compiler_flags": " -std=c++20;-Wall;-Wextra;-Werror;-Wno-error=restrict;SODIUM_STATIC=1",
+        "compiler_version": "11.4.0"
+    },
+    "build_date": "1734001238",
+    "build_rev": "cc9aaac37a9a6b7efeb3c4394a97522a600a1758",
+    "build_version": "0.99.1",
+    "device_id": "bH77vXakM0Lkgt8VcDOGZVW3"
+}
+```
+
+### ConfigurationChecksums
+
+Metric node that defines checksums of configuration files in the C2 protocol.
+
+```
+"configurationChecksums": {
+    "SHA256": {
+        "TestC2Metrics.yml": "9af6589bf7729bb88857aafe98cea4f41df049725401b5f0ded0a7b949d9b90c",
+        "minifi.properties": "06fb9f4730e3db7d0a0a1ee606a7de3fee5813edf42eab140616e8a2995072df"
+    }
+},
+```
+
+### DeviceInfoNode
+
+Contains information about the device the agent is running on.
+
+```
+"deviceInfo": {
+    "systemInfo": {
+        "cpuLoadAverage": 1.271484375,
+        "cpuUtilization": 0.06179499754781756,
+        "machineArch": "x86_64",
+        "memoryUsage": 12681670656,
+        "operatingSystem": "Linux",
+        "physicalMem": 67081129984,
+        "vCores": 20
+    },
+    "networkInfo": {
+        "hostname": "ggyimesi-5570-ubuntu",
+        "ipAddress": "10.255.0.1"
+    },
+    "identifier": "16475557466943148337"
+}
+```
+
+### FlowInformation
+
+Contains information about the flow the agent is running, including the versioned flow snapshot URI, queues, components, and processor statuses.
+
+```
+"flowInfo": {
+    "versionedFlowSnapshotURI": {
+        "bucketId": "default",
+        "flowId": "96273342-b9fe-11ef-a0ad-10f60a596f64"
+    },
+    "queues": {
+        "8368e3c8-015a-1003-52ca-83af40ec1332": {
+            "dataSize": 40,
+            "dataSizeMax": 1048576,
+            "name": "GenerateFlowFile/success/LogAttribute",
+            "size": 4,
+            "sizeMax": 0,
+            "uuid": "8368e3c8-015a-1003-52ca-83af40ec1332"
+        }
+    },
+    "components": {
+        "LogAttribute": {
+            "running": true,
+            "uuid": "5128e3c8-015a-1000-79ca-83af40ec1990"
+        },
+        "GenerateFlowFile": {
+            "running": true,
+            "uuid": "4fe2d51d-076a-49b0-88de-5cf5adf52b8f"
+        },
+        "FlowController": {
+            "running": true,
+            "uuid": "2438e3c8-015a-1000-79ca-83af40ec1990"
+        }
+    },
+    "processorStatuses": [
+        {
+            "id": "5128e3c8-015a-1000-79ca-83af40ec1990",
+            "groupId": "2438e3c8-015a-1000-79ca-83af40ec1990",
+            "bytesRead": 0,
+            "bytesWritten": 0,
+            "flowFilesIn": 0,
+            "flowFilesOut": 0,
+            "bytesIn": 0,
+            "bytesOut": 0,
+            "invocations": 0,
+            "processingNanos": 0,
+            "activeThreadCount": -1,
+            "terminatedThreadCount": -1
+        },
+        {
+            "id": "4fe2d51d-076a-49b0-88de-5cf5adf52b8f",
+            "groupId": "2438e3c8-015a-1000-79ca-83af40ec1990",
+            "bytesRead": 0,
+            "bytesWritten": 40,
+            "flowFilesIn": 0,
+            "flowFilesOut": 4,
+            "bytesIn": 0,
+            "bytesOut": 40,
+            "invocations": 4,
+            "processingNanos": 2119148,
+            "activeThreadCount": -1,
+            "terminatedThreadCount": -1
+        }
+    ],
+    "flowId": "96273342-b9fe-11ef-a0ad-10f60a596f64"
+}
+```
+
+### QueueMetrics
+
+Contains information about the queues in the flow, including the contained data and number of flow files.
+
+```
+"QueueMetrics": {
+    "GenerateFlowFile/success/LogAttribute": {
+        "datasize": "40",
+        "datasizemax": "1048576",
+        "queued": "4",
+        "queuedmax": "0"
+    }
+}
+```
+
+### RepositoryMetrics
+
+Contains information about the repositories in the agent, including the number of entries, size, and whether the repository is full.
+
+
+```
+"RepositoryMetrics": {
+    "repo_name": {
+        "entryCount": 0,
+        "full": false,
+        "maxSize": 0,
+        "running": true,
+        "size": 0
+    },
+    "ff": {
+        "entryCount": 0,
+        "full": false,
+        "maxSize": 0,
+        "running": true,
+        "size": 0
+    },
+    "org::apache::nifi::minifi::core::repository::VolatileContentRepository": {
+        "entryCount": 4,
+        "full": false,
+        "maxSize": 7864320,
+        "running": true,
+        "size": 40
+    }
+}
+```
+
+### Processor Metric Response Nodes
+
+Each processor can have its own metrics. These metric nodes can be configured in the minifi.properties by requesting metrics in the <ProcessorTye>Metric format, for example GetTCPMetrics to request metrics for the GetTCP processors. Besides configuring processor metrics directly, they can also be configured using regular expressions with the `processorMetrics/` prefix. For example `processorMetrics/Get.*Metrics` will match all processor metrics that start with Get.
+
+```
+"GetTCPMetrics": {
+    "2438e3c8-015a-1000-79ca-83af40ec1991": {
+        "AverageOnTriggerRunTime": 0,
+        "AverageSessionCommitRunTime": 0,
+        "BytesRead": 0,
+        "BytesWritten": 0,
+        "IncomingBytes": 0,
+        "IncomingFlowFiles": 0,
+        "LastOnTriggerRunTime": 0,
+        "LastSessionCommitRunTime": 0,
+        "OnTriggerInvocations": 11,
+        "ProcessingNanos": 729328,
+        "TransferredBytes": 0,
+        "TransferredFlowFiles": 0
+    }
+}
+```