-
Notifications
You must be signed in to change notification settings - Fork 1
/
Copy pathmanifest.yml
525 lines (520 loc) · 22.7 KB
/
manifest.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
artifacts:
- name: OpcUaExtractorSetup-{version}.msi
path: artifacts/OpcUaExtractorSetup-{version}.msi
platform: windows
- name: opcua-extractor.win-x64.{version}.zip
path: artifacts/opcua-extractor.win-x64.{version}.zip
platform: windows
- name: opcua-extractor_{version}_amd64.deb
path: artifacts/opcua-extractor_{version}_amd64.deb
platform: linux
displayName: "Linux (Debian) installer"
- name: opcua-extractor.linux-x64.{version}.zip
path: artifacts/opcua-extractor.linux-x64.{version}.zip
platform: linux
extractor:
externalId: opcua
name: Cognite OPC UA Extractor
description: Extracts time series, events, and assets data via the OPC UA protocol.
documentation: >
The Cognite OPC UA extractor connects to the open OPC UA protocol and streams time series into the CDF time series service and events into the CDF events service.
It batch extracts the OPC UA node hierarchy into the CDF staging area or as CDF assets and relationships.
A local test server is available in github releases.
tags: ["opcua", "predictor", "kepware", "embedded"]
type: "global"
links:
- name: Github
url: "https://github.com/cognitedata/opcua-extractor-net"
type: "generic"
- name: About
url: "https://docs.cognite.com/cdf/integration/guides/extraction/opc_ua"
type: "externalDocumentation"
- name: Hardware and CDF requirements
url: "https://docs.cognite.com/cdf/integration/guides/extraction/opc_ua/opc_ua_hw_requirements"
type: "externalDocumentation"
- name: Server requirements
url: "https://docs.cognite.com/cdf/integration/guides/extraction/opc_ua/opc_ua_server_requirements"
type: "externalDocumentation"
- name: Setting up
url: "https://docs.cognite.com/cdf/integration/guides/extraction/opc_ua/opc_ua_setup"
type: "externalDocumentation"
- name: Command line interface
url: "https://docs.cognite.com/cdf/integration/guides/extraction/opc_ua/opc_ua_cli"
type: "externalDocumentation"
- name: Configuration
url: "https://docs.cognite.com/cdf/integration/guides/extraction/configuration/opcua"
type: "externalDocumentation"
- name: Metrics
url: "https://docs.cognite.com/cdf/integration/guides/extraction/opc_ua/opc_ua_metrics"
type: "externalDocumentation"
- name: Troubleshooting
url: "https://docs.cognite.com/cdf/integration/guides/extraction/opc_ua/opc_ua_troubleshooting"
type: "externalDocumentation"
image: opc-ua.png
documentation:
includeLeafNodes: false
preamble:
- docs/preamble.md
- docs/keyvault.md
- docs/header.md
schema:
rootSchema: schema/full_config.schema.json
additionalUris:
- "https://raw.githubusercontent.com/"
versions:
"2.32.1-beta4":
description: Abandon reconnect attempts on unknown exceptions.
changelog:
fixed:
- Fixed an issue where the extractor would get stuck in a reconnect loop.
"2.32.1-beta3":
description: Fix possible memory leak from live data when history is suspended.
changelog:
fixed:
- Fix possible memory leak related to datapoints being buffered to avoid data loss on history end.
"2.32.1-beta2":
description: Fix issue causing the extractor to not properly change redundant server in some cases.
changelog:
fixed:
- Fix issue causing extractor not to properly switch between redundant servers when service level is bad without changing.
"2.32.1-beta1":
description: Experimental changes to connection logic.
changelog:
fixed:
- Revamped connection logic to potentially fix a memory leak and a few issues related to connections not being properly reestablished.
"2.32.0":
description: Beta support for writing events to data modeling records.
changelog:
added:
- Beta support for writing events to data modeling records, enable by configuring `cognite.stream-records`.
"2.31.2":
description: Improve handling of rapid service level changes.
changelog:
fixed:
- Fixed a potential race condition when service level rapidly changes from good to bad and back.
"2.31.1":
description: Minor improvements to reconnect reliability with large, unstable servers.
changelog:
fixed:
- Avoid a potential race condition when reconnecting to the server.
"2.31.0":
description: Support for writing timeseries to the core data models.
changelog:
added:
- Support for writing time series to the core data models.
"2.30.4":
description: Fix some minor issues related to retrying failed pushes to CDF.
changelog:
fixed:
- Avoid unnecessarily repeating pushes of timeseries and references.
- Improve logging when the upload queue is full.
- Do not update whether or not an asset is a root node.
"2.30.3":
description: Fix various issues.
changelog:
fixed:
- Fix error when updating parent IDs.
- Avoid broken state when the local config buffer cannot be written to.
"2.30.2":
description: Set `ReturnBounds` to false in history read requests.
changelog:
fixed:
- No longer accidentally set `ReturnBounds` to true in history read raw requests.
"2.30.1":
description: Always set history end time.
changelog:
changed:
- History end time is now always set in read requests to the server, even if history.end-time is not configured.
"2.30.0":
description: Add support for configuring filters from a list of values.
changelog:
added:
- Add support for configuring filters from a list of values contained either directly in config, or in a local file.
"2.29.1":
description: Log more related to continuation points.
changelog:
changed:
- Added more logging related to continuation point and history termination.
"2.29.0":
description: Full release of status codes on datapoints.
changelog:
changed:
- It is now possible to ingest null values for datapoints if the status is bad.
added:
- Added a filter type for only including filtered nodes.
"2.28.1":
description: Fix bug causing the extractor to fail to retry token requests.
changelog:
fixed:
- Properly retry failed token requests.
"2.28.0":
description: Limit the number of nodes and edges the extractor will buffer at a time.
changelog:
changed:
- The extractor will now never buffer more than 1 million datapoints, or 100 000 events.
"2.27.6":
description: Fixes edge-case caused by writing to FDM and clean at the same time.
changelog:
fixed:
- Fixed extractor failing to upload to clean when some nodes were included from outside the main tree.
"2.27.5":
description: Minor documentation updates.
"2.27.4":
description: Minor fixes to the FDM target prototype.
changelog:
fixed:
- Fixed issue causing errors on duplicate references in FDM writer.
- Fixed bug causing requests to fail on marking many raw rows as deleted.
added:
- Added configuration for controlling FDM chunk sizes.
"2.27.3":
description: Documentation updates.
"2.27.2":
description: Improve reliability of history restart triggers.
"2.27.1":
description: Enforce start-time < end-time when reading history, fix subscription recreation bug.
changelog:
changed:
- The extractor will now fail on startup if start-time is configured to be greater than end-time.
fixed:
- Fix a bug related to subscription recreation.
"2.27.0":
description: Beta support for status codes in time series.
changelog:
added:
- "Beta support for status codes in time series. Note that this may change without being considered a breaking change in the extractor."
changed:
- "The data format for the local failure buffer has changed. Any existing buffered datapoints will be discarded when the extractor is updated."
"2.26.1":
description: Improve logging around subscription recreation.
"2.26.0":
description: Configurable grace period for recreating dead subscriptions.
changelog:
added:
- "subscriptions.recreate-subscription-grace-period for grace period when recreating subscriptions"
"2.25.1":
description: Be less aggressive with history restart on dead subscriptions.
changelog:
changed:
- Only restart history after a subscription recovery grace period.
"2.25.0":
description: Restart history more consistently and in more cases. Add support for azure key vault.
changelog:
changed:
- Restart history when subscriptions are recreated.
- Stop history when any condition for not running history is satisfied, then start it once all conditions are satisfied.
added:
- "Support for azure key vault. Secrets are injected by adding a `key-vault` section to the config, and using the !keyvault [secretname] tag"
"2.24.1":
description: Fix Archive Release Builds
changelog:
fixed:
- "Fix for missing files in release archive(s)"
"2.24.0":
description: "Bugfixes and improvements"
changelog:
fixed:
- "fix: Allow empty array dimensions"
"2.23.0":
description: Update to .NET 8
changelog:
changed:
- "Update to .NET 8"
fixed:
- "Fix issue caused by dotnet runtime inconsistency when using the docker image."
"2.22.2":
description: Shut down history when service level goes bad, to avoid gaps if it goes back to good while history is still running.
"2.22.1":
description: Fix issue causing the extractor to try to create very many monitored items at the same time.
changelog:
fixed:
- "Fix to issue related to recreating subscriptions with lots of monitored items. The extractor would sometimes try to create all monitored items in a single request, which would invariably fail."
"2.22.0":
description: Reliability improvements and some minor additions
changelog:
added:
- "Persist namespace publication date between extractor restarts"
- "Log request id by default on every request to CDF"
changed:
- "Always attempt to reconnect to the server on connection loss. force-restart now only makes it so that the extractor doesn't _retry_ on reconnects, and can be disabled in many more cases. It is no longer necessary for redundancy."
- "root-node no longer falls back on the objects folder if the namespace is invalid."
fixed:
- "Fix a bug causing asset IDs on timeseries to not work properly in some cases"
- "Properly clear the delete marker on resources that have been restored on the server"
- "Improve logging of errors if the extractor restarts due to exceeding history failure threshold"
- "Fix a bug causing max-node-parallelism to not be properly set from server limits. This is mostly relevant for extractor deployments not using config generated by the config tool"
"2.21.2":
description: Reliability improvements
changelog:
fixed:
- Fixed bug causing extractor to crash if connection failed while creating extractor.
- Fixed issue causing extractor to fail to load data from CDF if connection to the server was lost.
"2.21.1":
description: Retry on more status codes
changelog:
added:
- Retry on more status codes
changed:
- Allow configuring retry status codes using symbolic IDs and hex values.
"2.21.0":
description: Supports writing to all CDF destinations
changelog:
added:
- Support writing to any/all CDF resource without dependents on one another
changed:
- Metadata configuration options
"2.20.3":
description: Fix issue causing the extractor to fail to push datapoints on windows.
changelog:
fixed:
- "Fix FileNotFound issue when pushing datapoints on windows"
"2.20.2":
description: Improve error messages
"2.20.0":
description: Improve error handling during history read.
changelog:
added:
- "Added an `error-threshold` option which indicates the percent of history read requests that must fail before the entire read is considered failed."
"2.19.0":
description: Add extra metadata per type and remote support in docker container
changelog:
added:
- "Extra metadata per type"
- "Set the environment variable OPCUA_CONFIG_DIR=/config_remote to have the docker container use a remote config from env variable COGNITE_EXTRACTION_PIPELINE"
"2.18.0":
description: Options for configuring lifetime-count and keep-alive-count of subscriptions
changelog:
added:
- "subscriptions.lifetime-count setting maximum number of empty publish requests before subscription shutdown"
- "subscriptions.keep-alive-count setting maximum number of empty publish requests before a keep alive response is sent"
changed:
- "Delete subscriptions on session shutdown"
"2.17.0":
description: Various config changes and some internal changes
changelog:
deprecated:
- "Deprecated the debug option on cognite, influx, and mqtt."
- "Deprecated cognite.data-set-id, and cognite.data-set-external-id, in favor of cognite.data-set.id and cognite.data-set.external-id"
added:
- "Added a dry-run option, which serves the same purpose as debug but applies to the entire extractor."
- "Added --dry-run as a command line option."
changed:
- "Major internal changes. This should not be visible to users at all, except in some changed logs."
"2.16.1":
description: "Correctly restart history after reconnect due to service level threshold"
changelog:
fixed:
- "Fixed a bug causing the extractor to not properly detect that the server had changed when reconnecting due to service level change"
"2.16.0":
description: "Support for certificate authentication to CDF"
changelog:
added:
- "Support for using a private key and certificate when authenticating towards CDF"
"2.15.3":
description: Improve handling of incorrect proto node ids
changelog:
changed:
- "Incorrect proto node ids no longer default to anything, and are instead always skipped."
"2.15.2":
description: Add bill of materials
"2.15.1":
description: Fix connecting to redundant servers
changelog:
fixed:
- "Fix issue causing the extractor to only connect to alt endpoints"
"2.15.0":
description: Add live monitoring of ServiceLevel, variable subscription config
changelog:
deprecated:
- "source.sampling-interval and source.queue-length"
fixed:
- "Fix issue causing delete states to not be properly fetched"
added:
- "Support for monitoring ServiceLevel and not updating extraction state if it drops below a threshold"
- "subscriptions.sampling-interval and subscriptions.queue-length"
- "subscriptions.alternative-configs containing a list of alternative subscription configurations"
"2.14.1":
description: Avoid crashing on duplicated namespaces
changelog:
fixed:
- "Avoid crashing on duplicated namespaces with rebrowse triggers enabled"
"2.14.0":
description: Workaround for NAT, various fixes
changelog:
fixed:
- "Make sure delete states are saved correctly"
- "Fix handling of variable children when the parent variable is ignored"
added:
- "Support for overriding endpoints returned by discovery servers, to work around NAT"
"2.13.2":
description: Improve logging in rebrowse manager
changelog:
changed:
- "Improve logging in rebrowse manager"
"2.13.1":
description: Minor improvements to the service manager
changelog:
added:
- "A file opcuabinpath.txt in the same folder as the service manager can be used to set the binary path"
- "If there is no opcuabinpath.txt and no installation registry entry, the service manager defaults to ../bin/OpcUaExtractor.exe"
fixed:
- "Display expected path if the service manager cannot find the executable"
"2.13.0":
description: Update to .NET 7
changelog:
changed:
- "Update to .NET 7"
"2.12.0":
description: Add support for marking nodes as deleted
changelog:
added:
- "Support for marking nodes as deleted. See `extraction.deletes`"
"2.11.0":
description: Adds support for rebrowse triggers
changelog:
added:
- "Adds support for triggering rebrowse when updates are detected in NamespacePublicationDate"
"2.10.5":
description: Improve handling of unknown namespace index
changelog:
fixed:
- "Improve handling of unknown namespace index"
"2.10.4":
description: Fix to a few bugs
changelog:
fixed:
- "Fix config to not being properly reloaded"
- "Fix data-set-external-id not working correctly"
- "Fix an error when restarting the extractor"
- "Fix logging of retries"
"2.10.3":
description: Fix bug related to external server references
changelog:
fixed:
- "Properly ignore external server references"
"2.10.2":
description: Fix to retries on some operations
changelog:
fixed:
- "Fix retries not working on certain operations"
"2.10.1":
description: Fixes internal performance issue
changelog:
fixed:
- "Fix performance issues during start up"
"2.10.0":
description: Add more options for SSL certificates in the MQTT destination
changelog:
added:
- "Option to add an additional certificate authority cert in MQTT destination"
- "Option to skip certificate checks entirely in MQTT destination"
"2.9.2":
description: Improve configuration of certificates in docker containers
changelog:
changed:
- "Docker containers now use a local certificate directory for application certificates by default, they also use a static signature not dependent on the host machine."
"2.9.1":
description: Actual fix to missing `cognite` section bug.
changelog:
fixed:
- "Bug causing extractor to fail if the `cognite` section was missing in remote config."
"2.9.0":
description: Support for non-transparent server redundancy.
changelog:
added:
- Support for non-transparent server redundancy ref. OPC-UA standard chapter 4 6.6.2
- "`extraction-pipeline: external-id` as alias for `pipeline-id` in config, to align with python extractors."
fixed:
- "Bug causing extractor to fail if the `cognite` section was missing in remote config."
changed:
- Improved performance of force-restart, the extractor will no-longer rebrowse automatically if force-restart is enabled. Instead use restart-on-reconnect if that is desired.
"2.8.0":
description: Basic support for high availability.
changelog:
added:
- Configuration options to use Raw or a Redis store for high availability
- Support for remote configuration files
"2.8.0-beta2":
description: Utility release to publish Extractor to public Docker registry
"2.8.0-beta1":
description: Support for remote configuration files
changelog:
added:
- "Support for remote configuration files stored in extraction pipelines"
"2.7.1":
description: Minor fix to consistency of reference extraction
changelog:
fixed:
- "Improve consistency of OPC UA reference extraction"
"2.7.0":
description: Cron support, config tool improvements
changelog:
fixed:
- "Make docker image correctly use .NET 6"
- "Fix issue causing historie to not be properly read when getting nodes from CDF"
added:
- "Improved configuration tool logging"
- "Add support for cron expressions for certain configuration options"
- "Add option to run the extractor entirely without an OPC-UA server"
"2.6.1":
description: "Improved tracing and fix to issue with history"
changelog:
fixed:
- "Fix issue causing end time to not be properly set when reading history"
- "Add option to enable heavy tracing of OPC-UA operations"
"2.6.0":
description: "Improve logging, fixes to history on restart, callback for browsing"
changelog:
added:
- "Improvements to logging"
- "Option to convert timeseries values to events"
- "Option to set a maximum size for the failure buffer"
- "Option to call a CDF function as callback after browse"
fixed:
- "Fix issue where history would not be restarted after a normal reconnect"
"2.5.3":
description: "Bugfixes, handling of bad subscriptions, and performance improvements in pubsub. See changelog"
changelog:
added:
- "Improved error handling on json serialization"
- "Configurable certificate expiry"
- "Automatically recreate dead subscriptions"
changed:
- "Update OPC-UA SDK for better performance when using pubsub"
"2.5.2":
description: "Improvements to logging and performance. See changelog"
changelog:
added:
- "Improvements to logging"
- "Transformation to make a variable historizing"
- "Option to require Historizing for reading variable history"
- "Various performance improvements"
changed:
- "Update to .NET 6"
fixed:
- "Fix some various subtle bugs related to attribute reading"
"2.5.0":
description: "Numerous fixes and improvements. See changelog"
changelog:
changed:
- "Wait for configuration file to exist when starting extractor"
- "Add name of event type to CDF event metadata"
- "History node parallelism is now properly shared, so that the configuration option can reflect available continuation points directly"
- "Considerable internal refactoring, using infrastructure from extractor utils"
added:
- "Add support for using NodeSet2 files as source for nodes instead of the OPC-UA server"
- "Support for throttling browse operations"
- "Limit config values based on server capabilities"
- "Improve logging of exceptions"
- "Add support for extraction pipelines"
- "Add proof of concept for OPC-UA pubsub using exposed server configuration"
- "Log configuration file on startup"
- "Support for specifying timestamps using relative syntax like `2d-ago`, and timespans like `2m`"
- "Optionally map nested variables to CDF, instead of treating them as metadata"
- "Optionally estimate size of OPC-UA arrays based on value on startup"
- "Optionally log OPC-UA SDK trace"
- "Optionally browse in background when reading from CDF"
fixed:
- "Correctly handle nodes that are discovered during browse but do not actually exist"
- "Fix service manager"
- "Fix logging of transformations"