Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add the MetricsTracerFactory for client side metrics #1566

Open
wants to merge 85 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
85 commits
Select commit Hold shift + click to select a range
abb292b
Add the metrics tracer factory class
danieljbruce Jan 10, 2025
051b488
Add system tests for the client side metrics
danieljbruce Jan 10, 2025
1c49f86
Add open telemetry packages
danieljbruce Jan 10, 2025
7a5be3b
Add a metrics tracer factory
danieljbruce Jan 10, 2025
5390411
metadata experimentation
danieljbruce Jan 13, 2025
a7f2fd4
Pass metadata, status along
danieljbruce Jan 13, 2025
9e3e5f5
Get mapped entries
danieljbruce Jan 13, 2025
9f93172
Start collecting a few metrics in the metrics trac
danieljbruce Jan 13, 2025
b32aea0
on attempt start
danieljbruce Jan 14, 2025
750c1e7
Adding more metrics
danieljbruce Jan 15, 2025
b885126
Add support for application blocking latencies
danieljbruce Jan 15, 2025
5fb300b
Add a TODO for date wrapper
danieljbruce Jan 15, 2025
feb36e7
Add first unit test for the metrics tracer
danieljbruce Jan 16, 2025
8465b3a
Move the code for the TestMeterProvider to
danieljbruce Jan 16, 2025
7a97aab
Move the Date provider to a second file
danieljbruce Jan 16, 2025
51d3dd3
Fix attempt latencies bug
danieljbruce Jan 16, 2025
ee8c272
Add assertion check against text file
danieljbruce Jan 17, 2025
b7413e8
More realistic seconds increment
danieljbruce Jan 17, 2025
7c8877b
Remove imports
danieljbruce Jan 17, 2025
503a2a9
Adjust event timings to be more realistic
danieljbruce Jan 17, 2025
938cb2c
Remove only
danieljbruce Jan 17, 2025
854e1d1
Add comments to the table class
danieljbruce Jan 17, 2025
db7d1b1
More comments in table
danieljbruce Jan 17, 2025
ee67037
Remove TODO
danieljbruce Jan 17, 2025
ea2fbe2
Move observability options into a separate file
danieljbruce Jan 17, 2025
c22eb5b
inline definitions for the tabular api surface
danieljbruce Jan 17, 2025
a658a39
Comment source code out for now
danieljbruce Jan 17, 2025
960b402
Add abstractions for classes that have a logger
danieljbruce Jan 17, 2025
23a7c14
Generate documentation for meter provider
danieljbruce Jan 17, 2025
0ac6d15
Generate documentation for the date provider
danieljbruce Jan 17, 2025
bad23b2
Generate logger documentation
danieljbruce Jan 17, 2025
49bd7ca
Observability options documentation
danieljbruce Jan 17, 2025
129e8fd
Add more documentation for various MTF methods
danieljbruce Jan 17, 2025
052c7bb
Comment out Metrics
danieljbruce Jan 17, 2025
ac27a95
Add a bunch of TODOs in front of the comments
danieljbruce Jan 17, 2025
18c942e
Delete client-side-metrics file
danieljbruce Jan 17, 2025
7a3aabc
Revert "Delete client-side-metrics file"
danieljbruce Jan 17, 2025
5906c29
Revert "Revert "Delete client-side-metrics file""
danieljbruce Jan 17, 2025
be731af
Add headers
danieljbruce Jan 17, 2025
c26640f
Remove TODOs
danieljbruce Jan 17, 2025
3011e50
Add AttemptInfo to distinguish from OperationInfo
danieljbruce Jan 17, 2025
945f237
Adjust dimensions to match documentation
danieljbruce Jan 17, 2025
b04c3c4
Update tests with dimension metrics
danieljbruce Jan 17, 2025
2417e80
Revert "Revert "Revert "Delete client-side-metrics file"""
danieljbruce Jan 17, 2025
df59d88
Do some measurements
danieljbruce Jan 20, 2025
8ad51f5
Revert "Do some measurements"
danieljbruce Jan 20, 2025
6868f5a
Revert "Revert "Revert "Revert "Delete client-side-metrics file""""
danieljbruce Jan 20, 2025
7cc36a2
Add header
danieljbruce Jan 20, 2025
62a4b8b
Remove the TODOs
danieljbruce Jan 20, 2025
7c4f414
Add line back
danieljbruce Jan 20, 2025
83f53ae
Add comment
danieljbruce Jan 20, 2025
610eec0
Add version
danieljbruce Jan 20, 2025
a2b5951
Add version to client side metrics
danieljbruce Jan 20, 2025
5f67cad
linter
danieljbruce Jan 20, 2025
8f20c78
Generate documentation for AttemptInfo interface
danieljbruce Jan 20, 2025
9b1ba9d
Logger documentation
danieljbruce Jan 20, 2025
88e96c3
Generate more documentation
danieljbruce Jan 20, 2025
ed39628
Generate documentation
danieljbruce Jan 20, 2025
76b1249
Make sure test reports correct duration, zone
danieljbruce Jan 20, 2025
8d60cb1
Generate documentation for the dimensions to strin
danieljbruce Jan 20, 2025
19fef92
Add version to the dimensions
danieljbruce Jan 20, 2025
1ecfb1c
Fix the client name. The version is going to chan
danieljbruce Jan 20, 2025
d8a3960
Update the expected output file.
danieljbruce Jan 20, 2025
1d6b645
Fox bug, get cluster
danieljbruce Jan 20, 2025
acb1d3a
Add fake cluster to tests
danieljbruce Jan 20, 2025
c30b057
Remove console log
danieljbruce Jan 20, 2025
9ef079b
Generate more documentation
danieljbruce Jan 20, 2025
d5a0368
Require a call to fetch the project when using MT
danieljbruce Jan 20, 2025
ae532d8
use same date provider for all metrics tracers
danieljbruce Jan 20, 2025
b2bced9
In the metrics traceer, don’t fetch the project
danieljbruce Jan 20, 2025
e1dd61c
Remove only
danieljbruce Jan 20, 2025
cd97f35
Merge branch 'main' into 359913994-first-PR-add-infrastructure
danieljbruce Jan 20, 2025
9ec98df
Add open telemetry api
danieljbruce Jan 20, 2025
2457dbe
Merge branch '359913994-first-PR-add-infrastructure' of https://githu…
danieljbruce Jan 20, 2025
5a1a3aa
Add TestExecuteQuery_EmptyResponse to failures
danieljbruce Jan 20, 2025
1bd2d2b
TestExecuteQuery_SingleSimpleRow known failures
danieljbruce Jan 20, 2025
c2be338
Fix syntax in known failures
danieljbruce Jan 20, 2025
cd0d774
Add two tests to the known failures
danieljbruce Jan 20, 2025
e7caf36
TestSampleRowKeys_Retry_WithRetryInfo to known fai
danieljbruce Jan 20, 2025
7fd86d2
Change word dimensions to attributes
danieljbruce Jan 20, 2025
db05ff3
Change more docs to use Attributes instead of dim
danieljbruce Jan 20, 2025
9cc4b15
attributes
danieljbruce Jan 20, 2025
0142329
Test should use attributes as string
danieljbruce Jan 20, 2025
15d6e4a
For Windows replace carriage return
danieljbruce Jan 21, 2025
865529e
Update documentation with types
danieljbruce Jan 21, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 45 additions & 0 deletions common/client-side-metrics-attributes.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
// Copyright 2025 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

/**
* Attributes (labels) associated with a Bigtable metric. These
* attributes provide context for the metric values.
*/
export interface Attributes {
projectId: string;
instanceId: string;
table: string;
cluster?: string | null;
zone?: string | null;
appProfileId?: string;
methodName: string;
attemptStatus?: string;
finalOperationStatus?: string;
streamingOperation?: string;
clientName: string;
}

/**
* Converts an Attributes object to a string representation.
* This string representation is suitable for use as labels or tags.
* The order of attributes in the output string is fixed:
* projectId;instanceId;table;cluster;zone;appProfileId;methodName;attemptStatus;finalOperationStatus;streamingOperation;clientName
* If an attribute is null or undefined, the empty string is used.
* @param {Attributes} a The Attributes object to convert.
* @returns A string representation of the attribute.
*/
export function attributesToString(a: Attributes) {
const p = (attribute?: string | null) => (attribute ? attribute : '');
return `${p(a.projectId)};${p(a.instanceId)};${p(a.table)};${p(a.cluster)};${p(a.zone)};${p(a.appProfileId)};${p(a.methodName)};${p(a.attemptStatus)};${p(a.finalOperationStatus)};${p(a.streamingOperation)};nodejs-bigtable`;
}
52 changes: 52 additions & 0 deletions common/logger.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
// Copyright 2025 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

/**
* A simple logger interface for logging messages. Implementations of this interface
* can provide various logging mechanisms (e.g., console logging, file logging, etc.).
*/
interface ILogger {
log(message: string): void;
}

/**
* An abstract base class that provides a logger instance. Subclasses can use this logger
* for logging messages.
*/
export abstract class WithLogger {
protected logger: ILogger;
/**
* @param {ILogger} logger The logger instance to be used by this object.
*/
constructor(logger: ILogger) {
this.logger = logger;
}
}

/**
* An abstract base class that provides a logger instance and a name. Subclasses
* can use the logger for logging messages, incorporating the name for context.
*/
export abstract class WithLoggerAndName {
protected logger: ILogger;
protected name: string;
/**
* @param {ILogger} logger The logger instance to be used by this object.
* @param {string} name The name associated with this object.
*/
constructor(logger: ILogger, name: string) {
this.logger = logger;
this.name = name;
}
}
56 changes: 56 additions & 0 deletions common/test-date-provider.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
// Copyright 2025 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

import {WithLogger} from './logger';

/**
* A test implementation of a Date-like object. Used for testing purposes. It provides a
* getTime method that returns a pre-determined fake date value, allowing for
* deterministic testing of time-dependent functionality.
*/
class TestDateLike {
private fakeDate;
/**
* @param {number} fakeDate The fake date value to be returned by getTime(), in milliseconds.
*/
constructor(fakeDate: number) {
this.fakeDate = fakeDate;
}
/**
* Returns the fake date value that this object was created with.
* @returns {number} The fake date, in milliseconds.
*/
getTime() {
return this.fakeDate;
}
}

/**
* A test implementation of a DateProvider. Used for testing purposes. Provides
* a deterministic series of fake dates, with each call to getDate() returning a date 1000ms later than the last.
* Logs each date value returned for verification purposes.
*/
export class TestDateProvider extends WithLogger {
private dateCounter = 0;
/**
* Returns a new fake date 1000ms later than the last. Logs the date for test verification.
* @returns {TestDateLike} A fake date object.
*/
getDate() {
// The test assumes exactly 1s passes between each getDate call.
this.dateCounter = this.dateCounter + 1000;
this.logger.log(`getDate call returns ${this.dateCounter.toString()} ms`);
return new TestDateLike(this.dateCounter);
}
}
88 changes: 88 additions & 0 deletions common/test-meter-provider.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
// Copyright 2025 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

import {WithLogger, WithLoggerAndName} from './logger';
import {Attributes, attributesToString} from './client-side-metrics-attributes';

/**
* A test implementation of a MeterProvider. This MeterProvider is used for testing purposes.
* It doesn't send metrics to a backend, but instead logs metric updates for verification.
*/
export class TestMeterProvider extends WithLogger {
/**
* Returns a TestMeter, that logs metric updates for verification.
* @param {string} name The name of the meter.
* @returns {TestMeter}
*/
getMeter(name: string) {
return new TestMeter(this.logger, name);
}
}

/**
* A test implementation of a Meter. Used for testing purposes. It doesn't send metrics to a backend,
* but instead logs metric updates for verification.
*/
class TestMeter extends WithLoggerAndName {
/**
* Creates a test histogram. The TestHistogram logs when values are recorded.
* @param {string} instrument The name of the instrument.
* @returns {TestHistogram}
*/
createHistogram(instrument: string) {
return new TestHistogram(this.logger, `${this.name}:${instrument}`);
}
/**
* Creates a test counter. The TestCounter logs when values are added.
* @param {string} instrument The name of the instrument.
* @returns {TestCounter}
*/
createCounter(instrument: string) {
return new TestCounter(this.logger, `${this.name}:${instrument}`);
}
}

/**
* A test implementation of a Counter. Used for testing purposes. It doesn't send metrics to a backend,
* but instead logs value additions for verification.
*/
class TestCounter extends WithLoggerAndName {
/**
* Simulates adding a value to the counter. Logs the value and the counter name.
* @param {number} value The value to be added to the counter.
* @param {Attributes} attributes The attributes associated with the value.
*/
add(value: number, attributes: Attributes) {
this.logger.log(
`Value added to counter ${this.name} = ${value.toString()} with attributes ${attributesToString(attributes)}`
);
}
}

/**
* A test implementation of a Histogram. Used for testing purposes. It doesn't send metrics to a backend,
* but instead logs recorded values for verification.
*/
class TestHistogram extends WithLoggerAndName {
/**
* Simulates recording a value in the histogram. Logs the value and the histogram name.
* @param {number} value The value to be recorded in the histogram.
* @param {Attributes} attributes The attributes associated with the value.
*/
record(value: number, attributes: Attributes) {
this.logger.log(
`Value added to histogram ${this.name} = ${value.toString()} with attributes ${attributesToString(attributes)}`
);
}
}
5 changes: 5 additions & 0 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -47,9 +47,14 @@
"precompile": "gts clean"
},
"dependencies": {
"@google-cloud/opentelemetry-cloud-monitoring-exporter": "^0.20.0",
"@google-cloud/opentelemetry-resource-util": "^2.4.0",
"@google-cloud/precise-date": "^4.0.0",
"@google-cloud/projectify": "^4.0.0",
"@google-cloud/promisify": "^4.0.0",
"@opentelemetry/api": "^1.9.0",
"@opentelemetry/resources": "^1.30.0",
"@opentelemetry/sdk-metrics": "^1.30.0",
"arrify": "^2.0.0",
"concat-stream": "^2.0.0",
"dot-prop": "^6.0.0",
Expand Down
Loading
Loading