-
Notifications
You must be signed in to change notification settings - Fork 108
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix!: don't unwrap int, double by default, add a warning when a number is autoconverted to double or int #773
Changes from all commits
28c2327
8a5aba8
068b0bf
a7152f5
b125553
2d31c03
5b62746
39e618c
573880d
a2ba674
beab398
aa86a2f
1d80aea
b2f4e4d
3c6eb03
62f4f3e
2085357
b28df5f
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -52,27 +52,44 @@ export namespace entity { | |
* provided. | ||
* | ||
* @class | ||
* @param {number} value The double value. | ||
* @param {number|string} value The double value. | ||
* | ||
* @example | ||
* const {Datastore} = require('@google-cloud/datastore'); | ||
* const datastore = new Datastore(); | ||
* const aDouble = datastore.double(7.3); | ||
*/ | ||
export class Double { | ||
export class Double extends Number { | ||
private _entityPropertyName: string | undefined; | ||
type: string; | ||
value: number; | ||
constructor(value: number) { | ||
constructor(value: number | string | ValueProto) { | ||
super(typeof value === 'object' ? value.doubleValue : value); | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. What is the use case of constructing from a proto ? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Int has the same support from proto, and it is part of the entityFromEntityProto path. This code mirrors the behavior of our Int class. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I was wondering the same thing, the fact that we allow this in the There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I added a bunch more tests to There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. These tests and all other changes will be in a separate PR. |
||
|
||
/** | ||
* @name Double#type | ||
* @type {string} | ||
*/ | ||
this.type = 'DatastoreDouble'; | ||
|
||
this._entityPropertyName = | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Do we really need to keep the property name ? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Honestly, I am not sure what this is for, but I mirrored behavior in the Int implementation. Maybe @stephenplusplus recalls? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. If we're not 100% what it's here for, I'd be tempted to drop it (let's chat with @stephenplusplus offline). There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I haven't checked the use cases but it seems dangerous if an entity has the form const [entity] = await ds.get(...);
entity.oldValue = entity.newValue;
entity.newValue = ds.Int(123);
await ds.save(entity); I can only imagine that keeping the entity property name would cause problems as I am not sure if it is actually a problem in the current code but at least it makes it easy to shoot yourself in the foot. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'll remove this. The reason we have this for Int is because we use it when using the |
||
typeof value === 'object' ? value.propertyName : undefined; | ||
|
||
/** | ||
* @name Double#value | ||
* @type {number} | ||
*/ | ||
this.value = value; | ||
this.value = | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Why do need a value member ? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Possibly? This started as a string, but realized I could store this as a number. We store it as a string for Int mostly due to Int range differences. Datastore Doubles have 64-bit double precision, IEEE 754, as do Node.js Numbers. So I was able to change this type and simplify a bit. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. feels kind of like this should be private. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I don't know about that. If we make it private then that would be a breaking change. Consider the case where users are using |
||
typeof value === 'object' ? Number(value.doubleValue) : Number(value); | ||
} | ||
|
||
// eslint-disable-next-line @typescript-eslint/no-explicit-any | ||
valueOf(): any { | ||
return Number(this.value); | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Number is redundant with l 83. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Playing around with extending class F extends Number {
constructor(value) {
super(value)
}
blerg() {
console.info(this.value)
}
}
const f = new F(99);
console.info(f.valueOf())
f.blerg() outputs:
Edit: I see we're already extending There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. class F extends Number {
constructor(value) {
super(value);
}
blerg() {
console.info(Number(this));
}
}
const f = new F(99);
f.blerg(); // outputs 99 as expected There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Good to know. |
||
} | ||
|
||
toJSON(): any { | ||
return {type: this.type, value: this.value}; | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This seems quite dangerous if the goal of There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. can you expand? Sorry, I don't quite understand/follow. Also, There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I am really not sure to get the intent of the change right but my understanding of the current state is that you want to return If I get this right then However with this code:
While not related to
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It seems that returning wrapped numbers in |
||
} | ||
} | ||
|
||
|
@@ -475,7 +492,7 @@ export namespace entity { | |
* | ||
* @private | ||
* @param {object} valueProto The protobuf Value message to convert. | ||
* @param {boolean | IntegerTypeCastOptions} [wrapNumbers=false] Wrap values of integerValue type in | ||
* @param {boolean | IntegerTypeCastOptions} [wrapNumbers=true] Wrap values of integerValue type in | ||
* {@link Datastore#Int} objects. | ||
* If a `boolean`, this will wrap values in {@link Datastore#Int} objects. | ||
* If an `object`, this will return a value returned by | ||
|
@@ -523,10 +540,21 @@ export namespace entity { | |
} | ||
|
||
case 'doubleValue': { | ||
if (wrapNumbers === undefined) { | ||
wrapNumbers = true; | ||
} | ||
|
||
if (wrapNumbers) { | ||
return new entity.Double(valueProto); | ||
} | ||
return Number(value); | ||
} | ||
|
||
case 'integerValue': { | ||
if (wrapNumbers === undefined) { | ||
wrapNumbers = true; | ||
} | ||
|
||
return wrapNumbers | ||
? typeof wrapNumbers === 'object' | ||
? new entity.Int(valueProto, wrapNumbers).valueOf() | ||
|
@@ -580,23 +608,6 @@ export namespace entity { | |
return valueProto; | ||
} | ||
|
||
if (typeof value === 'number') { | ||
if (Number.isInteger(value)) { | ||
if (!Number.isSafeInteger(value)) { | ||
process.emitWarning( | ||
'IntegerOutOfBoundsWarning: ' + | ||
"the value for '" + | ||
property + | ||
"' property is outside of bounds of a JavaScript Number.\n" + | ||
"Use 'Datastore.int(<integer_value_as_string>)' to preserve accuracy during the upload." | ||
); | ||
} | ||
value = new entity.Int(value); | ||
} else { | ||
value = new entity.Double(value); | ||
} | ||
} | ||
|
||
if (isDsInt(value)) { | ||
valueProto.integerValue = value.value; | ||
return valueProto; | ||
|
@@ -607,6 +618,39 @@ export namespace entity { | |
return valueProto; | ||
} | ||
|
||
if (typeof value === 'number') { | ||
const integerOutOfBoundsWarning = | ||
crwilcox marked this conversation as resolved.
Show resolved
Hide resolved
|
||
"IntegerOutOfBoundsWarning: the value for '" + | ||
property + | ||
"' property is outside of bounds of a JavaScript Number.\n" + | ||
"Use 'Datastore.int(<integer_value_as_string>)' to preserve accuracy " + | ||
'in your database.'; | ||
|
||
const typeCastWarning = | ||
"TypeCastWarning: the value for '" + | ||
property + | ||
"' property is a JavaScript Number.\n" + | ||
"Use 'Datastore.int(<integer_value_as_string>)' or " + | ||
"'Datastore.double(<double_value_as_string>)' to preserve consistent " + | ||
'Datastore types in your database.'; | ||
|
||
if (Number.isInteger(value)) { | ||
if (!Number.isSafeInteger(value)) { | ||
process.emitWarning(integerOutOfBoundsWarning); | ||
} else { | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Do we need the else here ? Also maybe move There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. They are exclusive. They both recommend the same outcome, but I think having two separate warnings is a bit confusing. This one warning is it could be dangerous, the other is destructive. I had what you recommended earlier but it is very very noisy :) There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Note: added 'debouncing' you will only get a single typecast warning for the entity. that way it doesn't flood the warning feed. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It might be reasonable to augment the out of bounds warning to mention the type issue as well, since that'll still be an issue if they fix their number size issue. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Since we're debouncing the typescast warning any ways, I'd consider just dropping the if and printing both. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think printing both could get noisy as Chris said. I ended up going with @feywind's suggestion here. |
||
warn('TypeCastWarning', typeCastWarning); | ||
} | ||
value = new entity.Int(value); | ||
valueProto.integerValue = value.value; | ||
return valueProto; | ||
} else { | ||
warn('TypeCastWarning', typeCastWarning); | ||
value = new entity.Double(value); | ||
valueProto.doubleValue = value.value; | ||
return valueProto; | ||
} | ||
} | ||
|
||
if (isDsGeoPoint(value)) { | ||
valueProto.geoPointValue = value.value; | ||
return valueProto; | ||
|
@@ -667,6 +711,14 @@ export namespace entity { | |
throw new Error('Unsupported field value, ' + value + ', was provided.'); | ||
} | ||
|
||
const warningTypesIssued = new Set<string>(); | ||
const warn = (warningName: string, warningMessage: string) => { | ||
if (!warningTypesIssued.has(warningName)) { | ||
warningTypesIssued.add(warningName); | ||
process.emitWarning(warningMessage); | ||
} | ||
}; | ||
|
||
/** | ||
* Convert any entity protocol to a plain object. | ||
* | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -20,6 +20,7 @@ import * as yaml from 'js-yaml'; | |
import {Datastore, Index} from '../src'; | ||
import {google} from '../protos/protos'; | ||
import {Storage} from '@google-cloud/storage'; | ||
import {entity} from '../src/entity'; | ||
|
||
describe('Datastore', () => { | ||
const testKinds: string[] = []; | ||
|
@@ -67,11 +68,11 @@ describe('Datastore', () => { | |
publishedAt: new Date(), | ||
author: 'Silvano', | ||
isDraft: false, | ||
wordCount: 400, | ||
rating: 5.0, | ||
wordCount: new entity.Int({propertyName: 'wordCount', integerValue: 400}), | ||
rating: new entity.Double({propertyName: 'rating', doubleValue: 5.0}), | ||
likes: null, | ||
metadata: { | ||
views: 100, | ||
views: new entity.Int({propertyName: 'views', integerValue: 100}), | ||
}, | ||
}; | ||
|
||
|
@@ -268,6 +269,40 @@ describe('Datastore', () => { | |
await datastore.delete(postKey); | ||
}); | ||
|
||
it('should save/get an int-able double value via Datastore.', async () => { | ||
const postKey = datastore.key('Team'); | ||
const points = Datastore.double(2); | ||
await datastore.save({key: postKey, data: {points}}); | ||
let [entity] = await datastore.get(postKey, {wrapNumbers: true}); | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. As a TODO, I wish we had unit tests for the Int and Double types, vs., only integration tests. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think a good unit test corresponding to this one would be to mock out the |
||
// Expect content is stored in datastore as a double with wrapping to | ||
// return a wrapped double | ||
assert.strictEqual(entity.points.type, 'DatastoreDouble'); | ||
assert.strictEqual(entity.points.value, 2); | ||
|
||
[entity] = await datastore.get(postKey, {wrapNumbers: false}); | ||
// Verify that when requested with wrapNumbers false, we get a plain | ||
// javascript Number 2. | ||
assert.strictEqual(entity.points, 2); | ||
|
||
[entity] = await datastore.get(postKey); | ||
// Expect without any options, a wrapped double to be returned. | ||
assert.strictEqual(entity.points.type, 'DatastoreDouble'); | ||
assert.strictEqual(entity.points.value, 2); | ||
|
||
// Save the data again, reget, ensuring that along the way it isn't | ||
// somehow changed to another numeric type. | ||
await datastore.save(entity); | ||
[entity] = await datastore.get(postKey); | ||
// expect as we saved, that this property is still a DatastoreDouble. | ||
assert.strictEqual(entity.points.type, 'DatastoreDouble'); | ||
assert.strictEqual(entity.points.value, 2); | ||
|
||
// Verify that DatastoreDouble implement Number behavior | ||
assert.strictEqual(entity.points + 1, 3); | ||
|
||
await datastore.delete(postKey); | ||
}); | ||
|
||
it('should wrap specified properties via IntegerTypeCastOptions.properties', async () => { | ||
const postKey = datastore.key('Scores'); | ||
const largeIntValueAsString = '9223372036854775807'; | ||
|
@@ -513,7 +548,7 @@ describe('Datastore', () => { | |
}, | ||
}); | ||
const [entity] = await datastore.get(key); | ||
assert.strictEqual(entity.year, integerValue); | ||
assert.strictEqual(entity.year.valueOf(), integerValue); | ||
}); | ||
|
||
it('should save and decode a double', async () => { | ||
|
@@ -527,7 +562,7 @@ describe('Datastore', () => { | |
}, | ||
}); | ||
const [entity] = await datastore.get(key); | ||
assert.strictEqual(entity.nines, doubleValue); | ||
assert.strictEqual(entity.nines.valueOf(), doubleValue); | ||
}); | ||
|
||
it('should save and decode a geo point', async () => { | ||
|
@@ -885,7 +920,7 @@ describe('Datastore', () => { | |
datastore.get(key), | ||
]); | ||
assert.strictEqual(typeof deletedEntity, 'undefined'); | ||
assert.strictEqual(fetchedEntity.rating, 10); | ||
assert.strictEqual(fetchedEntity.rating.valueOf(), 10); | ||
}); | ||
|
||
it('should use the last modification to a key', async () => { | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are Double / Int included in this type ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
They ought to be, via the Number type. Though that isn't really a use path. Support for that could be added, but as you might detect a trend, this is a mirroring of the existing style we went with for Ints.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There are not because
number
vsNumber
see https://www.typescriptlang.org/play?#code/MYGwhgzhAECyCeA5aBTAHgFxQOwCY0QFcBbAIxQCdoBvAXwCh6AzQ7YDASwHttomwOIABTYAXNGwlyFAJQ160RdGA8IXECgB0ILgHMRMgNz0GzVu268uAaxHiiZSnOoKlK7Go3a9B46f6CIigA7nBIQjJG9DZBoQiIEVH0QA
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there a case where we'd want to construct a
Number
from anotherNumber
? If not, I'd start with accepting astring
,number
, and potentiallyValueProto
(as we are).There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It seems like there is never a common case where we would want to construct a Double from a Double / Int. Constructing a
Double
from aDouble
is just creating a copy and making a Double from an Int could encourage users to misuse the API by inadvertently changing the column type.