PPl flatten command
#784
Merged
PPl flatten command
#784
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lukasz-soszynski-eliatra
force-pushed
the
flatten
branch
2 times, most recently
from
a7a0677 to
d566d3f
Compare
YANG-DB
requested changes
Oct 21, 2024
lukasz-soszynski-eliatra
force-pushed
the
flatten
branch
5 times, most recently
from
955ffe3 to
8f2ade9
Compare
lukasz-soszynski-eliatra
requested review from
dai-chen,
rupal-bq,
mengweieric,
vamsimanohar,
penghuo,
seankao-az,
anirudha,
kaituo,
noCharger,
LantaoJin and
ykmr1224
as code owners
lukasz-soszynski-eliatra
force-pushed
the
flatten
branch
from
03222ce to
287ad3f
Compare
YANG-DB
requested changes
Oct 29, 2024
| private val planTransformer = new CatalystQueryPlanVisitor() | ||
| private val pplParser = new PPLSyntaxParser() | ||
|
|
||
| test("test fillnull only field") { |
There was a problem hiding this comment.
@lukasz-soszynski-eliatra plz change the test name to match the ppl query
There was a problem hiding this comment.
done
There was a problem hiding this comment.
@lukasz-soszynski-eliatra plz add more tests with different queries including
- stats (aggregations)
- parse (parsing of regExp)
- eval
There was a problem hiding this comment.
done
YANG-DB
requested changes
Oct 30, 2024
| super.beforeAll() | ||
|
|
||
| // Create test table | ||
| createSimpleNestedJsonContentTable(tempFile, testTable) |
There was a problem hiding this comment.
@lukasz-soszynski-eliatra plz add additional test from other types of tables or use cases:
- NestesTable
- structTable
- table with multiValue column (Array)
There was a problem hiding this comment.
done
|
@lukasz-soszynski-eliatra also plz rebase ... |
Signed-off-by: Lukasz Soszynski <[email protected]>
…hecks for logical plans. Signed-off-by: Lukasz Soszynski <[email protected]>
… tests Signed-off-by: Lukasz Soszynski <[email protected]>
lukasz-soszynski-eliatra
force-pushed
the
flatten
branch
from
aefe422 to
0a0e2c5
Compare
YANG-DB
approved these changes
Oct 30, 2024
LantaoJin
reviewed
Oct 31, 2024
| #### Data | ||
| | \_time | bridges | city | coor | country | | ||
| |---------------------|----------------------------------------------|---------|------------------------|---------------| | ||
| | 2024-09-13T12:00:00 | [{801, Tower Bridge}, {928, London Bridge}] | London | {35, 51.5074, -0.1278} | England | |
There was a problem hiding this comment.
Where does these example data come from? I suggest to create some example data by ourselves instead of copying from internet only if they are from public dataset. @YANG-DB please do not use internet data, specially documentation of other similar product, in github issue either.
There was a problem hiding this comment.
These data come from the issue #669 but were extended.
| @@ -53,6 +53,7 @@ commands | |||
| | renameCommand | |||
| | fillnullCommand | |||
| | fieldsummaryCommand | |||
| | flattenCommand | |||
| ; | |||
|
|
|||
| commandName | |||
There was a problem hiding this comment.
Please add FLATTEN keyword in the end of commandName section.
There was a problem hiding this comment.
Great job! LGTM except this^
There was a problem hiding this comment.
Corrected.
There was a problem hiding this comment.
Please add
FLATTENkeyword in the end ofcommandNamesection.
please add here
Comment on lines
+4
to
+6
| Using `flatten` command to flatten a field of type: | ||
| - `struct<?,?>` | ||
| - `array<struct<?,?>>` |
There was a problem hiding this comment.
Are these data_types what we expecting for this command? I thought the expected input is json string. But it's fine for now. We can enhance it later.
Signed-off-by: Lukasz Soszynski <[email protected]>
YANG-DB
requested changes
Oct 31, 2024
| @@ -53,6 +53,7 @@ commands | |||
| | renameCommand | |||
| | fillnullCommand | |||
| | fieldsummaryCommand | |||
| | flattenCommand | |||
| ; | |||
|
|
|||
| commandName | |||
There was a problem hiding this comment.
Please add
FLATTENkeyword in the end ofcommandNamesection.
please add here
YANG-DB
approved these changes
Oct 31, 2024
YANG-DB
added a commit
that referenced
this pull request
Nov 2, 2024
* update antlr grammar for (future) P1 command syntax Signed-off-by: YANGDB <[email protected]> * add trendline command Signed-off-by: YANGDB <[email protected]> * add expand command Signed-off-by: YANGDB <[email protected]> * add geoip command Signed-off-by: YANGDB <[email protected]> * PPl `flatten` command (#784) * The flatten command implemented Signed-off-by: Lukasz Soszynski <[email protected]> * The flatten command integration tests were extended with additional checks for logical plans. Signed-off-by: Lukasz Soszynski <[email protected]> * flatten, added more tests related to plan translation and integration tests Signed-off-by: Lukasz Soszynski <[email protected]> * Flatten command added to command names list. Signed-off-by: Lukasz Soszynski <[email protected]> --------- Signed-off-by: Lukasz Soszynski <[email protected]> * Extract source table names from mv query (#854) * add sourceTables to MV index metadata properties Signed-off-by: Sean Kao <[email protected]> * parse source tables from mv query Signed-off-by: Sean Kao <[email protected]> * test cases for parse source tables from mv query Signed-off-by: Sean Kao <[email protected]> * use constant for metadata cache version Signed-off-by: Sean Kao <[email protected]> * write source tables to metadata cache Signed-off-by: Sean Kao <[email protected]> * address comment Signed-off-by: Sean Kao <[email protected]> * generate source tables for old mv without new prop Signed-off-by: Sean Kao <[email protected]> * syntax fix Signed-off-by: Sean Kao <[email protected]> --------- Signed-off-by: Sean Kao <[email protected]> * Fallback to internal scheduler when index creation failed (#850) * Fallback to internal scheduler when index creation failed Signed-off-by: Louis Chu <[email protected]> * Fix IT Signed-off-by: Louis Chu <[email protected]> * Fix IOException Signed-off-by: Louis Chu <[email protected]> --------- Signed-off-by: Louis Chu <[email protected]> * New trendline ppl command (SMA only) (#833) * WIP trendline command Signed-off-by: Kacper Trochimiak <[email protected]> * wip Signed-off-by: Kacper Trochimiak <[email protected]> * trendline supports sorting Signed-off-by: Kacper Trochimiak <[email protected]> * run scalafmtAll Signed-off-by: Kacper Trochimiak <[email protected]> * return null when there are too few data points Signed-off-by: Kacper Trochimiak <[email protected]> * sbt scalafmtAll Signed-off-by: Kacper Trochimiak <[email protected]> * Remove WMA references Signed-off-by: Hendrik Saly <[email protected]> * trendline - sortByField as Optional<Field> Signed-off-by: Kacper Trochimiak <[email protected]> * introduce TrendlineStrategy Signed-off-by: Kacper Trochimiak <[email protected]> * keywordsCanBeId -> replace SMA with trendlineType Signed-off-by: Kacper Trochimiak <[email protected]> * handle trendline alias as qualifiedName instead of fieldExpression Signed-off-by: Kacper Trochimiak <[email protected]> * Add docs Signed-off-by: Hendrik Saly <[email protected]> * Make alias optional Signed-off-by: Hendrik Saly <[email protected]> * Adapt tests for optional alias Signed-off-by: Hendrik Saly <[email protected]> * Adden logical plan unittests Signed-off-by: Hendrik Saly <[email protected]> * Add missing license headers Signed-off-by: Hendrik Saly <[email protected]> * Fix docs Signed-off-by: Hendrik Saly <[email protected]> * numberOfDataPoints must be 1 or greater Signed-off-by: Hendrik Saly <[email protected]> * Rename TrendlineStrategy to TrendlineCatalystUtils Signed-off-by: Hendrik Saly <[email protected]> * Validate TrendlineType early and pass around enum type Signed-off-by: Hendrik Saly <[email protected]> * Add trendline chaining test Signed-off-by: Hendrik Saly <[email protected]> * Fix compile errors Signed-off-by: Hendrik Saly <[email protected]> * Fix imports Signed-off-by: Hendrik Saly <[email protected]> * Fix imports Signed-off-by: Hendrik Saly <[email protected]> --------- Signed-off-by: Kacper Trochimiak <[email protected]> Signed-off-by: Hendrik Saly <[email protected]> Co-authored-by: Kacper Trochimiak <[email protected]> * update iplocation antlr Signed-off-by: YANGDB <[email protected]> --------- Signed-off-by: YANGDB <[email protected]> Signed-off-by: Lukasz Soszynski <[email protected]> Signed-off-by: Sean Kao <[email protected]> Signed-off-by: Louis Chu <[email protected]> Signed-off-by: Kacper Trochimiak <[email protected]> Signed-off-by: Hendrik Saly <[email protected]> Co-authored-by: lukasz-soszynski-eliatra <[email protected]> Co-authored-by: Sean Kao <[email protected]> Co-authored-by: Louis Chu <[email protected]> Co-authored-by: Hendrik Saly <[email protected]> Co-authored-by: Kacper Trochimiak <[email protected]>
YANG-DB
added a commit
that referenced
this pull request
Nov 5, 2024
* update antlr grammar for (future) P1 command syntax Signed-off-by: YANGDB <[email protected]> * add trendline command Signed-off-by: YANGDB <[email protected]> * add expand command Signed-off-by: YANGDB <[email protected]> * add geoip command Signed-off-by: YANGDB <[email protected]> * PPl `flatten` command (#784) * The flatten command implemented Signed-off-by: Lukasz Soszynski <[email protected]> * The flatten command integration tests were extended with additional checks for logical plans. Signed-off-by: Lukasz Soszynski <[email protected]> * flatten, added more tests related to plan translation and integration tests Signed-off-by: Lukasz Soszynski <[email protected]> * Flatten command added to command names list. Signed-off-by: Lukasz Soszynski <[email protected]> --------- Signed-off-by: Lukasz Soszynski <[email protected]> * Extract source table names from mv query (#854) * add sourceTables to MV index metadata properties Signed-off-by: Sean Kao <[email protected]> * parse source tables from mv query Signed-off-by: Sean Kao <[email protected]> * test cases for parse source tables from mv query Signed-off-by: Sean Kao <[email protected]> * use constant for metadata cache version Signed-off-by: Sean Kao <[email protected]> * write source tables to metadata cache Signed-off-by: Sean Kao <[email protected]> * address comment Signed-off-by: Sean Kao <[email protected]> * generate source tables for old mv without new prop Signed-off-by: Sean Kao <[email protected]> * syntax fix Signed-off-by: Sean Kao <[email protected]> --------- Signed-off-by: Sean Kao <[email protected]> * Fallback to internal scheduler when index creation failed (#850) * Fallback to internal scheduler when index creation failed Signed-off-by: Louis Chu <[email protected]> * Fix IT Signed-off-by: Louis Chu <[email protected]> * Fix IOException Signed-off-by: Louis Chu <[email protected]> --------- Signed-off-by: Louis Chu <[email protected]> * New trendline ppl command (SMA only) (#833) * WIP trendline command Signed-off-by: Kacper Trochimiak <[email protected]> * wip Signed-off-by: Kacper Trochimiak <[email protected]> * trendline supports sorting Signed-off-by: Kacper Trochimiak <[email protected]> * run scalafmtAll Signed-off-by: Kacper Trochimiak <[email protected]> * return null when there are too few data points Signed-off-by: Kacper Trochimiak <[email protected]> * sbt scalafmtAll Signed-off-by: Kacper Trochimiak <[email protected]> * Remove WMA references Signed-off-by: Hendrik Saly <[email protected]> * trendline - sortByField as Optional<Field> Signed-off-by: Kacper Trochimiak <[email protected]> * introduce TrendlineStrategy Signed-off-by: Kacper Trochimiak <[email protected]> * keywordsCanBeId -> replace SMA with trendlineType Signed-off-by: Kacper Trochimiak <[email protected]> * handle trendline alias as qualifiedName instead of fieldExpression Signed-off-by: Kacper Trochimiak <[email protected]> * Add docs Signed-off-by: Hendrik Saly <[email protected]> * Make alias optional Signed-off-by: Hendrik Saly <[email protected]> * Adapt tests for optional alias Signed-off-by: Hendrik Saly <[email protected]> * Adden logical plan unittests Signed-off-by: Hendrik Saly <[email protected]> * Add missing license headers Signed-off-by: Hendrik Saly <[email protected]> * Fix docs Signed-off-by: Hendrik Saly <[email protected]> * numberOfDataPoints must be 1 or greater Signed-off-by: Hendrik Saly <[email protected]> * Rename TrendlineStrategy to TrendlineCatalystUtils Signed-off-by: Hendrik Saly <[email protected]> * Validate TrendlineType early and pass around enum type Signed-off-by: Hendrik Saly <[email protected]> * Add trendline chaining test Signed-off-by: Hendrik Saly <[email protected]> * Fix compile errors Signed-off-by: Hendrik Saly <[email protected]> * Fix imports Signed-off-by: Hendrik Saly <[email protected]> * Fix imports Signed-off-by: Hendrik Saly <[email protected]> --------- Signed-off-by: Kacper Trochimiak <[email protected]> Signed-off-by: Hendrik Saly <[email protected]> Co-authored-by: Kacper Trochimiak <[email protected]> * update iplocation antlr Signed-off-by: YANGDB <[email protected]> * update scala fmt style Signed-off-by: YANGDB <[email protected]> * `cidrmatch` ppl command add logical tests and docs (#865) * update logical tests and docs Signed-off-by: YANGDB <[email protected]> * update scala fmt style Signed-off-by: YANGDB <[email protected]> * fix type error Signed-off-by: YANGDB <[email protected]> --------- Signed-off-by: YANGDB <[email protected]> * Support Lambda and add related array functions (#864) * json function enhancement Signed-off-by: Heng Qian <[email protected]> * Add JavaToScalaTransformer Signed-off-by: Heng Qian <[email protected]> * Apply scalafmtAll Signed-off-by: Heng Qian <[email protected]> * Address comments Signed-off-by: Heng Qian <[email protected]> * Add IT and change to use the same function name as spark Signed-off-by: Heng Qian <[email protected]> * Address comments Signed-off-by: Heng Qian <[email protected]> * Add document and separate lambda functions from json functions Signed-off-by: Heng Qian <[email protected]> * Add lambda functions transform and reduce Signed-off-by: Heng Qian <[email protected]> * polish lambda function document Signed-off-by: Heng Qian <[email protected]> * polish lambda function document Signed-off-by: Heng Qian <[email protected]> * Minor fix Signed-off-by: Heng Qian <[email protected]> * Minor change to polish the documents Signed-off-by: Heng Qian <[email protected]> --------- Signed-off-by: Heng Qian <[email protected]> --------- Signed-off-by: YANGDB <[email protected]> Signed-off-by: Lukasz Soszynski <[email protected]> Signed-off-by: Sean Kao <[email protected]> Signed-off-by: Louis Chu <[email protected]> Signed-off-by: Kacper Trochimiak <[email protected]> Signed-off-by: Hendrik Saly <[email protected]> Signed-off-by: Heng Qian <[email protected]> Co-authored-by: lukasz-soszynski-eliatra <[email protected]> Co-authored-by: Sean Kao <[email protected]> Co-authored-by: Louis Chu <[email protected]> Co-authored-by: Hendrik Saly <[email protected]> Co-authored-by: Kacper Trochimiak <[email protected]> Co-authored-by: qianheng <[email protected]>
kenrickyap
pushed a commit
to Bit-Quill/opensearch-spark
that referenced
this pull request
Dec 11, 2024
* The flatten command implemented Signed-off-by: Lukasz Soszynski <[email protected]> * The flatten command integration tests were extended with additional checks for logical plans. Signed-off-by: Lukasz Soszynski <[email protected]> * flatten, added more tests related to plan translation and integration tests Signed-off-by: Lukasz Soszynski <[email protected]> * Flatten command added to command names list. Signed-off-by: Lukasz Soszynski <[email protected]> --------- Signed-off-by: Lukasz Soszynski <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
The
flattencommand is introducedIssues Resolved
#669
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.