UI: Complex data types support

[harshach]

Hi,
In the upcoming 0.5 release, we are working on supporting complex data types and nested structures.
Examples like the following:
array<struct<shipdate:timestamp, price:double precision>>
array<struct<nestedarray:array<struct<struct<morenestedarray:array<string>>>>>
struct<a: string, b: struct<c: integer, d: struct<e: string>>>
struct<street_name:varchar(24),zipcode:int,city:varchar(100),country:struct<country_code:int, name:varchar(100)>>
array<struct<product_id:varchar(24),price:int,onsale:boolean,tax:int,weight:int,others:int,vendor:varchar(64)>>

We would love to hear feedback from the community to understand the different types they may be using and any input on our APIs and schema to capture the complex data types.
Please take a look at our sample data and some of the complex data types we have

https://github.com/open-metadata/OpenMetadata/blob/main/ingestion/examples/sample_data/datasets/tables.json#L438

Here is the UI mock. Please give us any feedback you may have

@juliarvalenti Thanks for the great feedback you've provided on Data Profiler. It would be great to get your feedback on this mock if you got time.
@amiorin could you please provide feedback and any examples you can share of complex data types

[juliarvalenti]

Hi @harshach,

I love where this is going. The nested rows are much clearer here.

One minor question - I see in the description for shop_id that it is a foreignkey for dim_shop. This appears to be manually set by the user.

Is it possible, valuable, or in scope to use the Entity Reference or some other method to define this relationship? For example - since its a FK, we could have it with a hollow key icon on the left. location_id can have a filled-in key icon.

See the mockup below:

[shahsank3t]

@juliarvalenti
Agree with your suggestion. We are already showing icons for Primary Key, Unique, and Not Null so adding another icon to denote Foreign Key does make sense and is a must-have IMO.

[harshach]

@juliarvalenti in APIs and Schemas we have constraints both at column level and table level. https://github.com/open-metadata/OpenMetadata/blob/main/catalog-rest-service/src/main/resources/json/schema/entity/data/table.json#L79
We do capture them during the ingestion process.