Supporting HugeGraph load from GraphAr

[Thespica]

Information about how HugeGraph loads files.

Information below may help us understand and support loading of HugeGraph, but maybe have limited coverage. Supplements and suggestions are welcome!

Links

Doc

hugegraph(server)

hugegraph-toolchain

Related modules

• Loader(file level): A data import component that can transform data from various sources into vertices and edges and bulk import them into the graph database. (for small/mall to medium size graph datasets)
• Client(Java SDK level): HugeGraph client written in Java. Users can use the client to operate HugeGraph using Java code.
• Hubble: a graph management and analysis platform that provides features: graph data load, schema management, graph relationship analysis, and graphical display. (for large data load)

Concepts

Three kinds files

How to use hugegraph loader

HugeGraph loader support files or directories from local or HDFS.

• schema file, including: PropertyKey, VertexLabel, EdgeLabel and IndexLabel.
• data file(csv, txt, json, or HDFS files)
• mapping/struct file

Further consider: make GraphAr Java SDK supporting HDFS?
comment(@acezen): GraphAr and HDFS is decoupled, I think Java SDK is supporting HDFS since c++ allow to read data from hdfs.

Vertex/Edge ID Policy

Vertex id policy

1. AUTOMATIC ID Policy
2. PRIMARY_KEY ID policy
3. CUSTOMIZE_STRING ID Policy
4. CUSTOMIZE_NUMBER ID Policy

Edge id policy

The Edge Id of HugeGraph is composed of srcVertexId + edgeLabel + sortKey + tgtVertexId. And sortKey is an important concept of HugeGraph.

The edges of HugeGraph only support directed edges, and undirected edges can be realized by creating two edges, Out and In.

[SemyonSinchenko]

Reading from HDFS should work out of the box via GraphSr Spark (there is a Hadoop dependency) with org.apache.hadoop.fs.FileSystem

P.S. HDFS is a file system, not a file format.
P.P.S. org.apache.hadoop.fs.FileSystem can be used for both local and hdfs files. It can choose the right implementation of FileSystem based on URI (hdfs:///, file:/, s3:///, etc.)

[acezen]

According to the HugeGraph document, The HugeGraph Loader seems just for small or medium size graph datasets import. For larger dataset, there is a module call Hubble that support import data parallelly. Maybe it's more suitable than Loader for GraphAr.

[Thespica]

As I know, Hubble is a visual analysis platform, which is more convenient but has less function.

[acezen]

I think we can first compare the graph model specification of HugeGraph with GraphAr's specification, to figure out whether GraphAr can support HugeGraph's schema(if not, are there some new features that we can import to GraphAr that improve the ability).

And then, see how to construct HugeGraph's schema with GraphAr's infos for a graph dataset.

Finally, see how to process the files of GraphAr to construct the graph in HugeGraph.

[Thespica]

I think we can use their loader example.

[acezen]

yes, that would be a good example to start.

[Thespica]

Differences of schema and how to mapping

Repeat PropertyLabel

In HugeGraph, PropertyLabel belongs to graph, and can be reused by different vertices and edges. So PropertyLabels with same name is not allowed in a graph.

In GraphAr, a property only belongs to a vertex label or a edge label. So properties with same name(maybe different data type) may exist in a graph.

Possible mapping methods:

• Keep origin property name, merge properties with same name and type. Throw error when there are properties with same name but have different data type.
• Combine origin property name and data type as new property name.
• Combine vertex/edge label and origin property name as new property name.

Nullable properties(keys)

In HugeGraph, properties in a VertexLabel or EdgeLabel must been set as nullable or not, not nullable by default.

In GraphAr, we haven't record that information, which means that every not primary property is nullable, and each primary property is not nullable.

Mapping solution: primary property by default(not nullable); not primary property by nullable.

PrimaryKey of VertexLabel

In HugeGraph, there are 4 id strategies to choose primary key of each VertexLabel:

interface	idStrategy	description
useAutomaticId	AUTOMATIC	generate id automatically by Snowflake algorithm
useCustomizeStringId	CUSTOMIZE_STRING	passed id by user, must be string type
useCustomizeNumberId	CUSTOMIZE_NUMBER	passed id by user, must be number type
usePrimaryKeyId	PRIMARY_KEY	choose some important prop as primary key to splice id

In GraphAr, a specific vertex can been identified by its vertex label + internal id, or been identified by its vertex label + its primary key(if have).

Mapping solution: using PRIMARY_KEY id strategy by vertex label + internal id;

Direction of edge

HugeGraph have only directed edge.

GraphAr have directed edge and not directed edge.

Mapping solution: for each undirected edge, split to in and out two directed edges.

Frequency of edge

HugeGraph allow single and multiple frequency of same edge with different property, and must set sortKey for multiple frequency edge.

GraphAr allow an edge occur many times, but have no further recording about that.

It is a problem to deal with that. Beside, should find out how HugeGraph handled repeated edge when it was set as single, cover it or have other behaviors?

IndexLabel

IndexLabel in HugeGraph aims to increase the ability to access data, but will use more spaces and cost more time to create and maintain.

GraphAr don't have this part, so don't need to mapping.

(Un)ordered by src/dst of edge

Not sure about that, maybe related to data rather than schema?

What may we learn from schema of HugeGraph(need discussion)

Nullable properties(keys)

Record nullable of property in each vertex and edge, for three type:

• nullable
• not nullable, allow same property value
• not nullable, only allow unique property value(is_primary)

Index label

Like what HugeGraph do, it well help querying data efficiently. But need consider cost of spaces to record them and cost of time to maintain them.

User defined information

For graph/edge/vertex info, property info in a vertex/edge or in a graph?

[acezen]

hi, @Thespica, I think you can create some issue about the feature base on the discussion here.

[lixueclaire]

hi, @Thespica。I'd like to clarify that GraphAr supports the storage of multiple edges between the same source and destination nodes, allowing for the representation of edge frequencies.

[Thespica]

Thanks for your clarification!

Modified it.