whether to consider cache vertexDataFrame

[haohao0103]

The vertexDataFrame obtained when the VertexWriter is created is convenient for subsequent jobs to use, whether to consider cache;
com.alibaba.graphar.writer.VertexWriter.scala

[acezen]

It's maybe a good way to cache the DataFrame, but in writing process, the DataFrame is always been used only once, I not sure shall we need to cache it.

[haohao0103]

hi, thank you for your reply, Caching RDDS may seem unreasonable when we first created the VertextWriter, I think it makes more sense to cache chunks when creating the VertextWriter, because creating chunks is a shuffle operation and writing them out for each propertyGroup, which can happen multiple times. This is my understanding. I look forward to your reply.

[acezen]

sorry for reply this so late. It seems that cache the chunks is a good optimization for memory performance. And I think there are some other place that can apply the same optimization too. Can you help us to improve GraphAr with the optimization? look forward to your reply.

[haohao0103]

 Thanks, I am glad to do that. 没脑子? ***@***.***  

…

------------------ 原始邮件 ------------------ 发件人: "alibaba/GraphAr" ***@***.***>; 发送时间: 2024年1月12日(星期五) 晚上6:04 ***@***.***>; ***@***.******@***.***>; 主题: Re: [alibaba/GraphAr] whether to consider cache vertexDataFrame (Discussion #256) sorry for reply this so late. It seems that cache the chunks is a good optimization for memory performance. And I think there are some other place that can apply the same optimization too. Can you help us to improve GraphAr with the optimization? look forward to your reply. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>

[acezen]

The issue #322 has been created for this discussion, FYI.