Reading from Parquet

[space55]

Hello,

Are there any plans to support reading a Parquet file into a dataframe? I have a need for this and am evaluating this library to use in an application.

Thanks!

[pjebs]

I've had lots of people in the past people asking for exporting to parquet, which I implemented.
You're the first to ask about importing, but I had put it in my todo list in may.
I won't have time to implement it soon. However, you can issue as PR.

[pjebs]

Hmmm. I noticed in my TODO list (#17), there had been 3 thumbs up for that request.

[khughitt]

In case it helps, here is some code I wrote to read a parquet file into a DataFrame that you may be able to adapt in the meantime:

package main

import (
	dataframe "github.com/rocketlaunchr/dataframe-go"
	"github.com/xitongsys/parquet-go-source/local"
	"github.com/xitongsys/parquet-go/reader"
        "context"
	"runtime"
)

func loadEntriesParquet(inputParquet string) *dataframe.DataFrame {
	// create local parquet/reader instances
	entriesFr, err := local.NewLocalFileReader(inputParquet)

	if err != nil {
		log.Println("Can't open file")
	}

	entriesPr, err := reader.NewParquetColumnReader(entriesFr, int64(runtime.NumCPU()))

	if err != nil {
		log.Println("Unable to create parquet reader", err)
	}

	// determine numer of rows in input parquet file
	numRows := int64(entriesPr.GetNumRows())

	// read columns from parquet and use them to construct a DataFrame instance of the
	// same form
	var paths, titles, bodies, accesscounts, accessdates, createddates, deadlines, priorities, archived []interface{}

	paths, _, _, _ = entriesPr.ReadColumnByPath("parquet_go_root.path", numRows)
	titles, _, _, _ = entriesPr.ReadColumnByPath("parquet_go_root.title", numRows)
	bodies, _, _, _ = entriesPr.ReadColumnByPath("parquet_go_root.body", numRows)
	accesscounts, _, _, _ = entriesPr.ReadColumnByPath("parquet_go_root.accesscount", numRows)
	accessdates, _, _, _ = entriesPr.ReadColumnByPath("parquet_go_root.lastaccess", numRows)
	createddates, _, _, _ = entriesPr.ReadColumnByPath("parquet_go_root.datecreated", numRows)
	deadlines, _, _, _ = entriesPr.ReadColumnByPath("parquet_go_root.deadline", numRows)
	priorities, _, _, _ = entriesPr.ReadColumnByPath("parquet_go_root.priority", numRows)
	archived, _, _, _ = entriesPr.ReadColumnByPath("parquet_go_root.archived", numRows)

	entries := dataframe.NewDataFrame(
		dataframe.NewSeriesString("path", nil, paths...),
		dataframe.NewSeriesString("title", nil, titles...),
		dataframe.NewSeriesString("body", nil, bodies...),
		dataframe.NewSeriesInt64("accesscount", nil, accesscounts...),
		dataframe.NewSeriesInt64("lastaccess", nil, accessdates...),
		dataframe.NewSeriesInt64("datecreated", nil, createddates...),
		dataframe.NewSeriesInt64("deadline", nil, deadlines...),
		dataframe.NewSeriesInt64("priority", nil, priorities...),
		dataframe.NewSeriesInt64("archived", nil, archived...),
	)

	entriesPr.ReadStop()
	entriesFr.Close()

	// sort entries by date of creation
	sortKey := []dataframe.SortKey{
		{Key: "datecreated", Desc: true},
	}

	ctx := context.Background()
	entries.Sort(ctx, sortKey)

	return entries
}

Few comments:

1. I can't make any claims that it is the most efficient approach, and feedback is welcome, but at least this should do the job..
2. The function loads a parquet dataframe containing "entries", with an expected format.. I left a lot of the file-specific logic in there to provide examples of how to handle different variable types.
3. I also left some logic in the bottom to help sort the dataframe once it's been loaded, in case that is useful.

Cheers.

[pjebs]

Thanks @khughitt . I need to generalise it so that it works for anything parquet data.

[CeciliaCoelho]

@pjebs Did you manage to generalise it? Can't get it to work, getting this error.

[pjebs]

@CeciliaCoelho can you show me your code.

I was actually waiting for a response to these Qs: xitongsys/parquet-go#360

[CeciliaCoelho]

@CeciliaCoelho can you show me your code.

I was actually waiting for a response to these Qs: xitongsys/parquet-go#360

Getting a new error now. This was a CSV that I converted to parquet using python but wanted to open and use in Go because of efficiency.
The CSV was like this:

I have this code:

package main

import (
	"context"
	"log"
	"runtime"

	dataframe "github.com/rocketlaunchr/dataframe-go"
	"github.com/xitongsys/parquet-go-source/local"
	"github.com/xitongsys/parquet-go/reader"
)

func loadEntriesParquet(inputParquet string) *dataframe.DataFrame {
	// create local parquet/reader instances
	entriesFr, err := local.NewLocalFileReader(inputParquet)

	if err != nil {
		log.Println("Can't open file")
	}

	entriesPr, err := reader.NewParquetColumnReader(entriesFr, int64(runtime.NumCPU()))

	if err != nil {
		log.Println("Unable to create parquet reader", err)
	}

	// determine numer of rows in input parquet file
	numRows := int64(entriesPr.GetNumRows())

	// read columns from parquet and use them to construct a DataFrame instance of the
	// same form
	var id, name, res, spill, turb, pump []interface{}

	id, _, _, _ = entriesPr.ReadColumnByPath("parquet_go_root.id", numRows)
	name, _, _, _ = entriesPr.ReadColumnByPath("parquet_go_root.name", numRows)
	res, _, _, _ = entriesPr.ReadColumnByPath("parquet_go_root.res", numRows)
	spill, _, _, _ = entriesPr.ReadColumnByPath("parquet_go_root.spill", numRows)
	turb, _, _, _ = entriesPr.ReadColumnByPath("parquet_go_root.turb", numRows)
	pump, _, _, _ = entriesPr.ReadColumnByPath("parquet_go_root.pump", numRows)

	entries := dataframe.NewDataFrame(
		dataframe.NewSeriesString("id", nil, id...),
		dataframe.NewSeriesString("name", nil, name...),
		dataframe.NewSeriesString("res", nil, res...),
		dataframe.NewSeriesInt64("spill", nil, spill...),
		dataframe.NewSeriesInt64("turb", nil, turb...),
		dataframe.NewSeriesInt64("pump", nil, pump...),
	)

	entriesPr.ReadStop()
	entriesFr.Close()

	// sort entries by date of creation
	sortKey := []dataframe.SortKey{
		{Key: "datecreated", Desc: true},
	}

	ctx := context.Background()
	entries.Sort(ctx, sortKey)

	return entries
}

func main() {
	loadEntriesParquet("cascades2.parquet")
}

Now the error I'm getting is this:

[pjebs]

The "new" error is because you are trying to sort based on a Series called "datecreated" but you don't have such a field.

[CeciliaCoelho]

The "new" error is because you are trying to sort based on a Series called "datecreated" but you don't have such a field.

Oh right, didn't notice that. It's running now. Thanks :)
How do I print or access the dataframe? (bet it's a stupid question, sorry I'm a Golang newbie)

[pjebs]

the function returns a *dataframe.DataFrame object. You can see examples in the Readme.

However, when I look at the code, it's not efficient at loading the Dataframe with the data. I need to understand that Parquet package better before I can improve the code.

[pjebs]

Parquet importing is now supported (experimental): @CeciliaCoelho @khughitt @space55