Support Rapids cudf

[tebeka]

We'd like to support using Rapids cudf in frames.
Both on read and write.

Support

• Categoracial data
• strings (see here)
• Read/write
• Run on Iguazio platform

[tebeka]

After some try & error here's how I got a working cudf environment:

• Started Nvidia GPU Clould Image on GCE
• On the image docker run --runtime=nvidia --rm -it rapidsai/rapidsai:cuda9.2_ubuntu16.04 (base on this)
• In the docker shell: source activate rapids

[tebeka]

The idea is to add a frame_factory attribute to Python clients (idea from here and many other database drivers in Python).

By default frame_factory will pandas.DataFrame

[tebeka]

@yaronha said he perfers EC2 image.

[tebeka]

Quick try with EC2 nvidia image didn't go well. Going back to GCE and once the code works I'll work with @bivas to run it hosted

[tebeka]

rapids docker has an old version of pandas. Upgraded from 0.20.3 to 0.23.4 since pd.concat didn't have sort

[tebeka]

After updating pandas, had to add /conda/pkgs/nvstrings-0.0.3-cuda9.2_py35_18/lib/ to LD_LIBRARY_PATH

[tebeka]

See #175

[tebeka]

Managed to run on EC2 with NVIDIA Volta Deep Learning AMI 18.11 AMI

[tebeka]

Install document here, see also this issue.

[tebeka]

cudf 0.4.0 is out. Trying to see if it work with frames I can't get a working environment:

I'm working on "NVIDIA Volta Deep Learning AMI 18.11 AMI", and reading https://rapids.ai/documentation.html

• When trying to install with conda. I get a librmm error and after conda installing cudatoolkit=9.2 I get: 'libcudf.so': /home/ubuntu/miniconda3/lib/python3.6/site-packages/../../libcudf.so: undefined symbol: _ZN9NVStrings17create_from_indexEPSt4pairIPKcmEjb
• Then trying to use the rapids docker. The cudf version there is "'0+unknown'", when trying to conda install cudf=0.4.0 in the container I get permission denied. I can't upgrade pandas as well.

Asked help from Yuval Mazor [email protected] (NVIDIA Israel)

[tebeka]

cudf 0.5.0 is out. I'm able to run it on "NVIDIA Volta Deep Learning AMI 18.11 AMI" with miniconda:

$ curl -LO https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
$ bash Miniconda3-latest-Linux-x86_64.sh -b
$ export PATH=${HOME}/miniconda3/bin:${PATH}
$ conda install -c nvidia -c rapidsai -c pytorch -c numba -c conda-forge -c defaults cudf=0.5 cuml=0.5 python=3.6
$ conda install cudatoolkit=9.2

[tebeka]

Closing this one, opening #196 and #197