Setup

State “DONE” from “” [2014-12-24 Wed 13:45]

create a data directory
create a file and put in it the download links

projdir="$HOME/projects/2014-12-20_datascibowl/"
if [ -d "$projdir" ]
then
  echo "$projdir exists, entering it"
  cd "$projdir"
else
  echo "making $projdir and entering it"
  mkdir "$projdir" && cd "$_"
fi
if [-d "data"]
then
  echo "data dir exists, entering it"
  cd "data"
else
  echo "creating data directory and download file list"  
  mkdir "data" && cd "data"
  data_download.txt
  echo 'http://www.kaggle.com/c/datasciencebowl/download/train.zip' > data_download.txt
  echo 'http://www.kaggle.com/c/datasciencebowl/download/test.zip' >> data_download.txt
fi

Get data

State “DONE” from “” [2014-12-24 Wed 13:45]

# currently doesn't work, need to export browser cookies
# and use wget option --load-cookies
# http://askubuntu.com/questions/161778/how-do-i-use-wget-curl-to-download-from-a-site-i-am-logged-into
# only if files in data_download dont exist
cd "$HOME/projects/2014-12-20_datascibowl/data"
wget -nv -i data_download.txt

# again if they dont already exist, that's -n
cd ~/projects/2014-12-20_datascibowl/data
unzip -n train
unzip -n test

Ipython

State “DONE” from “TODO” [2014-12-20 Sat 22:03]

installed in arch, had a few dependencies, some from AUR I think, though that may have been for skimage

ipython2 notebook

opens ipython in web browser
from Help can see keyboard shortcuts, and links to libary documentation

Python/Dev Stuff [8/10]

Setup packages

State “DONE” from “” [2014-12-24 Wed 13:45]

python environment in arch
it is suggested to manage packages with the distro package manager see
from the tutorial it looks like we need
2.7
- extra
  - [X] sklearn scikit-learn
  - [X] matplotlib
  - [X] pylab
  - [X] numpy
  - [X] pandas
  - [X] scipy
  - [X] pillow
  - there were some more dependencies for iptyhon, and one more for a function called from the tutorials
  - imread was one of them
- defaults
  - glob
  - os
  - warnings
  - pylab in venv
- needed gcc-fortran for install scipy in venv

virtual env Research

State “DONE” from “TODO” [2014-12-23 Tue 01:24]

Setting up for python development on ArchLinux

State “DONE” from “NEXT” [2014-12-24 Wed 13:44]
State “NEXT” from “” [2014-12-24 Wed 12:47]

References
- pydocs
- archwiki

First we’re going to isntall pip and virtual env, we dont need pip as venvs install pip into the venv when created, but worth getting it for shell completion.

sudo pacman -S python3-pip
sudo pacman -S python3-virtualenv
# sudo pacman -S python2-pip
# sudo pacman -S python2-virtualenv

Make pip play nice with zsh.

pip completion --zsh >> ~/.zprofile

venvwrapper is a nubmer of shell scripts that simplify working with and tracking venvs. I regret it fixes the virtual env base path, which affects how I organize projects. Although venvs can be linked to project directory using mkproject, which satisfies my purposes.

#sudo pip install virtualenvwrapper
sudo pacman -S python-virtualenvwrapper # can create python2 venvs
echo "source /usr/local/bin/virtualenvwrapper.sh" >> ~/.zprofile
export WORKON_HOME=~/venvs
export PROJECT_HOME=~/projects
mkdir -p $WORKON_HOME
echo "export WORKON_HOME=$WORKON_HOME" >> ~/.zprofile
echo "export PROJECT_HOME=$PROJECT_HOME" >> ~/.zprofile

Head over to the project directory and setup a venv there, then activate it.

#virtualenv -p /usr/bin/python2.7 workspace_datascibowl
# mkvirtualenv -p /usr/bin/python2.7 2014-12-23_datascibowl

# -f makes the venv even if proj exists
mkproject -f -p /usr/bin/python2.7 2014-12-24_testdatascibowl

# check out your venvs
ls $WORKON_HOME

pip install git://github.com/yajiemiao/pdnn
# check out packages in venv
lssitepackages

deactivate deactivates the current venv. And can also navigate to the active venv with cdvirtualenv.

Settung up python dev on ArchLinux v2

State “DONE” from “NEXT” [2015-01-06 Tue 22:04]
State “NEXT” from “TODO” [2015-01-06 Tue 08:34]

missed the point, mkproject not working, it’s putting the venv in the project dir instead of the $WORKON_HOME
retested - mkproject works
for future reference, this is how to do this if there’s alredy a project dir
so how to do this? just make the venv and then associate it with a project dir?

mkvirtualenv -p /usr/bin/python2.7 2014-12-20_datascibowl
setvirtualenvproject  ~/projects/2014-12-20_datascibowl
#or 
mkvirtualenv -p /usr/bin/python2.7 -a ~/projects/2014-12-20_datascibowl  2014-12-20_datascibowl 
cdvirtualenv # takes to venv dir
cdproject # takes to project dir of active venv

setting up venv for distribution

State “DONE” from “NEXT” [2015-01-06 Tue 22:44]
State “NEXT” from “” [2015-01-06 Tue 22:43]

make requirements list

pip freeze > ~/venvs/2014-12-20_datascibowl/requirements.txt

cat $WORKON_HOME/2014-12-20_datascibowl/requirements.txt

pip install -r
ipython
pyzmq
jinja2
tornado

Pickle for saving python binary objects

State “DONE” from “NEXT” [2015-01-09 Fri 21:18]
State “NEXT” from “TODO” [2015-01-06 Tue 13:49]
State “NEXT” from “TODO” [2015-01-06 Tue 13:37]

why pickle?

Pickle lets you make persistent python objects. Intermediate results, such as feature matrices and model objects, may be costly to compute. It is often useful then to save them. This may also aid reproduction. Pickle allows for this by converting python objects to byte streams.

what can pickle do?

Pickle pickles (serializes) python objects into byte streams. It also unpicks (or unserializes) them, i.e. the byte stream is loaded back into a python object. The byte streams can be saved to persistent storage, in a database, or transmitted over a network.

If you know the nitty gritty send me a link at the contact page, twitter, or leave a comment.

What can’t be pickled?

Generally speaking, classes, functions, and methods.

how to use it?

For pickling (unpickling), create a pickle object and call its dump (load) method with a file connection.

import pickle

wrc_champions = {
  "Sebastien Loeb": [i+2004 for i in range(9)],
  "Juha Kankkunen": [1986,1987,1991,1993]
}

with open("wrc_champions.p", "w") as f:
  pickle.dump(wrc_champions, f)

import pickle
import pprint
import os
with open("wrc_champions.p") as f:
  wrc_champions = pickle.load(f)

pprint.pprint(wrc_champions)
os.remove("wrc_champions.p")

cPickle

The cPickle module has Pickler and Unpickler methods, which are up to 1000 times faster than the Pickle counterparts. I wonder what the limitations are to that. Although, cPickle ones can’t be subclassed, they suit most common purposes. There are other reasons to stick with pickle, though.

Caveats (Carrots?)

Dont eat pickles from untrusted sources, arbitrary python code can be executed in unpickling! Found here, along with more pertinents.

Sets and lists builting data structures

State “DONE” from “” [2014-12-24 Wed 20:50]

Sets are hashed iterable unorder lists of distinct items.

a = ["why'd", "you", "crisp", "greedo?"]
set_a = set(a)
type(a)
type(set_a)

print "List a:"
for item in a:
  print item

print "Set a:"
for item in set_a:
  print item

a.append(", han")
print a

set_b = (["why'd", "you", "crisp", "greedo?"])
print set_a.difference(set_b)

list comprehensions

State “DONE” from “TODO” [2015-01-09 Fri 21:21]

did an online python tutorial
dont even remember why!

NEXT [#A] Iterators

State “NEXT” from “” [2015-03-07 Sat 13:04]

NEXT [#B] generators

State “NEXT” from “TODO” [2015-01-09 Fri 21:21]

learn key python packages [2/6]

State “DONE” from “” [2015-02-15 Sun 19:40]

for this i changed the python command variable in emacs to use python2
also must use :results output tag instead of :results value, as the latter requires you return object

Numpy [2/3]

State “DONE” from “” [2014-12-21 Sun 21:14]

creating arrays

State “DONE” from “” [2015-01-06 Tue 22:47]

import numpy as np
x = np.array([[1, 2, 3], [4, 5, 6]], np.int32)
print type(x)
print x
print x.shape
print x[0,1]
print x[1,2]
print x[:,1]
print x[:1,]

slicing

State “DONE” from “” [2015-01-06 Tue 22:47]

is powerful, and slices are pointers, i.e. updating their contents updates the parent object
slice with array[i:j:k] where i is start index, j is stop, and k int > 0 is step
seems to be indexed as so

0 2 4

1 3 5

import numpy as np
x = np.array([[1, 2, 3], [4, 5, 6]], np.int32)
print x[1:5:2]
print x[:,1]
print x[:1,]

NEXT iteration

State “NEXT” from “” [2015-01-06 Tue 22:47]

linalg

Scipy

State “DONE” from “TODO” [2015-01-04 Sun 13:18]

mathematical functions built on numpy
read about it.
- stats functions,
- spatial data method such as delaunay triangulation,
- singal proc
- integration, ode
there is a tutorial

NEXT multidimensional image proc

State “NEXT” from “TODO” [2015-01-09 Fri 21:22]
State “NEXT” from “TODO” [2015-01-09 Fri 21:22]
State “NEXT” from “TODO” [2015-01-09 Fri 21:22]
State “NEXT” from “TODO” [2015-01-09 Fri 21:21]

i dont think
relevant for snodrofo

NEXT Pandas

State “NEXT” from “TODO” [2015-01-09 Fri 21:22]

sci kit learn (user guide)

State “DONE” from “TODO” [2015-01-06 Tue 08:36]

the help section on regression is easy as i already understand that, understanding the python part is straight forward
and the algorithm map is very useful

NEXT sci kit image

State “NEXT” from “TODO” [2015-01-09 Fri 21:22]

Hu central moment expirimentation [0/2]

State “DONE” from “CANCELLED” [2015-02-15 Sun 19:40]
State “CANCELLED” from “TODO” [2015-02-05 Thu 12:50]
- dnn

Features [1/2]

[X] try using moments 2-6, I htink 1-7 are not as meaninful
- no better
[ ] try permutation of the central moments combinations

NEXT Tests [0/1]

State “NEXT” from “” [2015-01-06 Tue 08:35]

[ ] try manually doing the central moments on a trivial image to test the reflect, translation, rotation invariance

ML procedure

State “NEXT” from “” [2015-01-09 Fri 22:52]

read data
save pickled raw data? maybe with cpickle
calculate features and create descriptive stats on them
save feature matrices
fit models

model infrastructure

how can we optimize the iteration
save intermediate models with pickle
bert is doing this

Write a class for this? work with bert

ugh, i like the functional/R paradigm of methods belong to functions! this hurts my brain
inside venv we shoudl have all we need to work on it, including the data and libraries
class is datascibowl
attributes
- training classes
  - nested dict with names of classes/subclasses?
methods
- scale image
- calculate feature
- preprocess

CANCELLED ML Stuff

State “CANCELLED” from “TODO” [2015-02-15 Sun 19:42]
- sticking with pylearn

ufldl tutorial for deep learning

Check the kaggle handwriting competition example

State “DONE” from “NEXT” [2015-01-04 Sun 12:39]
State “NEXT” from “TODO” [2014-12-25 Thu 16:25]

bert thinks may be some inspiration here
I am not seeing much other than tutorial/info on RF classifier
RF decision trees

CANCELLED make a uniform probability prediction

State “CANCELLED” from “NEXT” [2015-01-04 Sun 12:39]
trivial
Rescheduled from “2014-12-22 Mon” on [2014-12-25 Thu 16:29]
State “NEXT” from “TODO” [2014-12-21 Sun 21:13]

The idea here is just to learn a bit more about the data manipulations, processing, etc in python. Cancelled - trivial.

data

submission of form

imagefilename	class1 prob	class2 prob	…	classn prob
1.jpg	1/n	1/n	1/n	1/n

NEXT Understand the error/performance measure better

Preprocessing

we have to identify our features, at the same time as not introducing some censorship of the dater
tutorial uses
- thresholding on mean to reduce noise
- dilate to connect neighboring pixels
- segmention of connection regions - calculate their labels (e.g. region 1,2,3)
- apply the labels to original images

NEXT check prportion of obs with maxRegion [/]

State “NEXT” from “TODO” [2015-01-09 Fri 22:52]

when no maxregion, axis region ratio is 0, same with hu centres for now
[ ] get the proportion which are 0

feature brainstorming

Recall, manual feature selection is a difficult endeavour, rather search the feature space and have a generalization to create features, e.g. hu moments and thier interactions
aspect ratio
eccentricity
something about the protrusions
how about degree of transparency/variance of pixel values in feature
most of these i reckon are captured by the hu moments

hu central moments Research

State “DONE” from “” [2015-01-04 Sun 21:14]

and see Hu central moment expirimentation for tests to try
see the wiki here
image moments are certain averages of image pixel intensities, and functions of such first moments
this is defined for discrete greyscale data as

\begin{equation} M_ij = ∑_x ∑_y x^i y^j I(x,y) \end{equation}

they usually are formulated to have an attractive property, e.g. the hu central moments are rotation, translation

\begin{equation} I_1 = η₂₀ + η₀₂ \end{equation} \begin{equation} I_2 = (η₂₀ - η₀₂)^2 + 4η₁₁^2 \end{equation} \begin{equatio} I_3 = (η₃₀ - 3η₁₂)^2 + (3η₂₁ - η₀₃)^2 \end{equation} \begin{equatio} I_4 = (η₃₀ + η₁₂)^2 + (η₂₁ + η₀₃)^2 \end{equation} \begin{equatio} I_5 = (η₃₀ - 3η₁₂) (η₃₀ + η₁₂)[ (η₃₀ + η₁₂)^2 - 3 (η₂₁ + η₀₃)^2] + (3 η₂₁ - η₀₃) (η₂₁ + η₀₃)[ 3(η₃₀ + η₁₂)^2 - (η₂₁ + η₀₃)^2] \end{equation} \begin{equatio} I_6 = (η₂₀ - η₀₂)[(η₃₀ + η₁₂)^2 - (η₂₁ + η₀₃)^2] + 4η₁₁(η₃₀ + η₁₂)(η₂₁ + η₀₃) \end{equation} \begin{equatio} I_7 = (3 η₂₁ - η₀₃)(η₃₀ + η₁₂)[(η₃₀ + η₁₂)^2 - 3(η₂₁ + η₀₃)^2] - (η₃₀ - 3η₁₂)(η₂₁ + η₀₃)[3(η₃₀ + η₁₂)^2 - (η₂₁ + η₀₃)^2]. \end{equation}

how does this transfer to image/pattern recognition? Any image can be represented as a density function in x,y, which can further be represented by these moments. these moments should be invariant with respect to the images position in the visual field and pattern size.
so why does the python method for getting these moments take a region centroid? maybe it’s to mask out parts of the image, but in this case, where the rest of the image is blank, I don’t see it mattering
so, of these moments, 7 may be bad as it represents reflected images, and 1, the centroid, I don’t think is pertinent
what of the others? the original ieee transactions document
regardless, my question is, if i add crappy features, why is the classifier using them and scoring so poorly?!

NEXT image feature extraction

State “NEXT” from “TODO” [2015-01-06 Tue 13:40]

link to image feat extr section on scikitlearn feat extr page

nested model

There are many families of plankton, and there is some similarity with sub classes, can we capture this
Some of the subclasses are even the same as others, but from different views
- e.g. hydromedusae partial dark vs ShapeA
- but we still have to predict each class as given in the folders

Different number of samples per class

shall we predict smaller samples classes with less confidence?
or resample to same number?

feature vs model selection

I tried a few features, with unexpected (poor) and otherwise insignificant impact on the model performance

Lets consider something, about toying with features.

# https://docs.python.org/2/library/itertools.html#itertools.combinations
import itertools as iter

def count_iterable(i):
    return sum(1 for e in i)

num_vars = [x for x in range(10)]
num_combinations = {}

for vars in num_vars:
    var_list = [var for var in range(vars)]
    num_combinations[vars] = 0
    for subsets in [y for y in var_list]:
        combinations = iter.combinations(var_list, subsets)
        num_combinations[vars] += count_iterable(combinations)

print num_combinations

There we have it, even 4 or 5 features, playing with them manually becomes unruly.

With regression, we have some feature selection estimators, i.e. lasso.

What about classification… decision trees should only use the optimal split. But they’ll use some split even if there’s not a good one.

So, is it more effective to begin already with the model search? Or do we have to search the feature model space?

This already highlights the difficulty with lacking domain knowledge. For the driver telematics I have a much better chance and specifying a pretty full feature space, but for plankton ID and image processing, where I know so little… sheesh.

So, From here, given my efforts in toying with features have been unsuccessful, my focus goes towards trying different models and their parameter space. Then perhaps some progress can be made here.

hu central moments Research

copied from above
and see Hu central moment expirimentation for tests to try
see the wiki here
image moments are certain averages of image pixel intensities, and functions of such first moments
this is defined for discrete greyscale data as

\begin{equation} M_ij = ∑_x ∑_y x^i y^j I(x,y) \end{equation}

they usually are formulated to have an attractive property, e.g. the hu central moments are rotation, translation, reflection?

\begin{equation} I_1 = η₂₀ + η₀₂ \end{equation} \begin{equation} I_2 = (η₂₀ - η₀₂)^2 + 4η₁₁^2 \end{equation} \begin{equatio} I_3 = (η₃₀ - 3η₁₂)^2 + (3η₂₁ - η₀₃)^2 \end{equation} \begin{equatio} I_4 = (η₃₀ + η₁₂)^2 + (η₂₁ + η₀₃)^2 \end{equation} \begin{equatio} I_5 = (η₃₀ - 3η₁₂) (η₃₀ + η₁₂)[ (η₃₀ + η₁₂)^2 - 3 (η₂₁ + η₀₃)^2] + (3 η₂₁ - η₀₃) (η₂₁ + η₀₃)[ 3(η₃₀ + η₁₂)^2 - (η₂₁ + η₀₃)^2] \end{equation} \begin{equatio} I_6 = (η₂₀ - η₀₂)[(η₃₀ + η₁₂)^2 - (η₂₁ + η₀₃)^2] + 4η₁₁(η₃₀ + η₁₂)(η₂₁ + η₀₃) \end{equation} \begin{equatio} I_7 = (3 η₂₁ - η₀₃)(η₃₀ + η₁₂)[(η₃₀ + η₁₂)^2 - 3(η₂₁ + η₀₃)^2] - (η₃₀ - 3η₁₂)(η₂₁ + η₀₃)[3(η₃₀ + η₁₂)^2 - (η₂₁ + η₀₃)^2]. \end{equation}

how does this transfer to image/pattern recognition? Any image can be represented as a density function in x,y, which can further be represented by these moments. these moments should be invariant with respect to the images position in the visual field and pattern size.
so why does the python method for getting these moments take a region centroid? maybe it’s to mask out parts of the image, but in this case, where the rest of the image is blank, I don’t see it mattering
so, of these moments, 7 may be bad as it represents reflected images, and 1, the centroid, I don’t think is pertinent
what of the others? the original ieee transactions document
regardless, my question is, if i add crappy features, why is the classifier using them and scoring so poorly?!

python vs R

R	python	notes
dynamically typed	statically typed
caret	scikitlearn	sklearn better API
functional	object oriented	in the sense that methods
		belong to functions in R

Blog topics/Notes [39/59]

pylearn 2 convolutional neural networks setup

State “DONE” from “TODO” [2015-02-15 Sun 20:12]

setup pylearn 2 dependencies [2/3]

State “DONE” from “TODO” [2015-02-05 Thu 12:51]

pylearn 2 deps docs

theano, might want bleeding edge

State “DONE” from “NEXT” [2015-01-11 Sun 12:58]
State “NEXT” from “DONE” [2015-01-11 Sun 12:58]

install with pip
[X] run tests

import theano
theano.test()

I’m getting issues with the nosetest. S and K instead of ‘.’. What are these? Can’t find in docs.
S is skip.
K is knownfail.

PyYAML

State “DONE” from “NEXT” [2015-01-11 Sun 13:19]
State “NEXT” from “TODO” [2015-01-11 Sun 13:00]

pip install PyYAML

NEXT PIL for some image related functionality

State “NEXT” from “TODO” [2015-01-11 Sun 13:20]

install pylearn2

State “DONE” from “NEXT” [2015-01-11 Sun 13:40]
State “NEXT” from “” [2015-01-11 Sun 13:36]

clone from github
in venv

python setup.py develop

first tutorial pylearn2 in practice

State “DONE” from “NEXT” [2015-01-11 Sun 13:58]
State “NEXT” from “” [2015-01-11 Sun 13:03]

https://github.com/zygmuntz/pylearn2-practice
it ran. that was just to verify it works!

setup environment vars

State “DONE” from “NEXT” [2015-01-11 Sun 14:16]
State “NEXT” from “” [2015-01-11 Sun 13:00]

These are used for some tutorials and test.

export PYLEARN2_DATA_PATH=/data/lisa/data
export PATH=$PATH:???/pylearn2/scripts

mnist tutorial

State “DONE” from “NEXT” [2015-02-15 Sun 19:44]
State “NEXT” from “” [2015-01-11 Sun 13:00]

ok, we know that pylearn2 is working. it’d be good to go through the minst tutorial and get to grips with the more complex yaml/data.
[X] get mnist
[X] run the tutorial

setup pylearn2 for our data [3/3]

State “DONE” from “TODO” [2015-02-15 Sun 20:09]

Overview of setting up expiriment

State “DONE” from “NEXT” [2015-01-11 Sun 14:42]
State “NEXT” from “” [2015-01-11 Sun 14:28]

guide

Pylearn2 expiriments are python objects of the type pylearn2.train.Train.

a dataset, of type pylearn2.datasets.dataset.Dataset
a model, of type pylearn2.models.model.Model
a training algorithm, of type pylearn2.training_algorithms.training_algorithm.TrainingAlgorithm

They are setup via a yaml configuration file such as this:

“` !obj:pylearn2.train.Train { “dataset”: !obj:pylearn2.datasets.dense_design_matrix.DenseDesignMatrix &dataset { “X” : !obj:numpy.random.normal { ‘size’:[5,3] }, }, “model”: !obj:pylearn2.models.autoencoder.DenoisingAutoencoder { “nvis” : 3, “nhid” : 4, “irange” : 0.05, “corruptor”: !obj:pylearn2.corruption.BinomialCorruptor { “corruption_level”: 0.5, }, “act_enc”: “tanh”, “act_dec”: null, # Linear activation on the decoder side. }, “algorithm”: !obj:pylearn2.training_algorithms.sgd.SGD { “learning_rate” : 1e-3, “batch_size” : 5, “monitoring_dataset” : *dataset, “cost” : !obj:pylearn2.costs.autoencoder.MeanSquaredReconstructionError {}, “termination_criterion” : !obj:pylearn2.termination_criteria.EpochCounter { “max_epochs”: 1, }, }, “save_path”: “./garbage.pkl” } “`

It’s clear we need to setup each of these components.

vincent domoulin tutorial

State “DONE” from “NEXT” [2015-02-15 Sun 20:09]
State “NEXT” from “” [2015-01-11 Sun 16:34]

http://vdumoulin.github.io/articles/extending-pylearn2/

Dataset

State “DONE” from “NEXT” [2015-02-15 Sun 20:08]
State “NEXT” from “” [2015-01-11 Sun 14:41]

The data needs to be setup. Pylearn expects it as a pylearn data object. Useful source for thes is at

pylearn2/pylearn2/datasets/dense_design_matrix.py
pylearn2/pylearn2/datasets/dataset.py

The first thing to note is what is required of the dataset object.

getting the data in

State “DONE” from “” [2015-02-05 Thu 12:59]

Duplicate the minst data setup.

# clone my fork of pylearn git clone git://
git branch -c datascibowl
cdproject
cp pylearn2/datasets/mnist.py pylearn2/datasets/datascibowl.py
cp pylearn2/utils/mnist-ubyte.py pylearn2/datascibowl_reader/datascibowl.py

The main thing to do with data for pylearn2 is to subclass the module’s data objects. In this case, we are using denseDesignMatrix. Let’s look at the relevant info from the docstring:

class DenseDesignMatrix(Dataset):

    """

    A class for representing datasets that can be stored as a \
    dense design matrix (and optionally, associated targets).

    Parameters
    ----------
    topo_view : ndarray, optional
        Should be supplied if X is not.  An array whose first \
        dimension is of length number examples. The remaining \
        dimensions are examples with topological significance, \
        e.g. for images the remaining axes are rows, columns, \
        and channels.
    y : ndarray, optional

        Targets for each example (e.g., class ids, values to be predicted
        in a regression task).

        - 2D ndarray, data type optional:
            This is the most common format and can be used for a variety
            of problem types. Each row of the matrix becomes the target
            for a different example. Specific models / costs can interpret
            this target vector differently. For example, the `Linear`
            output layer for the `MLP` class expects the target for each
            example to be a vector of real-valued regression targets. (It
            can be a vector of size one if you only have one regression
            target). The `Softmax` output layer of the `MLP` class expects
            the target to be a vector of N elements, where N is the number
            of classes, and expects all but one of the elements to 0. One
            element should have value 1., and the index of this element
            identifies the target class.
    axes: tuple, optional
        The axes ordering of the provided topo_view. Must be some permutation
        of ('b', 0, 1, 'c') where 'b' indicates the axis indexing examples,
        0 and 1 indicate the row/cols dimensions and 'c' indicates the axis
        indexing color channels.
    y_labels : int, optional
        If y contains labels then y_labels must be passed to indicate the
        total number of possible labels e.g. 10 for the MNIST dataset
        where the targets are numbers. This will make the set use
        IndexSpace.
    """

So, to subclass denseDesignMatrix for the data in question, the following are needed:

topo_view
- an ndarray with the following format
(examples, parameters_indicating_topological_significance)
- where parameters_indicating_toplogical_significance describe the structure of the data. Rows, columns, and channels is given as an example. It is easy to imagine setting up data in 3d space and temperature x-coords,y coords,z coords, temperature, for example
- this is also what will actually hold the data in the examples space
- e.g. we have (n, x, y) where n are ixj matrices of pixel values and x,y are simply i,j respectively
- reshaping this to (n,x,y,1) signifies that n is comprised of 1 value, whereas n,x,y,3 would let us use, e.g., RGB
y
- targets, each row corresponds to the targets for each example
- for this example, this should be an ndarray with the number of elements that we have classes, and all but one element having value 0 (i.e. each example belongs to one class)
- for example, imagine we have 3 mutually exclusive classes to predict. y may look like

0	0	1
1	0	0
1	0	0
0	0	1
0	0	1

axes
- permutations of (b,0,1,c) where
- b indicates examples
- 0,1 indicates rows/columns(i dont understand this convention, perhaps 0th and 1st dimension)
- and c indicates the colors
y_labels
- an integer containing the total number of possible labels, if y contains labels

DataSciBowl options [2/2]

State “DONE” from “” [2015-02-06 Fri 15:45]

[X] train and test
- actually, test folder in data is for prediction only
- do not use ‘test’
[X] total number of images for start - stop

Test DataSciBowl class

State “DONE” from “” [2015-02-06 Fri 16:03]

I could use the datascibowl_reader easily enough, but couldn’t manage to test the dataset class itself.

I tried just copying the class def to ipython and instantiating, but it doesnt work. Which is so odd. Because creating a DenseDesignMatrix works just fine. Wtf. For now, I’m going to move on and see about setting up a model.

It was because of a bad path. Working fine now.

Model setup with pylearn2

State “DONE” from “” [2015-02-06 Fri 16:18]

In the pylearn directory scripts/ there are great tutorials.

With a dataset class, the next step is training a model. The convenient way to do this is with yaml docs. With some adaptation from existing examples, mine should look like:

cat sr_train.yaml

There’s quite a bit going on there; the dataset, model, and fitting algorithm are specified along with some information for test/validation data. Feel free to leave a comment if the syntax is confusing and I will try to clarify.

I want to highlight that several datasets can be used to monitor the fitting. Their names are arbitrary. I am using ‘train’ and ‘valid’, where valid is just a subset of the classified data reserved for testing.

monitoring_dataset : Dataset or dictionary, optional

If not specified, no monitoring is used. If specified to be a Dataset, monitor on that Dataset. If specified to be dictionary, the keys should be string names of datasets, and the values should be Datasets. All monitoring channels will be computed for all monitoring Datasets and will have the dataset name and an underscore prepended to them

Fit a model

State “DONE” from “NEXT” [2015-02-15 Sun 17:48]
State “NEXT” from “” [2015-02-06 Fri 16:37]

See softmax_regression.ipynb.

On first attempt I get an error.

ValueError: Input dimension mis-match

the google groups has seen such a thing before. A good next step is to play with the batch size, and to add theano flags exception_verbosity=high and optimizer=fast_compile or none to get a trace of which pylearn call is irritating theano.

I’m not sure where to add these.

The docs note that theano looks at $HOME/.theanorc

Write those options to that file and try again.

echo "[global]" > /home/joth/.theanorc
echo "exception_verbosity = high" >> /home/joth/.theanorc
echo "optimizer = fast_compile" >> /home/joth/.theanorc

Still no success. But now with more debug info.

I tried a few modifications to the sr_train.yaml to rule out plausible simple mistakes with no success.

Someone on the forums recommended me to try something. here’s the tread.

“One way to figure out what is happening would be to put a breakpoint in monitor.py a bit before line 255, and check what comes out of the data iterator (X), in particular the shape of all elements in the tuple, and if they seem to correspond to inputs or targets of examples in the dataset.”

Let’s look at the call method of monitor.py

def __call__(self):
    """
    Runs the model on the monitoring dataset in order to add one
    data point to each of the channels.
    """

    # If the channels have changed at all, we need to recompile the theano
    # functions used to compute them
    if self._dirty:
        self.redo_theano()

    datasets = self._datasets

    # Set all channels' val_shared to 0
    self.begin_record_entry()
    for d, i, b, n, a, sd, ne in safe_izip(datasets,
                                           self._iteration_mode,
                                           self._batch_size,
                                           self._num_batches,
                                           self.accum,
                                           self._rng_seed,
                                           self.num_examples):
        if isinstance(d, six.string_types):
            d = yaml_parse.load(d)
            raise NotImplementedError()

        # need to put d back into self._datasets
        myiterator = d.iterator(mode=i,
                                batch_size=b,
                                num_batches=n,
                                data_specs=self._flat_data_specs,
                                return_tuple=True,
                                rng=sd)

        # If self._flat_data_specs is empty, no channel needs data,
        # so we do not need to call the iterator in order to average
        # the monitored values across different batches, we only
        # have to call them once.
        if len(self._flat_data_specs[1]) == 0:
            X = ()
            self.run_prereqs(X, d)
            a(*X)

        else:
            actual_ne = 0
            for X in myiterator:
                # X is a flat (not nested) tuple
                self.run_prereqs(X, d)
                a(*X)
                actual_ne += self._flat_data_specs[0].np_batch_size(X)
            # end for X
            if actual_ne != ne:
                raise RuntimeError("At compile time, your iterator said "
                                   "it had %d examples total, but at "
                                   "runtime it gave us %d." %
                                   (ne, actual_ne))
    # end for d

    log.info("Monitoring step:")
    log.info("\tEpochs seen: %d" % self._epochs_seen)
    log.info("\tBatches seen: %d" % self._num_batches_seen)
    log.info("\tExamples seen: %d" % self._examples_seen)
    t = time.time() - self.t0
    for channel_name in sorted(self.channels.keys(),
                               key=number_aware_alphabetical_key):
        channel = self.channels[channel_name]
        channel.time_record.append(t)
        channel.batch_record.append(self._num_batches_seen)
        channel.example_record.append(self._examples_seen)
        channel.epoch_record.append(self._epochs_seen)
        val = channel.val_shared.get_value()
        channel.val_record.append(val)
        # TODO: use logging infrastructure so that user can configure
        # formatting
        if abs(val) < 1e4:
            val_str = str(val)
        else:
            val_str = '%.3e' % val

        log.info("\t%s: %s" % (channel_name, val_str))

For each dataset an iterator is set up. My work is breaking in the comparison of the first two items the iterator returns. It will also be useful then to look at the __next__ method for the default densedesignmatrix class and compare it to the mnist one.

In python,

import pdb
pdb.set_trace() # at break location, on/as line 256

Debug learning:

Some useful pdb commands

(p)rint print a variable

(s)tep to step forward one statement, whereas (n)ext will step forward a function call. If the line is not a function call, s and n are equivalent.

r continue until routine; lets you continue until the next return, great way to escape functions if you reckon the bug to not be present

(q)uit

commands to enter literal commands, enter end to stop, and n to step through the commands

And an excellent tutorial is here

That aside behind us, I got into the stack to check out what the iterator is returning. X[0] are examples and X[1] should be log probabilities for each class.

(Pdb) X[0].shape
(5000, 1024)
(Pdb) X[1].shape
(1, 121)

X[1] actually all have value 1.

Also, in X[0], the data may be degenerate. For most of the examples, the max value is 0. This is confirmed in the raw data.

image_mean = [np.reshape(i, (1, 1024)).mean() for i in test.X]
d = {x:image_mean.count(x) for x in image_mean}
# {0.0: 30215,
#  0.71068366555606532: 1,
#  0.72670476576861209: 1,
#  0.73890979243259802: 1, 
#  ...}

So something is wrong with the data.

The a(*X) call is also confusing to me. I asked again in the google group.

…

So some time later I fixed the data class. But still getting this error with the softmax. Urg.

So I took a look at MNIST example.

The data is here. Downloaded to the project dir and decompress.

cd /home/joth/projects/2014-12-20_datascibowl/data/mnist
for file in $(ls | grep .gz):
  do
    gzip -d $file
  done
ls

Eventually I noticed my data is transposed. I don’t think the transpose was needed in the class initialization.

And, with those fixed, the muthafucka works. Sadly, the performance is 10 times worse than on the minst data, which has best misclass rate of 0.07

epochs seen:  9
time trained:  458.503871202
ave_grad_mult : 3.07640978739
ave_grad_size : 0.0578164084249
ave_step_size : 0.168970324213
test_objective : 0.269887997532
test_y_col_norms_max : 5.97236412318
test_y_col_norms_mean : 5.28605773752
test_y_col_norms_min : 4.20493453263
test_y_max_max_class : 0.999999964196
test_y_mean_max_class : 0.914368745924
test_y_min_max_class : 0.232516262448
test_y_misclass : 0.0759
test_y_nll : 0.269887997532
test_y_row_norms_max : 1.61850785481
test_y_row_norms_mean : 0.483502165396
test_y_row_norms_min : 0.0
total_seconds_last_epoch : 52.954261
train_objective : 0.255329041014
train_y_col_norms_max : 5.97236412318
train_y_col_norms_mean : 5.28605773752
train_y_col_norms_min : 4.20493453263
train_y_max_max_class : 0.999999968304
train_y_mean_max_class : 0.911166781743
train_y_min_max_class : 0.233844764224
train_y_misclass : 0.07102
train_y_nll : 0.255329041014
train_y_row_norms_max : 1.61850785481
train_y_row_norms_mean : 0.483502165396
train_y_row_norms_min : 0.0
training_seconds_this_epoch : 48.921041

Shuffle the observations - keep it going forward

State “DONE” from “NEXT” [2015-02-16 Mon 10:36]

State “NEXT” from “” [2015-02-15 Sun 22:10]

deterministically
this step alone improves misclassification rate from ~1 to ~0.8

Centred observations

State “DONE” from “NEXT” [2015-02-16 Mon 11:41]
State “NEXT” from “” [2015-02-16 Mon 11:10]

this also improves misclass from about .75 to .65
probably worth keeping

Try SGD

State “DONE” from “NEXT” [2015-02-16 Mon 17:07]

State “NEXT” from “” [2015-02-16 Mon 16:30]

Module for performing batch gradient methods. Technically, SGD and BGD both work with any batch size, but SGD has no line search functionality and is thus best suited to small batches, while BGD supports line searches and thuse works best with large batches.

link proper link

How does this compare to BGD.

class pylearn2.training_algorithms.bgd.BGD( cost=None, batch_size=None, batches_per_iter=None, updates_per_batch=10, monitoring_batch_size=None, monitoring_batches=None, monitoring_dataset=None, termination_criterion=None, set_batch_size=False, reset_alpha=True, conjugate=False, min_init_alpha=0.001, reset_conjugate=True, line_search_mode=None, verbose_optimization=False, scale_step=1.0, theano_function_mode=None, init_alpha=None, seed=None) class pylearn2.training_algorithms.sgd.SGD( learning_rate, cost=None, batch_size=None, monitoring_batch_size=None, monitoring_batches=None, monitoring_dataset=None, monitor_iteration_mode=’sequential’, termination_criterion=None, update_callbacks=None, learning_rule=None, set_batch_size=False, train_iteration_mode=None, batches_per_iter=None, theano_function_mode=None, monitoring_costs=None, seed=[2012, 10, 5])Basically, in terms of parameters, we now have to choose a learning rate.

Bottou suggest expirimenting with different rates on small samples. This is a research topic on its own.

I tried learning rates of 0.01 and 0.001 with 128 batch size.

This is muuuuuch faster than BGD.

But got better results with BGD (i.e. misclass rate of 0.8x with SGD).

Get predictions on training set

State “DONE” from “NEXT” [2015-02-16 Mon 20:50]
State “NEXT” from “TODO” [2015-02-16 Mon 20:50]
State “NEXT” from “WAITING” [2015-02-15 Sun 22:17]
State “NEXT” from “” [2015-02-15 Sun 22:17]

so following the fastml, it’s pretty easy to setup predictions, see http://localhost:8888/notebooks/predict_tests.ipynb or below

import theano
from pylearn2.utils import serial

model_path = 'softmax_regression_best_shuf_ctr_03.pkl'
model = serial.load( model_path )

X = model.get_input_space().make_theano_batch()
Y = model.fprop( X )

# Y = T.argmax( Y, axis = 1 ) # if binary clsasification
f = theano.function( [X], Y )
y = f( x_test )

it’s simply a matter of getting x_test in ther
this can be done by modifying the datascibowl_reader slightly
for now I can test it on the training data in the ipy notebook
good news is, class probs are predicted for each observation

Setup validation set for prediction

State “DONE” from “NEXT” [2015-02-17 Tue 08:43]
State “NEXT” from “” [2015-02-16 Mon 20:46]

could setup a function/class to writie the output data
basically use what’s in http://localhost:8888/notebooks/test_data_setup_for_pred.ipynb
in place of the per directory image fiel reading in the DataSciBowl_Reader class and save to a .pkl
predict on the np array with the good BGD softmax

Saving large pickle/h5

doesn’t work
get numexpr Cython and tables, hdf5 from package manager
then see http://www.shocksolution.com/2010/01/storing-large-numpy-arrays-on-disk-python-pickle-vs-hdf5adsf/
which works nicely

import tables
h5file = tables.openFile('test.h5', mode='w', title="Test Array")
root = h5file.root
h5file.createArray(root, "test", a)
h5file.close()

Get predictions on validation set and submit

State “DONE” from “NEXT” [2015-02-17 Tue 22:47]
State “NEXT” from “” [2015-02-16 Mon 20:53]

https://groups.google.com/forum/#!searchin/pylearn-users/predict/pylearn-users/d2WUUdA2_go/G73pGmWqm38J
For mlp, this extends my simple example
- https://github.com/lisa-lab/pylearn2/blob/master/pylearn2/scripts/mlp/predict_csv.py
- just use classification prediction with float and row names (row names being the image file name)
make sure data preprocessed with same settings as the model was used to fit!! e.g. size, center

Implement the classifier cost function in monitoring [3/3]

State “DONE” from “NEXT” [2015-02-21 Sat 22:48]

Summary: Theano is cool. Read carefully. I will use the builtin NLL since mine may be comparitively slow. So forget the multiclass log loss branch.

From ian goodfellow on google groups, there are a few options:

-Write a TrainExtension that adds one by calling Monitor.add_channel -Subclass Model and override get_monitoring_channels -Subclass Cost and override get_monitoring_channels

If you’re interested in modifying the library itself, it could be a good idea to make ModelExtensions support adding monitoring channels to the model’s default get_monitoring_channels return value.

So, at /pylearn2/train_extensions there is an example of a channel added, roc_auc, which I will use as reference.

Test usage of extra monitoring channels via extensions

State “DONE” from “” [2015-02-19 Thu 07:57]

First I will simply try using that channel in a monitor. See the example in sr_train_07.yaml. It simply adds the following

   extensions: [
	!obj:pylearn2.train_extensions.roc_auc.RocAucChannel {
	    channel_name: 'roc_auc'
	}
   ],

Research monitoring channel implementation

State “DONE” from “NEXT” [2015-02-19 Thu 22:22]

State “NEXT” from “” [2015-02-19 Thu 20:10]

cool find, C-c C-x C-j goes to currently clocked in task

Here let’s take a look at the multiclass log loss function.

def multiclass_log_loss(y_true, y_pred, eps=1e-15):
    predictions = np.clip(y_pred, eps, 1 - eps)

    # normalize row sums to 1
    predictions /= predictions.sum(axis=1)[:, np.newaxis]

    actual = np.zeros(y_pred.shape)
    n_samples = actual.shape[0]
    actual[np.arange(n_samples), y_true.astype(int)] = 1
    vectsum = np.sum(actual * np.log(predictions))
    loss = -1.0 / n_samples * vectsum
    return loss

That’s straight forward enough. As for adding a monitoring extension to pylearn, am referring to the roc_auc. It is using y and y_hat. Exactly what are these and what format do they take so I can appropriately feed them to the MLL?

I put a pdb.set_trace in the roc_auc monitor to find out.

y = T.argmax(target, axis=1)
y_hat = model.fprop(state)[:, self.positive_class_index]

(Pdb) p type(y)
<class 'theano.tensor.var.TensorVariable'>
(Pdb) p type(y_hat)
<class 'theano.tensor.var.TensorVariable'>

So these are simply symbols.

Moving forward a few lines we come across

#...
    pos = T.eq(y, self.positive_class_index)
    neg = T.eq(y, self.negative_class_index)
    keep = T.add(pos, neg).nonzero()
    y = T.eq(y[keep], self.positive_class_index)
    y_hat = y_hat[keep]

roc_auc = RocAucScoreOp(self.channel_name_suffix)(y, y_hat)
roc_auc = T.cast(roc_auc, config.floatX)

Take note that for roc_auc, posotive and negative cases were identified.

RocAucScoreOp is a theano operation wrapper to sklearn’s roc_auc metric. In other words we need to get our metric to work using theano’s symbolic expressions.

That doesn’t do any calculcations yet, just further sets up the theano components. cast in the next line simply changes its type to a flot.

Finally, the tensor is passed to monitor.add_channel

model.monitor.add_channel(name=channel_name,
                          ipt=(state, target),
                          val=roc_auc,
                          data_specs=(m_space, m_source),
                          dataset=dataset)

with theano.pp we can nicely print the symbolic expressions. I didn’t import that so can’t do it right now.

Continuing with pdb

Our roc_auc is now a float operation, but it’s not being used yet.

Late hours approach and this is where I would retire. So I’ll make a leap to monitor.add_channel and see what’s going on. After all, that’s as far as I need to get.

add_channel(self, name, ipt, val, dataset=None, prereqs=None,
            data_specs=None):
"""
Asks the monitor to start tracking a new value.  Can be called
even after the monitor is already in use.

Parameters
----------
name : str
    The display name in the monitor.
ipt : tensor_like
    The symbolic tensor which should be clamped to the data.
    (or a list/tuple containing symbolic tensors, following the
    data_specs)
val : tensor_like
    The value (function of `ipt`) to be tracked.
dataset : pylearn2.datasets.Dataset
    Which dataset to compute this channel on
prereqs : list of callables that take a list of numpy tensors
    Each prereq must be called exactly once per each new batch
    of data drawn *from dataset* before the channel value is
    computed if two channels provide a prereq with exactly the
    same id, that prereq will only be called once
data_specs : (space, source) pair
    Identifies the order, format and semantics of ipt
"""

I only need to get the proper tensor expressions set up for ipt and val. And from the looks of things, I only need to adjust the theano wrapper of roc_auc to return an expression for MLL.

It would be good to see another example. But I can’t find. Hold on now - here’s the one for the misclassification rate.

if (targets is not None):
    if ((not self._has_binary_target) or
            self.binary_target_dim == 1):
        # if binary_target_dim>1, the misclass rate is ill-defined
        y_hat = T.argmax(state, axis=1)
        y = (targets.reshape(y_hat.shape)
             if self._has_binary_target
             else T.argmax(targets, axis=1))
        misclass = T.neq(y, y_hat).mean()
        misclass = T.cast(misclass, config.floatX)
        rval['misclass'] = misclass
    rval['nll'] = self.cost(Y_hat=state, Y=targets)

I think that exemplifies how straight forward this can be. The proportion of nonmatching elements of the observed y and predicted y_hat from the total number of observations is the misclassifcation rate.

For me, y_hat must be probabilities, not classes. Om that case it is easy, remove the argmax line, which decides assigns a class based on the maximum probability.

In other words, replace the

misclass = T.neq(y, y_hat).mean()

with a tensor expression for multiclass log loss. multiclass_log_loss

Implement theano multiclass log loss

State “DONE” from “NEXT” [2015-02-21 Sat 22:51]

State “NEXT” from “TODO” [2015-02-19 Thu 22:23]

So let’s break down the MLL. I went through it line by line in multiclass_log_loss_test.ipynb

def multiclass_log_loss(y_true, y_pred, eps=1e-15):
    # clip values to >0 and <1 - needed?
    predictions = np.clip(y_pred, eps, 1 - eps)

    # normalize row sums to 1
    predictions /= predictions.sum(axis=1)[:, np.newaxis]

    # array, same shape as predictions, with 'true' predictions
    actual = np.zeros(y_pred.shape)
    n_samples = actual.shape[0]
    actual[np.arange(n_samples), y_true.astype(int)] = 1

    # elementwise product of log(predictions) and true classes
    vectsum = np.sum(actual * np.log(predictions))

    loss = -1.0 / n_samples * vectsum
    return loss

See the theano tutorial for a start.

My last question - where is a function made from the tensor and evaluated?

Also, be careful of the shape of y, it may be a column/row.

Be aware of broadcasting, and the default tensor functionality, which should be enough for my purposes.

def multiclass_log_loss(y_true, y_pred):
    """Multi class version of Logarithmic Loss metric.
    https://www.kaggle.com/wiki/MultiClassLogLoss

    Parameters
    ----------
    y_true : array, shape = [n_samples]
            true class, intergers in [0, n_classes - 1)
    y_pred : array, shape = [n_samples, n_classes]

    Returns
    -------
    loss : float
    """
    y_pred = T.matrix(dtype='float32')
    y_true = T.matrix('y_true', dtype='int32')
    y_true_square = T.matrix('y_true_square', dtype='int32')
    eps = T.constant(1e-15, 'eps', dtype='float32')

    n_samples = y_pred.shape[0]
    y_pred_clip = T.clip(y_pred, eps, 1-eps)
    y_pred_row_sum = y_pred_clip.sum(axis=1)[:, np.newaxis]
    y_pred_norm = y_pred_clip / y_pred_row_sum
    y_true_square_ones = T.set_subtensor(y_true_square[T.arange(y_true.shape[0]), y_true.dimshuffle(1,0)], 1)
    y_pred_log = T.log(y_pred_norm)
    log_true_prod = y_true_square_ones * y_pred_log
    loss = -1.0 / n_samples * log_true_prod.sum()

    log_loss = function([y_pred, y_true, y_true_square], loss)

Where will we pass epsilon? We can put constants in our theano expressions

from theano import tensor as T, function
x = T.dmatrix('x')
s = 1 / (1 + T.exp(-x))
logistic = function([x], s)
logistic([[0, 1], [-1, -2]])

Well, I did implement MLL in theano expressions. But it turns out it was unnecessary. As the calculation need not be in theano expressions, as the values are passed through theano to the actual score function. Take a look

fileOne="./pylearn2_fork/pylearn2/train_extensions/multiclass_log_loss.py"
fileTwo="./pylearn2_fork/pylearn2/train_extensions/roc_auc.py"
diff -y $fileOne $fileTwo

It wasn’t too much to get this working. It was a lot to stomach the fact that my MLL matched the objective function - the negative log likelihood.

They are indeed one and the same. It even said so in the Kaggle

The metric is negative the log likelihood of the model

Oh well. I learned a great deal about Theano, at least.

Issues

One of the problems here is GPU processing is only working with 32 bit values.

Note

State “DONE” from “” [2015-02-21 Sat 22:53]

some NLL and MLL are same thing
but my NLL score on data is much better than my submission
indicates overfitting
image augmentation will help, along with the CNN

Make the workflow better

State “DONE” from “NEXT” [2015-02-22 Sun 00:35]
State “NEXT” from “” [2015-02-17 Tue 22:47]

Expiriments are defined with yaml. I run the yaml in an ipython notebook.

The best and last model are saved. Load up the best model and run the predictions.

Can I move these notebooks to a new directory? Where are paths ill coded? Data path is hard coded, doesn’t matter relative to ipynb paths.

Get running on GPU/desktop [3/3]

State “DONE” from “NEXT” [2015-02-23 Mon 21:20]
State “NEXT” from “TODO” [2015-02-22 Sun 00:36]

create venv
update requirements.txt, remove pylearn fork, ssh it over
install reqs
clone project repo
ssh data over
ssh theano config over, add gpu

Move existing work over

State “DONE” from “” [2015-02-22 Sun 17:34]

mkproject -p /usr/bin/python2.7 2014-12-20_datascibowl

pip freeze > requirements.txt
# delete fork of pylearn, we'll clone it later
scp requirements.txt jotham@192.168.0.13:/home/jotham/projects/2014-12-20_datascibowl/requirements.txt

pip install numpy # have to install this for scikit-image
pip install six # have to install this for scikit-image
pip install numexp # have to install this for scikit-image
pip install Cython # have to install this for scikit-image
sudo pacman -S hdf5 # for pyhdf
pip install -r requirements.txt
git clone git:@gitlab.com:Jotham/20141220_datascibowl.git

scp -r data jotham@192.168.0.13:/home/jotham/projects/2014-12-20_datascibowl/
scp -r competition_data jotham@192.168.0.13:/home/jotham/projects/2014-12-20_datascibowl/
scp ~/.theanorc jotham@192.168.0.13:/home/jotham/.theanorc

git clone git@github.com:jo-tham/pylearn2.git pylearn2_fork
cd pylearn2_fork 
python setyp.py develop

Setup theano with CUDA for gpu computing

State “DONE” from “NEXT” [2015-02-22 Sun 21:06]
State “NEXT” from “WAITING” [2015-02-22 Sun 21:06]
State “WAITING” from “” [2015-02-22 Sun 19:58]
- waiting for new kernel/nvidia drivers
- until then, cpu only

The guide is here. Rough steps are as follows.

CUDA, nvidia’s proprietary gear for general purpose computing on GPUs, ships with nvidia.

pacman -Qs nvidia

We also need the cuda toolkit

sudo pacman -S cuda

Cuda installs all components in the directory /opt/cuda. To compile CUDA code, add /opt/cuda/include to the include path in the compiler instructions. Knowing this we can update .theanorc.

cat ~/.theanorc

I tried this test of the theano authors, but it failed

from theano import function, config, shared, sandbox
import theano.tensor as T
import numpy
import time

vlen = 10 * 30 * 768  # 10 x #cores x # threads per core
iters = 1000

rng = numpy.random.RandomState(22)
x = shared(numpy.asarray(rng.rand(vlen), config.floatX))
f = function([], T.exp(x))
print f.maker.fgraph.toposort()
t0 = time.time()
for i in xrange(iters):
    r = f()
t1 = time.time()
print 'Looping %d times took' % iters, t1 - t0, 'seconds'
print 'Result is', r
if numpy.any([isinstance(x.op, T.Elemwise) for x in f.maker.fgraph.toposort()]):
    print 'Used the cpu'
else:
    print 'Used the gpu'

python theano_gpu_test.py
# WARNING (theano.sandbox.cuda): CUDA is installed, but
# device gpu is not available (error: Unable to get the
# number of gpus available: unknown error)

nvidia-smi

A similar problem is reported on the google groups.

I tried running nvidia-smi as sudo to initialize the device files. Things still didn’t work.

Well, it turns out the nvidia driver 346.35 may have some issues with current kernel

CURSES!

With a reboot, it pretends to be working, but appears to use the CPU anyway. WTH?@!

Using gpu device 0: GeForce GTX 550 Ti
[HostFromGpu(<CudaNdarrayType(float32, vector)>), Elemwise{exp,no_inplace}(HostFromGpu.0)]
Looping 1000 times took 5.19549107552 seconds
Result is [ 1.23178029  1.61879337  1.52278066 ...,  2.20771813  2.29967761
  1.62323284]
Used the cpu

So I try running theano’s test suite in ipynb_tests/theano_gpu.ipynb

Those tests should be run with cpu.

also,

pip install pycuda

Im at my ends. Why didn’t the wee test with GPU work?

Enough, I just try the softmax_03 with gpu. The cpu load is 100%…

This is not working…

Troubleshoot theano on gpu

State “DONE” from “NEXT” [2015-02-23 Mon 21:20]
State “NEXT” from “” [2015-02-22 Sun 21:06]

OK, I tried tons of stuff. In the end, with this suggestion and forcing the device, things worked. Suspicious as to whether pylearn really using GPU. How to know?

Friggin laptop was faster, 400 instead of 600 seconds per epoch (although this may be due to it reverting to cpu when gpu computation doesn’t work). Neeeeed GPU to work.

export PATH=/opt/cuda/bin:$PATH
export LD_LIBRARY_PATH=/opt/cuda/lib64:$LD_LIBRARY_PATH
cd /opt/cuda/bin
install-samples.sh /opt/cuda/samples2
cd /opt/cuda/samples2/...
make

Several warnings like this:

[@] mkdir -p ../../bin/x86_64/linux/release
[@] cp bilateralFilter ../../bin/x86_64/linux/release
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/3_Imaging/bilateralFilter'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/3_Imaging/imageDenoising'
>>> WARNING - libGL.so not found, refer to CUDA Samples release notes for how to find and install them. <<<
>>> WARNING - libGLU.so not found, refer to CUDA Samples release notes for how to find and install them. <<<
>>> WARNING - libX11.so not found, refer to CUDA Samples release notes for how to find and install them. <<<
>>> WARNING - libXi.so not found, refer to CUDA Samples release notes for how to find and install them. <<<
>>> WARNING - libXmu.so not found, refer to CUDA Samples release notes for how to find and install them. <<<
>>> WARNING - glu.h not found, refer to CUDA Samples release notes for how to find and install them. <<<
[@] /opt/cuda/bin/nvcc -ccbin g++ -I../../common/inc -m64 -gencode arch=compute_11,code=sm_11 -gencode arch=compute_20,code=sm_20 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_52,code=compute_52 -o bmploader.o -c bmploader.cpp
[@] /opt/cuda/bin/nvcc -ccbin g++ -I../../common/inc -m64 -gencode arch=compute_11,code=sm_11 -gencode arch=compute_20,code=sm_20 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_52,code=compute_52 -o imageDenoising.o -c imageDenoising.cu
[@] /opt/cuda/bin/nvcc -ccbin g++ -I../../common/inc -m64 -gencode arch=compute_11,code=sm_11 -gencode arch=compute_20,code=sm_20 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_52,code=compute_52 -o imageDenoisingGL.o -c imageDenoisingGL.cpp
[@] /opt/cuda/bin/nvcc -ccbin g++ -m64 -gencode arch=compute_11,code=sm_11 -gencode arch=compute_20,code=sm_20 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_52,code=compute_52 -o imageDenoising bmploader.o imageDenoising.o imageDenoisingGL.o -L../../common/lib/linux/x86_64 -lGL -lGLU -lX11 -lXi -lXmu -lglut -lGLEW
[@] mkdir -p ../../bin/x86_64/linux/release
[@] cp imageDenoising ../../bin/x86_64/linux/release
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/3_Imaging/imageDenoising'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/3_Imaging/histogram'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/3_Imaging/histogram'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/3_Imaging/cudaDecodeGL'
>>> WARNING - libGL.so not found, refer to CUDA Samples release notes for how to find and install them. <<<
>>> WARNING - libGLU.so not found, refer to CUDA Samples release notes for how to find and install them. <<<
>>> WARNING - libX11.so not found, refer to CUDA Samples release notes for how to find and install them. <<<
>>> WARNING - libXi.so not found, refer to CUDA Samples release notes for how to find and install them. <<<
>>> WARNING - libXmu.so not found, refer to CUDA Samples release notes for how to find and install them. <<<
>>> WARNING - glu.h not found, refer to CUDA Samples release notes for how to find and install them. <<<
[@] /opt/cuda/bin/nvcc -ccbin g++ -I../../common/inc -m64 -gencode arch=compute_11,code=compute_11 -o FrameQueue.o -c FrameQueue.cpp
[@] /opt/cuda/bin/nvcc -ccbin g++ -I../../common/inc -m64 -gencode arch=compute_11,code=compute_11 -o ImageGL.o -c ImageGL.cpp
[@] /opt/cuda/bin/nvcc -ccbin g++ -I../../common/inc -m64 -gencode arch=compute_11,code=compute_11 -o VideoDecoder.o -c VideoDecoder.cpp
[@] /opt/cuda/bin/nvcc -ccbin g++ -I../../common/inc -m64 -gencode arch=compute_11,code=compute_11 -o VideoParser.o -c VideoParser.cpp
[@] /opt/cuda/bin/nvcc -ccbin g++ -I../../common/inc -m64 -gencode arch=compute_11,code=compute_11 -o VideoSource.o -c VideoSource.cpp
[@] /opt/cuda/bin/nvcc -ccbin g++ -I../../common/inc -m64 -gencode arch=compute_11,code=compute_11 -o cudaModuleMgr.o -c cudaModuleMgr.cpp
[@] /opt/cuda/bin/nvcc -ccbin g++ -I../../common/inc -m64 -gencode arch=compute_11,code=compute_11 -o cudaProcessFrame.o -c cudaProcessFrame.cpp
[@] /opt/cuda/bin/nvcc -ccbin g++ -I../../common/inc -m64 -gencode arch=compute_11,code=compute_11 -o videoDecodeGL.o -c videoDecodeGL.cpp
[@] /opt/cuda/bin/nvcc -ccbin g++ -m64 -gencode arch=compute_11,code=compute_11 -o cudaDecodeGL FrameQueue.o ImageGL.o VideoDecoder.o VideoParser.o VideoSource.o cudaModuleMgr.o cudaProcessFrame.o videoDecodeGL.o -L../../common/lib/linux/x86_64 -lGL -lGLU -lX11 -lXi -lXmu -lglut -lGLEW -lcuda -lcudart -lnvcuvid
[@] mkdir -p ../../bin/x86_64/linux/release
[@] cp cudaDecodeGL ../../bin/x86_64/linux/release
[@] /opt/cuda/bin/nvcc -ccbin g++ -I../../common/inc -m64 -gencode arch=compute_11,code=compute_11 -o NV12ToARGB_drvapi64.ptx -ptx NV12ToARGB_drvapi.cu
[@] mkdir -p data
[@] cp -f NV12ToARGB_drvapi64.ptx ./data
[@] mkdir -p ../../bin/x86_64/linux/release
[@] cp -f NV12ToARGB_drvapi64.ptx ../../bin/x86_64/linux/release
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/3_Imaging/cudaDecodeGL'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/3_Imaging/dct8x8'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/3_Imaging/dct8x8'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/3_Imaging/SobelFilter'
>>> WARNING - libGL.so not found, refer to CUDA Samples release notes for how to find and install them. <<<
>>> WARNING - libGLU.so not found, refer to CUDA Samples release notes for how to find and install them. <<<
>>> WARNING - libX11.so not found, refer to CUDA Samples release notes for how to find and install them. <<<
>>> WARNING - libXi.so not found, refer to CUDA Samples release notes for how to find and install them. <<<
>>> WARNING - libXmu.so not found, refer to CUDA Samples release notes for how to find and install them. <<<
>>> WARNING - glu.h not found, refer to CUDA Samples release notes for how to find and install them. <<<
[@] /opt/cuda/bin/nvcc -ccbin g++ -I../../common/inc -m64 -gencode arch=compute_11,code=sm_11 -gencode arch=compute_20,code=sm_20 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_52,code=compute_52 -o SobelFilter.o -c SobelFilter.cpp
[@] /opt/cuda/bin/nvcc -ccbin g++ -I../../common/inc -m64 -gencode arch=compute_11,code=sm_11 -gencode arch=compute_20,code=sm_20 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_52,code=compute_52 -o SobelFilter_kernels.o -c SobelFilter_kernels.cu
[@] /opt/cuda/bin/nvcc -ccbin g++ -m64 -gencode arch=compute_11,code=sm_11 -gencode arch=compute_20,code=sm_20 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_52,code=compute_52 -o SobelFilter SobelFilter.o SobelFilter_kernels.o -L../../common/lib/linux/x86_64 -lGL -lGLU -lX11 -lXi -lXmu -lglut -lGLEW
[@] mkdir -p ../../bin/x86_64/linux/release
[@] cp SobelFilter ../../bin/x86_64/linux/release
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/3_Imaging/SobelFilter'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/3_Imaging/stereoDisparity'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/3_Imaging/stereoDisparity'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/3_Imaging/convolutionTexture'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/3_Imaging/convolutionTexture'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/3_Imaging/dwtHaar1D'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/3_Imaging/dwtHaar1D'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/3_Imaging/recursiveGaussian'
>>> WARNING - libGL.so not found, refer to CUDA Samples release notes for how to find and install them. <<<
>>> WARNING - libGLU.so not found, refer to CUDA Samples release notes for how to find and install them. <<<
>>> WARNING - libX11.so not found, refer to CUDA Samples release notes for how to find and install them. <<<
>>> WARNING - libXi.so not found, refer to CUDA Samples release notes for how to find and install them. <<<
>>> WARNING - libXmu.so not found, refer to CUDA Samples release notes for how to find and install them. <<<
>>> WARNING - glu.h not found, refer to CUDA Samples release notes for how to find and install them. <<<
[@] /opt/cuda/bin/nvcc -ccbin g++ -I../../common/inc -m64 -gencode arch=compute_11,code=sm_11 -gencode arch=compute_20,code=sm_20 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_52,code=compute_52 -o recursiveGaussian.o -c recursiveGaussian.cpp
[@] /opt/cuda/bin/nvcc -ccbin g++ -I../../common/inc -m64 -gencode arch=compute_11,code=sm_11 -gencode arch=compute_20,code=sm_20 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_52,code=compute_52 -o recursiveGaussian_cuda.o -c recursiveGaussian_cuda.cu
[@] /opt/cuda/bin/nvcc -ccbin g++ -m64 -gencode arch=compute_11,code=sm_11 -gencode arch=compute_20,code=sm_20 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_52,code=compute_52 -o recursiveGaussian recursiveGaussian.o recursiveGaussian_cuda.o -L../../common/lib/linux/x86_64 -lGL -lGLU -lX11 -lXi -lXmu -lglut -lGLEW
[@] mkdir -p ../../bin/x86_64/linux/release
[@] cp recursiveGaussian ../../bin/x86_64/linux/release
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/3_Imaging/recursiveGaussian'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/3_Imaging/convolutionFFT2D'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/3_Imaging/convolutionFFT2D'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/3_Imaging/HSOpticalFlow'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/3_Imaging/HSOpticalFlow'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/3_Imaging/boxFilter'
>>> WARNING - libGL.so not found, refer to CUDA Samples release notes for how to find and install them. <<<
>>> WARNING - libGLU.so not found, refer to CUDA Samples release notes for how to find and install them. <<<
>>> WARNING - libX11.so not found, refer to CUDA Samples release notes for how to find and install them. <<<
>>> WARNING - libXi.so not found, refer to CUDA Samples release notes for how to find and install them. <<<
>>> WARNING - libXmu.so not found, refer to CUDA Samples release notes for how to find and install them. <<<
>>> WARNING - glu.h not found, refer to CUDA Samples release notes for how to find and install them. <<<
[@] /opt/cuda/bin/nvcc -ccbin g++ -I../../common/inc -m64 -gencode arch=compute_11,code=sm_11 -gencode arch=compute_20,code=sm_20 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_52,code=compute_52 -o boxFilter.o -c boxFilter.cpp
[@] /opt/cuda/bin/nvcc -ccbin g++ -I../../common/inc -m64 -gencode arch=compute_11,code=sm_11 -gencode arch=compute_20,code=sm_20 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_52,code=compute_52 -o boxFilter_cpu.o -c boxFilter_cpu.cpp
[@] /opt/cuda/bin/nvcc -ccbin g++ -I../../common/inc -m64 -gencode arch=compute_11,code=sm_11 -gencode arch=compute_20,code=sm_20 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_52,code=compute_52 -o boxFilter_kernel.o -c boxFilter_kernel.cu
[@] /opt/cuda/bin/nvcc -ccbin g++ -m64 -gencode arch=compute_11,code=sm_11 -gencode arch=compute_20,code=sm_20 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_52,code=compute_52 -o boxFilter boxFilter.o boxFilter_cpu.o boxFilter_kernel.o -L../../common/lib/linux/x86_64 -lGL -lGLU -lX11 -lXi -lXmu -lglut -lGLEW
[@] mkdir -p ../../bin/x86_64/linux/release
[@] cp boxFilter ../../bin/x86_64/linux/release
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/3_Imaging/boxFilter'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/3_Imaging/convolutionSeparable'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/3_Imaging/convolutionSeparable'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/3_Imaging/dxtc'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/3_Imaging/dxtc'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/4_Finance/quasirandomGenerator'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/4_Finance/quasirandomGenerator'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/4_Finance/SobolQRNG'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/4_Finance/SobolQRNG'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/4_Finance/MonteCarloMultiGPU'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/4_Finance/MonteCarloMultiGPU'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/4_Finance/binomialOptions'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/4_Finance/binomialOptions'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/4_Finance/BlackScholes'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/4_Finance/BlackScholes'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/5_Simulations/particles'
>>> WARNING - libGL.so not found, refer to CUDA Samples release notes for how to find and install them. <<<
>>> WARNING - libGLU.so not found, refer to CUDA Samples release notes for how to find and install them. <<<
>>> WARNING - libX11.so not found, refer to CUDA Samples release notes for how to find and install them. <<<
>>> WARNING - libXi.so not found, refer to CUDA Samples release notes for how to find and install them. <<<
>>> WARNING - libXmu.so not found, refer to CUDA Samples release notes for how to find and install them. <<<
>>> WARNING - glu.h not found, refer to CUDA Samples release notes for how to find and install them. <<<
[@] /opt/cuda/bin/nvcc -ccbin g++ -I../../common/inc -m64 -gencode arch=compute_11,code=sm_11 -gencode arch=compute_13,code=sm_13 -gencode arch=compute_20,code=sm_20 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_52,code=compute_52 -o particleSystem.o -c particleSystem.cpp
[@] /opt/cuda/bin/nvcc -ccbin g++ -I../../common/inc -m64 -gencode arch=compute_11,code=sm_11 -gencode arch=compute_13,code=sm_13 -gencode arch=compute_20,code=sm_20 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_52,code=compute_52 -o particleSystem_cuda.o -c particleSystem_cuda.cu
[@] /opt/cuda/bin/nvcc -ccbin g++ -I../../common/inc -m64 -gencode arch=compute_11,code=sm_11 -gencode arch=compute_13,code=sm_13 -gencode arch=compute_20,code=sm_20 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_52,code=compute_52 -o particles.o -c particles.cpp
[@] /opt/cuda/bin/nvcc -ccbin g++ -I../../common/inc -m64 -gencode arch=compute_11,code=sm_11 -gencode arch=compute_13,code=sm_13 -gencode arch=compute_20,code=sm_20 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_52,code=compute_52 -o render_particles.o -c render_particles.cpp
[@] /opt/cuda/bin/nvcc -ccbin g++ -I../../common/inc -m64 -gencode arch=compute_11,code=sm_11 -gencode arch=compute_13,code=sm_13 -gencode arch=compute_20,code=sm_20 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_52,code=compute_52 -o shaders.o -c shaders.cpp
[@] /opt/cuda/bin/nvcc -ccbin g++ -m64 -gencode arch=compute_11,code=sm_11 -gencode arch=compute_13,code=sm_13 -gencode arch=compute_20,code=sm_20 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_52,code=compute_52 -o particles particleSystem.o particleSystem_cuda.o particles.o render_particles.o shaders.o -L../../common/lib/linux/x86_64 -lGL -lGLU -lX11 -lXi -lXmu -lglut -lGLEW
[@] mkdir -p ../../bin/x86_64/linux/release
[@] cp particles ../../bin/x86_64/linux/release
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/5_Simulations/particles'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/5_Simulations/smokeParticles'
>>> WARNING - libGL.so not found, refer to CUDA Samples release notes for how to find and install them. <<<
>>> WARNING - libGLU.so not found, refer to CUDA Samples release notes for how to find and install them. <<<
>>> WARNING - libX11.so not found, refer to CUDA Samples release notes for how to find and install them. <<<
>>> WARNING - libXi.so not found, refer to CUDA Samples release notes for how to find and install them. <<<
>>> WARNING - libXmu.so not found, refer to CUDA Samples release notes for how to find and install them. <<<
>>> WARNING - glu.h not found, refer to CUDA Samples release notes for how to find and install them. <<<
[@] /opt/cuda/bin/nvcc -ccbin g++ -I../../common/inc -m64 -gencode arch=compute_11,code=sm_11 -gencode arch=compute_13,code=sm_13 -gencode arch=compute_20,code=sm_20 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_52,code=compute_52 -o GLSLProgram.o -c GLSLProgram.cpp
[@] /opt/cuda/bin/nvcc -ccbin g++ -I../../common/inc -m64 -gencode arch=compute_11,code=sm_11 -gencode arch=compute_13,code=sm_13 -gencode arch=compute_20,code=sm_20 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_52,code=compute_52 -o ParticleSystem.o -c ParticleSystem.cpp
[@] /opt/cuda/bin/nvcc -ccbin g++ -I../../common/inc -m64 -gencode arch=compute_11,code=sm_11 -gencode arch=compute_13,code=sm_13 -gencode arch=compute_20,code=sm_20 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_52,code=compute_52 -o ParticleSystem_cuda.o -c ParticleSystem_cuda.cu
[@] /opt/cuda/bin/nvcc -ccbin g++ -I../../common/inc -m64 -gencode arch=compute_11,code=sm_11 -gencode arch=compute_13,code=sm_13 -gencode arch=compute_20,code=sm_20 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_52,code=compute_52 -o SmokeRenderer.o -c SmokeRenderer.cpp
[@] /opt/cuda/bin/nvcc -ccbin g++ -I../../common/inc -m64 -gencode arch=compute_11,code=sm_11 -gencode arch=compute_13,code=sm_13 -gencode arch=compute_20,code=sm_20 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_52,code=compute_52 -o SmokeShaders.o -c SmokeShaders.cpp
[@] /opt/cuda/bin/nvcc -ccbin g++ -I../../common/inc -m64 -gencode arch=compute_11,code=sm_11 -gencode arch=compute_13,code=sm_13 -gencode arch=compute_20,code=sm_20 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_52,code=compute_52 -o framebufferObject.o -c framebufferObject.cpp
[@] /opt/cuda/bin/nvcc -ccbin g++ -I../../common/inc -m64 -gencode arch=compute_11,code=sm_11 -gencode arch=compute_13,code=sm_13 -gencode arch=compute_20,code=sm_20 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_52,code=compute_52 -o particleDemo.o -c particleDemo.cpp
[@] /opt/cuda/bin/nvcc -ccbin g++ -I../../common/inc -m64 -gencode arch=compute_11,code=sm_11 -gencode arch=compute_13,code=sm_13 -gencode arch=compute_20,code=sm_20 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_52,code=compute_52 -o renderbuffer.o -c renderbuffer.cpp
[@] /opt/cuda/bin/nvcc -ccbin g++ -m64 -gencode arch=compute_11,code=sm_11 -gencode arch=compute_13,code=sm_13 -gencode arch=compute_20,code=sm_20 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_52,code=compute_52 -o smokeParticles GLSLProgram.o ParticleSystem.o ParticleSystem_cuda.o SmokeRenderer.o SmokeShaders.o framebufferObject.o particleDemo.o renderbuffer.o -L../../common/lib/linux/x86_64 -lGL -lGLU -lX11 -lXi -lXmu -lglut -lGLEW
[@] mkdir -p ../../bin/x86_64/linux/release
[@] cp smokeParticles ../../bin/x86_64/linux/release
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/5_Simulations/smokeParticles'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/5_Simulations/oceanFFT'
>>> WARNING - libGL.so not found, refer to CUDA Samples release notes for how to find and install them. <<<
>>> WARNING - libGLU.so not found, refer to CUDA Samples release notes for how to find and install them. <<<
>>> WARNING - libX11.so not found, refer to CUDA Samples release notes for how to find and install them. <<<
>>> WARNING - libXi.so not found, refer to CUDA Samples release notes for how to find and install them. <<<
>>> WARNING - libXmu.so not found, refer to CUDA Samples release notes for how to find and install them. <<<
>>> WARNING - glu.h not found, refer to CUDA Samples release notes for how to find and install them. <<<
[@] /opt/cuda/bin/nvcc -ccbin g++ -I../../common/inc -m64 -gencode arch=compute_11,code=sm_11 -gencode arch=compute_20,code=sm_20 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_52,code=compute_52 -o oceanFFT.o -c oceanFFT.cpp
[@] /opt/cuda/bin/nvcc -ccbin g++ -I../../common/inc -m64 -gencode arch=compute_11,code=sm_11 -gencode arch=compute_20,code=sm_20 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_52,code=compute_52 -o oceanFFT_kernel.o -c oceanFFT_kernel.cu
[@] /opt/cuda/bin/nvcc -ccbin g++ -m64 -gencode arch=compute_11,code=sm_11 -gencode arch=compute_20,code=sm_20 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_52,code=compute_52 -o oceanFFT oceanFFT.o oceanFFT_kernel.o -L../../common/lib/linux/x86_64 -lGL -lGLU -lX11 -lXi -lXmu -lglut -lGLEW -lcufft
[@] mkdir -p ../../bin/x86_64/linux/release
[@] cp oceanFFT ../../bin/x86_64/linux/release
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/5_Simulations/oceanFFT'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/5_Simulations/fluidsGL'
>>> WARNING - libGL.so not found, refer to CUDA Samples release notes for how to find and install them. <<<
>>> WARNING - libGLU.so not found, refer to CUDA Samples release notes for how to find and install them. <<<
>>> WARNING - libX11.so not found, refer to CUDA Samples release notes for how to find and install them. <<<
>>> WARNING - libXi.so not found, refer to CUDA Samples release notes for how to find and install them. <<<
>>> WARNING - libXmu.so not found, refer to CUDA Samples release notes for how to find and install them. <<<
>>> WARNING - glu.h not found, refer to CUDA Samples release notes for how to find and install them. <<<
[@] /opt/cuda/bin/nvcc -ccbin g++ -I../../common/inc -m64 -gencode arch=compute_11,code=sm_11 -gencode arch=compute_20,code=sm_20 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_52,code=compute_52 -o fluidsGL.o -c fluidsGL.cpp
[@] /opt/cuda/bin/nvcc -ccbin g++ -I../../common/inc -m64 -gencode arch=compute_11,code=sm_11 -gencode arch=compute_20,code=sm_20 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_52,code=compute_52 -o fluidsGL_kernels.o -c fluidsGL_kernels.cu
[@] /opt/cuda/bin/nvcc -ccbin g++ -m64 -gencode arch=compute_11,code=sm_11 -gencode arch=compute_20,code=sm_20 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_52,code=compute_52 -o fluidsGL fluidsGL.o fluidsGL_kernels.o -L../../common/lib/linux/x86_64 -lGL -lGLU -lX11 -lXi -lXmu -lglut -lGLEW -lcufft
[@] mkdir -p ../../bin/x86_64/linux/release
[@] cp fluidsGL ../../bin/x86_64/linux/release
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/5_Simulations/fluidsGL'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/5_Simulations/nbody'
>>> WARNING - libGL.so not found, refer to CUDA Samples release notes for how to find and install them. <<<
>>> WARNING - libGLU.so not found, refer to CUDA Samples release notes for how to find and install them. <<<
>>> WARNING - libX11.so not found, refer to CUDA Samples release notes for how to find and install them. <<<
>>> WARNING - libXi.so not found, refer to CUDA Samples release notes for how to find and install them. <<<
>>> WARNING - libXmu.so not found, refer to CUDA Samples release notes for how to find and install them. <<<
>>> WARNING - glu.h not found, refer to CUDA Samples release notes for how to find and install them. <<<
[@] /opt/cuda/bin/nvcc -ccbin g++ -I../../common/inc -m64 -ftz=true -gencode arch=compute_11,code=sm_11 -gencode arch=compute_13,code=sm_13 -gencode arch=compute_20,code=sm_20 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_52,code=compute_52 -o bodysystemcuda.o -c bodysystemcuda.cu
[@] /opt/cuda/bin/nvcc -ccbin g++ -I../../common/inc -m64 -ftz=true -gencode arch=compute_11,code=sm_11 -gencode arch=compute_13,code=sm_13 -gencode arch=compute_20,code=sm_20 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_52,code=compute_52 -o nbody.o -c nbody.cpp
[@] /opt/cuda/bin/nvcc -ccbin g++ -I../../common/inc -m64 -ftz=true -gencode arch=compute_11,code=sm_11 -gencode arch=compute_13,code=sm_13 -gencode arch=compute_20,code=sm_20 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_52,code=compute_52 -o render_particles.o -c render_particles.cpp
[@] /opt/cuda/bin/nvcc -ccbin g++ -m64 -gencode arch=compute_11,code=sm_11 -gencode arch=compute_13,code=sm_13 -gencode arch=compute_20,code=sm_20 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_52,code=compute_52 -o nbody bodysystemcuda.o nbody.o render_particles.o -L../../common/lib/linux/x86_64 -lGL -lGLU -lX11 -lXi -lXmu -lglut -lGLEW
[@] mkdir -p ../../bin/x86_64/linux/release
[@] cp nbody ../../bin/x86_64/linux/release
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/5_Simulations/nbody'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/6_Advanced/lineOfSight'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/6_Advanced/lineOfSight'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/6_Advanced/alignedTypes'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/6_Advanced/alignedTypes'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/6_Advanced/fastWalshTransform'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/6_Advanced/fastWalshTransform'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/6_Advanced/sortingNetworks'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/6_Advanced/sortingNetworks'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/6_Advanced/cdpQuadtree'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/6_Advanced/cdpQuadtree'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/6_Advanced/StreamPriorities'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/6_Advanced/StreamPriorities'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/6_Advanced/newdelete'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/6_Advanced/newdelete'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/6_Advanced/mergeSort'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/6_Advanced/mergeSort'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/6_Advanced/eigenvalues'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/6_Advanced/eigenvalues'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/6_Advanced/shfl_scan'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/6_Advanced/shfl_scan'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/6_Advanced/reduction'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/6_Advanced/reduction'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/6_Advanced/radixSortThrust'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/6_Advanced/radixSortThrust'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/6_Advanced/threadFenceReduction'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/6_Advanced/threadFenceReduction'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/6_Advanced/ptxjit'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/6_Advanced/ptxjit'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/6_Advanced/scan'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/6_Advanced/scan'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/6_Advanced/scalarProd'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/6_Advanced/scalarProd'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/6_Advanced/FDTD3d'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/6_Advanced/FDTD3d'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/6_Advanced/matrixMulDynlinkJIT'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/6_Advanced/matrixMulDynlinkJIT'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/6_Advanced/simpleHyperQ'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/6_Advanced/simpleHyperQ'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/6_Advanced/interval'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/6_Advanced/interval'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/6_Advanced/concurrentKernels'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/6_Advanced/concurrentKernels'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/6_Advanced/FunctionPointers'
>>> WARNING - libGL.so not found, refer to CUDA Samples release notes for how to find and install them. <<<
>>> WARNING - libGLU.so not found, refer to CUDA Samples release notes for how to find and install them. <<<
>>> WARNING - libX11.so not found, refer to CUDA Samples release notes for how to find and install them. <<<
>>> WARNING - libXi.so not found, refer to CUDA Samples release notes for how to find and install them. <<<
>>> WARNING - libXmu.so not found, refer to CUDA Samples release notes for how to find and install them. <<<
>>> WARNING - glu.h not found, refer to CUDA Samples release notes for how to find and install them. <<<
[@] /opt/cuda/bin/nvcc -ccbin g++ -I../../common/inc -m64 -gencode arch=compute_20,code=sm_20 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_52,code=compute_52 -o FunctionPointers.o -c FunctionPointers.cpp
[@] /opt/cuda/bin/nvcc -ccbin g++ -I../../common/inc -m64 -gencode arch=compute_20,code=sm_20 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_52,code=compute_52 -o FunctionPointers_kernels.o -c FunctionPointers_kernels.cu
[@] /opt/cuda/bin/nvcc -ccbin g++ -m64 -gencode arch=compute_20,code=sm_20 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_52,code=compute_52 -o FunctionPointers FunctionPointers.o FunctionPointers_kernels.o -L../../common/lib/linux/x86_64 -lGL -lGLU -lX11 -lXi -lXmu -lglut -lGLEW
[@] mkdir -p ../../bin/x86_64/linux/release
[@] cp FunctionPointers ../../bin/x86_64/linux/release
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/6_Advanced/FunctionPointers'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/6_Advanced/cdpBezierTessellation'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/6_Advanced/cdpBezierTessellation'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/6_Advanced/segmentationTreeThrust'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/6_Advanced/segmentationTreeThrust'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/6_Advanced/threadMigration'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/6_Advanced/threadMigration'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/6_Advanced/cdpAdvancedQuicksort'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/6_Advanced/cdpAdvancedQuicksort'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/6_Advanced/cdpLUDecomposition'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/6_Advanced/cdpLUDecomposition'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/6_Advanced/transpose'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/6_Advanced/transpose'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/7_CUDALibraries/imageSegmentationNPP'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/7_CUDALibraries/imageSegmentationNPP'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/7_CUDALibraries/simpleCUFFT_callback'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/7_CUDALibraries/simpleCUFFT_callback'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/7_CUDALibraries/conjugateGradient'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/7_CUDALibraries/conjugateGradient'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/7_CUDALibraries/freeImageInteropNPP'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/7_CUDALibraries/freeImageInteropNPP'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/7_CUDALibraries/MC_EstimatePiQ'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/7_CUDALibraries/MC_EstimatePiQ'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/7_CUDALibraries/simpleCUFFT_2d_MGPU'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/7_CUDALibraries/simpleCUFFT_2d_MGPU'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/7_CUDALibraries/MersenneTwisterGP11213'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/7_CUDALibraries/MersenneTwisterGP11213'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/7_CUDALibraries/simpleCUFFT_MGPU'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/7_CUDALibraries/simpleCUFFT_MGPU'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/7_CUDALibraries/grabcutNPP'
>>> WARNING - libGL.so not found, refer to CUDA Samples release notes for how to find and install them. <<<
>>> WARNING - libGLU.so not found, refer to CUDA Samples release notes for how to find and install them. <<<
>>> WARNING - libX11.so not found, refer to CUDA Samples release notes for how to find and install them. <<<
>>> WARNING - libXi.so not found, refer to CUDA Samples release notes for how to find and install them. <<<
>>> WARNING - libXmu.so not found, refer to CUDA Samples release notes for how to find and install them. <<<
>>> WARNING - glu.h not found, refer to CUDA Samples release notes for how to find and install them. <<<
[@] /opt/cuda/bin/nvcc -ccbin g++ -I../../common/inc -I../common/UtilNPP -I../common/FreeImage/include -m64 -gencode arch=compute_11,code=sm_11 -gencode arch=compute_20,code=sm_20 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_52,code=compute_52 -o GrabCut.o -c GrabCut.cpp
[@] /opt/cuda/bin/nvcc -ccbin g++ -I../../common/inc -I../common/UtilNPP -I../common/FreeImage/include -m64 -gencode arch=compute_11,code=sm_11 -gencode arch=compute_20,code=sm_20 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_52,code=compute_52 -o GrabcutGMM.o -c GrabcutGMM.cu
[@] /opt/cuda/bin/nvcc -ccbin g++ -I../../common/inc -I../common/UtilNPP -I../common/FreeImage/include -m64 -gencode arch=compute_11,code=sm_11 -gencode arch=compute_20,code=sm_20 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_52,code=compute_52 -o GrabcutHistogram.o -c GrabcutHistogram.cu
[@] /opt/cuda/bin/nvcc -ccbin g++ -I../../common/inc -I../common/UtilNPP -I../common/FreeImage/include -m64 -gencode arch=compute_11,code=sm_11 -gencode arch=compute_20,code=sm_20 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_52,code=compute_52 -o GrabcutMain.o -c GrabcutMain.cpp
[@] /opt/cuda/bin/nvcc -ccbin g++ -I../../common/inc -I../common/UtilNPP -I../common/FreeImage/include -m64 -gencode arch=compute_11,code=sm_11 -gencode arch=compute_20,code=sm_20 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_52,code=compute_52 -o GrabcutUtil.o -c GrabcutUtil.cu
[@] /opt/cuda/bin/nvcc -ccbin g++ -m64 -gencode arch=compute_11,code=sm_11 -gencode arch=compute_20,code=sm_20 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_52,code=compute_52 -o grabcutNPP GrabCut.o GrabcutGMM.o GrabcutHistogram.o GrabcutMain.o GrabcutUtil.o -L../../common/lib/linux/x86_64 -lGL -lGLU -lX11 -lXi -lXmu -lglut -lGLEW -L../common/FreeImage/lib -L../common/FreeImage/lib/linux -L../common/FreeImage/lib/linux/x86_64 -lnppi -lnppc -lfreeimage
[@] mkdir -p ../../bin/x86_64/linux/release
[@] cp grabcutNPP ../../bin/x86_64/linux/release
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/7_CUDALibraries/grabcutNPP'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/7_CUDALibraries/simpleCUFFT'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/7_CUDALibraries/simpleCUFFT'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/7_CUDALibraries/cuHook'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/7_CUDALibraries/cuHook'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/7_CUDALibraries/conjugateGradientUM'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/7_CUDALibraries/conjugateGradientUM'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/7_CUDALibraries/boxFilterNPP'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/7_CUDALibraries/boxFilterNPP'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/7_CUDALibraries/MC_EstimatePiInlineQ'
"/opt/cuda"/bin/nvcc -ccbin g++ -I../../common/inc  -m64     -gencode arch=compute_13,code=sm_13 -gencode arch=compute_20,code=sm_20 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_52,code=compute_52 -o main.o -c src/main.cpp
nvcc warning : The 'compute_11', 'compute_12', 'compute_13', 'sm_11', 'sm_12', and 'sm_13' architectures are deprecated, and may be removed in a future release.
"/opt/cuda"/bin/nvcc -ccbin g++ -I../../common/inc  -m64     -gencode arch=compute_13,code=sm_13 -gencode arch=compute_20,code=sm_20 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_52,code=compute_52 -o piestimator.o -c src/piestimator.cu
nvcc warning : The 'compute_11', 'compute_12', 'compute_13', 'sm_11', 'sm_12', and 'sm_13' architectures are deprecated, and may be removed in a future release.
"/opt/cuda"/bin/nvcc -ccbin g++ -I../../common/inc  -m64     -gencode arch=compute_13,code=sm_13 -gencode arch=compute_20,code=sm_20 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_52,code=compute_52 -o test.o -c src/test.cpp
nvcc warning : The 'compute_11', 'compute_12', 'compute_13', 'sm_11', 'sm_12', and 'sm_13' architectures are deprecated, and may be removed in a future release.
"/opt/cuda"/bin/nvcc -ccbin g++   -m64       -gencode arch=compute_13,code=sm_13 -gencode arch=compute_20,code=sm_20 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_52,code=compute_52 -o MC_EstimatePiInlineQ main.o piestimator.o test.o  -lcurand
nvcc warning : The 'compute_11', 'compute_12', 'compute_13', 'sm_11', 'sm_12', and 'sm_13' architectures are deprecated, and may be removed in a future release.
mkdir -p ../../bin/x86_64/linux/release
cp MC_EstimatePiInlineQ ../../bin/x86_64/linux/release
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/7_CUDALibraries/MC_EstimatePiInlineQ'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/7_CUDALibraries/MC_SingleAsianOptionP'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/7_CUDALibraries/MC_SingleAsianOptionP'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/7_CUDALibraries/simpleDevLibCUBLAS'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/7_CUDALibraries/simpleDevLibCUBLAS'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/7_CUDALibraries/jpegNPP'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/7_CUDALibraries/jpegNPP'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/7_CUDALibraries/MC_EstimatePiInlineP'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/7_CUDALibraries/MC_EstimatePiInlineP'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/7_CUDALibraries/histEqualizationNPP'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/7_CUDALibraries/histEqualizationNPP'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/7_CUDALibraries/randomFog'
>>> WARNING - libGL.so not found, refer to CUDA Samples release notes for how to find and install them. <<<
>>> WARNING - libGLU.so not found, refer to CUDA Samples release notes for how to find and install them. <<<
>>> WARNING - libX11.so not found, refer to CUDA Samples release notes for how to find and install them. <<<
>>> WARNING - libXi.so not found, refer to CUDA Samples release notes for how to find and install them. <<<
>>> WARNING - libXmu.so not found, refer to CUDA Samples release notes for how to find and install them. <<<
>>> WARNING - glu.h not found, refer to CUDA Samples release notes for how to find and install them. <<<
[@] /opt/cuda/bin/nvcc -ccbin g++ -I../../common/inc -m64 -Xcompiler -fpermissive -gencode arch=compute_11,code=compute_11 -o randomFog.o -c randomFog.cpp
[@] /opt/cuda/bin/nvcc -ccbin g++ -I../../common/inc -m64 -Xcompiler -fpermissive -gencode arch=compute_11,code=compute_11 -o rng.o -c rng.cpp
[@] /opt/cuda/bin/nvcc -ccbin g++ -m64 -gencode arch=compute_11,code=compute_11 -o randomFog randomFog.o rng.o -L../../common/lib/linux/x86_64 -lGL -lGLU -lX11 -lXi -lXmu -lglut -lGLEW -lcurand
[@] mkdir -p ../../bin/x86_64/linux/release
[@] cp randomFog ../../bin/x86_64/linux/release
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/7_CUDALibraries/randomFog'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/7_CUDALibraries/MC_EstimatePiP'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/7_CUDALibraries/MC_EstimatePiP'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/7_CUDALibraries/batchCUBLAS'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/7_CUDALibraries/batchCUBLAS'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/7_CUDALibraries/conjugateGradientPrecond'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/7_CUDALibraries/conjugateGradientPrecond'
make[1]: Entering directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/7_CUDALibraries/simpleCUBLAS'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/opt/cuda/samples2/NVIDIA_CUDA-6.5_Samples/7_CUDALibraries/simpleCUBLAS'
Finished building CUDA samples

I force the GPU, and it accepts

THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32,force_device=True python theano_gpu_test.py
#Using gpu device 0: GeForce GTX 550 Ti
#[GpuElemwise{exp,no_inplace}(<CudaNdarrayType(float32, vector)>), HostFromGpu(GpuElemwise{exp,no_inplace}.0)]
#Looping 1000 times took 0.438356876373 seconds
#Result is [ 1.23178029  1.61879349  1.52278066 ...,  2.20771813  2.29967761
#  1.62323296]
#Used the gpu

So I add that to my .theanorc and try fitting model

Doesn’t seem to work. Suuuper slow still. Could just be settings. See Tune GPU.

WTH …

So, theano is running on gpu using force flag
Tiny bit of memory load, and when I try two processes to use gpu I get this error

INFO (theano.gof.compilelock): Waiting for existing lock by process '9511' (I am process '7923')
INFO:theano.gof.compilelock:Waiting for existing lock by process '9511' (I am process '7923')
INFO (theano.gof.compilelock): To manually release the lock, delete /home/jotham/.theano/compiledir_Linux-3.18.6-1-ARCH-x86_64-with-glibc2.2.5--2.7.9-64/lock_dir
INFO:theano.gof.compilelock:To manually release the lock, delete /home/jotham/.theano/compiledir_Linux-3.18.6-1-ARCH-x86_64-with-glibc2.2.5--2.7.9-64/lock_dir

Theano tests ran with following output

THEANO_FLAGS=mode=FAST_RUN,device=cpu,floatX=float32 python -c 'import theano; theano.test()' >> theano_test.txt

THEANO_FLAGS=mode=FAST_RUN,device=cpu,floatX=float32 python -c 'import theano; theano.test()' >> theano_test.txt
........................................E.............../home/jotham/venvs/2014-12-20_datascibowl/lib/python2.7/site-packages/theano/compile/tests/test_inplace_opt_for_value.py:170: UserWarning: theano modules are deprecated and will be removed in release 0.7
  super(ExampleRNN, self).__init__()
/home/jotham/venvs/2014-12-20_datascibowl/lib/python2.7/site-packages/theano/tensor/subtensor.py:110: FutureWarning: comparison to `None` will result in an elementwise object comparison in the future.
  start in [None, 0] or
/home/jotham/venvs/2014-12-20_datascibowl/lib/python2.7/site-packages/theano/tensor/subtensor.py:114: FutureWarning: comparison to `None` will result in an elementwise object comparison in the future.
  stop in [None, length, maxsize] or
...................................................................................................../home/jotham/venvs/2014-12-20_datascibowl/lib/python2.7/site-packages/theano/scan_module/scan_perform_ext.py:85: RuntimeWarning: numpy.ndarray size changed, may indicate binary incompatibility
  from scan_perform.scan_perform import *
................................................................................................................................................/home/jotham/venvs/2014-12-20_datascibowl/lib/python2.7/site-packages/theano/tensor/opt.py:2165: FutureWarning: comparison to `None` will result in an elementwise object comparison in the future.
  if (replace_x == replace_y and
......................................EE..................................................................E............................./home/jotham/venvs/2014-12-20_datascibowl/lib/python2.7/site-packages/theano/tensor/subtensor.py:190: FutureWarning: comparison to `None` will result in an elementwise object comparison in the future.
  if stop in [None, maxsize]:
...........E..............E................................................/home/jotham/venvs/2014-12-20_datascibowl/lib/python2.7/site-packages/theano/sparse/basic.py:87: FutureWarning: comparison to `None` will result in an elementwise object comparison in the future.
  return numpy.all(a == b)
................./home/jotham/venvs/2014-12-20_datascibowl/lib/python2.7/site-packages/scipy/sparse/data.py:63: ComplexWarning: Casting complex values to real discards the imaginary part
  return self._with_data(self.data.astype(t))
/home/jotham/venvs/2014-12-20_datascibowl/lib/python2.7/site-packages/theano/sparse/tests/test_basic.py:2128: ComplexWarning: Casting complex values to real discards the imaginary part
  expected = data.toarray().astype(o_dtype)
...../home/jotham/venvs/2014-12-20_datascibowl/lib/python2.7/site-packages/scipy/sparse/compressed.py:698: SparseEfficiencyWarning: Changing the sparsity structure of a csc_matrix is expensive. lil_matrix is more efficient.
  SparseEfficiencyWarning)
/home/jotham/venvs/2014-12-20_datascibowl/lib/python2.7/site-packages/scipy/sparse/compressed.py:698: SparseEfficiencyWarning: Changing the sparsity structure of a csr_matrix is expensive. lil_matrix is more efficient.
  SparseEfficiencyWarning)
...................................................................................................../home/jotham/venvs/2014-12-20_datascibowl/lib/python2.7/site-packages/theano/tensor/subtensor.py:1469: UserWarning: DEPRECATION WARNING: AdvancedSubtensor1, you are using an old interface to the sparse grad. You should use theano.sparse_grad(a_tensor[an_int_vector]). 
  "DEPRECATION WARNING: AdvancedSubtensor1, you are using"
.................................................................................................................................................................................................................................INFO (theano.gof.compilelock): Waiting for existing lock by process '21429' (I am process '9511')
INFO (theano.gof.compilelock): To manually release the lock, delete /home/jotham/.theano/compiledir_Linux-3.18.6-1-ARCH-x86_64-with-glibc2.2.5--2.7.9-64/lock_dir
............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................E........................................................................................................................................................................................................................................................................................................................E..EEEE.E............/home/jotham/venvs/2014-12-20_datascibowl/lib/python2.7/site-packages/theano/compile/function_module.py:579: ComplexWarning: Casting complex values to real discards the imaginary part
  outputs = self.fn()
....t......................................................E........................../home/jotham/venvs/2014-12-20_datascibowl/lib/python2.7/site-packages/theano/tensor/tests/test_naacl09.py:69: UserWarning: RandomStreams is deprecated and will be removed in release 0.7. Use shared_randomstreams.RandomStreams or MRG_RandomStreams instead.
  self.random = T.randomstreams.RandomStreams()
E.....................................................................................................E......E.............................................................................................................................................../home/jotham/venvs/2014-12-20_datascibowl/lib/python2.7/site-packages/theano/tensor/var.py:359: FutureWarning: comparison to `None` will result in an elementwise object comparison in the future.
  if arg != numpy.newaxis:
......................................................................................................EE.............................................
======================================================================
ERROR: test_none (theano.compile.tests.test_function_module.T_function)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jotham/venvs/2014-12-20_datascibowl/lib/python2.7/site-packages/theano/compile/tests/test_function_module.py", line 42, in test_none
    raise KnownFailureTest('See #254: Using None as function output leads to [] return value')
KnownFailureTest: See #254: Using None as function output leads to [] return value

======================================================================
ERROR: test002_generator_one_scalar_output (theano.sandbox.scan_module.tests.test_scan.TestScan)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jotham/venvs/2014-12-20_datascibowl/lib/python2.7/site-packages/theano/sandbox/scan_module/tests/test_scan.py", line 474, in test002_generator_one_scalar_output
    raise KnownFailureTest('Work-in-progress sandbox ScanOp is not fully '
KnownFailureTest: Work-in-progress sandbox ScanOp is not fully functional yet

======================================================================
ERROR: test003_one_sequence_one_output_and_weights (theano.sandbox.scan_module.tests.test_scan.TestScan)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jotham/venvs/2014-12-20_datascibowl/lib/python2.7/site-packages/theano/sandbox/scan_module/tests/test_scan.py", line 512, in test003_one_sequence_one_output_and_weights
    raise KnownFailureTest('Work-in-progress sandbox ScanOp is not fully '
KnownFailureTest: Work-in-progress sandbox ScanOp is not fully functional yet

======================================================================
ERROR: test_alloc_inputs2 (theano.scan_module.tests.test_scan.T_Scan)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jotham/venvs/2014-12-20_datascibowl/lib/python2.7/site-packages/theano/scan_module/tests/test_scan.py", line 2844, in test_alloc_inputs2
    "This tests depends on an optimization for scan "
KnownFailureTest: This tests depends on an optimization for scan that has not been implemented yet.

======================================================================
ERROR: test_infershape_seq_shorter_nsteps (theano.scan_module.tests.test_scan.T_Scan)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jotham/venvs/2014-12-20_datascibowl/lib/python2.7/site-packages/theano/scan_module/tests/test_scan.py", line 3040, in test_infershape_seq_shorter_nsteps
    raise KnownFailureTest('This is a generic problem with infershape'
KnownFailureTest: This is a generic problem with infershape that has to be discussed and figured out

======================================================================
ERROR: test_outputs_info_not_typed (theano.scan_module.tests.test_scan.T_Scan)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jotham/venvs/2014-12-20_datascibowl/lib/python2.7/site-packages/numpy/testing/decorators.py", line 213, in knownfailer
    raise KnownFailureTest(msg)
KnownFailureTest: This test fails because not typed outputs_info are always gived the smallest dtype. There is no upcast of outputs_info in scan for now.

======================================================================
ERROR: test_arithmetic_cast (theano.tensor.tests.test_basic.test_arithmetic_cast)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jotham/venvs/2014-12-20_datascibowl/lib/python2.7/site-packages/theano/tensor/tests/test_basic.py", line 5583, in test_arithmetic_cast
    raise KnownFailureTest('Known issue with '
KnownFailureTest: Known issue with numpy >= 1.6.x see #761

======================================================================
ERROR: test_abs_grad (theano.tensor.tests.test_complex.TestRealImag)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jotham/venvs/2014-12-20_datascibowl/lib/python2.7/site-packages/numpy/testing/decorators.py", line 213, in knownfailer
    raise KnownFailureTest(msg)
KnownFailureTest: Complex grads not enabled, see #178

======================================================================
ERROR: test_complex_grads (theano.tensor.tests.test_complex.TestRealImag)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jotham/venvs/2014-12-20_datascibowl/lib/python2.7/site-packages/numpy/testing/decorators.py", line 213, in knownfailer
    raise KnownFailureTest(msg)
KnownFailureTest: Complex grads not enabled, see #178

======================================================================
ERROR: test_mul_mixed (theano.tensor.tests.test_complex.TestRealImag)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jotham/venvs/2014-12-20_datascibowl/lib/python2.7/site-packages/numpy/testing/decorators.py", line 213, in knownfailer
    raise KnownFailureTest(msg)
KnownFailureTest: Complex grads not enabled, see #178

======================================================================
ERROR: test_mul_mixed0 (theano.tensor.tests.test_complex.TestRealImag)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jotham/venvs/2014-12-20_datascibowl/lib/python2.7/site-packages/numpy/testing/decorators.py", line 213, in knownfailer
    raise KnownFailureTest(msg)
KnownFailureTest: Complex grads not enabled, see #178

======================================================================
ERROR: test_mul_mixed1 (theano.tensor.tests.test_complex.TestRealImag)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jotham/venvs/2014-12-20_datascibowl/lib/python2.7/site-packages/numpy/testing/decorators.py", line 213, in knownfailer
    raise KnownFailureTest(msg)
KnownFailureTest: Complex grads not enabled, see #178

======================================================================
ERROR: test_polar_grads (theano.tensor.tests.test_complex.TestRealImag)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jotham/venvs/2014-12-20_datascibowl/lib/python2.7/site-packages/numpy/testing/decorators.py", line 213, in knownfailer
    raise KnownFailureTest(msg)
KnownFailureTest: Complex grads not enabled, see #178

======================================================================
ERROR: test_gradient (theano.tensor.tests.test_fourier.TestFourier)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jotham/venvs/2014-12-20_datascibowl/lib/python2.7/site-packages/numpy/testing/decorators.py", line 213, in knownfailer
    raise KnownFailureTest(msg)
KnownFailureTest: Complex grads not enabled, see #178

======================================================================
ERROR: theano.tensor.tests.test_naacl09.test_naacl_model
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jotham/venvs/2014-12-20_datascibowl/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/jotham/venvs/2014-12-20_datascibowl/lib/python2.7/site-packages/theano/tensor/tests/test_naacl09.py", line 613, in test_naacl_model
    raise KnownFailureTest("Deprecated compile.module fails to "
KnownFailureTest: Deprecated compile.module fails to give a sensible warning when updates to a variable have the wrong type

======================================================================
ERROR: theano.tensor.tests.test_opt.test_log_add
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jotham/venvs/2014-12-20_datascibowl/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/jotham/venvs/2014-12-20_datascibowl/lib/python2.7/site-packages/theano/tensor/tests/test_opt.py", line 1508, in test_log_add
    raise KnownFailureTest(('log(add(exp)) is not stabilized when adding '
KnownFailureTest: log(add(exp)) is not stabilized when adding more than 2 elements, see #623

======================================================================
ERROR: Currently Theano enable the constant_folding optimization before stabilization optimization.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jotham/venvs/2014-12-20_datascibowl/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/jotham/venvs/2014-12-20_datascibowl/lib/python2.7/site-packages/theano/tensor/tests/test_opt.py", line 3068, in test_constant_get_stabilized
    "Theano optimizes constant before stabilization. "
KnownFailureTest: Theano optimizes constant before stabilization. This breaks stabilization optimization in some cases. See #504.

======================================================================
ERROR: test_dot (theano.tests.test_rop.test_RopLop)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jotham/venvs/2014-12-20_datascibowl/lib/python2.7/site-packages/theano/tests/test_rop.py", line 277, in test_dot
    self.check_rop_lop(tensor.dot(self.x, W), self.in_shape)
  File "/home/jotham/venvs/2014-12-20_datascibowl/lib/python2.7/site-packages/theano/tests/test_rop.py", line 191, in check_rop_lop
    raise KnownFailureTest("Rop doesn't handle non-differentiable "
KnownFailureTest: Rop doesn't handle non-differentiable inputs correctly. Bug exposed by fixing Add.grad method.

======================================================================
ERROR: test_elemwise0 (theano.tests.test_rop.test_RopLop)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jotham/venvs/2014-12-20_datascibowl/lib/python2.7/site-packages/theano/tests/test_rop.py", line 280, in test_elemwise0
    self.check_rop_lop((self.x + 1) ** 2, self.in_shape)
  File "/home/jotham/venvs/2014-12-20_datascibowl/lib/python2.7/site-packages/theano/tests/test_rop.py", line 191, in check_rop_lop
    raise KnownFailureTest("Rop doesn't handle non-differentiable "
KnownFailureTest: Rop doesn't handle non-differentiable inputs correctly. Bug exposed by fixing Add.grad method.

----------------------------------------------------------------------
Ran 2441 tests in 3104.784s

Run CNN mnist tutorial

State “DONE” from “WAITING” [2015-02-24 Tue 06:59]
State “WAITING” from “” [2015-02-23 Mon 22:05]
- running

power went out :(
epochs were somewhat fast, however

Test whole ipynb flow on desktop

State “DONE” from “NEXT” [2015-02-23 Mon 22:07]
State “NEXT” from “” [2015-02-22 Sun 21:17]

did this on softreg shuf_ctr_sgd_10
see ipynb_work

ls ipynb_work

change MonitorBasedSaveBest to use nll

State “DONE” from “NEXT” [2015-02-23 Mon 22:08]
State “NEXT” from “” [2015-02-22 Sun 21:18]

See Test whole ipynb flow on desktop

Don’t hardcode path into datasetclass

State “DONE” from “NEXT” [2015-02-24 Tue 07:16]
State “NEXT” from “” [2015-02-22 Sun 21:21]

use pylearn_data_path

Ask how to know if pylearn really using GPU?

State “DONE” from “WAITING” [2015-02-25 Wed 19:38]
State “WAITING” from “DONE” [2015-02-23 Mon 22:03]
- see if anything
State “DONE” from “” [2015-02-23 Mon 22:03]

could try a job with and without gpu flags
https://groups.google.com/forum/#!topic/pylearn-users/aZ43lHLpDT0

Use CNN [4/4]

State “DONE” from “NEXT” [2015-03-02 Mon 19:44]
State “NEXT” from “TODO” [2015-02-25 Wed 08:16]
State “NEXT” from “” [2015-02-16 Mon 11:46]

layers
some more tips and tricks

first check out multilayer perceptrons

State “DONE” from “TODO” [2015-02-25 Wed 08:19]

http://localhost:8888/notebooks/pylearn2_fork/pylearn2/scripts/tutorials/multilayer_perceptron/multilayer_perceptron.ipynb
The MLP makes a weak assumption on the model, that, generally speaking, the inputs map to outputs via the composition of several functions.
A related issue with MLPs is that they have many configuration options. The model itself imposes design decisions such as what type of function to use for each layer, the dimensionality of each layer. Also, the log likelihood is no longer generally concave, so the choice of optimization procedure matters more than it did with softmax regression. These configuration options are known as “hyperparameters.” Choosing the right hyperparameters is an open and exciting research problem

To show how the MLP is specified via the yaml

Model: !obj:pylearn2.models.mlp.MLP { layers: [ !obj:pylearn2.models.mlp.Sigmoid { layer_name: ‘h0’, dim: 500, sparse_init: 15, }, !obj:pylearn2.models.mlp.Softmax { layer_name: ‘y’, n_classes: 10, irange: 0. } ], nvis: 784, },

see

ls ipynb_work/*mlp*

Fit a MLP

State “DONE” from “NEXT” [2015-02-25 Wed 22:19]
State “NEXT” from “” [2015-02-25 Wed 08:19]

Some new parameters to define

Model MLP layer_name dim sparse_init n_classes irange

Algorithm BGD updates per batch conjugate

with an interlude for emacs greatness: this failed as there was tabs in the yaml. C-x h (highlight all) and M-x untabify <RET> to replace all tabs with whitespace.

cat ipynb_work/mlp_01.yaml

Score still bad, 8.xx. CNN should provide a substantial improvement.

Research on CNN application

State “DONE” from “” [2015-02-25 Wed 19:38]

See krizhevsky et al

Here is a good wiki on softmax
- it’s use is for mutually exclusive multiclass problems
- in the 2 class problem it simplifies to logistic regression
The parameters are learned by optimizing the cost function using an itertive algorithm, such as gradient descent. Note that such an optimization algorithm can be used for many cost functions, such as those applying to K-means, SVM, Lasso.
- the cost function for softmax with weight decay, is some equation
- bottou provides some good info on SGD
  - it should be used when training time is the bottleneck
- more from bottou, SGD with mnist as example
Convolutional layers?
Kernals and neurons?
Dropout
softmax params
- We trained our models using stochastic gradient descent with a batch size of 128 examples, momentum of 0.9, and weight decay of 0.0005.

then cnn

State “DONE” from “NEXT” [2015-03-02 Mon 19:44]
State “NEXT” from “” [2015-02-25 Wed 19:38]

http://deeplearning.net/tutorial/lenet.html great explanation
http://localhost:8888/notebooks/pylearn2_fork/pylearn2/scripts/tutorials/convolutional_network/convolutional_network.ipynb
example settings
- https://groups.google.com/forum/#!topic/pylearn-users/ZVbIQGh0YMQ
CNN is simply MLP with slight translation invariance
this is good for our image recognition problem
this is set up similar to MLP, but we use ‘input_space’ instead of ‘nvis’, which preserves topological significance. nvis is shorthand for vectorspace(n), i.e. right now my images are being converted to vector representation! humbug
the yaml setup file in the CNN tutorial is useful to refer to,
- conv rectified layers are used
- and

cat ipynb_work/cnn_01.yaml

Predict CNN

State “DONE” from “TODO” [2015-03-04 Wed 22:45]

had issues with shape
for some reason pdb.set_trace not working

Check out google group

From here:

According to your yaml the images are 48x48x1 so I think you just need use the command np.reshape(x, size=(B, 48, 48, 1), where B is the number of examples in the file, after the data has been loaded into x and before and before the line y = f(x).

My data is already in the proper shape in the pickled file. I don’t think this will help.

This sounds promising:

Rather than calling f directly on x_test, I think you have to get the data be converted to the format expected by your model, which is some kind of masked array.

The easiest way would probably be to have your test data in a Dataset object, like the training data is, and access batches of data through an iterator built by the dataset.

For instance:

it = test_set.iterator(mode='sequential', batch_size=...,
                       data_specs=(model.get_input_source(),
                                   model.get_input_space()))

y_pred = []
for batch in it:
  y_pred.extend(f(batch))

AND

Please also check whether X is a tuple (most likely yes) or not. If yes, then you need to create function using this statement: f = theano.function(X, Y) - note the lack of brackets around X - it is a tuple already and contains two symbolic input variables required.

Also you may use the following code to call the function:

f(*test_set.iterator(mode=’sequential’, num_batches=1, data_specs=(model.get_input_space(), model.get_input_source())).next())[0]

f will return tuple of (data,mask). mask will be all ‘1’ and you probably don’t need it.

X is not a tuple.

May need to try what he’s suggested with the iterator. One thing to do is try predicting one image. It did work, see cnn_00_predict.

So we can build a class similar to DataSciBowl which reads in the validation data and builds an iterator for it.

This shows a working example for predict with cnn:

x= [[[[180.05], [80.5]]]] 
for layer in train.model.layers:
    print("{0}:{1}".format(layer.layer_name, layer.get_param_values()))
    X = layer.get_input_space().make_theano_batch()
    Y = layer.fprop(X)
    f = function([X], Y)
    y = f(x)
    print ("input:{0}".format(x))    
    print ("output:{0}".format(y))    
    x = y
    print("---------------------------------------------------")

Also, beware the batch size.

[#A] Prediction validation

State “DONE” from “NEXT” [2015-03-07 Sat 13:13]
State “NEXT” from “TODO” [2015-03-04 Wed 23:06]
State “NEXT” from “” [2015-03-04 Wed 22:51]

seeing large difference in scores on validation vs test set.
need to verify
see this thread on forums
- verify structure/transformations of inputs
- verify rows and columns of outputs

output format

State “DONE” from “” [2015-03-05 Thu 08:03]

the columns are ordered differently
how does kaggle expect the data? in some type of order or it uses column names? it says it uses column names, so i should be ok there

verify column names order corresponds to order of prediction

State “DONE” from “” [2015-03-07 Sat 12:58]

My columns and the sample submission columns are in a different order
futhermore, my columns and the ‘labels’ as seen in ipynb_tests/test_data_setup_for_pred are in a different order
where can I get the order of the classes in the model?
in the datascibowl_reader util, i loop through the folders
the folders have the following order based on the list of folders

’../data/datascibowl/train/euphausiids_young’, ‘../data/datascibowl/train/hydromedusae_narcomedusae’, ‘../data/datascibowl/train/copepod_calanoid_small_longantennae’, ‘../data/datascibowl/train/appendicularian_s_shape’, ‘../data/datascibowl/train/decapods’,

whereas my predictions do not
In fact, in test_data_setup_for_pred alone I can see that the order of the predictions is different than that of the labels
is the order of the list preserved through pickling and unpickling?
- apparently
is the order preserved in the prediction script?
- let’s check
somehow it wasn’t! see the comparison in cnn_00_predict

verify row names

State “DONE” from “” [2015-03-07 Sat 12:58]

also didn’t match
how did this happen

[#A] Dropout/maxout research

State “DONE” from “NEXT” [2015-03-08 Sun 15:34]
State “NEXT” from “TODO” [2015-03-08 Sun 15:28]
State “NEXT” from “” [2015-03-07 Sat 13:49]

there’s 2 places to see examples here pylearn2/scripts/papers/maxout|dropout see also, maxout, and dropout examples
maxout is a model layer
dropout is a cost function, when used on a layer it is applied everywhere
maxout (at least in 2013) only worked with SGD algorithm
Geoff Hinton, who came up multiple times at a dinner last night where 4/7 attendees were practicing machine learning, authored the widely recognized paper on dropout.
In simple terms, this prevents overfitting by omitting hidden features with probability 0.5 and some observed elements in each training case. The results is similar to averaging models over random samples of the observations, while requiring less resources to fit many models.
pylearn2.costs.mlp.dropout
dropout performs better under some constraints, which maxout takes advantage of, as found by Goodfellow et al
pylearn2.models.maxout.Maxout
- a hidden layer which does max pooling over groups of linear units
pylearn2.models.maxout.MaxoutConvC01B
pylearn2.models.maxout.MaxoutLocalC01B

Research Learning rate/momentum

State “DONE” from “” [2015-03-08 Sun 18:20]

here’s a good overview
learning rate is the size of the jump taken in the parameter space and momentum accelerates the learning rate when the previous change in the weight update was in the same direction
as for the actual values, grid search? some standard? Something for another day.

Update theano

State “DONE” from “NEXT” [2015-03-08 Sun 18:21]
State “NEXT” from “” [2015-03-08 Sun 15:55]

/home/jotham/projects/2014-12-20_datascibowl/pylearn2_fork/pylearn2/sandbox/cuda_convnet/__init__.py:66: UserWarning: You are using probably a too old Theano version. That will cause compilation crash. If so, update Theano.

Run with dropout/maxout

State “DONE” from “NEXT” [2015-03-08 Sun 20:27]
State “NEXT” from “” [2015-03-08 Sun 15:55]

Improve sample/Augmentation [1/4]

State “DONE” from “TODO” [2015-03-10 Tue 22:35]

see 4.1 in krizhevsky et al

[#A] Resample training set to even obs/class

State “DONE” from “NEXT” [2015-03-10 Tue 22:35]
State “NEXT” from “TODO” [2015-03-07 Sat 13:23]
State “NEXT” from “” [2015-02-27 Fri 15:23]

lets ahve a test/validation set which takes evently from the classes
could use with berts random transformer to set number of transformations to max(n_obs_class)
but we really lose out with the random transformations, well, i suppose it depends on structure of validation set (i.e. what orientations are represented there compared to training set)

CANCELLED use larger images while we’re at it, 48x48

State “CANCELLED” from “NEXT” [2015-03-10 Tue 22:35]
State “NEXT” from “” [2015-03-07 Sat 13:47]

waiting, didnt offer much improvement when maxout used

Continue learning

State “DONE” from “TODO” [2015-03-08 Sun 18:24]

furthermore, I see that the maxout examples use continued learning, what is this?
ah, i think it’s used for if you want to interrupt learning or just after the stopping criterion is reached, this can be invoked to decide if learning should, guess … continue!

try random transform via block and transformerdataset

State “DONE” from “” [2015-03-12 Thu 06:15]

miserable score

online augmentation

State “DONE” from “NEXT” [2015-03-12 Thu 08:53]
State “NEXT” from “” [2015-03-10 Tue 23:07]

https://github.com/benanne/kaggle-galaxies/blob/master/realtime_augmentation.py
see the commit

try just rotation; no translations

State “DONE” from “NEXT” [2015-03-12 Thu 21:37]
State “NEXT” from “” [2015-03-12 Thu 08:38]

i think i got this wrong
removed the signal condition
when i reran visualization in random_transformation_tests_0 some of the images were just lack
and even without translations, sometimes the shape was pretty much absent from the image

Try transformations without class balancing

State “DONE” from “NEXT” [2015-03-12 Thu 21:39]
State “NEXT” from “” [2015-03-10 Tue 23:07]

see here, class balancing may be problematic

try random 90 deg warm online augmentation

State “DONE” from “NEXT” [2015-03-12 Thu 23:43]
State “NEXT” from “” [2015-03-12 Thu 21:38]

keep the signal condition
run the tests
score still plummetter

try thr 3 orthogonal transform

State “DONE” from “NEXT” [2015-03-12 Thu 23:55]
State “NEXT” from “” [2015-03-12 Thu 08:53]

put in Datascibowl dataset class
the validation set score is extremely variable
may have to relax the parameters slightly
7e14 after 4 eopchs
WTF is that

Verify the 3 orthogonal rotations

State “DONE” from “” [2015-03-14 Sat 15:15]

see ipynb_test/datascibowlreader_vanilla_augmentation
affine use radians, was assumin degrees
it also requires some parameters other than rotate; i made some naive assumptions about how it works

Retry 3 orthogonal augmentations

State “DONE” from “NEXT” [2015-03-14 Sat 19:50]
State “NEXT” from “” [2015-03-14 Sat 15:16]

still no good, even with the fix

NEXT Parameter research

State “NEXT” from “” [2015-03-14 Sat 22:31]

momentum and init
- http://www.cs.toronto.edu/~fritz/absps/momentum.pdf
convolutional layers
- http://deeplearning.net/software/pylearn2/library/models.html#id10

Vincent Domulin’s pylearn 2 blog posts

http://vdumoulin.github.io/articles/pylearn2-jobman/
http://vdumoulin.github.io/articles/extending-pylearn2/
very useful way to move forward with pylearn 2

parameter grid search, see pylearn2-jobman from domulin

[#A] Retry 3 orthogonl augment with different model

State “DONE” from “NEXT” [2015-03-14 Sat 22:31]
State “NEXT” from “TODO” [2015-03-14 Sat 19:50]

see cnn_maxout_augment_rotate_06.yaml
- increase batch size to 256 and learning rate to .05

NEXT split into training and sample before augment

State “NEXT” from “” [2015-03-10 Tue 23:05]

NEXT Train model with class balancing

State “NEXT” from “” [2015-03-10 Tue 22:37]

may need to make some adjustments for memory usage
- validation set from original images only - not augmented?
can we skip monitoring of the training set?

NEXT retry prediction on whole test set

State “NEXT” from “” [2015-03-08 Sun 20:26]

with theano RC
otherwise, use iterator with larger pred batches, cause the individual is so slow

[#B] Shuffle and subset image paths

State “NEXT” from “” [2015-02-27 Fri 15:15]

not images themselves/resulting array
i.e. don’t unnecessarily read too much in
then can naively apply transformations
if doing non orthogonal rotations, have to fill introduced space with mean (for centred) or 0

[#B] parameter tuning

-

[#C] Profile GPU

link

[#C] Remove the unnecessary reshapes

State “NEXT” from “” [2015-02-16 Mon 10:06]

also test reshape
does

A A A

B B B

reshape(6,1) become

A A A B B B

and that reshape(3,2) become

A A A

B B B

?

[#C] Threshold observations?

State “NEXT” from “” [2015-02-16 Mon 11:11]

I thought softmax didn’t learn 0 and 1 well, this should perform poorly

[#A] Validate the process for generating validation image and column labels

somehow things got fucked up in the original data prediction
better investigate that failure to know how to avoid it next time

Review image proc presentation

simple BGD and SGD again on softmax again

using augmented data
on laptop

other stopping riles

epochs
other monitor costs

Predict MLP with argmax

State “NEXT” from “TODO” [2015-03-03 Tue 22:05]

classify and see score
probably shitier
low priority, MLL/NLL penalizes for such boldness

Compare time to read images vs load h5 file

see Saving large pickle/h5

Pylearn2 models and official docs

algorithm
- bgd
  - batch size

Setting this to 1 results in pure stochastic gradient descent whereas setting it to the size of the training set effectively results in batch gradient descent.
- softmax parameters
  - batch
  - irange

Understand the fitting monitor

State “NEXT” from “” [2015-02-15 Sun 22:14]

The fitting monitor gives some information that should be understood. Here’s helpful link

Monitoring step:
	Epochs seen: 3
	Batches seen: 15
	Examples seen: 75000
	ave_grad_mult: 0.0497610855166
	ave_grad_size: 1.33559131637
	ave_step_size: 0.0652578686429
	total_seconds_last_epoch: 410.991061
	train_objective: 3.63486511009
	train_y_col_norms_max: 0.743025824372
	train_y_col_norms_mean: 0.17295732139
	train_y_col_norms_min: 0.0269976684984
	train_y_max_max_class: 0.48181040514
	train_y_mean_max_class: 0.0885340497259
	train_y_min_max_class: 0.0411385993209
	train_y_misclass: 0.8402
	train_y_nll: 3.63486511009
	train_y_row_norms_max: 0.270664018449
	train_y_row_norms_mean: 0.066049282967
	train_y_row_norms_min: 0.0319228798405
	training_seconds_this_epoch: 400.303218

epochs

batches

examples

ave_grad mult size

ave_step_size

objective

train_y_ col|row_norms_ max mean min max_max_class mean_max_class min_max_class misclass nll

full batch size

-https://groups.google.com/forum/#!topic/pylearn-users/kM1PDr0jlm8

Fit more models/Combinations to try

State “NEXT” from “” [2015-02-15 Sun 22:14]

parameter space
- how to save performance vs. settings
model
algorithm
data augmentation

CANCELLED [#C] Tune GPU

State “CANCELLED” from “TODO” [2015-03-08 Sun 18:26]
- got bleeding edge theano and performance satisfactory
State “NEXT” from “” [2015-02-23 Mon 22:05]

It seems slow with BGD but fast with SGD.

See this thread. And that one. Finally, this one may also help.

Get the bleeding edge theano.

pip uninstall theano
pip install --upgrade --no-deps git+git://github.com/Theano/Theano.git

Try adding this to .theanorc. It will confirm that gpu is working. Two, it offers a speedup.

allow_gc = False

Try (‘c’, 1, 0, ‘b’) axis order.

Post competition

Setup a few python dev env things for emacs

consider making it extensible to remote machines with just shell access

http://emacswiki.org/emacs/PythonProgrammingInEmacs https://www.youtube.com/watch?v=0cZ7szFuz18 https://github.com/AndreaCrotti/minimal-emacs-configuration

cusedit ido mode windowmove key bindings shift

yasnippet snippets for classes, r functions, for example

also very much need the package which fixes the custom-set-variable bullshit

Misc Learned

Pushing to different remotes with magit

C-u C-u P P

Files

notes.org

Latest commit

History

notes.org

File metadata and controls

Setup

Get data

Ipython

Python/Dev Stuff [8/10]

Setup packages

virtual env Research

Setting up for python development on ArchLinux

Settung up python dev on ArchLinux v2

setting up venv for distribution

Pickle for saving python binary objects

why pickle?

what can pickle do?

What can’t be pickled?

how to use it?

cPickle

Caveats (Carrots?)

Sets and lists builting data structures

list comprehensions

NEXT [#A] Iterators

NEXT [#B] generators

learn key python packages [2/6]

Numpy [2/3]

creating arrays

slicing

NEXT iteration

linalg

Scipy

NEXT multidimensional image proc

NEXT Pandas

sci kit learn (user guide)

NEXT sci kit image

Hu central moment expirimentation [0/2]

Features [1/2]

NEXT Tests [0/1]

ML procedure

model infrastructure

Write a class for this? work with bert

CANCELLED ML Stuff

ufldl tutorial for deep learning

Check the kaggle handwriting competition example

CANCELLED make a uniform probability prediction

data

NEXT Understand the error/performance measure better

Preprocessing

NEXT check prportion of obs with maxRegion [/]

feature brainstorming

hu central moments Research

NEXT image feature extraction

nested model

Different number of samples per class

feature vs model selection

hu central moments Research

python vs R

Blog topics/Notes [39/59]

pylearn 2 convolutional neural networks setup

setup pylearn 2 dependencies [2/3]

theano, might want bleeding edge

PyYAML

NEXT PIL for some image related functionality

install pylearn2

first tutorial pylearn2 in practice

setup environment vars

mnist tutorial

setup pylearn2 for our data [3/3]

Overview of setting up expiriment

vincent domoulin tutorial

Dataset

getting the data in

DataSciBowl options [2/2]

Test DataSciBowl class

Model setup with pylearn2

Fit a model

Shuffle the observations - keep it going forward

Centred observations