Skip to content

Latest commit

 

History

History

images

README.md

Building your own images

Setup

To build your own images, install docker on your computer:

sudo apt install docker.io

and login to our docker image repository:

docker login ic-registry.epfl.ch

Afterwards, clear the plain-text password file:

rm ~/.docker/config.json

Dockerfiles

An image is specified by a directory containing a file named Dockerfile. Here is the Dockerfile documentation.

At the beginning, we say on top of which file we build:

FROM base_image_tag

Other common operations are:

  • run commands with RUN:
RUN apt-get update \
	&& apt-get --no-install-recommends -y install sudo htop tmux locate mc less \
	&& apt-get clean
  • set environment variables with ENV:
ENV AUTO_SHUTDOWN_TIME 1h
  • copy files from the current directory
COPY local_file /opt/lab/file_inside_image

Existing images

You can view the existing images available in our repository at https://ic-registry.epfl.ch/ (login to see CVLab's images). There are also many publicly available images at the Docker Hub.

I prepared some images which will hopefully be of use to you (click to see Dockerfiles):

lab-base

ic-registry.epfl.ch/cvlab/lis/lab-base:cuda10.1-devel
ic-registry.epfl.ch/cvlab/lis/lab-base:cpu (same)

Basic utilities and the user-setup system. You can make additional setup steps by putting a .sh script in /opt/lab/setup_steps. The setup steps are run in alphabetical order, hence the nubmers at the start.

lab-pytorch-cuda-ext

ic-registry.epfl.ch/cvlab/lis/lab-pytorch-cuda-ext

  • usual Python numeric libs
  • Jupyter
  • PyTorch and accessories
  • OpenCV

lab-pytorch-extra

ic-registry.epfl.ch/cvlab/lis/lab-pytorch-extra

Extra libs on top of PyTorch

lab-python-ml

ic-registry.epfl.ch/cvlab/lis/lab-python-ml

  • TensorFlow
  • GluonCV - collection of trained networks for various vision tasks

lab-pytorch-apex

ic-registry.epfl.ch/cvlab/lis/lab-pytorch-apex

Extra libs on top of PyTorch

lab-colmap

ic-registry.epfl.ch/cvlab/lis/lab-pytorch-apex-colmap

Adds COLMAP multi-view 3d reconstruction.

If you need additional software installed, please let me know, or create your own image on top, as described below.

py38src branch - why are we building python from source?

  • At the time, our image is based on nvidia/cuda:10.2-cudnn7-devel-ubuntu18.04. There is a newer one on ubuntu20.04, but it has CUDA 11 which PyTorch is not ready for, additionally our drivers are for 10.2 CUDA. That image is on the bionic ubuntu version.
  • The default python3 package on bionic is 3.6 and this works. In this branch however we want newer python - 3.8 at the time.
  • There is a python3.8 package on bionic, however there is a packaging bug - this installation is missing the disutils module from the standard library [discussion of the problem]. The lack of distutils causes failure to install PIP. The module distutils is normally found in a package python3-distutils, but on bionic there is no python3.8-distutils.
  • There is a PPA deadsnakes with more python builds for various ubuntu versions. However, the 3.8 installation on ubuntu fails to load the ssl module - perhaps they did not install the ssl during the build. The lack of ssl causes an error when downloading packages over HTTPS.
  • Hence we build python from source, based on this dockerfile of python image

Extending the image

In case you want to extend the image, please see this example lab-python-extra Dockerfile. Here we install libraries from the repositories, but it is possible to do much more.

# start from the base image
FROM ic-registry.epfl.ch/cvlab/lis/lab-python-ml:cuda10

# extra linux libraries
RUN apt-get update \
	&& apt-get install --no-install-recommends -y git \
	&& apt-get clean

# extra python libraries
RUN pip --no-cache-dir install \
	natsort jinja2

Once your docker file is ready, build the image with docker build. Assuming it is in lab-python-extra/Dockerfile, the command is:

docker build lab-python-extra -t ic-registry.epfl.ch/cvlab/my_user_name/something:label_name

You can test the image locally, for example with:

docker run -it ic-registry.epfl.ch/cvlab/my_user_name/something:label_name /bin/bash

or using this provided test environment if you are familiar with docker-compose.

Once the image is built, we push it to the repository:

docker push ic-registry.epfl.ch/cvlab/my_user_name/something:label_name

Now you can use the image in your pods!

Multi-stage builds

If your software needs to be compiled, you may benefit from multi-stage builds: this involves creating a temporary container with the build tools where the compilation takes place, then we only copy the results of the compilation to the output image. This saves space in the output image and allows the build process to be cached.

Maintaining the images

To build the COLMAP image, download the source first - link and destination listed in build.sh.

The images are built by build.sh.

At this stage, they are in the :dev tag, and can be tested before being used by everyone.

Release using publish.sh which relabels from :dev to :latest.

Server

The build is done on iccvlabsrv23, where I have the permission to launch docker.

On this server, the default docker network "bridge" does not give us access to the outside net. This is fixed by --network="host".