tutorial docker charliecloud

How to build a docker image then charliecloud
guide
Author

Audrey Lapendry

Published

February 15, 2024

Hello everyone!

I’m sharing below the steps I followed to generate a Docker image, then Charliecloud, in order to use it on the CBPsmn. Don’t forget to adapt the commands according to what you want to do (for example, you need to replace the alapendr).

Before starting, make sure you are a member of the abc group on the CBPsmn so you can add your Charliecloud image once created. To do this, check if the abc group is listed when you run the command groups on the CBPsmn. If not, request to be added to the abc group by submitting a request via this link.

1. Creating a Docker image from a Dockerfile locally

To start, I worked locally to create a Docker image of the FitHiC2 software. After several tests, here’s the Dockerfile code that resulted in the successful creation of the FitHiC2 image:

FROM python:3.8
LABEL maintainer="Audrey Lapendry <audrey.lapendry@ens-lyon.fr"

WORKDIR /app

RUN apt-get update && apt-get install -y git

RUN pip install numpy==1.14.*
RUN pip install scipy==1.10.*
RUN pip install scikit-learn==0.19.*
RUN pip install sortedcontainers==2.0.*
RUN pip install matplotlib==2.2.*

RUN git clone https://github.com/ay-lab/fithic.git .

RUN apt-get install -y build-essential libgsl-dev

RUN chmod +x /app/fithic/utils/*.py
RUN chmod +x /app/fithic/fithic.py
RUN mkdir /scratch ########## Important line to add for CBPsmn ##########

CMD ["bash"]

To build the Docker image from the Dockerfile, execute the following command:
sudo docker build -t alapendr/fithic .

Then, launch the container image you created with this command to verify that the installation went as expected:
sudo docker run -it alapendr/fithic

2. Adding the Docker image created locally to Docker Hub

Then, add the image you created to Docker Hub by following these steps: 1. Create a Docker Hub account here if you don’t already have one 2. Click on “Create Repository”, add a name, a description, and leave the visibility as public 3. Go back to the folder where your Dockerfile is located locally. Then, run the following two commands to add the image to Docker Hub: 4. sudo docker login -u alapendr # Your Docker Hub password will be requested 5. sudo docker push alapendr/fithic

3. Retrieving the Docker image from Docker Hub on PSMN, then converting to Charliecloud

As Docker is not compatible with the use of a compute cluster (since it requires superuser privileges), we will convert the Docker image to Charliecloud, which is suitable for this use. To do this: 1. Connect to the CBPsmn using your credentials 2. Set up Charliecloud if not already done:

  • To do this, open your ~/.bashrc file with a text editor, such as nano, using the following command: nano ~/.bashrc.
  • Add the following line to the file:
    PATH=/Xnfs/abc/charliecloud_bin/:/Xnfs/abc/nextflow_bin:$PATH
  • Save the changes and exit the text editor
  • To apply the changes, run the following command: source ~/.bashrc
    You will then be able to use Charliecloud on your CBPsmn account.
  1. Retrieve the Docker image from Docker Hub and convert it to Charliecloud:
  • Access your directory on the CBPsmn where you want to work:
    cd /scratch/Bio/alapendr/
  • If not already done, create a working directory:
    mkdir fithic_analysis
  • Then create a file specifying the Docker image you want to download from Docker Hub:
    echo "alapendr/fithic:latest" > fithic_analysis/fithic_docker.nf
  • Download the Docker image with this command:
    /Xnfs/abc/nextflow_bin/pull_ch_images_locally.sh fithic_analysis
  • The image has been downloaded here /Xnfs/abc/charliecloud/img/alapendr%fithic+latest

4. Using the Charliecloud image in a SLURM job

Now that you have your Charliecloud image, you can use it as you wish. Here’s an example SLURM script to launch a Charliecloud container, via the ch-run command, and run a FitHiC analysis:

#SBATCH --ntasks=1
#SBATCH --partition=Lake
#SBATCH --cpus-per-task=1
#SBATCH --mem-per-cpu=32G
#SBATCH --time=02:00:00
#SBATCH --job-name=fithic
#SBATCH --output=/scratch/Bio/alapendr/fithic_launch/log/test.out
#SBATCH --error=/scratch/Bio/alapendr/fithic_launch/log/test.err

ch-run --write --bind=/scratch/:/scratch/ /Xnfs/abc/charliecloud/img/alapendr%fithic+latest /scratch/Bio/alapendr/fithic_launch/src/fithic_launcher.sh /scratch/Bio/alapendr/fithic_launch/data/test/4DNFI1MIOZ4H_100000.txt.gz /scratch/Bio/alapendr/fithic_launch/data/hg38_chr_len.txt 100000 /scratch/Bio/alapendr/fithic_launch/results/tmp/ /scratch/Bio/alapendr/fithic_launch/results/results/ 4DNFI1MIOZ4H

I hope these explanations have been helpful. Feel free to ask me any questions or for further clarification.

Have a great day, Audrey

2024/04/02 addition by Laurent Modolo Steps to follow to use a charliecloud container on the PSMN:

  1. look for the image in the /Xnfs/abc/charliecloud/img/ folder
  2. use the command ch-run --write --bind=/scratch/:/scratch/ --bind=/Xnfs:/Xnfs to be able to write in the /scratch and /Xnfs folder from within the container
  3. build a command of the following form : ch-run + container/path + -- command withing the container