Skip to content

Latest commit

 

History

History
276 lines (211 loc) · 8.97 KB

README.md

File metadata and controls

276 lines (211 loc) · 8.97 KB

ROF Camera Trap Data Analysis Research Compedium

Dependencies:

  • docker
  • nvidia-container-toolkit

Report(s) and website

The reports directory currently contains the 📁 paper directory with the Quarto source document for the GnC report. Includes code to reproduce the figures and tables generated by the analysis.

Website is under construction.

How to run in your browser or download and run locally

This research compendium has been developed using the statistical programming language R. To work with the compendium, you will need installed on your computer the R software itself and optionally RStudio Desktop.

You can download the compendium as a zip from from this URL: master.zip. After unzipping: - open the .Rproj file in RStudio - run devtools::install() to ensure you have the packages this analysis depends on (also listed in the DESCRIPTION file). - finally, open analysis/paper/paper.Rmd and knit to produce the paper.docx, or run rmarkdown::render("analysis/paper/paper.qmd") in the R console

How to cite

Please cite this compendium as:

Lucet, Valentin; Stewart, Frances et al., (2024). Compendium of R code and data for ROF Camera Trap Data Analysis - Preliminary Report. Accessed 31 Jan 2024. Online at https://doi.org/xxx/xxx

Main Workflow

  1. At the terminal, clone this repository.
git clone https://github.com/StewartWILDlab/rofcamtrap
  1. Build the docker image of the computing environment, by running the build script. This will also build the apptainer image needed on HPC.
rofcamtrap/dockerfiles/build.sh
  1. Run docker image, with the proper volumes. We currently have two separate storage volumes, for each of the two camera retrievals that took place. We also make sure to hook the rofcamtrap folder as a volume.
# -e DISABLE_AUTH=true --shm-size 50G
docker run \
  -v "$(pwd):/workspace/rofcamtrap" \
  -v "/media/vlucet/TrailCamST/TrailCamStorage:/workspace/storage/TrailCamStorage" \
  -v "/media/vlucet/TrailCamST/TrailCamStorage_2:/workspace/storage/TrailCamStorage_2" \
  --gpus all \
  -it rofcamtrap
  1. Activate ENV, then run mega detector on images using the bash script.
mamba activate cameratraps-detector
rofcamtrap/scripts/bash/camtrap.sh \
  -b "/workspace/git" \
  -s "/workspace/storage/TrailCamStorage_2" \
  -m "/workspace/models/md_v5a.0.0.pt" \
  -o "/workspace/rofcamtrap/1_MegaDetector/0_outputs/TrailCamStorage_2" \
  md
  1. Run the repeat detector using the bash script, and remove all instances of trues positives.
rofcamtrap/scripts/bash/camtrap.sh \
  -b "/workspace/git" \
  -s "/workspace/storage/TrailCamStorage_2" \
  -i "/workspace/rofcamtrap/1_MegaDetector/0_outputs/TrailCamStorage_2" \
  repeat-detect
  1. Patch MD’s output with the repeat detect results.
rofcamtrap/scripts/bash/camtrap.sh \
  -b "/workspace/git" \
  -s "/workspace/storage/TrailCamStorage_2" \
  -i "/workspace/rofcamtrap/1_MegaDetector/0_outputs/TrailCamStorage_2" \
  -o "/workspace/rofcamtrap/1_MegaDetector/1_outputs_no_repeats/TrailCamStorage_2" \
  repeat-remove
  1. Optionally, write out the visualizations of the detections.
rofcamtrap/scripts/bash/camtrap.sh \
  -b "/workspace/git" \
  -s "/workspace/storage/TrailCamStorage_2" \
  -i "/workspace/rofcamtrap/1_MegaDetector/1_outputs_no_repeats/TrailCamStorage_2" \
  -o "/workspace/rofcamtrap/1_MegaDetector/2_visualize/TrailCamStorage_2" \
  viz
  1. Switch environments
mamba deactivate 
cd mdtools
poetry shell
cd ../
  1. Convert to coco and ls [and 3rd format?].
rofcamtrap/scripts/bash/camtrap.sh \
  -b "/workspace/git" \
  -s "/workspace/storage/TrailCamStorage_2" \
  -i "/workspace/rofcamtrap/1_MegaDetector/1_outputs_no_repeats/TrailCamStorage_2" \
  -o "/workspace/rofcamtrap/2_LabelStudio/0_inputs/TrailCamStorage_2" \
  repeat-convert
  1. Crop annotations.
rofcamtrap/scripts/bash/camtrap.sh \
  -b "/workspace/git" \
  -s "/workspace/storage/TrailCamStorage_2" \
  -i "/workspace/rofcamtrap/2_LabelStudio/0_inputs/TrailCamStorage_2" \
  crop

Label studio outputs

  1. Enter the container running the label studio app and output ths number of projects
docker exec -it label-studio-app-1 bash
curl -X GET http://localhost:8080/api/projects/?page_size=1000 -H 'Authorization: Token INSERT_TOKEN' -o files/outputs/project_counts.json
  1. On the instance outside the container, run the Python script to extract projects id
import json

with open('data/outputs/project_counts.json', 'r') as file:
  data = json.load(file)
  
ids = [data['results'][i]['id'] for i in range(len(data['results']))]

with open('data/outputs/project_ids.txt', 'w') as file:
  for the_id in ids:
        file.write(str(the_id) + '\n')
  1. Back in the container
arr=(192 191 190 189 188 187 186 185 184 183 182 181 180 179 178 177 176 175 174 173 172 171 170 86 85 84 83 82 81 80 79 78 77 76 75 74 73 72 65 64 63 62 61 60 59 58 57 55 54 53 52 51 50 49 48 47 46 44 43 42 41 40 39 37 35 33 32 31 30 28 25 23 21 20 19 18 17 16 15 13 10);
for id in ${arr[@]}; do         
  # url="http://localhost:8080/api/projects/${id}/export?exportType=JSON&download_all_tasks=true";
  url="http://localhost:8080/api/projects/${id}/export?exportType=JSON";
  curl -X GET "$url" -H 'Authorization: Token 3135fdd1f4a5b9b3630b69011ec4d70e7800c41d'\
    -o "files/outputs/ls/output_file_$id.json";
done
  1. Back outside the container, check for file numbers
ls data/outputs/ls # 82 Jan 29 2024
  1. Copy to local machine
scp -i ssh_key/arbutus_def_fstewart_prod '[email protected]:~/data/outputs/ls/*' rofcamtrap/2_LabelStudio/1_outputs_downloaded/
  1. Process those outputs
rofcamtrap/scripts/bash/camtrap.sh \
  -i "/workspace/rofcamtrap/2_LabelStudio/1_outputs_downloaded/" \
  -o "/workspace/rofcamtrap/2_LabelStudio/2_outputs_processed" \
  post

Processing detections & annotations

See the quarto documents in the analysis directory.

Classifier training workflow

On beluga, we use the apptainer image instead.

. /workspace/conda/etc/profile.d/conda.sh 
. /workspace/conda/etc/profile.d/mamba.sh
PATH="$PATH:$HOME/.local/bin"
mamba activate cameratraps-detector
rofcamtrap/scripts/bash/camtrap.sh -b "/workspace/git" -s "/workspace/storage/my_passport_images" -m "/workspace/models/md_v5a.0.0.pt" md

$ mkdir -p /scratch/$USER/apptainer/{cache,tmp}
$ export APPTAINER_CACHEDIR="/scratch/$USER/apptainer/cache"
$ export APPTAINER_TMPDIR="/scratch/$USER/apptainer/tmp"

salloc --time=00:05:00 --mem=4G --ntasks=1 --gpus-per-task=1 --cpus-per-task=1 --account=rrg-fstewart

apptainer shell --nv -C -B "$(pwd):/workspace/rofcamtrap"  -B "/media/vlucet/TrailCamST/TrailCamStorage:/workspace/storage/TrailCamStorage"  -B "/media/vlucet/My Passport/Images:/workspace/storage/my_passport_images" rofcamtrap.sif

apptainer shell --nv -C -B "$(pwd):/workspace/rofcamtrap" -B "/home/vlucet/projects/rrg-fstewart/vlucet:/workspace/project/" rofcamtrap.sif

Species classifier

  1. Crop all annotations using the functions in the R package. TODO: first local then containerized (code now written for locals)
# Read in the annotations
anns <- readRDS("data/objects/annotations_wide_noempty.rds")
rofcamtrap::crop_from_annotations(anns, 
                                  base_dir = "/media/vlucet/TrailCamST1/",
                                  out_dir = "data/images/cropped/")

We need to download the models… TBC.