Merge pull request #26 from osg-htc/xalim-exercises

Make exercises public

docs/materials/checkpoint/part1-ex1-checkpointing.md

@@ -26,7 +26,7 @@ To get set up:
 1.  Download the Python script that is the main executable for this exercise:
 
         :::console
-        user@server $ wget https://raw.githubusercontent.com/osg-htc/user-school-2022/main/src/checkpointing/fibonacci.py
+        user@server $ wget http://proxy.chtc.wisc.edu/SQUID/osg-school-2024/fibonacci.py
 
 1.  If you want to run the script directly, make it executable first:
 

docs/materials/data/part1-ex1-data-needs.md

@@ -168,4 +168,4 @@ Up next!
 --------
 
 Next you will create a HTCondor submit script to transfer the Blast input files in order to run Blast on a worker nodes.
-[Next Exercise](../part1-ex2-file-transfer)
+[Next Exercise](part1-ex2-file-transfer.md)

docs/materials/data/part1-ex2-file-transfer.md

@@ -28,7 +28,7 @@ previous exercise.
 
 ### Review: HTCondor File Transfer
 
-![OSG data transfer](../files/osgus18-day4-part2-ex2-data-transfer.jpg)
+![OSG data transfer](files/osgus18-day4-part2-ex2-data-transfer.jpg)
 
 Recall that OSG does **NOT** have a shared filesystem! Instead,
 HTCondor *transfers* your executable and input files (specified with
@@ -59,7 +59,7 @@ queue
 
 ### Implement file compression
 
-In our first blast job from the Software exercises ([1.1](../../software/part1-ex1-download)), the database files in the `pdbaa` directory were all transferred, as is, but we
+In our first blast job from the Software exercises ([1.1](../software/part4-ex1-download.md)), the database files in the `pdbaa` directory were all transferred, as is, but we
 could instead transfer them as a single, compressed file using `tar`.
 For this version of the job, let's compress our blast database files to send them to the submit node as a single
 `tar.gz` file (otherwise known as a tarball), by following the below steps:
@@ -123,7 +123,7 @@ It's good to start with values that are a little higher than you think a test jo
     files (hint: the log file only exists on the submit node).
 -   Whether you'd like to request some extra memory or disk space, just in case
 
-Look at the `log` file for your `blastx` job from Software exercise ([1.1](../../software/part1-ex1-download)), and compare the memory and disk "Usage" to what you predicted
+Look at the `log` file for your `blastx` job from Software exercise ([1.1](../software/part4-ex1-download.md)), and compare the memory and disk "Usage" to what you predicted
 from the files.
 Make sure to update the submit file with more accurate memory and disk requests. You may still want to request slightly
 more than the job actually used. 
@@ -136,7 +136,7 @@ It should take a few minutes to complete, and then you can check to make sure th
 `pdbaa` database files) were copied back at the end of the job.
 
 Run a **`du -sh`** on the directory with this job's input.
-How does it compare to the directory from Software exercise ([1.1](../../software/part1-ex1-download)), and why?
+How does it compare to the directory from Software exercise ([1.1](../software/part4-ex1-download.md)), and why?
 
 transfer\_output\_files
 -----------------------
@@ -217,11 +217,11 @@ Conclusions
 
 In this exercise, you:
 
--   Used your data requirements knowledge from the [previous exercise](../part1-ex1-data-needs) to write a job.
+-   Used your data requirements knowledge from the [previous exercise](part1-ex1-data-needs.md) to write a job.
 -   Executed the job on a remote worker node and took note of the data usage.
 -   Used `transfer_input_files` to transfer inputs
 -   Used `transfer_output_files` to transfer outputs
 -   Used `transfer_output_remaps` to map outputs to a different destination
 
-When you've completed the above, continue with the [next exercise](../part1-ex3-blast-split).
+When you've completed the above, continue with the [next exercise](part1-ex3-blast-split.md).
 

docs/materials/data/part1-ex3-blast-split.md

@@ -16,7 +16,7 @@ Setup
 1. Log in to `ap40.uw.osg-htc.org`
 
 1. Create a directory for this exercise named `blast-split` and change into it.
-1. Copy over the following files from the [previous exercise](../part1-ex2-file-transfer):
+1. Copy over the following files from the [previous exercise](part1-ex2-file-transfer.md):
     - Your submit file
     - `blastx`
     - `pdbaa_files.tar.gz`
@@ -72,7 +72,7 @@ First, you'll create a new submit file that passes the input filename as an argu
 filenames.
 Follow the below steps:
 
-1. Copy the submit file from the [previous exercise](../part1-ex2-file-transfer) to a new file called `blast_split.sub` and modify the "queue" line of the submit file to the
+1. Copy the submit file from the [previous exercise](part1-ex2-file-transfer.md) to a new file called `blast_split.sub` and modify the "queue" line of the submit file to the
    following:
 
         queue inputfile matching mouse_rna.fa.*

docs/materials/data/part2-ex1-osdf-inputs.md

@@ -18,7 +18,7 @@ OSDF is connected to a distributed set of caches spread across the U.S.
 They are connected with high bandwidth connections to each other, and to the data origin servers, where your data is
 originally placed.
 
-![OSDF Map](../files/osgus19-day4-part2-CacheLocations.png)
+![OSDF Map](files/osgus19-day4-part2-CacheLocations.png)
 
 Setup
 -----
@@ -93,5 +93,5 @@ The next time you use OSDF after the school, remember to first check for old fil
 Next exercise
 -------------
 
-Once completed, move onto the next exercise: [Using OSDF for outputs](../part2-ex2-osdf-outputs)
+Once completed, move onto the next exercise: [Using OSDF for outputs](part2-ex2-osdf-outputs.md)
 

docs/materials/data/part2-ex2-osdf-outputs.md

@@ -7,7 +7,7 @@ Data Exercise 2.2: Using OSDF for outputs
 
 In this exercise, we will run a multimedia program that converts and manipulates video files.
 In particular, we want to convert large `.mov` files to smaller (10-100s of MB) `mp4` files.
-Just like the Blast database in the [previous exercise](../part2-ex1-osdf-inputs), these video
+Just like the Blast database in the [previous exercise](part2-ex1-osdf-inputs.md), these video
 files are potentially too large to send to jobs using HTCondor's default file transfer for 
 inputs/outputs, so we will use OSDF.
 

docs/materials/index.md

@@ -3,26 +3,26 @@
 ## School Overview and Intro
 
 View the slides:
-[PDF](welcome/files/osgus23-day1-part1-welcome-timc.pdf)
+[Slides coming soon]
 
 ## Intro to HTC and HTCondor Job Execution
 
 ### Intro to HTC Slides
 
-Intro to HTC: [PDF](htcondor/files/osgus23-intro-to-htc.pdf)
+Intro to HTC: [Slides coming soon]
 
-Worksheet: [PDF](htcondor/files/osgus23-htc-worksheet.pdf)
+Worksheet: [Slides coming soon]
 
 ### Intro to HTCondor Slides
 
-View the slides: [PDF](htcondor/files/osgus23-htc-htcondor.pdf)
+View the slides: [Slides coming soon]
 <!--
 [PowerPoint](htcondor/files/osgus22-htc-htcondor.pptx))
 -->
 
 ### Intro Exercises 1: Running and Viewing Simple Jobs (Strongly Recommended)
 
-- [Exercise 1.1: Log in to the local submit machine and look around](htcondor/part1-ex1-login)
+- [Exercise 1.1: Log in to the local submit machine and look around](htcondor/part1-ex1-login.md)
 - [Exercise 1.2: Experiment with HTCondor commands](htcondor/part1-ex2-commands.md)
 - [Exercise 1.3: Run jobs!](htcondor/part1-ex3-jobs.md)
 - [Exercise 1.4: Read and interpret log files](htcondor/part1-ex4-logs.md)
@@ -37,7 +37,7 @@ View the slides: [PDF](htcondor/files/osgus23-htc-htcondor.pdf)
 
 ## Intro to HTCondor Multiple Job Execution
 
-View the Slides ([PDF](htcondor/files/osgus23-htc-htcondor-multiple-jobs.pdf))
+View the Slides: [Slides coming soon]
 
 
 ### Intro Exercises 2: Running Many HTC Jobs (Strongly Recommended)
@@ -51,7 +51,7 @@ View the Slides ([PDF](htcondor/files/osgus23-htc-htcondor-multiple-jobs.pdf))
 ## OSG
 
 View the slides:
-[PDF](osg/files/osgus23-day2-part1-osg-timc.pdf)
+[Slides coming soon]
 
 ### OSG Exercises: Comparing PATh and OSG (Strongly Recommended)
 
@@ -62,8 +62,7 @@ View the slides:
 
 ## Troubleshooting
 
-Slides: ([PDF](troubleshooting/files/OSGUS2023_troubleshooting.pdf),
-[PowerPoint](troubleshooting/files/OSGUS2023_troubleshooting.pptx))
+Slides: [Slides coming soon]
 
 ### Troubleshooting Exercises: 
 
@@ -72,7 +71,7 @@ Slides: ([PDF](troubleshooting/files/OSGUS2023_troubleshooting.pdf),
 
 ## Software
 
-Slides: [PDF](software/files/osgus23-software.pdf)
+Slides: [Slides coming soon]
 
 ### Software Exercises 1: Exploring Containers
 
@@ -106,9 +105,7 @@ Slides: [PDF](software/files/osgus23-software.pdf)
 
 ## Data
 
-View the slides
-([PDF](data/files/osgus23-data.pdf),
-[PowerPoint](data/files/osgus23-data.pptx))
+View the slides: [Slides coming soon]
 
 ### Data Exercises 1: HTCondor File Transfer (Strongly Recommended)
 
@@ -124,8 +121,7 @@ View the slides
 
 ## Scaling Up
 
-View the slides
-([PDF](scaling/files/osgus23-scaling-out.pdf))
+View the slides: [Slides coming soon]
 
 ### Scaling Up Exercises
 
@@ -136,9 +132,7 @@ View the slides
 
 ## Workflows with DAGMan
 
-View the slides
-([PDF](workflows/files/osgus23-dagman.pdf),
-[PowerPoint](workflows/files/osgus23-dagman.pptx))
+View the slides: [Slides coming soon]
 
 ### DAGMan Exercises 1
 
@@ -167,27 +161,24 @@ END EXTRA TOPICS THAT ARE NOT READY YET -->
 
 ### Self-checkpointing for long-running jobs
 
-View the slides
-([PDF](checkpoint/files/OSGUS2023_checkpointing.pdf),[PPT](checkpoint/files/OSGUS2023_checkpointing.pptx))
+View the slides: [Slides coming soon]
 
 -   [Exercise 1.1: Trying out self-checkpointing](checkpoint/part1-ex1-checkpointing.md)
 
 
 ### Special Environments
 
-View the slides
-([PDF](special/files/osgus23-special.pdf),
-[PowerPoint](special/files/osgus23-special.pptx))
+View the slides: [Slides coming soon]
 
 ### Special Environments Exercises 1
 
 - [Exercise 1.1: GPUs](special/part1-ex1-gpus.md)
 
 ### Introduction to Research Computing Facilitation
 
-View the slides: [PDF](facilitation/files/osgus23-facilitation-campuses.pdf)
+View the slides: [Slides coming soon]
 
 ## Final Talks
 
-*   Philosophy: (slides coming soon)
-*   Final thoughts: [PDF](final/files/osgus23-day5-part6-forward-timc.pdf)
+*   Philosophy: [Slides coming soon]
+*   Final thoughts: [Slides coming soon]

docs/materials/osg/part1-ex3-hardware-diffs.md

@@ -108,7 +108,7 @@ Now you will do essentially the same thing on the OSPool.
 
 1.  Log in or switch to `ap40.uw.osg-htc.org`
 
-1.  Copy the `osg-ex13` directory from the [section above](#checking-chtc-memory-availability)
+1.  Copy the `osg-ex13` directory from the [section above](#checking-path-memory-availability)
     from `ap1.facility.path-cc.io` to `ap40.uw.osg-htc.org`
 
     If you get stuck during the copying process, refer to [OSG exercise 1.1](part1-ex1-login-scp.md).

docs/materials/software/part1-ex3-docker-jobs.md

@@ -18,7 +18,7 @@ Create Local Copy of Docker Container
 While it is technically possible to use a Docker container directly in a job, 
 there are some good reasons for converting it to a local Apptainer container first. 
 We'll do this with the same `python:3.10` Docker container we used in the 
-[first exercise](../part1-ex1-run-apptainer). 
+[first exercise](part1-ex1-run-apptainer.md). 
 
 To convert the Docker container to a local Apptainer container, run: 
 
@@ -31,15 +31,15 @@ second argument is what we're building from (in this case, Docker).
 Submit File and Executable
 -------------------
 
-1.  Make a copy of your submit file from the [previous container exercise](../part1-ex2-apptainer-jobs) or build from an existing submit file. 
+1.  Make a copy of your submit file from the [previous container exercise](part1-ex2-apptainer-jobs.md) or build from an existing submit file. 
 
 1.  Add the following lines to the submit file or modify existing lines to match the lines below: 
 
 		:::file
 		universe = container
 		container_image = local-py310.sif
 
-1.  Use the same executable as the [previous exercise](../part1-ex2-apptainer-jobs). 
+1.  Use the same executable as the [previous exercise](part1-ex2-apptainer-jobs.md). 
 
 1. Once these steps are done, submit the job. You might get a warning about using OSDF for container transfers - ignore this warning for now.
 

docs/materials/software/part1-ex4-apptainer-build.md

@@ -66,7 +66,7 @@ Note that we are starting with the same `ubuntu` base we used in previous
 exercises. The `%post` statement includes our installation commands, including 
 updating the `pip` and `numpy` packages, and then using `pip` to install `cowsay`.
 
-To learn more about definition files, see [Exercise 3.1](../part3-ex1-apptainer-recipes)
+To learn more about definition files, see [Exercise 3.1](part3-ex1-apptainer-recipes.md)
 
 Build the Container
 -------------------
@@ -78,7 +78,7 @@ Once the definition file is complete, we can build the container.
 		:::console
 		$ apptainer build py-cowsay.sif py-cowsay.def
 
-As with the Docker image in the [previous exercise](../part1-ex3-docker-jobs), 
+As with the Docker image in the [previous exercise](part1-ex3-docker-jobs.md), 
 the first argument is the name to give to the newly create image file and the 
 second argument is how to build the container image - in this case, the definition file. 
 
@@ -87,7 +87,7 @@ Testing the Image Locally
 -------------------
 
 1. Do you remember how to interactively test an image? Look back 
-at [Exercise 1.1](../part1-ex1-run-apptainer) and guess what command would 
+at [Exercise 1.1](part1-ex1-run-apptainer.md) and guess what command would 
 allow us to test our new container. 
 
 1. Try running: 

docs/materials/software/part1-ex5-pick-an-option.md

@@ -45,25 +45,25 @@ Choose a Strategy
 		* [jupyter](https://hub.docker.com/u/jupyter)
 		* [nvidia](https://hub.docker.com/u/nvidia)
 		* (and many more!)
-  If yes, try using this container first, as shown in [Exercise 1.2](../part1-ex2-apptainer-jobs) and [Exercise 1.3](../part1-ex3-docker-jobs)
+  If yes, try using this container first, as shown in [Exercise 1.2](part1-ex2-apptainer-jobs.md) and [Exercise 1.3](part1-ex3-docker-jobs.md)
 
 1. Is there a simple download or easy compilation process? If so, can you 
  download the software and use it via a wrapper script? See the exercises from 
- Part 4 ([Download Software Files](../part4-ex1-download), 
- [Use a Wrapper Script](../part4-ex2-wrapper), 
- [Wrapper Script Arguments](../part4-ex3-arguments)). To learn more about using 
- this approach for specific softwares, see the examples in [Part 5](../../index.html#-software-exercises-5-compiled-software-examples). 
+ Part 4 ([Download Software Files](part4-ex1-download.md), 
+ [Use a Wrapper Script](part4-ex2-wrapper.md), 
+ [Wrapper Script Arguments](part4-ex3-arguments.md)). To learn more about using 
+ this approach for specific softwares, see the examples in [Part 5](../index.md#software-exercises-5-compiled-software-examples-optional). 
 
-1. Are you using conda? See the specific example in [Exercise 5.3](../part5-ex3-conda)
+1. Are you using conda? See the specific example in [Exercise 5.3](part5-ex3-conda.md)
 
 1. If neither of the above options works (which may be true for more software!), you 
   may want to build your own container. 
     1. If you want to just use this container on the OSPool, build an 
-    Apptainer container as described in [Exercise 1.4](../part1-ex4-apptainer-build) and 
-    with more information in [Exercise 3.1](../part3-ex1-apptainer-recipes)
+    Apptainer container as described in [Exercise 1.4](part1-ex4-apptainer-build.md) and 
+    with more information in [Exercise 3.1](part3-ex1-apptainer-recipes.md)
     1. If you want to use the container on your own computer or share with 
     others who would use it on a laptop or desktop, look at the Docker container 
-    example in [Exercise 3.2](../part3-ex2-docker-build). 
+    example in [Exercise 3.2](part3-ex2-docker-build.md). 
 
 Don't do ALL of the software exercises in parts 3 - 5! Instead, choose the section(s) 
 that makes sense based on how you want to manage your software. Talk to the School 
@@ -73,5 +73,5 @@ Create an Executable
 ---------------------
 
 Regardless of which approach you use, check out 
-the [Build an HTC-Friendly Executable](../part2-ex1-build-executable) exercise
+the [Build an HTC-Friendly Executable](part2-ex1-build-executable.md) exercise
 for some tips on how to make your script more robust and easy to use with multiple jobs. 

docs/materials/software/part3-ex2-docker-build.md

@@ -119,7 +119,7 @@ command line:
 	Docker Hub username and password and then try the push again. 
 
 1. Once your container image is in DockerHub, you can use it in jobs as described 
-in [Exercise 1.3](../part1-ex3-docker-jobs). 
+in [Exercise 1.3](part1-ex3-docker-jobs.md). 
 
 > Thanks to [Josh Karpel](https://github.com/JoshKarpel/osg-school-example-dockerfile) for 
 providing the original sample `Dockerfile`!

docs/materials/software/part4-ex1-download.md

@@ -23,21 +23,21 @@ reference database, the BLAST program.
 "blast executable or "download blast software".  Hopefully these
 searches will lead you to a BLAST website page that looks like this:
 
-    ![BLAST landing page](../files/part1-ex1-blast-landing-page.png)
+    ![BLAST landing page](files/part1-ex1-blast-landing-page.png)
 
 1.  Click on the title that says ["Download
-BLAST"](../files/part1-ex1-blast-front-page.png) and then look for the
+BLAST"](files/part1-ex1-blast-front-page.png) and then look for the
 link that has the [latest installation and source
-code](../files/part1-ex1-blast-dl-page.png).  
+code](files/part1-ex1-blast-dl-page.png).  
 
 	This will either open a page in a web browser that looks like this: 
 
-	![Download page](../files/part1-ex1-blast-dl-list.png)
+	![Download page](files/part1-ex1-blast-dl-list.png)
 
 	Or you will be asked to open the link in your file browser (choose the 
 	Connect as Guest option): 
 
-	![Download Folder](../files/part1-ex1-blast-dl-folder.png)
+	![Download Folder](files/part1-ex1-blast-dl-folder.png)
 
 	In either case, you should end up on a
 	page with a list of each version of BLAST that is available for
@@ -50,9 +50,9 @@ look at the list of downloads and try to determine which one you want.**
 1.  Based on our operating system, we want to use the Linux binary,
 which is labelled with the `x64-linux` suffix. 
 
-	![BLAST download page](../files/part1-ex1-blast-dl-list-linux.png)
+	![BLAST download page](files/part1-ex1-blast-dl-list-linux.png)
 
-	![BLAST download folder](../files/part1-ex1-blast-dl-folder-linux.png)
+	![BLAST download folder](files/part1-ex1-blast-dl-folder-linux.png)
 
 	All the other links are either for source code or other operating
 systems.