Add CI for checking for broken links manually, weekly and in PRs

.github/workflows/README.md

@@ -27,7 +27,7 @@ Format:
     - job: assign_to_board
         - Assigns issues and PRs to either Content Development board or BSSw Internal board based on labels ("content: *" or "scope: site-internal")
 * notify-external-contrib.yml (Notify external contributions)
-    - https://github.com/betterscientificsoftware/bssw.io/blob/0e1ba1664239ab6097e903f8bb94fef08576f103/.github/workflows/notify-external-contrib.yml#L6-L12
+    - <https://github.com/betterscientificsoftware/bssw.io/blob/0e1ba1664239ab6097e903f8bb94fef08576f103/.github/workflows/notify-external-contrib.yml#L6-L12>
     - job: notify-external-contributions
         - Label and send email to bssw-editorial-list for externally opened issues, prs and discussions
 

.github/workflows/check-urls.yml

@@ -0,0 +1,161 @@
+name: Check URLs
+
+on:
+  workflow_dispatch:
+  schedule:
+    - cron: '17 5 * * 0' # 5:17 AM every Sunday
+  pull_request:
+    branches: [ main ]
+env:
+  ignore_file_patterns: |
+    docs
+    images
+    utils
+    Events
+
+jobs:
+  check-urls:
+    runs-on: ubuntu-latest
+
+    steps:
+
+    - name: Set up Python
+      uses: actions/setup-python@v2
+      with:
+        python-version: '3.9'
+
+    - name: Install dependencies
+      run: |
+        python -m pip --no-cache-dir --disable-pip-version-check install --upgrade pip
+        python -m pip --no-cache-dir --disable-pip-version-check install linkchecker
+
+    - name: Reformat environment variables
+      id: setup_vars
+      run: |
+        tmp=$(echo "${{ env.ignore_file_patterns }}" | tr '\n' ' ')
+        echo "ignore_file_patterns=$tmp" >> $GITHUB_OUTPUT
+
+    - name: Checkout Repo for PR branch
+      if: ${{ github.event_name == 'pull_request' }}
+      uses: actions/checkout@v4
+
+    - name: Checkout Repo for link check
+      if: ${{ github.event_name != 'pull_request' }}
+      uses: actions/checkout@v4
+      with:
+        ref: 'sched-link-checks'
+
+    - name: Sync main to link check branch
+      if: ${{ github.event_name != 'pull_request' }}
+      run: | 
+        git config user.name 'github-actions'
+        git config user.email '[email protected]'
+        git fetch origin main
+        git merge origin/main --no-edit -X thiers
+        git push origin sched-link-checks
+
+    - name: Get Changed Files (for PRs)
+      if: ${{ github.event_name == 'pull_request' }}
+      id: changed-files
+      uses: tj-actions/changed-files@v42
+      with:
+        separator: ' '
+
+    - name: Generate lists of files to check and ignore
+      id: file_list
+      run: |
+        if [ "${{ github.event_name }}" = "pull_request" ]; then
+          echo "files=${{ steps.changed-files.outputs.all_changed_files }}" >> $GITHUB_OUTPUT
+          echo "ignore_file_patterns=" >> $GITHUB_OUTPUT
+        else
+          echo "files=" >> $GITHUB_OUTPUT
+          echo "ignore_file_patterns=${{ steps.setup_vars.outputs.ignore_file_patterns }}" >> $GITHUB_OUTPUT
+        fi
+
+    - name: Check URLs in selected files
+      run: |
+        for f in ${{ steps.file_list.outputs.files }}; do
+            if [ "${f##*.}" != "md" ]; then
+                continue
+            fi
+            for ef in ${{ steps.file_list.outputs.ignore_file_patterns }}; do
+                if [ "$ef" = "$f" ]; then
+                    continue 2 # ignore this file
+                fi
+                fd=$(echo $f | cut -d'/' -f1)
+                if [ "$ef" = "$fd" ]; then
+                    continue 2 # ignore this dir
+                fi
+            done
+            linkchecker -f utils/LinkChecker/.linkcheckerrc file://$(pwd)/$f >> linkchecker.out || true
+            cat linkchecker.out >> linkchecker-all.out
+        done
+
+    - name: Process log
+      run: |
+        python utils/LinkChecker/cklcresults.py ${{ github.event_name }}
+
+    - name: Upload artifact
+      if: ${{ github.event_name == 'pull_request' }}
+      uses: actions/upload-artifact@v4
+      with:
+        name: bad-links
+        path: bad_links.txt
+
+    - name: Update link logs
+      if: ${{ github.event_name != 'pull_request' }}
+      run: |
+        git commit -m 'Update link logs'
+        git push origin sched-link-checks
+
+#
+# Keep the recurring failures and definitely bad lists in repo on
+# branch manage-broken-links
+#
+# Download those files before startin
+#
+# If a link "works" (200) remove it from "recurring failures" list
+# If a link "does not work" (!= 200)
+#    - if it is already on recurring failures list
+#       - if it is too old, flag it as "definitely bad", else nothing
+#       
+#    - if it is not already on persistent failures list, add it to "new" and "persistent failures" list and date it
+
+# 
+# Upload the recurring failures and definitely bad lists to somehwere
+# Report success if definitely bad list is empty, otherwise failure
+# generate email with links or actual data
+#
+# you have to use file:// on command-line to checker
+
+# bare URLs in markdown are not actually links and will not be checked. Many
+# markdown renderers and browsers will recognize these and handle them as links
+# but that is by convention only. There is no markdown standard for how bare
+# URLs in markdown are handled. The only standard is to enclose them in `<` and
+# `>` chars.
+
+#
+# Description:
+#
+# Triggers in one of three ways; 1) manually, 2) scheduled weekly Sunday's 5:17 AM
+# or 3) pull request
+#
+# Stores ignore pattern cases in env. variables and then reformats those (because
+# they have newlines) into a comma-separated single line string that can be digested
+# as inputs to other actions.
+#
+# For PRs, uses changed-files action to get list of changed files and passes this
+# to urlchecker via `include_files param. Also, ignore file patterns is set to 
+# empty string for PRs because we think URLs anywhere in PRs should be checked.
+#
+# For scheduled or manual triggers, uses fact that empty `include_files` param
+# causes urlchecker to process *all* files that match in `file_type` param but do
+# not match any `exclude_files` patterns. These file patterns for exclude work
+# more or less like file globs. So, specifying the initial part of the string
+# for a file (path) name is sufficient to ignore the file.
+#
+# We include Events in file patterns to ignore because of all content we host,
+# we suspect Event URLs are the most likely to go stale rather quickly **and** because
+# the URL validness is important only during the short window prior to the event.
+# That said, we don't want to ignore Events in PRs and we do not as per above.
+#

Articles/Blog/2020-01-usrse.md

@@ -31,9 +31,7 @@ fundamental part of their work.
 In the United States and globally, over the past two to three years (except the UK, which seems to be
 a few years ahead of the trend), the use of the term RSE has exploded. The RSE
 role was formally recognized in 2019 by the European Commission in a  report
-“[Recognising the Importance of Software in Research - Research Software
-Engineers (RSEs), a UK
-Example](https://ec.europa.eu/info/sites/info/files/research_and_innovation/importance_of_software_in_research.pdf)”
+“[Recognising the Importance of Software in Research - Research Software Engineers (RSEs), a UK Example](https://op.europa.eu/s/yA6b)”
 and by  UNESCO in the report “[Paris Call: Software Source Code as Heritage
 for Sustainable
 Development](https://en.unesco.org/foss/paris-call-software-source-code)”. As an 

Articles/Blog/2020-11-PSIP4HDF5.md

@@ -55,7 +55,7 @@ In April of 2020, the
 [LLNL WSC](https://wci.llnl.gov/about-us/weapon-simulation-and-computing) Research Coordination Council
 established a $150K contract with [The HDF Group](https://www.hdfgroup.org)
 (THG), the organization responsible for developing and maintaining
-[HDF5](https://portal.hdfgroup.org/display/HDF5/HDF5), to employ PSIP to effect some
+[HDF5](https://docs.hdfgroup.org/hdf5/develop/index.html), to employ PSIP to effect some
 software process improvement goals (IGs). Interested readers are welcome to read the
 [full report](https://www.osti.gov/biblio/1698291-psip-hdf5pilot-project-final-report).
 
@@ -101,7 +101,7 @@ and create PTCs:
 * **Improvement Goal #3 (IG3)**: Define and adopt a set of consistent coding standards ([PTC](https://bssw-psip.github.io/ptc-catalog/catalog/THGCodingStandards.html))
 
 THG staff members then self-organized around the IG(s) they were passionate about drafting and refining
-PTCs (see images below). Developing quality [PTCs](https://bssw-psip.github.io/practice-guides/how_to_create_ptc.html)
+PTCs (see images below). Developing quality [PTCs](https://bssw-psip.github.io/practice-guides/pages/how_to_create_ptc.html)
 is essential but also nontrivial.
 It can require facilitation by an experienced but objective third party. The challenge is in ensuring
 that each step has a good definition of "done" and also represents a value-added improvement in

Articles/Blog/2021-09-CollegevilleReportDay1.md

@@ -6,7 +6,7 @@
  - <img src='https://github.com/betterscientificsoftware/bssw.io/raw/main/images/Blog_2109_Collegeville1.png' />
  <!-- will updated hero image -->
 
-#### Contributed by: [Cody Balos](https://github.com/balos1), [Jed Brown](https://github.com/jedbrown), [Gerasimos Chourdakis](https://github.com/MakisH), [Ben Cowan](https://github.com/benc303), [Anshu Dubey](https://github.com/adubey64), [Vadim Dyadechko](https://github.com/vdayadechko), [Robert Jacob](https://github.com/rljacob), [Sarah Knepper](https://github.com/sknepper), [Jay Lofstead](https://github.com/gflofst), [Lois Curfman McInnes](https://github.com/curfman), [Reed Milewicz](https://github.com/rmmilewi), [Jacob Moxley](https://github.com/jmox0351), [Todd Munson](https://github.com/tmunson), [Sarah Osborn](https://github.com/osborn9), [Jim Pivarski](https://github.com/jpivarski), [Elaine Raybourn](https://github.com/elainraybourn), [Barry Smith](https://github.com/BarrySmith), [James Willenbring](https://github.com/jwillenbring), [Ulrike Meier Yang](https://github.com/ulrikeyang), [Sam Yates](https://github.com/halfflat), [Michael A. Heroux](https://github.com/maherou), and [Johanna Cohoon](https://github.com/jlcohoon)
+#### Contributed by: [Cody Balos](https://github.com/balos1), [Jed Brown](https://github.com/jedbrown), [Gerasimos Chourdakis](https://github.com/MakisH), [Ben Cowan](https://github.com/benc303), [Anshu Dubey](https://github.com/adubey64), [Vadim Dyadechko](https://github.com/vdyadechko), [Robert Jacob](https://github.com/rljacob), [Sarah Knepper](https://github.com/sknepper), [Jay Lofstead](https://github.com/gflofst), [Lois Curfman McInnes](https://github.com/curfman), [Reed Milewicz](https://github.com/rmmilewi), [Jacob Moxley](https://github.com/jmox0351), [Todd Munson](https://github.com/tmunson), [Sarah Osborn](https://github.com/osborn9), [Jim Pivarski](https://github.com/jpivarski), [Elaine Raybourn](https://github.com/elaineraybourn), [Barry Smith](https://github.com/BarrySmith), [James Willenbring](https://github.com/jwillenbring), [Ulrike Meier Yang](https://github.com/ulrikeyang), [Sam Yates](https://github.com/halfflat), [Michael A. Heroux](https://github.com/maherou), and [Johanna Cohoon](https://github.com/jlcohoon)
 
 #### Publication date: September 20, 2021
 

Articles/Blog/2021-12-sc21-swe-cse-bof.md

@@ -4,7 +4,7 @@
 
  - <img src='https://github.com/betterscientificsoftware/bssw.io/raw/main/images/Blog_2112_SC21.png' />
 
-#### Contributed by: [David E. Bernholdt](https://github.com/bernhold), [Michael Bader](https://github.com/baderml), Michelle Barker, Ben Brown, [Anshu Dubey](https://github.com/adubey64), [Nasir Eisty](https://github.com/neistyS), [Sandra Gesing](https://github.com/sandragesing), [Patricia Grubel](https://github.com/pagrubel), [Rinku Gupta](https://github.com/rinkug), [Michael A. Heroux](https://github.com/maherou), [Saswata Hier-Majumder](https://github.com/sashgeophysics), [Axel Huebl](https://github.com/ax3l), [Mozhgan Kabiri Chimeh](https://github.com/mozhgan-kch), [Daniel S. Katz](https://github.com/danielskatz), [Tomislav Maric](https://github.com/tmaric), [Lois Curfman McInnes](https://github.com/curfman), Bill Miller, [Manish Parashar](https://github.com/parasharmanish), [Ulf D. Schiller](https://github.com/uschille), [Jean Sexton](https://github.com/jmsexton03), [Peter Vaillancourt](https://github.com/sk8forether), Marion Weinzierl, [Yo Yehudi](https://github.com/yochannah)
+#### Contributed by: [David E. Bernholdt](https://github.com/bernhold), [Michael Bader](https://github.com/baderml), Michelle Barker, Ben Brown, [Anshu Dubey](https://github.com/adubey64), [Nasir Eisty](https://github.com/neisty), [Sandra Gesing](https://github.com/sandragesing), [Patricia Grubel](https://github.com/pagrubel), [Rinku Gupta](https://github.com/rinkug), [Michael A. Heroux](https://github.com/maherou), [Saswata Hier-Majumder](https://github.com/sashgeophysics), [Axel Huebl](https://github.com/ax3l), [Mozhgan Kabiri Chimeh](https://github.com/mozhgan-kch), [Daniel S. Katz](https://github.com/danielskatz), [Tomislav Maric](https://github.com/tmaric), [Lois Curfman McInnes](https://github.com/curfman), Bill Miller, [Manish Parashar](https://github.com/parasharmanish), [Ulf D. Schiller](https://github.com/uschille), [Jean Sexton](https://github.com/jmsexton03), [Peter Vaillancourt](https://github.com/sk8forether), Marion Weinzierl, [Yo Yehudi](https://github.com/yochannah)
 
 #### Publication date: December 21, 2021
 
@@ -16,12 +16,12 @@ We’ve settled on a format for the BoF that includes  3-minute lightning talks,
 
 We had six lightning talks this year, covering a wide range of topics with speakers from the US, UK, Germany, and Australia. 
 
-* [Ecosystems are the Future!](https://betterscientificsoftware.github.io/swe-cse-bof/2021-11-sc21-bof/01-brown-ecosystems.pdf) by Benjamin Brown (Office of Advanced Scientific Computing Research (ASCR), Office of Science, U.S. Dept. of Energy).  Ben discussed the importance of ecosystems in his vision for the future of the ASCR high performance computing and networking user facilities ([ALCF](https://www.alcf.anl.gov/), [NERSC](https://www.nersc.gov/), [OLCF](https://www.olcf.ornl.gov/), and[ ESnet](https://www.es.net/)) and highlighted scientific software as one of the key ecosystems.  He closed with the message “Software ecosystems are research infrastructure!”
+* [Ecosystems are the Future!](https://betterscientificsoftware.github.io/swe-cse-bof/assets/2021-11-sc21-bof/01-brown-ecosystems.pdf) by Benjamin Brown (Office of Advanced Scientific Computing Research (ASCR), Office of Science, U.S. Dept. of Energy).  Ben discussed the importance of ecosystems in his vision for the future of the ASCR high performance computing and networking user facilities ([ALCF](https://www.alcf.anl.gov/), [NERSC](https://www.nersc.gov/), [OLCF](https://www.olcf.ornl.gov/), and [ESnet](https://www.es.net/)) and highlighted scientific software as one of the key ecosystems.  He closed with the message “Software ecosystems are research infrastructure!”
 * [Open Source for Researchers](https://doi.org/10.5281/zenodo.5655022) by Yo Yehudi (Wellcome Trust). Yo encouraged us to up our game for our open source software projects, making them more open and more accessible to others.  She touched on readme files, roadmaps, contributor guides, codes of conduct, a requested citation, contact information, and using an issue tracker and also suggested some resources to help folks get started contributing to open source software.
-* [The Internat. CSE Master Program at TUM](https://betterscientificsoftware.github.io/swe-cse-bof/2021-11-sc21-bof/03-bader-masterprogram.pdf) by Michael Bader (Technical University of Munich, TUM).  Michael described the International Master’s program in Computational Science and Engineering, which has been offered at TUM since 2001, currently serving approximately 50 students per year.  The 4-semester program combines classes in computer science, numerical analysis, and scientific computing - with one of the key challenges being how to guide students from various backgrounds towards becoming experts in software development for supercomputing applications.
-* [Senior Level RSE career paths (with an s)](https://betterscientificsoftware.github.io/swe-cse-bof/2021-11-sc21-bof/04-katz-seniorrse.pdf) by Daniel S. Katz (University of Illinois at Urbana-Champaign). Dan presented ideas to help define paths for career progression for research software engineers (RSEs), looking particularly at offering a richer set of opportunities at senior levels to allow RSEs to explore different roles, emphasizing different skills.
-* [FAIR 4 Research Software (FAIR4RS)](https://betterscientificsoftware.github.io/swe-cse-bof/2021-11-sc21-bof/05-barker-fair4rs.pdf) by Michelle Barker (Research Software Alliance). Michelle presented the emerging idea of applying the FAIR principles (findability, accessibility, interoperability, reusability) to research software, noting “Software is not just another type of data.”  The Research Data Alliance, FORCE11, and the Research Software Alliance are working together to develop the FAIR4RS principles and guidelines for implementing them.
-* [Highlights from the IEEE CS Ad Hoc Committee on Open Science & Reproducibility](https://betterscientificsoftware.github.io/swe-cse-bof/2021-11-sc21-bof/06-parashar-openscience.pdf) by Manish Parashar (University of Utah). In 2019, the National Academies of Science, Engineering, and Medicine (NASEM) published a report on Reproducibility and Replicability in Science. Manish described work by the IEEE Computer Society, building upon the NASEM report, to develop an action plan to improve and recognize reproducibility in the society’s publications, conferences, and through its technical committees.
+* [The Internat. CSE Master Program at TUM](https://betterscientificsoftware.github.io/swe-cse-bof/assets/2021-11-sc21-bof/03-bader-masterprogram.pdf) by Michael Bader (Technical University of Munich, TUM).  Michael described the International Master’s program in Computational Science and Engineering, which has been offered at TUM since 2001, currently serving approximately 50 students per year.  The 4-semester program combines classes in computer science, numerical analysis, and scientific computing - with one of the key challenges being how to guide students from various backgrounds towards becoming experts in software development for supercomputing applications.
+* [Senior Level RSE career paths (with an s)](https://betterscientificsoftware.github.io/swe-cse-bof/assets/2021-11-sc21-bof/04-katz-seniorrse.pdf) by Daniel S. Katz (University of Illinois at Urbana-Champaign). Dan presented ideas to help define paths for career progression for research software engineers (RSEs), looking particularly at offering a richer set of opportunities at senior levels to allow RSEs to explore different roles, emphasizing different skills.
+* [FAIR 4 Research Software (FAIR4RS)](https://betterscientificsoftware.github.io/swe-cse-bof/assets/2021-11-sc21-bof/05-barker-fair4rs.pdf) by Michelle Barker (Research Software Alliance). Michelle presented the emerging idea of applying the FAIR principles (findability, accessibility, interoperability, reusability) to research software, noting “Software is not just another type of data.”  The Research Data Alliance, FORCE11, and the Research Software Alliance are working together to develop the FAIR4RS principles and guidelines for implementing them.
+* [Highlights from the IEEE CS Ad Hoc Committee on Open Science & Reproducibility](https://betterscientificsoftware.github.io/swe-cse-bof/assets/2021-11-sc21-bof/06-parashar-openscience.pdf) by Manish Parashar (University of Utah). In 2019, the National Academies of Science, Engineering, and Medicine (NASEM) published a report on Reproducibility and Replicability in Science. Manish described work by the IEEE Computer Society, building upon the NASEM report, to develop an action plan to improve and recognize reproducibility in the society’s publications, conferences, and through its technical committees.
 
 ### Breakout discussions