Skip to content
This repository has been archived by the owner on Aug 26, 2020. It is now read-only.

Commit

Permalink
Merged develop for release 0.9
Browse files Browse the repository at this point in the history
Release 0.9
  • Loading branch information
luisfabib authored May 7, 2020
2 parents 846abe2 + 69ff9a8 commit 03fed72
Show file tree
Hide file tree
Showing 787 changed files with 37,178 additions and 67,299 deletions.
27 changes: 15 additions & 12 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,13 @@ on:
branches:
- master
- release
pull_request:
branches:
- master

jobs:
build:
name: CI
name: CI (tests)
runs-on: ubuntu-latest
steps:
- name: Checkout trigger commit
Expand Down Expand Up @@ -37,35 +40,35 @@ jobs:
- name: Transfer source files to CI-server
if: success()
run: |
scp -o StrictHostKeyChecking=no -i ~/sshkey -P ${{ secrets.GATEWAY_PORT }} -r ../DeerAnalysis2 ${{ secrets.GATEWAY_HOST }}:/home/lufa/scp_traffic/
ssh -o StrictHostKeyChecking=no -i ~/sshkey -p ${{ secrets.GATEWAY_PORT }} ${{ secrets.GATEWAY_HOST }} scp -r /home/lufa/scp_traffic/DeerAnalysis2 ${{ secrets.CISERVER_HOST }}:/home/deeranalysis/
ssh -o StrictHostKeyChecking=no -i ~/sshkey -p ${{ secrets.GATEWAY_PORT }} ${{ secrets.GATEWAY_HOST }} "rm -f -r /home/lufa/scp_traffic/DeerAnalysis2"
scp -o StrictHostKeyChecking=no -i ~/sshkey -P ${{ secrets.GATEWAY_PORT }} -r ../DeerLab ${{ secrets.GATEWAY_HOST }}:/home/lufa/scp_traffic/
ssh -o StrictHostKeyChecking=no -i ~/sshkey -p ${{ secrets.GATEWAY_PORT }} ${{ secrets.GATEWAY_HOST }} scp -r /home/lufa/scp_traffic/DeerLab ${{ secrets.CISERVER_HOST }}:~
ssh -o StrictHostKeyChecking=no -i ~/sshkey -p ${{ secrets.GATEWAY_PORT }} ${{ secrets.GATEWAY_HOST }} "rm -f -r /home/lufa/scp_traffic/DeerLab"
- name: Run test suite connected to CI-Server
if: success()
run: |
ssh -tt -o StrictHostKeyChecking=no -i ~/sshkey -p ${{ secrets.GATEWAY_PORT }} -t ${{ secrets.GATEWAY_HOST }} ssh ${{ secrets.CISERVER_HOST }} << EOF
#Run test suite
cd DeerAnalysis2/build
matlab -nodesktop -r "datestsuite;exit"
cd DeerLab/build
matlab -nodesktop -r "dltestsuite;exit"
python3 uploadS3.py --keyfile ~/.ssh/aws_access_keys.txt --file "coverage_badge.json" --bucket deershields
python3 uploadS3.py --keyfile ~/.ssh/aws_access_keys.txt --file "testsuite_badge.json" --bucket deershields
#Transfer log file to the gateway
scp -o StrictHostKeyChecking=no -i ~/.ssh/gateway_key -P ${{ secrets.GATEWAY_PORT }} ./datestsuite.error ${{ secrets.GATEWAY_HOST }}:/home/lufa/scp_traffic/ || true
scp -o StrictHostKeyChecking=no -i ~/.ssh/gateway_key -P ${{ secrets.GATEWAY_PORT }} ./dltestsuite.error ${{ secrets.GATEWAY_HOST }}:/home/lufa/scp_traffic/ || true
#Delete the source files
cd /home/deeranalysis/
rm -f -r DeerAnalysis2
cd ~
rm -f -r DeerLab
EOF
- name: Disconnect CI-server
if: success()
run: |
#Transfer log file to GitHub VM
scp -o StrictHostKeyChecking=no -i ~/sshkey -P ${{ secrets.GATEWAY_PORT }} ${{ secrets.GATEWAY_HOST }}:/home/lufa/scp_traffic/datestsuite.error ~/ || true
ssh -o StrictHostKeyChecking=no -i ~/sshkey -p ${{ secrets.GATEWAY_PORT }} ${{ secrets.GATEWAY_HOST }} "rm -f /home/lufa/scp_traffic/datestsuite.error" || true
scp -o StrictHostKeyChecking=no -i ~/sshkey -P ${{ secrets.GATEWAY_PORT }} ${{ secrets.GATEWAY_HOST }}:/home/lufa/scp_traffic/dltestsuite.error ~/ || true
ssh -o StrictHostKeyChecking=no -i ~/sshkey -p ${{ secrets.GATEWAY_PORT }} ${{ secrets.GATEWAY_HOST }} "rm -f /home/lufa/scp_traffic/dltestsuite.error" || true
rm -f ~/sshkey
- name: All tests passed successfully
if: success()
run: |
! [ -f ~/datestsuite.error ] && echo "Test suite passed" || (echo "Test suite failed" && deerfailed)
! [ -f ~/dltestsuite.error ] && echo "Test suite passed" || (echo "Test suite failed" && deerfailed)
- name: Disconnect VPN connection
if: always()
run: |
Expand Down
77 changes: 77 additions & 0 deletions .github/workflows/ci_dev.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
name: Scheduled Continuous Integration

on:
schedule:
# Run every day at 6:30 AM
- cron: '30 6 * * *'

jobs:
build:
name: Scheduled CI (dev)
runs-on: ubuntu-latest
steps:
- name: Checkout trigger commit
uses: actions/checkout@v2
with:
ref: develop
- name: Install dependencies
run: |
sudo apt-get install -y openvpn openconnect
- name: Prepare VM
if: success()
run: |
rm -f -r ./.git
echo "${{ secrets.GATEWAY_KEY }}" > ~/sshkey
sudo chmod 600 ~/sshkey
- name: Establish VPN connection
if: success()
run: |
# Force returning packets to be routed over public interface and not over the VPN
sudo ip rule add from $(ip route get 1 | grep -Po '(?<=src )(\S+)') table 128
sudo ip route add table 128 to $(ip route get 1 | grep -Po '(?<=src )(\S+)')/32 dev $(ip -4 route ls | grep default | grep -Po '(?<=dev )(\S+)')
sudo ip route add table 128 default via $(ip -4 route ls | grep default | grep -Po '(?<=via )(\S+)')
# Start network tunnel (TUN) device
sudo openvpn --mktun --dev tun1
sudo ifconfig tun1 up
# Launch VPN connection
echo "${{ secrets.VPN_PASSWD }}" | sudo openconnect ${{ secrets.VPN_SERVER }} --user=${{ secrets.VPN_USER }} --background --passwd-on-stdin
- name: Transfer source files to CI-server
if: success()
run: |
scp -o StrictHostKeyChecking=no -i ~/sshkey -P ${{ secrets.GATEWAY_PORT }} -r ../DeerLab ${{ secrets.GATEWAY_HOST }}:/home/lufa/scp_traffic/
ssh -o StrictHostKeyChecking=no -i ~/sshkey -p ${{ secrets.GATEWAY_PORT }} ${{ secrets.GATEWAY_HOST }} scp -r /home/lufa/scp_traffic/DeerLab ${{ secrets.CISERVER_HOST }}:~
ssh -o StrictHostKeyChecking=no -i ~/sshkey -p ${{ secrets.GATEWAY_PORT }} ${{ secrets.GATEWAY_HOST }} "rm -f -r /home/lufa/scp_traffic/DeerLab"
- name: Run test suite connected to CI-Server
if: success()
run: |
ssh -tt -o StrictHostKeyChecking=no -i ~/sshkey -p ${{ secrets.GATEWAY_PORT }} -t ${{ secrets.GATEWAY_HOST }} ssh ${{ secrets.CISERVER_HOST }} << EOF
#Run test suite
cd DeerLab/build
matlab -nodesktop -r "dltestsuite;exit"
python3 uploadS3.py --keyfile ~/.ssh/aws_access_keys.txt --file "coverage_badge.json" --bucket deershields
python3 uploadS3.py --keyfile ~/.ssh/aws_access_keys.txt --file "testsuite_badge.json" --bucket deershields
#Transfer log file to the gateway
scp -o StrictHostKeyChecking=no -i ~/.ssh/gateway_key -P ${{ secrets.GATEWAY_PORT }} ./dltestsuite.error ${{ secrets.GATEWAY_HOST }}:/home/lufa/scp_traffic/ || true
#Delete the source files
cd ~
rm -f -r DeerLab
EOF
- name: Disconnect CI-server
if: success()
run: |
#Transfer log file to GitHub VM
scp -o StrictHostKeyChecking=no -i ~/sshkey -P ${{ secrets.GATEWAY_PORT }} ${{ secrets.GATEWAY_HOST }}:/home/lufa/scp_traffic/dltestsuite.error ~/ || true
ssh -o StrictHostKeyChecking=no -i ~/sshkey -p ${{ secrets.GATEWAY_PORT }} ${{ secrets.GATEWAY_HOST }} "rm -f /home/lufa/scp_traffic/dltestsuite.error" || true
rm -f ~/sshkey
- name: All tests passed successfully
if: success()
run: |
! [ -f ~/dltestsuite.error ] && echo "Test suite passed" || (echo "Test suite failed" && deerfailed)
- name: Disconnect VPN connection
if: always()
run: |
#Ensure the VPN is disconnected by killing openconnect
sudo pkill openconnect
#Remove TUN device
sudo ifconfig tun1 down
sudo openvpn --rmtun --dev tun1
Original file line number Diff line number Diff line change
@@ -1,41 +1,47 @@
name: Webpage update

on:
push:
branches:
- master
paths:
- 'docsrc/**'
- '.github/**'

jobs:

build:
name: Build
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v1
- name: Set up Python 3.7
uses: actions/setup-python@v1
with:
python-version: 3.7
- name: Install dependencies
run: |
python -m pip install --upgrade pip
python -m pip install sphinx==1.8.0
python -m pip install sphinx_rtd_theme
python -m pip install sphinxcontrib-matlabdomain
python -m pip install sphinxcontrib-httpdomain
python -m pip install boto3
sudo apt install texlive-extra-utils
sudo apt-get install texlive-latex-extra
- name: Run Sphinx Builder
run: |
cd docsrc
sphinx-build -E -b html ./source ../docs
- name: Connect and update DeerAnalysis AWS S3 Bucket
run: |
cd build
python -m synchS3 -k ${{ secrets.S3_ACCESS }} -s ${{ secrets.S3_SECRETACCESS }} -d "../docs" --bucket deeranalysis.org
name: Docs Build & Deployment

on:
push:
branches:
- master
- hotfixes
paths:
- 'build/**'
- 'docsrc/**'

schedule:
# Run once a week on Sunday at 12:00 PM
- cron: '0 12 * * 0'

jobs:

deploy:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v1
- name: Set up Python 3.7
uses: actions/setup-python@v1
with:
python-version: 3.7
- name: Install dependencies
run: |
python -m pip install --upgrade pip
python -m pip install sphinx==1.8.0
python -m pip install sphinx_rtd_theme
python -m pip install sphinxcontrib-matlabdomain
python -m pip install sphinxcontrib-httpdomain
sudo apt install texlive-extra-utils
sudo apt-get install texlive-latex-extra
- name: Build multi-version docs
run: |
cd build
python -m multiversedocs
cd ..
- name: Deploy to GH-Pages
uses: peaceiris/actions-gh-pages@v3
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
publish_dir: ./multidocs
7 changes: 5 additions & 2 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,9 @@ docsrc/aws_access_keys.txt
docsrc/__pycache__/
build/.ssh/
tutorials/**/*.pdf
tutorials/**/*.m
tutorials/**/*.mlx
tests/_tutorials
tests/_tutorials/**
tests/_tutorials/**
build/__pycache__/multiversedocs.cpython-37.pyc
multidocs/**/*
build/__pycache__/multiversedocs_testing.cpython-37.pyc
32 changes: 16 additions & 16 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,29 +1,28 @@

<p align="center">
<img src="./docsrc/source/logo_dark.png" alt="DeerLab Logo" width="60%"></img>
<img src="./docsrc/source/_static/logo_dark.png" alt="DeerLab Logo" width="60%"></img>
</p>
</div>

<p align="center">
<img src="https://img.shields.io/github/issues-raw/luisfabib/deerlab?style=flat"></img>
<img src="https://img.shields.io/badge/MATLAB-R2016b--R2019b-brightgreen?style=flat"></img>
<img src="https://img.shields.io/github/downloads/luisfabib/deerlab/total?style=flat"></img>
<img src="https://github.com/luisfabib/deerlab/workflows/Webpage%20update/badge.svg?style=flat-square"></img>
<img src="https://img.shields.io/github/issues-raw/JeschkeLab/DeerLab?style=flat"></img>
<img src="https://img.shields.io/badge/MATLAB-R2017a--R2020a-brightgreen?style=flat"></img>
<img src="https://img.shields.io/github/downloads/JeschkeLab/DeerLab/total?style=flat"></img>
<img src="https://img.shields.io/endpoint?url=https%3A%2F%2Fdeershields.s3.eu-central-1.amazonaws.com%2Fcoverage_badge.json"></img>
<img src="https://img.shields.io/endpoint?url=https%3A%2F%2Fdeershields.s3.eu-central-1.amazonaws.com%2Ftestsuite_badge.json"></img>
</p>

### About
The DeerLab software package is a MATLAB toolbox for the analysis of data from DEER (double electron-electron resonance) spectroscopy and similar dipolar spectroscopy techniques (DQC, RIDME, SIFTER). The main homepage can be found at www.deeranalysis.org. This is the GitHub repository of the DeerLab source code, including instructions for compiling and installing DeerLab.
The DeerLab software package is a MATLAB toolbox for the analysis of data from DEER (double electron-electron resonance) spectroscopy and similar dipolar EPR spectroscopy techniques (RIDME, DQC, SIFTER). The main homepage can be found at [jeschkelab.github.io/DeerLab](https://jeschkelab.github.io/DeerLab/). This is the GitHub repository of the DeerLab source code, including instructions for compiling and installing DeerLab.

It consists of a collection of functions that perform single processing or fitting tasks. They can be combined in scripts to generate custom analysis workflows.
It consists of a collection of functions that perform modelling, processing or fitting tasks. They can be combined in scripts to build custom data analysis workflows.

To model distance distributions, DeerLab supports two types of model classes and associated workflows: parameter-free models (as used in Tikhonov regularization) as well as a series of parameterized models (mutli-Gaussians etc). It also provides a selection of background models. There are functions for generating synthetic datasets as well as for fitting and analyzing experimental data sets.
To model distance distributions, DeerLab supports two types of model classes and associated workflows: parameter-free models (as used in Tikhonov regularization) as well as a series of parameterized models (mutli-Gaussians etc). It also provides a selection of background and experiment models. There are functions for generating synthetic datasets as well as for fitting and analyzing experimental data sets.

### Requirements
The application programming interface (API) of DeerLab requires the following products:
DeerLab requires the following products:

* MATLAB (oldest version supported R2016b) (see <https://ch.mathworks.com/products/matlab.html>)
* MATLAB (R2016b or newer) (see <https://ch.mathworks.com/products/matlab.html>)

Optional functionality may require the following products:

Expand All @@ -48,20 +47,21 @@ In order for MATLAB to access the DeerLab API functions, the path to the DeerLab
2) Add the following lines of code:

addpath('mypath/DeerLab/functions')
addpath('mypath/DeerLab/functions/models')

3) Save ``startup.m`` and restart MATLAB.

### License

The DeerLab package is licensed under the MIT License. The "package" consists of the DOE ([functions/](https://github.com/luisfabib/deerlab/tree/master/functions)), documentation source ([docsrc/](https://github.com/luisfabib/deerlab/tree/master/docsrc)), tutorial scripts ([tutorials/](https://github.com/luisfabib/deerlab/tree/master/tutorials)), test suite ([tests/](https://github.com/luisfabib/deerlab/tree/master/tests)) and pipeline scripts ([.github/workflows](https://github.com/luisfabib/deerlab/tree/master/.github/workflows)). See below for exceptions.

Copyright (c) 2019: Luis Fabregas, Stefan Stoll, Gunnar Jeschke, and [other contributors](https://github.com/luisfabib/deerlab/contributors).
The DeerLab toolbox is licensed under the MIT License. The complete toolbox consists of the functions ([functions/](https://github.com/JeschkeLab/DeerLab/tree/master/functions)), documentation source ([docsrc/](https://github.com/JeschkeLab/DeerLab/tree/master/docsrc)), tutorial scripts ([tutorials/](https://github.com/JeschkeLab/DeerLab/tree/master/tutorials)), test suite ([tests/](https://github.com/JeschkeLab/DeerLab/tree/master/tests)), and pipeline scripts ([.github/workflows](https://github.com/JeschkeLab/DeerLab/tree/master/.github/workflows)). See below for exceptions.

DeerLab includes code from the following projects, which have their own licenses:
- [datahash.m](https://www.mathworks.com/matlabcentral/fileexchange/31272-datahash) (Hash-key generator by Jan Simon) [BSD]
- [fresnelS.m, fresnelC.m](https://www.mathworks.com/matlabcentral/fileexchange/28765-fresnels-and-fresnelc) (Efficient and accurate Fresnel integrals by John D'Errico) [BSD]
- [fminsearchcon.m](https://www.mathworks.com/matlabcentral/fileexchange/8277-fminsearchbnd-fminsearchcon) (Bound constrained optimization using fminsearch by John D'Errico) [BSD]
- [nlsqbnd.m](https://www.mathworks.com/matlabcentral/fileexchange/8277-fminsearchbnd-fminsearchcon) (Non-linear least squares solver with box constraints by Alain Barraud) [BSD]

- [nlsqbnd.m](https://ch.mathworks.com/matlabcentral/fileexchange/23621-nlsqbnd) (Non-linear least squares solver with box constraints by Alain Barraud) [BSD]
- [golden.m](https://www.mathworks.com/matlabcentral/fileexchange/25919-golden-section-method-algorithm) (Golden Section method algorithm by Katarzyna Zarnowiec) [BSD]
- [jacobianest.m](https://www.mathworks.com/matlabcentral/fileexchange/13490-adaptive-robust-numerical-differentiation) (Adaptive Robust Numerical Differentiation by John D'Errico) [BSD]
- [kde.m](https://ch.mathworks.com/matlabcentral/fileexchange/14034-kernel-density-estimator) (Kernel Density Estimator by Zdravko Botev) [BSD]
- [LevenbergMarquardt.m, jacobiansimple.m](https://ch.mathworks.com/matlabcentral/fileexchange/53449-levenberg-marquardt-toolbox)(Levenberg-Marquardt & Jacobian toolbox by Alexander Dentler)[BSD]

Copyright (c) 2019-2020: Luis Fabregas, Stefan Stoll, Gunnar Jeschke, and [other contributors](https://github.com/JeschkeLab/DeerLab/contributors).
54 changes: 0 additions & 54 deletions build/compiletutorials.m

This file was deleted.

Loading

0 comments on commit 03fed72

Please sign in to comment.