some updates on the suite

Docs/UsersGuide/Regression/test_suite.tex

@@ -26,66 +26,68 @@ \section{Overview}
 \begin{figure}[t]
 \centering
 \includegraphics[width=5.0in]{testsuite}
-\caption{\label{fig:test_suite_main} Main test suite results page.  Each 
+\caption{\label{fig:test_suite_main} Main test suite results page.  Each
 row indicates a single test suite run, arranged by date, and each column
 indicates a different test problem. }
 \end{figure}
 
 \section{Test Suite Inputs File}
 
-The inputs file for the test suite separates the problems into blocks.
-The header of a problem block has the form {\tt [problem-name]}.
-Beneath each problem block, there are a number of options set for each
-problem.  A separate heading, {\tt [main]}, is used for the suite-wide
-options.
+The inputs file for the test suite separates the problems into
+sections (blocks).  There are several required sections: {\tt main},
+{\tt BoxLib}, and {\tt source}, along with at least one problem
+definition.  Some applications use source from multiple repositories,
+so extra source repos are named {\tt extra-}{\ttfamily {\em sourcename}}.
+
+Each of these section names appears in a heading delineated with {\tt [ ]}.
+Beneath the heading a number of options that apply to that section.
 
 \subsection{Suite-wide options}
 
 An example of the {\tt [main]} block from {\tt Maestro-tests.ini} is:
 \begin{lstlisting}
 [main]
-boxLibDir      = /home/zingale/testing/BoxLib/
-sourceDir      = /home/zingale/testing/MAESTRO/
-testTopDir     = /home/zingale/testing/
-webTopDir      = /home/www/Maestro/test-suite/
+testTopDir     = /home/zingale/gfortran-testing/
+webTopDir      = /home/www/Maestro/test-suite/gfortran-suite/
 
 sourceTree = F_Src
-FCOMP = gfortran
-numMakeJobs = 4
 
 suiteName = Maestro
+
 reportActiveTestsOnly = 1
 
-extSrcDir = /home/zingale/testing/AstroDev/
-extSrcCompString = ASTRODEV_DIR
+FCOMP = gfortran
+add_to_f_make_command = TEST=t FSANITIZER=t
 
-extraBuildDir = /home/zingale/testing/MAESTRO_Exec/
-extraBuildDirCompString = MAESTRO_HOME
+numMakeJobs = 4
+
+purge_output = 1
 
-# MPIcommand should use the placeholders:
-#   @host@ to indicate where to put the hostname to run on
-#   @nprocs@ to indicate where to put the number of processors
-#   @command@ to indicate where to put the command to run
-#
-# only tests with useMPI = 1 will run in parallel
-# nprocs is problem dependent and specified in the individual problem
-# sections.
+summary_job_info_field1 = Network
+summary_job_info_field2 = EOS
 
 MPIcommand = mpiexec -n @nprocs@ @command@
 
-# force double precision plotfiles regardless of what the inputs file says
 globalAddToExecString = --single_prec_plotfiles F
 \end{lstlisting}
 
-The first group of options define the necessary paths:
+The first group of options define the directories that the test suite
+uses:
 \begin{itemize}
-\item {\tt boxLibDir}: the location of the \boxlib\ source
-\item {\tt sourceDir}: the top-level source directory (for \maestro\ in this case)
 \item {\tt testTopDir}: the directory that the suite should use as its
   root directory for running the tests
 \item {\tt webTopDir}: the directory in which to place the web output.
 \end{itemize}
 
+For output, some descriptive options come next:
+\begin{itemize}
+\item {\tt suiteName}: the prefix to use on output directories---it
+  does not need to match the program name
+\item {\tt reportActiveTestsOnly}: whether to show all the tests on
+  the web output known  to the suite throughout its history, or
+  only the ones that are currently defined in the input file.
+\end{itemize}
+
 The source tree for \boxlib\ (Fortran or C++) and compilers to use are set
 as:
 \begin{itemize}
@@ -97,40 +99,13 @@ \subsection{Suite-wide options}
   compiler stays consistent in the tests.
 \item {\tt COMP}: sets the C++ compiler to use---only needed for C++
   \boxlib\ codes
+\item {\tt add\_to\_f\_make\_command}: this is an options to the make
+  command we wanted added for every test.  Here we turn on the
+  \boxlib\ {\tt TEST} features.
 \item {\tt numMakeJobs}: tells the suite how many parallel build
   jobs to do at once (through the {\tt make -j N} command)
 \end{itemize}
 
-For output, some descriptive options come next:
-\begin{itemize}
-\item {\tt suiteName}: the prefix to use on output directories---it
-  does not need to match the program name
-\item {\tt reportActiveTestsOnly}: whether to show all the tests on
-  the web output known  to the suite throughout its history, or
-  only the ones that are currently defined in the input file.
-\end{itemize}
-
-Some codes use a collection of git repos to manage the code, and
-require additional source repos or build directories.  These
-options allow for that:
-\begin{itemize}
-\item {\tt extSrcDir}: an additional git repo containing source code
-  that should be compiled
-\item {\tt extStrCompString}: the name of an environment variable to
-  set on the make line to point to the source.  Here, by setting
-  this to {\tt ASTRODEV\_DIR}, the {\tt make} command will have
-  {\tt ASTRODEV\_DIR=\itshape{extSrcDir}} included, substituting
-  in the value of {\tt extSrcDir}
-\item {\tt extraBuildDir}: an additional git repo that has problem
-  directories we will compile in
-\item {\tt extraBuildDirCompString}: since we are no longer compiling
-  in the {\tt sourceDir}, we need to tell {\tt Make} where to find the
-  original source.  We set this to the environment variable the {\tt
-    GNUmakefile} uses to refer to the main source directory.  Here,
-  setting this to {\tt MAESTRO\_HOME} will add {\tt
-    MAESTRO\_HOME=\itshape{sourceDir}} to the {\tt make} command
-\end{itemize}
-
 Finally, there are several parameters that describe how to run the code
 \begin{itemize}
 \item {\tt MPIcommand}: specify the generic manner in which to run an
@@ -150,8 +125,47 @@ \subsection{Suite-wide options}
 \end{itemize}
 
 There are a few additionally options, including those for sending
-e-mail upon test completion---see {\tt Castro-tests.ini} for these.
+e-mail upon test completion or posting to a slack channel---see the full {\tt Maestro-tests.ini} for these.
+
+\subsection{Sources}
+
+The test suite expects that all the source code is in a git repo (or
+repos).  The most important section is {\tt [BoxLib]}:
+\begin{lstlisting}
+[BoxLib]
+dir = /home/zingale/gfortran-testing/BoxLib/
+branch = development
+\end{lstlisting}
+This simply sets what directory to use for BoxLib and what git branch.
+For applications, we will also have a main source---this will be the
+default location that the suite looks to for building tests:
+\begin{lstlisting}
+[source]
+dir = /home/zingale/gfortran-testing/MAESTRO/
+branch = development
+\end{lstlisting}
+Any additional git repos that are needed can also be specified, e.g.,
+{\sf Maestro} needs an additional repository called {\tt
+  Microphysics}:
+\begin{lstlisting}
+[extra-Microphysics]
+dir = /home/zingale/gfortran-testing/Microphysics/
+branch = "development"
+comp_string = MICROPHYSICS_HOME=@self@
+\end{lstlisting}
+Here we see an additional option: {\tt comp\_string} will always be
+added to the make line, and in this case will set the variable {\tt
+  MICROPHYSICS\_HOME} to the {\tt dir} listed in this section.
+
+If an extra directory should also be used as a place to build tests,
+then you need to supply the option {\tt build=1} in the section.
 
+When the test suite is run, it will do a pull in each of the git repos
+defined in the source sections and then checkout the requested branch.
+It will also compare the branch name against a default branch (which
+defaults to {\tt "development"}, and if any branch is not the default,
+the master test webpage will put an asterisk next to the date to
+indicate that it used an alternate branch.
 
 \subsection{Problem-specific options}
 
@@ -167,7 +181,7 @@ \subsection{Problem-specific options}
 dim = 2
 doVis = 1
 visVar = "tfromp"
-compileTest = 0 
+compileTest = 0
 restartTest = 0
 useMPI = 1
 numprocs = 4
@@ -231,7 +245,7 @@ \subsection{Problem-specific options}
   to the test suite webpage.  You need to set {\tt visVar} to the name
   of the plotfile variable to visualize.  An image of that field from
   the last plotfile will be appended to the problem's test webpage.
-  
+
 \item {\tt compareFile}: Ordinarily, the test suite uses the last
   plotfile output to compare to.  To force the comparison to a
   specific file, set {\tt compareFile} to the name of the file to
@@ -243,7 +257,14 @@ \subsection{Problem-specific options}
   string to add to the {\tt make} line that compiles the problem.
 \end{itemize}
 
-
+If the test requires the source to be build in a directory other 
+than that referred to by the {\tt [source]} section, then you
+need to add the option
+\begin{lstlisting}
+extra_build_dir = name
+\end{lstlisting}
+where {\tt name} is the part of the section {\tt [extra-name]} 
+that defines the repo.
 
 \section{Initializing the Test Suite}
 
@@ -325,9 +346,7 @@ \section{Updating Benchmarks}
 update.  Subsequent runs of the test suite will use the new benchmarks.
 If you only want to update the benchmarks of a single test, then you
 can use the ``{\tt --single\_test test}'' flag on the commandline, where
-{\tt test} is the name of the test to update.  
+{\tt test} is the name of the test to update.
 
 Finally, if you just want to rerun the tests the previously failed,
 you can use the {\tt --redo\_failed} option.
-
-