Skip to content

Commit

Permalink
Update OCCA Backend (#1072)
Browse files Browse the repository at this point in the history
* Update OCCA memory interop call.

* Removes deprecated kernelBuilder calls; builds directly from the device.

* Uses `std::memset` and `std:memcpy`.

* Uses `std::to_string` instead of internal `occa::toString`.

* Uses `std::memcpy`.

* Sets kernel properties.

* Removes deprecated kernelBuilder calls; builds directly from the device.

* Removes deprecated kernelBuild call; builds directly from the device.

* Uses `std::memcpy`.

* Removes deprecated calls to `occa::linalg`.

* Add registration and device configuration for DPC++ and OpenCL backends.

* Add DPC++ and OpenCL backends to makefile.

* Spelling.

* Configure build flags for oneAPI compilers.

* Correctly set mode in `occa::json` object.

* Add missing functions to OCCA CeedVector implementation.

* Adds missing call to `setValueKernel` in OCCA CeedVector impl.

* Gets occa device function name from the CeedQFunction.

* Adds OpenCL and DPC++ to backends list.

* Uses unique kernel name for OCCA qFunction kernels.

* Adds a dummy `ceed.h` header in include in OCCA kernels.

* Rewrite arrays of structs in format that OCCA can handle.

* Adds stubs for missing functions in OCCA qfunctioncontext.

* Includes the cmath header when compiling C++ code.

* Add stubs for missing OCCA backend LinearAssembleXXX functions.

* Adds missing functions to OCCA implemenation of qFunctionContext.

* Removes math function headers which were causing OCCA JIT failures.

* Rewrite arrays of structs in format that OCCA can handle.

* Rewrites fluids example qfunctions to be compatible with OCCA.

* Fixes array dimensions in mass2dbuild.

* Rewrites advection problem kernels to work with OCCA.

* Rewrites blasius problem kernels to work with OCCA.

* Rewrites channel problem kernels to work with OCCA.

* Rewrites dirichlet bc kernels to work with OCCA.

* Rewrites newtonian kernels to work with OCCA.

* Rewrites setupgeo kernels to be compatible with OCCA.

* Rewrites stabilization kernels to be compatible with OCCA.

* Rewrites stg kernels to be compatible with OCCA.

* Adds occa backends to tests for the fluids example.

* doc - update OCCA info in release notes + README

* ci - run with OCCA v1.4

* occa - update copyright boilerplate

* occa - drop unused define

* ci - fix OCCA install

* wip

* ci - fix occa skip list

* make/ci - fix use of OCCA_DIR/bin/occa

* makefile - minor style

* ci - re-enable OCCA dir caching

* doc - update release notes to mention OCCA QF workarounds

Co-authored-by: Kris Rowe <[email protected]>
  • Loading branch information
jeremylt and kris-rowe authored Oct 13, 2022
1 parent 9e201c8 commit 0be03a9
Show file tree
Hide file tree
Showing 56 changed files with 712 additions and 969 deletions.
10 changes: 3 additions & 7 deletions .gitlab-ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -22,20 +22,16 @@ noether-cpu:
- cd .. && export XSMM_HASH=44433be9426eddaed88415646c15b3bcc61afc85 && { [[ -d libxsmm-$XSMM_HASH ]] || { curl -L https://github.com/libxsmm/libxsmm/archive/$XSMM_HASH.tar.gz -o xsmm.tar.gz && tar zvxf xsmm.tar.gz && rm xsmm.tar.gz && make -C libxsmm-$XSMM_HASH -j$(nproc); }; } && export XSMM_DIR=$PWD/libxsmm-$XSMM_HASH && cd libCEED
- echo "-------------- LIBXSMM -------------" && basename $XSMM_DIR
# -- OCCA v1.1.0
- cd .. && export OCCA_VERSION=occa-1.1.0 OCCA_OPENCL_ENABLED=0 && { [[ -d $OCCA_VERSION ]] || { git clone --depth 1 --branch v1.1.0 https://github.com/libocca/occa.git $OCCA_VERSION && sed -i '/sysctl.h/d' $OCCA_VERSION/src/tools/sys.cpp && make -C $OCCA_VERSION -j$(nproc); }; } && export OCCA_DIR=$PWD/$OCCA_VERSION && cd libCEED
- echo "-------------- OCCA ----------------" && make -C $OCCA_DIR info
- cd .. && export OCCA_VERSION=occa-1.4.0 && { [[ -d $OCCA_VERSION ]] || { git clone --depth 1 --branch v1.4.0 https://github.com/libocca/occa.git $OCCA_VERSION && cd $OCCA_VERSION && export ENABLE_OPENCL="OFF" ENABLE_DPCPP="OFF" ENABLE_HIP="OFF" ENABLE_CUDA="OFF" && ./configure-cmake.sh && cmake --build build --parallel $NPROC_CPU && cmake --install build && cd ..; }; } && export OCCA_DIR=$PWD/$OCCA_VERSION/install && cd libCEED
- echo "-------------- OCCA ----------------" && git -C $OCCA_DIR describe --tags && LD_LIBRARY_PATH=$OCCA_DIR/lib $OCCA_DIR/bin/occa info
script:
- rm -f .SUCCESS
# libCEED
- make configure OPT='-O -march=native -ffp-contract=fast'
# Note: OCCA backends currently disabled in CI
- BACKENDS_CPU=$(OCCA_DIR= make info-backends-all | grep -o '/cpu[^ ]*')
- BACKENDS_CPU=$(make info-backends-all | grep -o '/cpu[^ ]*')
- echo "-------------- libCEED -------------" && make info
- echo "-------------- BACKENDS_CPU --------" && echo $BACKENDS_CPU
- make -j$NPROC_CPU
# Remove OCCA after verifying it compiles
# CodeCov of OCCA backend not useful since testing is intentionally disabled
- export OCCA_DIR= && make -j$NPROC_CPU -W build/interface/ceed-register.o
# -- libCEED only tests
- echo "-------------- core tests ----------"
- echo '[{"subject":"/","metrics":[{"name":"Transfer Size (KB)","value":"19.5","desiredSize":"smaller"},{"name":"Speed Index","value":0,"desiredSize":"smaller"},{"name":"Total Score","value":92,"desiredSize":"larger"},{"name":"Requests","value":4,"desiredSize":"smaller"}]}]' > performance.json
Expand Down
19 changes: 13 additions & 6 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -64,11 +64,11 @@ MPI ?= 1
# CEED_DIR env for NEK5K testing
export CEED_DIR = $(abspath .)

# XSMM_DIR env variable should point to XSMM master (github.com/hfp/libxsmm)
# XSMM_DIR env variable should point to XSMM main (github.com/hfp/libxsmm)
XSMM_DIR ?= ../libxsmm

# OCCA_DIR env variable should point to OCCA master (github.com/libocca/occa)
OCCA_DIR ?= ../occa
# OCCA_DIR env variable should point to OCCA main (github.com/libocca/occa)
OCCA_DIR ?= ../occa/install

# env variable MAGMA_DIR can be used too
MAGMA_DIR ?= ../magma
Expand All @@ -92,27 +92,33 @@ AFLAGS = -fsanitize=address #-fsanitize=undefined -fno-omit-frame-pointer

# Note: Intel oneAPI C/C++ compiler is now icx/icpx
CC_VENDOR := $(subst icc_orig,icc,$(subst oneAPI,icc,$(firstword $(filter gcc clang icc icc_orig oneAPI XL,$(subst -, ,$(shell $(CC) --version))))))
FC_VENDOR := $(if $(FC),$(firstword $(filter GNU ifort XL,$(shell $(FC) --version 2>&1 || $(FC) -qversion))))
FC_VENDOR := $(if $(FC),$(firstword $(filter GNU ifort ifx XL,$(shell $(FC) --version 2>&1 || $(FC) -qversion))))

# Default extra flags by vendor
MARCHFLAG.gcc := -march=native
MARCHFLAG.clang := $(MARCHFLAG.gcc)
MARCHFLAG.icc :=
MARCHFLAG.oneAPI := $(MARCHFLAG.clang)
OMP_SIMD_FLAG.gcc := -fopenmp-simd
OMP_SIMD_FLAG.clang := $(OMP_SIMD_FLAG.gcc)
OMP_SIMD_FLAG.icc := -qopenmp-simd
OMP_SIMD_FLAG.oneAPI := $(OMP_SIMD_FLAG.clang)
OPT.gcc := -ffp-contract=fast
OPT.clang := $(OPT.gcc)
OPT.oneAPI := $(OPT.clang)
CFLAGS.gcc := -fPIC -std=c99 -Wall -Wextra -Wno-unused-parameter -MMD -MP
CFLAGS.clang := $(CFLAGS.gcc)
CFLAGS.icc := $(CFLAGS.gcc)
CFLAGS.oneAPI := $(CFLAGS.clang)
CFLAGS.XL := -qpic -MMD
CXXFLAGS.gcc := -fPIC -std=c++11 -Wall -Wextra -Wno-unused-parameter -MMD -MP
CXXFLAGS.clang := $(CXXFLAGS.gcc)
CXXFLAGS.icc := $(CXXFLAGS.gcc)
CXXFLAGS.oneAPI := $(CXXFLAGS.clang)
CXXFLAGS.XL := -qpic -std=c++11 -MMD
FFLAGS.GNU := -fPIC -cpp -Wall -Wextra -Wno-unused-parameter -Wno-unused-dummy-argument -MMD -MP
FFLAGS.ifort := -fPIC -cpp
FFLAGS.ifx := $(FFLAGS.ifort)
FFLAGS.XL := -qpic -ffree-form -qpreprocess -qextname -MMD

# This check works with compilers that use gcc and clang. It fails with some
Expand Down Expand Up @@ -369,9 +375,10 @@ endif
# OCCA Backends
OCCA_BACKENDS = /cpu/self/occa
ifneq ($(wildcard $(OCCA_DIR)/lib/libocca.*),)
OCCA_MODES := $(shell $(OCCA_DIR)/bin/occa modes)
OCCA_MODES := $(shell LD_LIBRARY_PATH=$(OCCA_DIR)/lib $(OCCA_DIR)/bin/occa modes)
OCCA_BACKENDS += $(if $(filter OpenMP,$(OCCA_MODES)),/cpu/openmp/occa)
# OCCA_BACKENDS += $(if $(filter OpenCL,$(OCCA_MODES)),/gpu/opencl/occa)
OCCA_BACKENDS += $(if $(filter dpcpp,$(OCCA_MODES)),/gpu/dpcpp/occa)
OCCA_BACKENDS += $(if $(filter OpenCL,$(OCCA_MODES)),/gpu/opencl/occa)
OCCA_BACKENDS += $(if $(filter HIP,$(OCCA_MODES)),/gpu/hip/occa)
OCCA_BACKENDS += $(if $(filter CUDA,$(OCCA_MODES)),/gpu/cuda/occa)

Expand Down
4 changes: 3 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -158,6 +158,7 @@ There are multiple supported backends, which can be selected at runtime in the e
| `/*/occa` | Selects backend based on available OCCA modes | Yes |
| `/cpu/self/occa` | OCCA backend with serial CPU kernels | Yes |
| `/cpu/openmp/occa` | OCCA backend with OpenMP kernels | Yes |
| `/cpu/dpcpp/occa` | OCCA backend with CPC++ kernels | Yes |
| `/gpu/cuda/occa` | OCCA backend with CUDA kernels | Yes |
| `/gpu/hip/occa`~ | OCCA backend with HIP kernels | Yes |

Expand Down Expand Up @@ -198,8 +199,9 @@ For example:
The `/*/occa` backends rely upon the [OCCA](http://github.com/libocca/occa) package to provide cross platform performance.
To enable the OCCA backend, the environment variable `OCCA_DIR` must point to the top-level OCCA directory, with the OCCA library located in the `${OCCA_DIR}/lib` (By default, `OCCA_DIR` is set to `../occa`).
OCCA version 1.4.0 or newer is required.

Additionally, users can pass specific OCCA device properties after setting the CEED resource.
Users can pass specific OCCA device properties after setting the CEED resource.
For example:

> - `"/*/occa:mode='CUDA',device_id=0"`
Expand Down
2 changes: 1 addition & 1 deletion backends/ceed-backend-list.h
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ MACRO(CeedRegister_Magma, 2, "/gpu/cuda/magma", "/gpu/hip/magma")
MACRO(CeedRegister_Magma_Det, 2, "/gpu/cuda/magma/det", "/gpu/hip/magma/det")
MACRO(CeedRegister_Memcheck_Blocked, 1, "/cpu/self/memcheck/blocked")
MACRO(CeedRegister_Memcheck_Serial, 1, "/cpu/self/memcheck/serial")
MACRO(CeedRegister_Occa, 4, "/cpu/self/occa", "/cpu/openmp/occa", "/gpu/hip/occa", "/gpu/cuda/occa")
MACRO(CeedRegister_Occa, 6, "/cpu/self/occa", "/cpu/openmp/occa", "/gpu/dpcpp/occa", "/gpu/opencl/occa", "/gpu/hip/occa", "/gpu/cuda/occa")
MACRO(CeedRegister_Opt_Blocked, 1, "/cpu/self/opt/blocked")
MACRO(CeedRegister_Opt_Serial, 1, "/cpu/self/opt/serial")
MACRO(CeedRegister_Ref, 1, "/cpu/self/ref/serial")
Expand Down
17 changes: 4 additions & 13 deletions backends/occa/ceed-occa-basis.cpp
Original file line number Diff line number Diff line change
@@ -1,18 +1,9 @@
// Copyright (c) 2019, Lawrence Livermore National Security, LLC.
// Produced at the Lawrence Livermore National Laboratory. LLNL-CODE-734707.
// All Rights reserved. See files LICENSE and NOTICE for details.
// Copyright (c) 2017-2022, Lawrence Livermore National Security, LLC and other CEED contributors.
// All Rights Reserved. See the top-level LICENSE and NOTICE files for details.
//
// This file is part of CEED, a collection of benchmarks, miniapps, software
// libraries and APIs for efficient high-order finite element and spectral
// element discretizations for exascale applications. For more information and
// source code availability see http://github.com/ceed
// SPDX-License-Identifier: BSD-2-Clause
//
// The CEED research is supported by the Exascale Computing Project 17-SC-20-SC,
// a collaborative effort of two U.S. Department of Energy organizations (Office
// of Science and the National Nuclear Security Administration) responsible for
// the planning and preparation of a capable exascale ecosystem, including
// software, applications, hardware, advanced system engineering and early
// testbed platforms, in support of the nation's exascale computing imperative.
// This file is part of CEED: http://github.com/ceed

#include "ceed-occa-basis.hpp"
#include "ceed-occa-tensor-basis.hpp"
Expand Down
17 changes: 4 additions & 13 deletions backends/occa/ceed-occa-basis.hpp
Original file line number Diff line number Diff line change
@@ -1,18 +1,9 @@
// Copyright (c) 2019, Lawrence Livermore National Security, LLC.
// Produced at the Lawrence Livermore National Laboratory. LLNL-CODE-734707.
// All Rights reserved. See files LICENSE and NOTICE for details.
// Copyright (c) 2017-2022, Lawrence Livermore National Security, LLC and other CEED contributors.
// All Rights Reserved. See the top-level LICENSE and NOTICE files for details.
//
// This file is part of CEED, a collection of benchmarks, miniapps, software
// libraries and APIs for efficient high-order finite element and spectral
// element discretizations for exascale applications. For more information and
// source code availability see http://github.com/ceed
// SPDX-License-Identifier: BSD-2-Clause
//
// The CEED research is supported by the Exascale Computing Project 17-SC-20-SC,
// a collaborative effort of two U.S. Department of Energy organizations (Office
// of Science and the National Nuclear Security Administration) responsible for
// the planning and preparation of a capable exascale ecosystem, including
// software, applications, hardware, advanced system engineering and early
// testbed platforms, in support of the nation's exascale computing imperative.
// This file is part of CEED: http://github.com/ceed

#ifndef CEED_OCCA_BASIS_HEADER
#define CEED_OCCA_BASIS_HEADER
Expand Down
17 changes: 4 additions & 13 deletions backends/occa/ceed-occa-ceed-object.cpp
Original file line number Diff line number Diff line change
@@ -1,18 +1,9 @@
// Copyright (c) 2019, Lawrence Livermore National Security, LLC.
// Produced at the Lawrence Livermore National Laboratory. LLNL-CODE-734707.
// All Rights reserved. See files LICENSE and NOTICE for details.
// Copyright (c) 2017-2022, Lawrence Livermore National Security, LLC and other CEED contributors.
// All Rights Reserved. See the top-level LICENSE and NOTICE files for details.
//
// This file is part of CEED, a collection of benchmarks, miniapps, software
// libraries and APIs for efficient high-order finite element and spectral
// element discretizations for exascale applications. For more information and
// source code availability see http://github.com/ceed
// SPDX-License-Identifier: BSD-2-Clause
//
// The CEED research is supported by the Exascale Computing Project 17-SC-20-SC,
// a collaborative effort of two U.S. Department of Energy organizations (Office
// of Science and the National Nuclear Security Administration) responsible for
// the planning and preparation of a capable exascale ecosystem, including
// software, applications, hardware, advanced system engineering and early
// testbed platforms, in support of the nation's exascale computing imperative.
// This file is part of CEED: http://github.com/ceed

#include "ceed-occa-ceed-object.hpp"
#include "ceed-occa-context.hpp"
Expand Down
17 changes: 4 additions & 13 deletions backends/occa/ceed-occa-ceed-object.hpp
Original file line number Diff line number Diff line change
@@ -1,18 +1,9 @@
// Copyright (c) 2019, Lawrence Livermore National Security, LLC.
// Produced at the Lawrence Livermore National Laboratory. LLNL-CODE-734707.
// All Rights reserved. See files LICENSE and NOTICE for details.
// Copyright (c) 2017-2022, Lawrence Livermore National Security, LLC and other CEED contributors.
// All Rights Reserved. See the top-level LICENSE and NOTICE files for details.
//
// This file is part of CEED, a collection of benchmarks, miniapps, software
// libraries and APIs for efficient high-order finite element and spectral
// element discretizations for exascale applications. For more information and
// source code availability see http://github.com/ceed
// SPDX-License-Identifier: BSD-2-Clause
//
// The CEED research is supported by the Exascale Computing Project 17-SC-20-SC,
// a collaborative effort of two U.S. Department of Energy organizations (Office
// of Science and the National Nuclear Security Administration) responsible for
// the planning and preparation of a capable exascale ecosystem, including
// software, applications, hardware, advanced system engineering and early
// testbed platforms, in support of the nation's exascale computing imperative.
// This file is part of CEED: http://github.com/ceed

#ifndef CEED_OCCA_CEEDOBJECT_HEADER
#define CEED_OCCA_CEEDOBJECT_HEADER
Expand Down
17 changes: 4 additions & 13 deletions backends/occa/ceed-occa-context.cpp
Original file line number Diff line number Diff line change
@@ -1,18 +1,9 @@
// Copyright (c) 2019, Lawrence Livermore National Security, LLC.
// Produced at the Lawrence Livermore National Laboratory. LLNL-CODE-734707.
// All Rights reserved. See files LICENSE and NOTICE for details.
// Copyright (c) 2017-2022, Lawrence Livermore National Security, LLC and other CEED contributors.
// All Rights Reserved. See the top-level LICENSE and NOTICE files for details.
//
// This file is part of CEED, a collection of benchmarks, miniapps, software
// libraries and APIs for efficient high-order finite element and spectral
// element discretizations for exascale applications. For more information and
// source code availability see http://github.com/ceed
// SPDX-License-Identifier: BSD-2-Clause
//
// The CEED research is supported by the Exascale Computing Project 17-SC-20-SC,
// a collaborative effort of two U.S. Department of Energy organizations (Office
// of Science and the National Nuclear Security Administration) responsible for
// the planning and preparation of a capable exascale ecosystem, including
// software, applications, hardware, advanced system engineering and early
// testbed platforms, in support of the nation's exascale computing imperative.
// This file is part of CEED: http://github.com/ceed

#include "ceed-occa-context.hpp"

Expand Down
17 changes: 4 additions & 13 deletions backends/occa/ceed-occa-context.hpp
Original file line number Diff line number Diff line change
@@ -1,18 +1,9 @@
// Copyright (c) 2019, Lawrence Livermore National Security, LLC.
// Produced at the Lawrence Livermore National Laboratory. LLNL-CODE-734707.
// All Rights reserved. See files LICENSE and NOTICE for details.
// Copyright (c) 2017-2022, Lawrence Livermore National Security, LLC and other CEED contributors.
// All Rights Reserved. See the top-level LICENSE and NOTICE files for details.
//
// This file is part of CEED, a collection of benchmarks, miniapps, software
// libraries and APIs for efficient high-order finite element and spectral
// element discretizations for exascale applications. For more information and
// source code availability see http://github.com/ceed
// SPDX-License-Identifier: BSD-2-Clause
//
// The CEED research is supported by the Exascale Computing Project 17-SC-20-SC,
// a collaborative effort of two U.S. Department of Energy organizations (Office
// of Science and the National Nuclear Security Administration) responsible for
// the planning and preparation of a capable exascale ecosystem, including
// software, applications, hardware, advanced system engineering and early
// testbed platforms, in support of the nation's exascale computing imperative.
// This file is part of CEED: http://github.com/ceed

#ifndef CEED_OCCA_CONTEXT_HEADER
#define CEED_OCCA_CONTEXT_HEADER
Expand Down
18 changes: 4 additions & 14 deletions backends/occa/ceed-occa-cpu-operator.cpp
Original file line number Diff line number Diff line change
@@ -1,19 +1,9 @@
// Copyright (c) 2020, Lawrence Livermore National Security, LLC.
// Produced at the Lawrence Livermore National Laboratory. LLNL-CODE-734707.
// All Rights reserved. See files LICENSE and NOTICE for details.
// Copyright (c) 2017-2022, Lawrence Livermore National Security, LLC and other CEED contributors.
// All Rights Reserved. See the top-level LICENSE and NOTICE files for details.
//
// This file is part of CEED, a collection of benchmarks, miniapps, software
// libraries and APIs for efficient high-order finite element and spectral
// element discretizations for exascale applications. For more information and
// source code availability see http://github.com/ceed
// SPDX-License-Identifier: BSD-2-Clause
//
// The CEED research is supported by the Exascale Computing Project 17-SC-20-SC,
// a collaborative effort of two U.S. Department of Energy organizations (Office
// of Science and the National Nuclear Security Administration) responsible for
// the planning and preparation of a capable exascale ecosystem, including
// software, applications, hardware, advanced system engineering and early
// testbed platforms, in support of the nation's exascale computing imperative.
#define CEED_DEBUG_COLOR 12
// This file is part of CEED: http://github.com/ceed

#include "ceed-occa-cpu-operator.hpp"
#include "ceed-occa-elem-restriction.hpp"
Expand Down
17 changes: 4 additions & 13 deletions backends/occa/ceed-occa-cpu-operator.hpp
Original file line number Diff line number Diff line change
@@ -1,18 +1,9 @@
// Copyright (c) 2020, Lawrence Livermore National Security, LLC.
// Produced at the Lawrence Livermore National Laboratory. LLNL-CODE-734707.
// All Rights reserved. See files LICENSE and NOTICE for details.
// Copyright (c) 2017-2022, Lawrence Livermore National Security, LLC and other CEED contributors.
// All Rights Reserved. See the top-level LICENSE and NOTICE files for details.
//
// This file is part of CEED, a collection of benchmarks, miniapps, software
// libraries and APIs for efficient high-order finite element and spectral
// element discretizations for exascale applications. For more information and
// source code availability see http://github.com/ceed
// SPDX-License-Identifier: BSD-2-Clause
//
// The CEED research is supported by the Exascale Computing Project 17-SC-20-SC,
// a collaborative effort of two U.S. Department of Energy organizations (Office
// of Science and the National Nuclear Security Administration) responsible for
// the planning and preparation of a capable exascale ecosystem, including
// software, applications, hardware, advanced system engineering and early
// testbed platforms, in support of the nation's exascale computing imperative.
// This file is part of CEED: http://github.com/ceed

#ifndef CEED_OCCA_CPU_OPERATOR_HEADER
#define CEED_OCCA_CPU_OPERATOR_HEADER
Expand Down
Loading

0 comments on commit 0be03a9

Please sign in to comment.