Getting Started

Note: The current CMakefile and relevant instructions are mainly tested on Linux-based systems including Windows Subsystem for Linux

Dependencies

The following prerequisites and dependencies are required for building OPS. Building each of the backends is optional and depends on the hardware and/or capabilities you will be targeting.

CMake

CMake 3.18 or newer is required for using the CMake build system. If the latest version is not installed/shipped by default, it can be downloaded from https://cmake.org/download/, e.g., using the following script.

version=3.19.0
wget https://github.com/Kitware/CMake/releases/download/v$version/cmake-$version-Linux-x86_64.sh
# Assume that CMake is going to be installed at /usr/local/cmake
cmake_dir=/usr/local/cmake
# sudo is not necessary for directories in user space.
sudo mkdir $cmake_dir
sudo sh ./cmake-$version-Linux-x86_64.sh --prefix=$cmake_dir  --skip-license
sudo ln -s $cmake_dir/bin/cmake /usr/local/bin/cmake

Python

Python 3.8 or newer is required for the OPS code generator (ops_translator). How the virtual environment is set up depends on which build system you use:

CMake build — the venv is created automatically during the CMake configure step (no manual action needed). The entire ops_translator/ tree is copied to ${CMAKE_INSTALL_PREFIX}/translator/ops_translator/ and setup_venv_cmake.sh runs automatically to create:

${CMAKE_INSTALL_PREFIX}/translator/ops_translator/ops_venv/

If the venv already exists at that path it is left untouched.

Makefile build — the venv is created under ops_translator/.python/ by running:

cd ops_translator
make python        # creates .python/ venv and installs all dependencies

To force a clean rebuild:

cd ops_translator
make clean         # removes the .python/ venv
make python        # rebuilds from scratch

The Makefile uses whatever python3 is on your PATH — on HPC systems this is typically loaded via a module, e.g.:

module load python/3.9.7

Note for HPC systems (Makefile build): Some module-provided Python builds lack SSL support (e.g. compiled against an older libssl.so.1.1 no longer present on the system). The Makefile automatically detects this and falls back to /usr/bin/python3 to create the venv, which typically has working SSL. The resulting venv is otherwise identical.

HDF5

HDF5 is required for parts of IO functionalities. The CMake build system uses the parallel version by default even for sequential codes, and automatically identifies the library. If the automatic process fails, the path to the parallel HDF5 library can be specified by using -DHDF5_ROOT.

CUDA Backend

The CUDA backend targets NVIDIA GPUs with a compute capability of 3.0 or greater. The CMake build system will detect the toolkit automatically. If the automatic process fails, the build system will compile the library without CUDA support. Please use -DCUDA_TOOLKIT_ROOT_DIR to manually specify the path.

HIP Backend

The HIP backend targets AMD GPUs and NVIDIA GPUs which are supported by HIP - either through its CUDA support or the ROCm stack (tested with >=3.9).

SYCL Backend

The SYCL backend targets Intel, AMD, and NVIDIA GPUs. It is supported via Intel OneAPI (tested with >=2024.2.0), Intel’s public LLVM, and hipSYCL (>=0.9.1). It runs on Intel CPUs and GPUs using OpenCL and Level Zero, and on AMD and NVIDIA GPUs through the LLVM fork and hipSYCL. hipSYCL also provides OpenMP support for a wide range of CPU architectures.

Supported Features

  • Precision Support in OPS - OPS allows data to be stored and operated on in single (float), double (double), and half (__fp16 or half) precision. Support for half-precision operations may depend on the capabilities of the target hardware and backend (e.g., SYCL, CUDA, OpenMP). A test application demonstrating the use of half-precision datasets—Laplace2D—is available at OPS/apps/c/halfprecision.

Tridiagonal Solver Backend

To use the tridiagonal solver OPS API in applications and build example applications such as adi, adi_burger and adi_burger_3D, the open source tridiagonal solver (scalar) library needs to be cloned and built from the Tridsolver repository.

git clone https://github.com/OP-DSL/tridsolver.git

Details on building the scalar tridiagonal solver library can be found in the README file located at the appropriate subdirectory.

Obtaining OPS

The latest OPS source code can be obtained by cloning the OPS repository using

git clone https://github.com/OP-DSL/OPS.git

Build OPS

Using CMake

Build library and example applications together

Create a build directory, and run CMake (version 3.18 or newer)

mkdir build
cd build
# See below for CMake options
cmake ${PATH_TO_OPS} -DBUILD_OPS_APPS=ON -DOPS_TEST=ON -DAPP_INSTALL_DIR=$HOME/OPS-APP -DCMAKE_INSTALL_PREFIX=$HOME/OPS-INSTALL -DGPU_NUMBER=1
make # IEEE=1 enables IEEE flags in compiler
make install # sudo is needed if a directory like /usr/local/ is chosen.

After installation, the library and the Python translator can be found at the directory specified by CMAKE_INSTALL_PREFIX, together with the executable files for applications at APP_INSTALL_DIR.

Build library and example applications separately

In this mode, the library can be first built and installed as

mkdir build
cd build
# See below for CMake options
cmake ${PATH_TO_OPS}   -DCMAKE_INSTALL_PREFIX=$HOME/OPS-INSTALL
make # IEEE=1 enables IEEE flags in compiler
make install # sudo is needed if a system directory is chosen

Then the application can be built as:

mkdir appbuild
cd appbuild
# See below for CMake options
cmake ${PATH_TO_APPS} -DOPS_INSTALL_DIR=$HOME/OPS-INSTALL -DOPS_TEST=ON -DAPP_INSTALL_DIR=$HOME/OPS-APP -DGPU_NUMBER=1
make # IEEE=1 this option is important for applications to get accurate results

CMake options

  • -DCMAKE_BUILD_TYPE=Release - enable optimizations

  • -DBUILD_OPS_APPS=ON - build example applications (Library CMake only)

  • -DOPS_TEST=ON - enable the tests

  • -DCMAKE_INSTALL_PREFIX= - specify the installation directory for the library (/usr/local by default, Library CMake only)

  • -DAPP_INSTALL_DIR= - specify the installation directory for the applications ($HOME/OPS-APPS by default)

  • -DGPU_NUMBER= - specify the number of GPUs used in the tests

  • -DOPS_INSTALL_DIR= - specify where the OPS library is installed (Application CMake only, see here)

  • -DOPS_VERBOSE_WARNING=ON - show verbose output during the building process

Using Makefiles

OPS can also be built using Makefiles located in the ops directory and in each example application’s directory.

Set up environment variables:

  • OPS_COMPILER - compiler to be used (Currently supports Intel, PGI and Cray compilers, but others can be easily incorporated by extending the Makefiles used in step 2 and 3)

  • OPS_INSTALL_PATH - Installation directory of OPS/ops

  • CUDA_INSTALL_PATH - Installation directory of CUDA, usually /usr/local/cuda (to build CUDA libs and applications)

  • MPI_INSTALL_PATH - Installation directory of MPI (to build MPI based distributed memory libs and applications)

  • HDF5_INSTALL_PATH - Installation directory of HDF5 (to support HDF5 based File I/O)

See example scripts (e.g. source_intel_2021.3_pythonenv, source_pgi_nvhpc_23_pythonenv, source_amd_rocm-5.4.3_pythonenv) under OPS/scripts that set up the environment for building with various compilers (Intel, PGI, ROCm).

Build back-end library

For C/C++ back-end use the Makefile under OPS/ops/c (modify the Makefile if required). The library will be built in OPS/ops/c/lib

cd $OPS_INSTALL_PATH/c
make

For Fortran back-end use the Makefile under OPS/ops/fortran (modify the Makefile if required). The library will be built in OPS/ops/fortran/lib

cd $OPS_INSTALL_PATH/fortran
make

Build example applications

Once the back-end libraries are built, the example applications can be compiled for different supported architectures based on the configured environment using a simple make command: For example, to build CloverLeaf_3D under OPS/apps/c/CloverLeaf_3D:

cd OPS/apps/c/CloverLeaf_3D/
make