Getting Started
Note: The current CMakefile and relevant instructions are mainly tested on linux-based systems including Windows Subsystem for Linux
Dependencies
The following prerequisites and dependencies are required for building OPS. Building each of the backends are optional and depends on the hardware and/or capabilities you will be targeting.
CMake
CMake 3.18 or newer is required for using the CMake building system. If the latest version is not installed/shipped by default, it can be downloaded from https://cmake.org/download/, e.g., using the following script.
version=3.19.0
wget https://github.com/Kitware/CMake/releases/download/v$version/cmake-$version-Linux-x86_64.sh
# Assume that CMake is going to be installed at /usr/local/cmake
cmake_dir=/usr/local/cmake
# sudo is not necessary for directories in user space.
sudo mkdir $cmake_dir
sudo sh ./cmake-$version-Linux-x86_64.sh --prefix=$cmake_dir --skip-license
sudo ln -s $cmake_dir/bin/cmake /usr/local/bin/cmake
Python
The Python dependencies (primarily used for the OPS code generator) are best installed by setting up a virtual environment so that required packages can be installed without superuser privileges. To set up the Python virtual environment and install the required dependant packages, ensure that you have Python3.9 or a more recent version with pip installed.
Detailed instructions for installing virtual environment using pip can be found here
Execute following after cloning the OPS repository (see below) to install required packages. Note OPS_INSTALL_PATH
is the installation directory of OPS/ops:
#Install virtual environment using pip (if not installed earlier)
#Please set the OPS_INSTALL_PATH variable before running following commands
python3 -m pip install --user virtualenv
mkdir -p $OPS_INSTALL_PATH/../ops_translator/ops_venv
python3 -m venv $OPS_INSTALL_PATH/../ops_translator/ops_venv
source $OPS_INSTALL_PATH/../ops_translator/ops_venv/bin/activate
python3 -m pip install --upgrade pip
python3 -m pip install -r $OPS_INSTALL_PATH/../ops_translator/requirements.txt
python3 -m pip install --force-reinstall libclang==16.0.6
These instructions can be executed by running the script OPS/ops_translator/setup_venv.sh
file.
After successfully setting up the Python virtual environment and installing the required dependent packages using the above instructions, you will need to activate the virtual environment by source $OPS_INSTALL_PATH/../ops_translator/ops_venv/bin/activate
every time you want to use the code generator. Activating the virtual environment ensures that the code generator and its dependencies are isolated from the system-wide Python installation, avoiding conflicts and ensuring proper execution.
HDF5
HDF5 is required for parts of IO functionalities. The CMake build system uses the parallel version by default even for sequential codes, and automatically identify the library. If the automatic process fails, the path to the parallel HDF5 library can be specified by using -DHDF5_ROOT
.
CUDA Backend
The CUDA backend targets NVIDIA GPUs with a compute capability of 3.0 or greater. The CMake build system will detect the tookit automatically. If the automatic process fails, the build system will compile the library without the CUDA support. Please use -DCUDA_TOOLKIT_ROOT_DIR
to manually specify the path.
HIP Backend
The HIP backend targets AMD GPUs and NVIDIA GPUs which are supported by HIP - either through its CUDA support or the ROCm stack (tested with >=3.9).
SYCL Backend
The SYCL backend is currently in development and only working without MPI. It has been tested with Intel OneAPI (>=2021.1), Intel’s public LLVM version, and hipSYCL (>=0.9.1), and runs on Intel CPUs and GPUs through Intel’s OpenCL and Level Zero, NVIDIA and AMD GPUs both with the LLVM fork as well as hipSYCL. hipSYCL’s OpenMP support covers most CPU architectures too.
Tridiagonal Solver Backend
To use the tridiagonal solver OPS API in applications and build example applications such as adi
, adi_burger
and adi_burger_3D
the open source tridiagonal solver (scalar) library needs to be cloned and built from the Tridsolver repository.
git clone https://github.com/OP-DSL/tridsolver.git
Details on building scalar tridiagonal solver library can be found in the README file located at the appropriate subdirectory.
Obtaining OPS
The latest OPS source code can be obtained by cloning the OPS repository using
git clone https://github.com/OP-DSL/OPS.git
Build OPS
Using cmake
Build library and example applications together
Create a build directory, and run CMake (version 3.18 or newer)
mkdir build
cd build
# Please see below for CMake options
cmake ${PATH_TO_OPS} -DBUILD_OPS_APPS=ON -DOPS_TEST=ON -DAPP_INSTALL_DIR=$HOME/OPS-APP -DCMAKE_INSTALL_PREFIX=$HOME/OPS-INSTALL -DGPU_NUMBER=1
make # IEEE=1 enable IEEE flags in compiler
make install # sudo is needed if a directory like /usr/local/ is chosen.
After installation, the library and the python translator can be found at the direcory specified by CMAKE_INSTALL_PREFIX
, together with the executable files for applications at APP_INSTALL_DIR
.
Build library and example applications separately
In this mode, the library can be firstly built and installed as
mkdir build
cd build
# Please see below for CMake options
cmake ${PATH_TO_OPS} -DCMAKE_INSTALL_PREFIX=$HOME/OPS-INSTALL
make # IEEE=1 enable IEEE flags in compiler
make install # sudo is needed if a system direction is chosen,
Then the application can be built as:
mkdir appbuild
cd appbuild
# Please see below for CMake options
cmake ${PATH_TO_APPS} -DOPS_INSTALL_DIR=$HOME/OPS-INSTALL -DOPS_TEST=ON -DAPP_INSTALL_DIR=$HOME/OPS-APP -DGPU_NUMBER=1
make # IEEE=1 this option is important for applications to get accurate results
cmake options
-DCMAKE_BUILD_TYPE=Release
- enable optimizations-DBUILD_OPS_APPS=ON
- build example applications (Library CMake only)-DOPS_TEST=ON
- enable the tests-DCMAKE_INSTALL_PREFIX=
- specify the installation direction for the library (/usr/local
by default, Library CMake only)-DAPP_INSTALL_DIR=
- specify the installation direction for the applications ($HOME/OPS-APPS
by default)-DGPU_NUMBER=
- specify the number of GPUs used in the tests-DOPS_INSTALL_DIR=
- specify where the OPS library is installed (Application CMake only, see here)-DOPS_VERBOSE_WARNING=ON
- show verbose output during building process
Using Makefiles
Set up environmental variables:
OPS_COMPILER
- compiler to be used (Currently supports Intel, PGI and Cray compilers, but others can be easily incorporated by extending the Makefiles used in step 2 and 3)OPS_INSTALL_PATH
- Installation directory of OPS/opsCUDA_INSTALL_PATH
- Installation directory of CUDA, usually/usr/local/cuda
(to build CUDA libs and applications)OPENCL_INSTALL_PATH
- Installation directory of OpenCL, usually/usr/local/cuda
for NVIDIA OpenCL implementation (to build OpenCL libs and applications)MPI_INSTALL_PATH
- Installation directory of MPI (to build MPI based distributed memory libs and applications)HDF5_INSTALL_PATH
- Installation directory of HDF5 (to support HDF5 based File I/O)
See example scripts (e.g. source_intel
, source_pgi_15.10
, source_cray
) under OPS/ops/scripts
that sets up the environment for building with various compilers (Intel, PGI, Cray).
Build back-end library
For C/C++ back-end use Makefile under OPS/ops/c
(modify Makefile if required). The libraries will be built in OPS/ops/c/lib
cd $OPS_INSTALL_PATH/c
make
For Fortran back-end use Makefile under OPS/ops/fortran
(modify Makefile if required). The libraries will be built in OPS/ops/fortran/lib
cd $OPS_INSTALL_PATH/fortran
make
Build exampe applications
For example to build CloverLeaf_3D under OPS/apps/c/CloverLeaf_3D
cd ../apps/c/Cloverleaf_3D/
make