# Getting Started

**Note: The current CMakefile and relevant instructions are mainly tested on Linux-based systems including Windows Subsystem for Linux**

## Dependencies

The following prerequisites and dependencies are required for building OPS. Building each of the **backends** is optional and depends on the hardware and/or capabilities you will be targeting. 

  **CMake**

CMake 3.18 or newer is required for using the CMake build system. If the latest version is not installed/shipped by default, it can be downloaded from https://cmake.org/download/, e.g., using the following script.
  ```bash
  version=3.19.0
  wget https://github.com/Kitware/CMake/releases/download/v$version/cmake-$version-Linux-x86_64.sh
  # Assume that CMake is going to be installed at /usr/local/cmake
  cmake_dir=/usr/local/cmake
  # sudo is not necessary for directories in user space.
  sudo mkdir $cmake_dir
  sudo sh ./cmake-$version-Linux-x86_64.sh --prefix=$cmake_dir  --skip-license
  sudo ln -s $cmake_dir/bin/cmake /usr/local/bin/cmake
  ```

 **Python**

Python 3.8 or newer is required for the OPS code generator (`ops_translator`). How the virtual environment is set up depends on which build system you use:

**CMake build** — the venv is created automatically during the CMake configure step (no manual action needed). The entire `ops_translator/` tree is copied to `${CMAKE_INSTALL_PREFIX}/translator/ops_translator/` and `setup_venv_cmake.sh` runs automatically to create:
```
${CMAKE_INSTALL_PREFIX}/translator/ops_translator/ops_venv/
```
If the venv already exists at that path it is left untouched.

**Makefile build** — the venv is created under `ops_translator/.python/` by running:
```bash
cd ops_translator
make python        # creates .python/ venv and installs all dependencies
```
To force a clean rebuild:
```bash
cd ops_translator
make clean         # removes the .python/ venv
make python        # rebuilds from scratch
```
The Makefile uses whatever `python3` is on your `PATH` — on HPC systems this is typically loaded via a module, e.g.:
```bash
module load python/3.9.7
```

> **Note for HPC systems (Makefile build):** Some module-provided Python builds lack SSL support (e.g. compiled against an older `libssl.so.1.1` no longer present on the system). The Makefile automatically detects this and falls back to `/usr/bin/python3` to create the venv, which typically has working SSL. The resulting venv is otherwise identical.

 **HDF5**

[HDF5](https://www.hdfgroup.org/solutions/hdf5) is required for parts of IO functionalities. The CMake build system **uses the parallel version by default** even for sequential codes, and automatically identifies the library. If the automatic process fails, the path to the parallel HDF5 library can be specified by using `-DHDF5_ROOT`.

 **CUDA Backend**
 
The [CUDA](https://developer.nvidia.com/cuda-downloads) backend targets NVIDIA GPUs with a compute capability of 3.0 or greater. The CMake build system will detect the toolkit automatically. If the automatic process fails, the build system will compile the library without CUDA support. Please use `-DCUDA_TOOLKIT_ROOT_DIR` to manually specify the path.

 **HIP Backend**

The HIP backend targets AMD GPUs and NVIDIA GPUs which are supported by HIP - either through its CUDA support or the [ROCm](https://rocmdocs.amd.com/en/latest/) stack (tested with >=3.9). 

 **SYCL Backend**

The [SYCL](https://www.khronos.org/sycl/) backend targets Intel, AMD, and NVIDIA GPUs. It is supported via Intel OneAPI (tested with >=2024.2.0), Intel’s public LLVM, and hipSYCL (>=0.9.1). It runs on Intel CPUs and GPUs using OpenCL and Level Zero, and on AMD and NVIDIA GPUs through the LLVM fork and hipSYCL. hipSYCL also provides OpenMP support for a wide range of CPU architectures.

 **Supported Features**

* `Precision Support in OPS` - OPS allows data to be stored and operated on in single (float), double (double), and half (__fp16 or half) precision. Support for half-precision operations may depend on the capabilities of the target hardware and backend (e.g., SYCL, CUDA, OpenMP). A test application demonstrating the use of half-precision datasets—Laplace2D—is available at `OPS/apps/c/halfprecision`.

 **Tridiagonal Solver Backend**

To use the tridiagonal solver OPS API in applications and build example applications such as `adi`, `adi_burger` and `adi_burger_3D`, the open source tridiagonal solver (scalar) library needs to be cloned and built from the [Tridsolver repository](https://github.com/OP-DSL/tridsolver). 
```bash
git clone https://github.com/OP-DSL/tridsolver.git
```
Details on building the scalar tridiagonal solver library can be found in the [README](https://github.com/OP-DSL/tridsolver/blob/master/scalar/README) file located at the appropriate subdirectory.

## Obtaining OPS
The latest OPS source code can be obtained by cloning the [OPS repository](https://github.com/OP-DSL/OPS) using
```bash
git clone https://github.com/OP-DSL/OPS.git
```
    
## Build OPS
### Using CMake
#### Build library and example applications together

  Create a build directory, and run CMake (version 3.18 or newer)
  ```bash
  mkdir build
  cd build
  # See below for CMake options
  cmake ${PATH_TO_OPS} -DBUILD_OPS_APPS=ON -DOPS_TEST=ON -DAPP_INSTALL_DIR=$HOME/OPS-APP -DCMAKE_INSTALL_PREFIX=$HOME/OPS-INSTALL -DGPU_NUMBER=1
  make # IEEE=1 enables IEEE flags in compiler
  make install # sudo is needed if a directory like /usr/local/ is chosen.
  ```
After installation, the library and the Python translator can be found at the directory specified by `CMAKE_INSTALL_PREFIX`, together with the executable files for applications at `APP_INSTALL_DIR`.

####  Build library and example applications separately

In this mode, the library can be first built and installed as

```bash
mkdir build
cd build
# See below for CMake options
cmake ${PATH_TO_OPS}   -DCMAKE_INSTALL_PREFIX=$HOME/OPS-INSTALL
make # IEEE=1 enables IEEE flags in compiler
make install # sudo is needed if a system directory is chosen
```
Then the application can be built as:

```bash
mkdir appbuild
cd appbuild
# See below for CMake options
cmake ${PATH_TO_APPS} -DOPS_INSTALL_DIR=$HOME/OPS-INSTALL -DOPS_TEST=ON -DAPP_INSTALL_DIR=$HOME/OPS-APP -DGPU_NUMBER=1
make # IEEE=1 this option is important for applications to get accurate results
```
<!-- #### Tests

A few tasks for testing codes can be run by
```bash
make test
```
The current tests are mainly based on the applications.
-->

#### CMake options

  * `-DCMAKE_BUILD_TYPE=Release` - enable optimizations
  * `-DBUILD_OPS_APPS=ON` - build example applications (Library CMake only)
  * `-DOPS_TEST=ON` - enable the tests
  * `-DCMAKE_INSTALL_PREFIX=` - specify the installation directory for the library (`/usr/local` by default, Library CMake only)
  * `-DAPP_INSTALL_DIR=` - specify the installation directory for the applications (`$HOME/OPS-APPS` by default)
  * `-DGPU_NUMBER=` - specify the number of GPUs used in the tests
  * `-DOPS_INSTALL_DIR=` - specify where the OPS library is installed (Application CMake only, see [here](#build-the-library-and-example-applications-separately))
  * `-DOPS_VERBOSE_WARNING=ON` - show verbose output during the building process
  <!-- * `-DHDF5_PREFER_PARALLEL=ON` - build using parallel HDF5, rather than serial HDF5 libraries -->
  <!-- * `-DBUILD_OPS_FORTRAN=ON` - enable building OPS Fortran libraries. -->

<!-- 1. Set up environment variables:
* `CUDA_PATH` - Installation directory of CUDA, usually `/usr/local/cuda` (to build CUDA libs and applications, only needed if CUDA cannot be found in standard locations, or to enable OpenCL)
* `MPI_HOME` - Installation directory of MPI (to build MPI based distributed memory libs and applications) only needed if MPI not installed in standard locations
* `HDF5_ROOT` - Installation directory of HDF5 (to support HDF5 based File I/O) if HDF5 not installed in standard location -->

### Using Makefiles

OPS can also be built using Makefiles located in the ops directory and in each example application's directory. 

#### Set up environment variables:

  * `OPS_COMPILER` - compiler to be used (Currently supports Intel, PGI and Cray compilers, but others can be easily incorporated by extending the Makefiles used in step 2 and 3)
  * `OPS_INSTALL_PATH` - Installation directory of OPS/ops
  * `CUDA_INSTALL_PATH` - Installation directory of CUDA, usually `/usr/local/cuda` (to build CUDA libs and applications)
  * `MPI_INSTALL_PATH` - Installation directory of MPI (to build MPI based distributed memory libs and applications)
  * `HDF5_INSTALL_PATH` - Installation directory of HDF5 (to support HDF5 based File I/O)

See example scripts (e.g. `source_intel_2021.3_pythonenv`, `source_pgi_nvhpc_23_pythonenv`, `source_amd_rocm-5.4.3_pythonenv`) under `OPS/scripts` that set up the environment for building with various compilers (Intel, PGI, ROCm).

#### Build back-end library
For C/C++ back-end use the Makefile under `OPS/ops/c` (modify the Makefile if required). The library will be built in `OPS/ops/c/lib`
```bash
cd $OPS_INSTALL_PATH/c
make
```
For Fortran back-end use the Makefile under `OPS/ops/fortran` (modify the Makefile if required). The library will be built in `OPS/ops/fortran/lib`
```bash
cd $OPS_INSTALL_PATH/fortran
make
```
#### Build example applications
Once the back-end libraries are built, the example applications can be compiled for different supported architectures based on the configured environment using a simple `make` command:
For example, to build CloverLeaf_3D under `OPS/apps/c/CloverLeaf_3D`:
```bash  
cd OPS/apps/c/CloverLeaf_3D/
make
```  
<!---#### Makefile options