Basic Usage

Single versus Double Precision binaries

Since v1.4, nanoFluidX has been delivered in both Single (SP) and Double (DP) Precision. Single Precision is sufficient for most industrial cases. However, for specific simulations Single Precision is insufficient and Double Precision should be used. Typically this arises where there is a very high velocity combined with a comparatively small dx, which leads to a very small timestep and the need to resolve very small changes in particle position and velocity accurately.

The Single Precision binary contains a warning message about possible precision issues, with estimations of the smallest position changes and velocities that will be resolved. It is up to you to acknowledge the warning message and decide whether the values are acceptable for your simulation or if it will be necessary to use Double Precision.

Performance of Single versus Double Precision binaries varies greatly depending on the type of GPU, but typically run times for Double Precision will be at least 2x slower than Single Precision. The latest generation of NVIDIA Tesla GPUs significantly improved both Single and Double Precision performance, but be aware that older generations may be closer to 3x slower for DP simulations.

Restriction: Check that your GPU model can support Double Precision computation. Some Quadro and RTX models have severely compromised DP performance (Over 10x slower than SP) or have no DP capability at all.

NVIDIA SMI

Similar to using the command top to see the currently running jobs on the CPU, NVIDIA provides the tool nvidia-smi that shows the actual usage of the GPUs. The currently occupied GPUs can be detected if some jobs are running on that machine.

Environment setup

After the installation of nanoFluidX, it is necessary to set up the system variable environment. You can source the environment script that is provided with the installation and located in the installation folder.

When sourcing the set_nFX_environment.sh, use the source command. Do not execute using sh.

Running the set_nFX_environment.sh script is necessary every time your session is interrupted, either by a restart, shut down, or some other case which results in resetting the environment variables.

Alternatively, you can add a command to source the set_nFX_environment.sh script in your .bashrc file. This mitigates the need for explicit sourcing of the environment script for every active session.

When you carry out the ldd command on the nanoFluidX binary, all libraries regarding CUDA, OpenMPI, and Altair licensing should point to the respective subfolders in the ./libs folder inside of the root nanoFluidX folder.

If you are not using the bash shell, but rather csh or tcsh, there is a set_nFX_environment.csh script included in the package that you can use in the same manner as the .sh script.

Launch and Stop a Simulation

The command to start a simulation with single precision binary is:

$ CUDA_VISIBLE_DEVICES=0 mpiexec –np 1 $nFX_SP –i test.cfg

The first command, CUDA_VISIBLE_DEVICES, chooses the GPU-devices that are used to start this job. By using this keyword, it is possible to avoid running different jobs on the same card. Invoking a simulation without this command will always result in running the job on device number zero. For multiple cards, list the device numbers separated with a comma and set the number of processes to -np #N. Ensure the number of processes -np matches the number of selected GPUs. For example, if you want to run on three GPUs, the command should look like:

$ CUDA_VISIBLE_DEVICES=0,1,2 mpiexec –np 3 $nFX_SP –i test.cfg

Note: The CPU is used for nanoFluidX to control the simulation, the actual work is done on the GPU.

To launch a simulation, a configuration file has to be passed to the binary via the option -i, followed by the configuration file name. This file and the corresponding geometry file need to be placed in the same folder where the simulation is started. If you would like to run a Double Precision binary, the $nFX_SP should be replaced with $nFX_DP.

nanoFluidX is a multi-GPU-capable CFD code which implies that a simulation has to be started using the mpiexec -np N command, where N is the number of GPUs used. This is the case even for a single-GPU run since the general implementation expects a parallel environment. The CPU-GPU pinning is one-to-one, where each mpi-rank owns one GPU.

Important: nanoFluidX treats all problems as three-dimensional. Consequently, two-dimensional problems are set up simply by using a zero or constant third vector component in all directional quantities.

To appropriately stop a simulation, you can create an empty file titled SAVEABORTFILE in the root folder of the running case. nanoFluidX will pick up on the newly created file in the folder, stop the simulation and write the flow fields at the given time step and restart file.

Try to simulate one of the example cases provided with the binary.

Provide a .zip Archive to Launch a Simulation

Considering the growing number of files required to run a nanoFluidX simulation and to simplify the transfer process of these files to the production environment, nanoFluidX and all the auxiliary tools offer an archive input mode using libzip library. The following operations are carried out when archive input files are used:

The archive file is extracted to the same directory as the archive file, preserving relative directory structure.
Any files with the same name as the ones in the archive are overwritten. This is to ensure extracted files match the archive contents at all times.
In multinode cases, the archive is extracted only after detecting a local file system. Naturally, the archive file must be available on the local file system of the node.
The simulation runs based on the files extracted from the archive.

The archive file passed to the binaries for execution must satisfy the following conditions:

The archive must be in zip format using the Deflate compression method, however, a .zip extension is not required. Using .cfg as archive extension leads to errors.
The .cfg file inside the archive must have the same name as the archive file except with a .cfg extension. For example, an archive named archive.xyz must contain a cfg file named archive.cfg.

When using an archive file, consider the following:

As the extracted files overwrite any files with the same name already on the disk, you must be careful about any changes made to these files on the disk. For example, changes done to a .cfg file for a recon run, as these will be lost.
To keep the command line options simple, we have refrained from adding switches for introducing warning or error messages before overwriting files.
Running a recon simulation using archive files is not blocked by the solver. However, it is recommended to carry out recon simulations directly via the .cfg file.
The solver does not remove the archive or the extracted files from the disk. This is deliberate to simplify the relaunch via archive or modifying of the files necessary for a recon simulation.
As both the archive and extracted files are present on the disk, more disk space is required.
Extraction of the archive adds to the startup time of the simulation. This time might be significant for very large cases and depends on the disk type. In a small non-exhaustive test, extracting a 1.5GB archive (3.2GB uncompressed) on a 5400rpm HDD took 1’53” while the same operation took 27” on an SSD.

To pass an archive file to nanoFluidX, use -k or --archive commandline options as

$ CUDA_VISIBLE_DEVICES=0 mpiexec –np 1 $nFX_SP –k archive.zip

or

$ CUDA_VISIBLE_DEVICES=0 mpiexec –np 1 $nFX_SP --archive archive.zip

The same options (-k or --archive) apply for nFX[p]. To use nFX[c] in prep archive mode, pass the archive file via --nfxpk or --nFXp_k.

Since the file extensions used by nanoFluidX (and its auxiliaries) are not fixed, there is no way to determine whether an archive file or a .cfg file was intended to be passed in. As a result, the errors resulting from mismatch between the commandline option and the file passed in (for example, passing a .cfg file with -k option or passing an archive file with -i option, may not directly point to the issue.