Supported GPUs

GPU accelerated PMEMD has been implemented using CUDA and thus will only run on NVIDIA GPUs at present. Due to accuracy concerns with pure single precision the code uses a custom designed hybrid single / double / fixed precision model termed SPFP. This places the requirement that the GPU hardware supports both double precision and integer atomics meaning only GPUs with hardware revision 2.0 and later can be used. Support for hardware revision 1.3 was present in previous versions of the code but for code complexity and maintenance reasons has been deprecated in AMBER 14. At the time of Amber's release the following cards are supported (* = untested):

Caution: With the exception of Exxact AMBER certified cards the GTX-780Ti cards are NOT recommended at this time due to instability and numerical accuracy during AMBER MD simulations which we have tracked down to a specific hardware design flaw. In collaboration with Exxact we have developed a custom GTX780Ti bios that circumvents this hardware flaw and makes the cards fully stable for CUDA workloads.

GTX-Titan and GTX-780 cards require NVIDIA Driver version >= 319.60.

GTX-Titan-Black Edition cards require NVIDIA Driver version >= 337.09 .or. 331.79 or later.

GTX-970 / 980 / Titan-X cards are supported but require at least CUDA v6.5.

Other cards not listed here may also be supported as long as they correctly implement the Hardware Revision 2.0, 3.0, 3.5, 5.0 or 5.5 specifications.

Note that you should ensure that all GPUs on which you plan to run PMEMD are connected to PCI-E 2.0 x16 lane slots or better, especially when running in parallel across multiple GPUs which really needs PCI-E 3.0 x16 with peer to peer support for reasonable scaling. If this is not the case then you will likely see degraded performance, although this effect is lessened in serial if you write to the mdout or mdcrd files infrequently (e.g. every 2000 steps or so). Scaling over multiple GPUs within a single node is possible, if all are in x16 or better slots and can communicate via peer to peer [i.e. connected to the same physical processor socket]. It is also possible to run over multiple nodes using infiniband but not recommended except for loosely coupled runs such as REMD. The main advantage of AMBER's approach to GPU implementation over other implementations such as NAMD and Gromacs is that it is possible to run multiple single GPU runs on a single node with little or no slow down. For example a node with 4 GTX-Titan-X cards could run 4 individual AMBER DHFR 4fs NVE calculations all at the same time without slowdown providing an aggregate throughput in excess of 1250ns/day.

To simplify the process of selecting supported hardware we have teamed with Exxact Corp to provide AMBER Certified Life Sciences GPU computing solutions which are now in use in >80 major universities worldwide as well as multiple major pharma and biotech companies. For more information please refer to the Recommended Hardware section.

Optimum Hardware Designs for Running GPU Amber are provided on the Recommended Hardware page.

System Size Limits

In order to obtain the extensive speedups that we see with GPUs it is critical that all of the calculation take place on the GPU within the GPU memory. This avoids the performance hit that one takes copying to and from the GPU and also allows us to achieve extensive speedups for realistic size systems. This avoids the need to create systems with millions of atoms to show reasonable speedups even when sampling lengths are unrealistic. This unfortunately means that the entire calculation must fit within the GPU memory. Additionally we make use of a number of scratch arrays to achieve high performance. This means that the GPU memory usage can actually be higher than a typical CPU run. It also means, due to the way we had to initially implement parallel GPU support that the memory usage per GPU does NOT decrease as you increase the number of GPUs. This is something we hope to fix in the future but for the moment the atom count limitations imposed on systems by the GPU memory is roughly constant whether you run in serial or in parallel.

Since, unlike CPUs it is not possible to add more memory to a GPU (without replacing it entirely) and there is no concept of swap as there is on the CPU the size of the GPU memory imposes hard limits on the number of atoms supported in a simulation. Early on within the mdout file you will find information on the GPU being used and an estimate of the amount of GPU and CPU memory required:

|------------------- GPU DEVICE INFO --------------------
|
| CUDA Capable Devices Detected: 1
| CUDA Device ID in use: 0
| CUDA Device Name: Tesla C2070
| CUDA Device Global Mem Size: 6143 MB
| CUDA Device Num Multiprocessors: 14
| CUDA Device Core Freq: 1.15 GHz
|
|--------------------------------------------------------

...

| GPU memory information:
| KB of GPU memory in use: 4638979
| KB of CPU memory in use: 790531

The reported GPU memory usage is likely an underestimate and meant for guidance only to give you an idea of how close you are to the GPU's memory limit. Just because it is less than the available Device Global Mem Size does not necessarily mean that it will run. You should also be aware that the GPU's available memory is reduced by 1/9th if you have ECC turned on.

Memory usage is affected by the run parameters. In particular the size of the cutoff, larger cutoffs needing more memory, and the ensemble being used. Additionally the physical GPU hardware affects memory usage since the optimizations used are non-identical for different GPU types. Typically, for PME runs, memory usage runs:

NPT > NVT > NVE
NTT=3 > NTT=1 == NTT=2 > NTT=0
Barostat=1 > Barostat=2

Use of restraints etc will also increase the amount of memory in use. As will the density of your system. The higher the density the more pairs per atom there are and thus the more GPU memory will be required. The following table provides an approximate UPPER BOUND to the number of atoms supported as a function of GPU model. These numbers were estimated using boxes of TIP3P water (PME) and solvent caps of TIP3P water (GB). These had lower than optimum densities and so you may find you are actually limited for dense solvated proteins to around 20% less than the numbers here. Nevertheless these should provide reasonable estimates to work from.

All numbers are for SPFP precision and are approximate limits. The actual limits will depend on system density, simulation settings etc. These numbers are thus designed to serve as guidelines only.

Explicit Solvent (PME)

8 angstrom cutoff. Cubic box of TIP3P Water, NTT=3, NTB=2, NTP=1,NTF=2,NTC=2,DT=0.002.

Implicit Solvent (GB)

No cutoff, Sphere of TIP3P Water (for testing purposes only), NTT=3, NTB=0, NTF=2, NTC=2, DT=0.002, IGB=1

GPU		GB (max atoms)
type	memory	SPFP
GTX580	3.0 GB	1,195,000
M2090 (ecc off)	6.0 GB	1,740,000
GTX680	2.0 GB	960,000
K10 (ecc off) / GTX780 / GTX980	4.0 GB	1,310,000
K20X (ecc off) / GTX-Titan / GTX-Titan-Black	6.0 GB	1,750,000
K40 / K80 / M40 (ecc off) / GTX-Titan-X / GTX980TI	12.0 GB	3,350,000

^

Accuracy Considerations

The nature of current generation GPUs is such that single precision arithmetic is considerably faster (>23x for K10, >2x for K20) than double precision arithmetic. This poses an issue when trying to obtain good performance from GPUs. Traditionally the CPU code in Amber has always used double precision throughout the calculation. While this full double precision approach has been implemented in the GPU code it gives very poor performance. Thus the AMBER GPU implementation was initially written to use a combination of single and double precision, termed hybrid precision (SPDP), that is discussed in further detail in the references provided at the beginning of this page. This approach used single precision for individual calculations within the simulation but double precision for all accumulations. It also used double precision for shake calculations and for other parts of the code where loss of precision was deemed to be unacceptable. Tests have shown that energy conservation is equivalent to the full double precision code and specific ensemble properties, such as order parameters, match the full double precision CPU code. The user should understand though that this approach leads to divergence between GPU and CPU simulations, similar to that observed when running the CPU code across different processor counts in parallel but occurring more rapidly. For this reason the GPU test cases are more sensitive to rounding differences caused by hardware and compiler variations and will likely require manual inspection of the test case diff files in order to verify that the installation is providing correct results. There can also be differences between CPU and GPU runs and between GPU and GPU on different GPU models for runs that rely on the random number stream. For example NTT=2 and NTT=3. This is because the random number stream is different between CPUs and GPUs.

With AMBER 14 the single and double precision models (SPSP, SPDP and DPDP) have been deprecated and replaced with a new hybrid model that combines single precision calculation with fixed precision accumulation. This is described in detail in the following manuscript and tests have shown that it provides accuracy that is as good or better than the original SPDP model. This new precision model is supported on v2.0 hardware and newer and is especially beneficial on Kepler I (3.0) hardware.

Scott Le Grand; Andreas W. Goetz; & Ross C. Walker* "SPFP: Speed without compromise - a mixed precision model for GPU accelerated molecular dynamics simulations.", Comp. Phys. Comm, 2013, 184, pp374-380, DOI: 10.1016/j.cpc.2012.09.022

While the default precision model is currently the new SPFP model there is also a DPFP model available which provides full double precision calculation with bitwise reproducible fixed precision accumulation to facilitate advanced testing and comparison. The precision models supported in AMBER 14, and determined at compile time as described later, are:

• SPFP - (Default) Uses a combination of single precision for calculation and fixed (integer) precision for accumulation. This approach provides optimum speedup on Kepler cards, minimizes memory overhead and provides greater net precision than the original SPDP model. It is designed for optimum performance on hardware revision 2.0 or later.
   
• DPFP - Uses double precision (and double precision equivalent fixed precision) for the entire calculation. This provides for careful regression testing against the CPU code. It makes no additional approximations above and beyond the CPU implementation and would be the model of choice if performance was not a consideration. On v2.0 NVIDIA hardware (e.g. M2090) the performance is approximately half that of the SPFP model while on v3.0 NVIDIA hardware (e.g. K10) the performance is substantially less than the SPFP model.

Recommendation for Minimization

One limitation of the SPFP precision model is that force can be truncated if they overflow the fixed precision representation. This should never be a problem during MD simulations for any well behaved system. However, for minimization or very early in the heating phase it can present a problem. This is especially true if two atoms are close to each other and thus have large VDW repulsions. It is recommended therefore that for minimization you use the CPU version of the code. Only in situations where you are confident the structure is reasonable, for example if it was a snapshot from dynamics, should you use the GPU code (SPFP) for minimization.

^

Installation and Testing

The single GPU version of PMEMD is called pmemd.cuda while the multi-GPU version is called pmemd.cuda.MPI. These are built separately from the standard serial and parallel installations. Before attempting to build the GPU versions of PMEMD you should have built and tested at least the serial version of Amber and preferably the parallel version as well. This will help ensure that basic issues relating to standard compilation on your hardware and operating system do not lead to confusion with GPU related compilation and testing problems. You should also be reasonably familiar with Amber's compilation and test procedures. The minimum requirements for building the GPU version of PMEMD are, as of the AMBER 14 release:

• NVIDIA Toolkit v5.0, 5.5, 6.0, 6.5 or 7.5. (v6.5 or later required for Maxwell [GTX970/980/Titan-X] cards)
   
• NVIDIA GPU supporting Hardware Revision 2.0, 3.0, 3.5 and later.
   
• NVIDIA CUDA Driver v319.60 or later (v337.09 or later required for GTX-780Ti and GTX-Titan-Black)
   
• MPI library for parallel GPU. (MVAPICH2 v1.8 or later / MPICH2 v1.4p1 or later recommended)

It is assumed that you have already correctly installed and tested CUDA support on your GPU. Before attempting to build pmemd.cuda you should make sure you have correctly installed both the NVIDIA Toolkit (nvcc compiler) and a CUDA supporting NVIDIA driver. You may want to try downloading the NVIDIA CUDA SDK (available from http://www.nvidia.com/ or included as 'samples' in the toolkit v5.0 or later) and see if you can build that. Additionally the environment variable CUDA_HOME should be set to point to your NVIDIA Toolkit installation and $CUDA_HOME/bin/ should be in your path. At the time of release CUDA 5.0 or later is required with 5.0, 5.5, 6.0, 6.5 and 7.5 having been tested and officially supported. Note toolkit version 7.0 is not recommended due to performance regression and a bug in the GPU selection mechanism. As such 7.0 is considered unsupported. The current recommended version is CUDA 7.5.

Building and Testing the Default SPFP Precision Model

Single GPU (pmemd.cuda)

Assuming you have a working CUDA installation you can build the single GPU version, pmemd.cuda, using the default SPFP precision model as follows:

cd $AMBERHOME
make clean
./configure -cuda gnu    (or intel)
make install

Next you can run the tests using the default GPU (the one with the largest memory) with:

make test.cuda

The majority of these tests should pass. However, given the parallel nature of GPUs, meaning the order of operation is not well defined, the limited precision of the SPFP precision model, and variations in the random number generator on different GPU hardware, it is not uncommon for there to be several possible failures. You may also see some tests, particularly the cellulose test, fail on GPUs with limited memory. You should inspect the diff file created in the $AMBERHOME/logs/test_amber_cuda/ directory to manually verify any possible failures. Differences which occur on only a few lines and are minor in nature can be safely ignored. Any large differences, or if you are unsure, should be posted to the Amber mailing list for comment.

Multiple GPU (pmemd.cuda.MPI)

Once you have built and tested the serial GPU version you can optionally build the parallel version (if you have multiple GPUs of the same model). Unlike the CPU code it is not necessary to build the parallel version of the GPU code in order to access specific simulation options (except REMD). Thus you only need to build the parallel GPU code if you plan to run a single calculation across multiple GPUs. AMBER 12 had poor multi-GPU scaling for GPUs later than the Fermi (C2050/2070/2090) generation. This scaling has been improved significantly in AMBER 14 via the use of peer to peer communication over the PCI-E bus. While the underlying communication between GPUs is peer to peer if supported the code still requires MPI in order to run.

No special software beyond CUDA 5.0 and an MPI library is required to support peer to peer. For details on determining which GPUs can operate in peer to peer mode please see the multi-GPU section of the running on GPUs section below. The instructions here assume that you can already successfully build the MPI version of the CPU code. If you cannot, then you should focus on solving this before you move on to attempting to build the parallel GPU version.

The parallel GPU version of the code works using MPI v1 or later.

You can build the multi-GPU code as follows:

cd $AMBERHOME
make clean
./configure -cuda -mpi gnu    (or intel)
make install

Next you can run the tests using GPUs enumerated sequentially within a node (if you have multiple nodes or more complex GPU setups within a node then you should refer to the discussion below on running on multiple GPUs):

export DO_PARALLEL='mpirun -np 2' # for bash/sh
setenv DO_PARALLEL 'mpirun -np 2' # for csh/tcsh
./test_amber_cuda_parallel.sh

Building non-standard Precision Models

You can build different precision models as described below. However, be aware that this is meant largely as a debugging and testing issue and NOT for running production calculations. Please post any questions or comments you may have regarding this to the Amber mailing list. You select which precision model to compile as follows:

cd $AMBERHOME
make clean
./configure -cuda_DPFP gnu
make install

This will produce executables named pmemd.cuda_XXXX where XXXX is the precision model selected at configure time (SPFP or DPFP). You can then test this on the GPU with the most memory as follows:

cd $AMBERHOME/test/
./test_amber_cuda.sh DPFP (to test the DPFP precision model)

Testing Alternative GPUs

Should you wish to run the serial GPU tests on a GPU different from the one with the most memory (and lowest GPU ID if more than one identical GPU exists) then you can provide this by setting the CUDA_VISIBLE_DEVICES environment variable. For example, to test the GPU with ID = 2 and the default SPFP precision model you would specify:

cd $AMBERHOME
export CUDA_VISIBLE_DEVICES=2
make test.cuda

^