Amber masthead
Filler image AmberTools18 Amber18 Manuals Tutorials Force Fields Contacts History
Filler image

Useful links:

Amber Home
Download Amber
Installation
News
Amber Citations
GPU Support
Features
Get Started
Benchmarks
Hardware
Logistics
Patches
Intel Support
Updates
Mailing Lists
For Educators
File Formats

PMEMD runs on many models of NVIDIA GPUs

| Building Your Own System | AMBER Certified Hardware Solutions |

GPU accelerated PMEMD has been implemented using CUDA and thus will only run on NVIDIA GPUs at present. The code uses a custom designed hybrid single / double / fixed precision model termed SPFP which requires a minimum hardware revision or 3.0 (Kepler, GTX680). For price and performance reasons at this time we generally recommend the GeForce cards over the more expensive Tesla or Quadro variants. There are no issues, accuracy or otherwise, with running AMBER on GeForce GPUs.

In addition to the general information presented here, Ross Walker has prepared sections with details on building your own GPU machine for Amber use, or for obtaining certified machines from Exxact Corporation which come with AMBER 18, and other Life Sciences or Deep Learning Software as requested, preinstalled and validated.

At the time of writing the following cards are supported by AMBER 18:

  • Hardware Version 7.5 (Turing)
    • RTX-2080Ti
    • RTX-2080
    • RTX-2070(untested)
  • Hardware Version 7.0 (Volta V100)
    • Titan-V
    • V100
    • Quadro GV100
  • Hardware Version 6.1 (Pascal GP102/104)
    • Titan-XP [aka Pascal Titan-X]
    • GTX-1080TI / 1080 / 1070 / 1060
    • Quadro P6000 / P5000
    • P4 / P40
  • Hardware Version 6.0 (Pascal P100/DGX-1)
    • Quadro GP100 (with optional NVLink)
    • P100 12GB / P100 16GB / DGX-1
  • Hardware Version 5.0 / 5.5 (Maxwell)
    • M4, M40, M60
    • GTX-Titan-X
    • GTX970 / 980 / 980Ti
    • Quadro cards supporting SM5.0 or 5.5
  • Hardware Version 3.0 / 3.5 (Kepler I / Kepler II)
    • Tesla K20 / K20X / K40 / K80
    • Tesla K10 / K8
    • GTX-Titan / GTX-Titan-Black / GTX-Titan-Z
    • GTX770 / 780
    • GTX670 / 680 / 690
    • Quadro cards supporting SM3.0 or 3.5

While we recommend CUDA 9.1 or 9.2 for the best speed of the resulting executables, the following compilers revisions are the minimum requirements for different tiers of hardware:

  • Turing (SM_75) based cards require CUDA 9.2 or later.
  • Volta (V100 - SM_70) based cards require CUDA 9.0 or later.
  • Pascal (GP102/104 - SM_61) based cards (GTX-1080TI / 1080 / 1070 / 1060 and Titan-XP) require CUDA 8.0 or later.
  • GTX-1080 cards require NVIDIA Driver version >= 367.27 for reliable numerical results.
  • GTX-Titan and GTX-780 cards require NVIDIA Driver version >= 319.60 for correct numerical results.
  • GTX-780Ti cards require a modified Bios from Exxact Corp to give correct numerical results.
  • GTX-Titan-Black Edition cards require NVIDIA Driver version >= 337.09 or 331.79 or later for correct numerical results.

Other cards not listed here may also be supported as long as they implement Hardware Revision 3.0, 3.5, 5.0, 5.5, 6.0, 6.1, 7.0 or 7.5 specifications.

Note that you should ensure that all GPUs on which you plan to run PMEMD are connected to PCI-E 2.0 x16 lane slots or better. If this is not the case then you will likely see degraded performance, although this effect is lessened in serial if you write to the mdout or mdcrd files infrequently (e.g. every 2000 steps or so). Scaling over multiple GPUs within a single node is not really feasible anymore for PME calculations given the interconnect performance has not kept pace with improvements in the individual GPU performance. However, it is still possible to get good multi-GPU scaling for implicit solvent calculations larger than 2500 atoms if all GPUs are in x16 or better slots and can communicate via peer to peer (i.e. connected to the same physical processor socket).

For a detailed writeup on PCI-E layouts in modern hardware and the variations in peer to peer support see the following write-up: [Exploring the complexities of PCI-E connectivity]. It is also possible to run over multiple nodes, although you are unlikely to see any performance benefit and thus it is not recommended except for loosely coupled runs such as REMD. The main advantage of AMBER's approach to GPU implementation over other implementations such as NAMD and Gromacs is that it is possible to run multiple single GPU runs on a single node with little or no slow down. For example, a node with 4 RTX2080TI GPUs cards can run 4 individual AMBER DHFR 4fs NVE calculations all at the same time without slowdown providing an aggregate throughput in excess of 2500ns/day.

Building Your Own System

By Ross Walker (ross __at__ rosswalker.co.uk)

If you are happy putting together your own machines from individual components then you can build unbelievably fast AMBER GPU machines for very little money. Your main considerations are a suitable motherboard, a processor with at least 1 core per GPU and a power supply beefy enough to run everything. Simple 2 or 3 GPU systems can be built for around $3500 INCLUDING THE GPUS! Here's a recommended shopping list of parts for building reliable high performing AMBER GPU machines. This machine runs the DHFR NVE HMR 4fs benchmark at over 700ns/day using just one of the GPUs! The system as specced can support up to 3 GPUs, with a 1600W power supply although we do not recommend going beyond 2GPUs in such a system (you can actually fit 4 in but I have seen issues with overheating with all 4 GPUs in use, and there is limited clearance for the 4th GPU). With 2 GPUs you can run two calculations all at the same time (one on each GPU) without impacting performance.

Amazon

Prices current as of Oct 2018
(Hover mouse over links for current prices)

1 x Nanoxia Deep Silence 3 Mid Tower Case ~ $89.99
1 x EVGA Supernova P2 80 Plus Platinum Rated 1600-Watt Modular ATX Power Supply ~ $366.46
1 x ASUS Z270-WS LGA1151 DDR4 ATX Motherboard ~ $368.37
1 x Samsung 860 EVO 1TB 2.5 Inch SATA III Internal SSD ~ $162.99
1 x Crucial 32GB Kit (8GBx4) DDR4 2666 MT/s (PC4-21300) ~ $271.99
1 x Intel Core i7-9700K Desktop Processor 8 Cores up to 4.9 GHz Turbo ~ $409.99
1 x EVGA CLC 240 Liquid/Water CPU Cooler ~ $79.99
2 x ZOTAC Gaming GeForce RTX 2080 Blower 8GB ~ $769.99

Total Price: ~ $3289.76 for 1 machine [2 GPUs] (as of Oct 2018)

AMBER Certified Hardware Solutions

By Ross Walker (ross _at_ rosswalker.co.uk)

In order to make AMBER GPU computing as simple and cost effective as possible we work with Exxact Corporation to provide a number of pre-configured, fully warranted [even with GeForce cards] and optimized turn-key desktop and cluster solutions specifically designed for running AMBER simulations. These are discussed in more detail below with the most up to date configurations available on Exxact's AMBER MD Workstation page. Recent work with Exxact has also extended this to optimized solutions for a wide range of life science applications, GPU Accelerated Cryo-EM with Relion and GPU accelerated NVIDIA Digits Dev Boxes for machine learning.

AMBER Certification

One of the biggest challenges with GPU computing is knowing what the optimal configuration is. If you listen to NVIDIA Sales Reps or you go to a tier one vendor such as Dell or HPe you will likely end up paying a large amount for a sub-optimal machine. To make things as simple as possible we designed the AMBER certification process which involves offering turn-key solutions that conform to the following:

  1. Technical and sales personnel trained by AMBER developers and familiar with AMBER requirements.
  2. Vendor staff have direct link to AMBER developers for technical support and troubleshooting.
  3. All hardware specifications, and custom requests, are approved and tested by AMBER developers.
  4. AMBER developer approved installation, configuration and testing including applying latest updates.
  5. Example submission scripts and pre-configured batch queuing systems (clusters) and automatic AMBER update scripts are provided.
  6. A fully configured serial, parallel and GPU AMBER computing environment is provided for all users.
  7. 24 hour individual GPU burn-in and full numerical validation using AMBER developer designed GPU test suite.
  8. Comprehensive benchmark report and performance validation for all GPUs.
  9. Full vendor 3 year warranty on all components (including GeForce and Tesla GPUs).
  10. All systems are verified personally by an AMBER developer before shipping.
  11. AMBER 18 preinstalled, validated and benchmarked.

The goal of this program is to make it simple to purchase optimum reliable and cost effective AMBER GPU computing solutions without the need for an understanding of GPU or CPU hardware. If you know how to run simulations with AMBER then you will be able to run simulations immediately after powering up an AMBER certified system without any required configuration or installation procedures. Support can also be provided for equipment requests in proposals with text describing optimum hardware-software co-design available as needed. Due to the success of this program it has recently been extended to a range of life science applications in the form of the Exxact Life Sciences Certified GPU Computing Program.

Exxact AMBER Certified MD Workstation and SimCluster

The main driving force behind the AMBER GPU development has always been to bring supercomputer like performance to individual desktops at a price that is appropriate for the widest range of researchers possible. The motivation is maximizing the amount and quality of the science that can be done rather than chasing artificially large grand challenge problems with massive supercomputers. Think of it as Molecular Dynamics for the 99%.

To make it as simple as possible for AMBER users to purchase optimal workstations and small clusters for running GPU AMBER (and regular CPU AMBER simulations as well) we have teamed up with Exxact Corporation to co-design a series of machines that provide, in our opinion, the optimum price performance ratio within three specific categories:

  1. Individual Workstations in the $2,500 to $10,000 range. These use GeForce gaming cards but in our experience and that of a large number of users provide excellent reliability and unparalleled performance.
     
  2. Individual high end workstations and rack mount nodes in the $5,000 to $18,000 range. These machines use either GeForce cards (RTX2080TI, Titan-XP [Pascal], GTX-1080TI etc) or, if requested, professional Tesla boards (P100, V100 etc) and provide very high GPU densities (up to 8 GPUs in a single box).
     
  3. Small clusters. These can be custom built for just about any price range and can accommodate either the enterprise Tesla boards (P100, V100 etc) or the very cost effective GeForce cards (Titan-XP [Pascal], RTX2080TI, GTX-1080, GTX-1080TI etc) providing stunning performance for extremely reasonable prices.

The following are three example machine configurations, co-designed in collaboration with Exxact. It is also possible to order these machines configured for a range of life sciences applications in addition to AMBER. For more details please contact Ross Walker (ross _at_ rosswalker.co.uk) or Nick Chen (nick@exxactcorp.com) mentioning that you are interested in GPU computing solutions for running AMBER.

Exxact - AMBER Certified Workstations

AMBER Certified
Entry-Level Workstation
AMBER Certified
Mid-Level Workstation
AMBER Certified
High-End Workstation

Ideal for Graduate Students

Ideal for Researchers

Maximum Performance

• 1x Intel Core i9-7900X CPU
• 1 or 2 x RTX 2080TI or 2080 GPUs
• 32 GB system memory
• AMBER18 preinstalled, tested & optimized
• CentOS 7
• 3 year warranty

• 2x Intel Silver 4110 CPUs
• 2 to 4 x RTX 2080TI, or P100/V100
• 64 GB system memory
• AMBER18 preinstalled, tested & optimized
• CentOS 7
• 3 year warranty

• 2x Intel Gold 5120 CPUs
• 4x GTX 1080TI or P100/V100
• 64 GB system memory
• AMBER18 preinstalled, tested & optimized
• CentOS 7
• 3 year warranty

~$4499 ~$7499 ~$9499
Rack mount nodes and cluster configurations available on request.
Example Specs

Disclosure: Exxact contribute to funding AMBER GPU development and research.

"How's that for maxed out?"

Last modified: Oct 29, 2018