AMBER 16 GPU
ACCELERATION SUPPORT

| Background | Authorship & Support | Supported Features | Supported GPUs |
| Accuracy Considerations | Installation and Testing | Running GPU Accelerated Simulations |
| Considerations for Maximizing GPU Performance | Benchmarks |
| Recommended Hardware & Test Drives |
| Return to Main Amber Page |

This page describes AMBER 16 GPU support.
If you are using AMBER 14 please see the archived AMBER 14 page here.

Recommended Hardware

by Ross Walker.

| Preconfigured Turn-Key AMBER Certified Tesla and GeForce Machines |
| Free Test Drives | Building your own Machine |

The AMBER GPU implementation has been designed to work on a broad range of hardware. Essentially the only thing you need is a NVIDIA GPU supporting hardware revision 2.0, 3.0, 3.5, 5.0 or later. However, there are some considerations when it comes to maximizing performance both in serial and parallel.

This page has been designed to help you select suitable hardware configurations for running AMBER. If you want a simple, hassle free, solution to GPU computing with AMBER we recommend the Exxact AMBER Certified Solutions described below. These have been developed jointly between Exxact and the AMBER developers specifically to make GPU MD simulations as fast and hassle free as possible. They are shipped with AMBER 16 preinstalled (AMBER 16 license required) and include full GPU validation, 3 years warranty on all components and a full benchmark report. This program has recently been extended to Exxact Certified Solutions for a range of life sciences applications, Cryo-EM and also to Machine Learning and Deep Neural Networks.
 

 

Serial

In serial the performance for each independent AMBER GPU job is, assuming NTPR, NTWX etc are large enough, mostly independent of the underlying CPU, motherboard chipset, the PCI-E bandwidth and the number of GPUs per node. At a minimum you need as least one free CPU core per GPU. If you are building small desktops to run serial calculations then multiple GPUs per node will be the most cost effective. Ideally you should still try to keep the GPUs on x16 PCI-E slots and make sure your power supply is sufficient to power all the GPUs under full load, and that you have adequate cooling.

Parallel

If you want to use multiple GPUs in parallel, rather than one simulation per GPU, then considerations change to the available bandwidth in the node (attempting to run across multiple nodes is not recommended except for Replica Exchange based simulations). With AMBER 16 the ideal specification for performance is 2 or 4 GPUs per node (8 GPUs is supported but needs server class hardware and 240V power) all in PCI-E Gen 3 x16 slots (or better). AMBER 16 uses peer to peer communication to provide optimum multi-GPU scaling. At the time of writing no standard motherboards exist that support more than two way peer to peer at full x16 speed (but we have a new unique custom-built system from Exxact that supports 4 and 8-way P2P simulations - see below). In most 4 GPU systems including the standard Exxact 4 GPU systems you are thus limited to the following combinations of runs [4 x 1 GPU; or 2 x 2 GPU; or 2 x 1 GPU + 1 x 2 GPU]. With an traditional 8 GPU system you would be limited to the following combinations [8 x 1 GPU; or 4 x 2 GPU; or 6 x 1 GPU + 1 x 2 GPU; or 4 x 1 GPU + 2 x 2 GPU; or 2 x 1 GPU + 3 x 2 GPU]. The new 8 and 10 GPU Peer to Peer solutions will add 2 x 4 GPU (and 1 x 8 GPU in AMBER 16 for large GB calculations) to those options.

Pre-configured AMBER Certified Optimal Solutions

In order to make AMBER GPU computing as simple and cost effective as possible we have teamed up with Exxact Corporation to provide a number of pre-configured, fully warranted [even with GeForce cards] and optimized turn-key desktop and cluster solutions specifically designed for running AMBER simulations. These are discussed in more detail below with the most up to date configurations available on Exxact's AMBER MD Workstation page. Recent work with Exxact has also extended this to optimized solutions for a wide range of life science applications, GPU Accelerated Cryo-EM with Relion and GPU accelerated NVIDIA Digits Dev Boxes for machine learning.

AMBER Certification

One of the biggest challenges with GPU computing is knowing what the optimal configuration is. If you go to a tier one vendor such as Dell you will likely end up paying a large amount for a sub-optimal machine. To make things as simple as possible we designed the AMBER certification process which is currently offered as sole source by Exxact Corporation. This certification involves offering turn-key solutions that conform to the following:

  1. Technical and sales personnel trained by AMBER developers and familiar with AMBER requirements.
  2. Vendor staff have direct link to AMBER developers for technical support and troubleshooting.
  3. All hardware specifications, and custom requests, are approved and tested by AMBER developers.
  4. AMBER developer approved installation, configuration and testing including applying latest updates.
  5. Example submission scripts and pre-configured batch queuing systems (clusters) and automatic AMBER update scripts are provided.
  6. A fully configured serial, parallel and GPU AMBER computing environment is provided for all users.
  7. 24 hour individual GPU burn-in and full numerical validation using AMBER developer designed GPU test suite.
  8. Comprehensive benchmark report and performance validation for all GPUs.
  9. Full vendor 3 year warranty on all components (including GeForce and Tesla GPUs).
  10. All systems are verified personally by an AMBER developer before shipping.
  11. All AMBER v14 systems shipped after Dec 20th 2015 include a specially optimized version of AMBER 14 that provides performance equivalent to AMBER 16 (~15% better performance than that available on comparable hardware from other vendor). Systems can be selected with either AMBER 14 or 16 preinstalled.

The goal of this program is to make it simple to purchase optimum reliable and cost effective AMBER GPU computing solutions without the need for an understanding of GPU or CPU hardware. If you know how to run simulations with AMBER then you will be able to run simulations immediately after powering up an AMBER certified system without any required configuration or installation procedures. Support can also be provided for equipment requests in proposals with text describing optimum hardware-software co-design available as needed. Due to the success of this program it has recently been extended to a range of life science applications in the form of the Exxact Life Sciences Certified GPU Computing Program.

 

Exxact AMBER Certified MD Workstation and SimCluster

The main driving force behind the AMBER GPU development has always been to bring supercomputer like performance to individual desktops at a price that is appropriate for the widest range of researchers possible. The motivation is maximizing the amount and quality of the science that can be done rather than chasing artificially large grand challenge problems with massive supercomputers. Think of it as Molecular Dynamics for the 99%.

To make it as simple as possible for AMBER users to purchase optimal workstations and small clusters for running GPU AMBER (and regular CPU AMBER simulations as well) we have teamed up with Exxact Corporation to co-design a series of machines that provide, in our opinion, the optimum price performance ratio within three specific categories:

  1. Individual Workstations in the $2000 to $8000 range. These use GeForce gaming cards but in our experience and that of a large number of users provide excellent reliability and unparalleled performance.
     
  2. Individual high end workstations in the $5000 to $15000 range. These machines use either GeForce cards (Titan-XP [Pascal], GTX-Titan-X, GTX-1080 etc) or, if requested, professional Tesla boards (K40, K80 etc) and provide very high GPU densities (up to 8 GPUs in a single box).
     
  3. Small clusters. These can be custom built for just about any price range and can accommodate either the enterprise Tesla boards (K40, K80 etc) or the very cost effective GeForce cards (Titan-XP [Pascal], GTX-Titan-X, GTX-1080 etc) providing stunning performance for extremely reasonable prices.

The following are three example machine configurations, co-designed in collaboration with Exxact. These are the machines that were used to obtain the benchmarks shown on the AMBER GPU benchmark page. They come as AMBER certified platforms, which includes AMBER 16 fully installed, tested and configured (AMBER 16 license required). They carry full 3 year warranties (even when configured with GeForce cards), are 'burnt in' for a minimum of 24 hours using AMBER 16 to verify performance, reliability and numerical correctness of all hardware (this is what makes it possible to offer reliable GeForce solutions) and can be customized as desired.

It is also possible to order these machines configured for a range of life sciences applications in addition to AMBER. For more details please contact Ross Walker (ross _at_ rosswalker.co.uk) or Mike Chen (mchen@exxactcorp.com) mentioning that you are interested in GPU computing solutions for running AMBER.

Exxact - AMBER Certified Workstations

AMBER Certified
Entry-Level Workstation
AMBER Certified
Mid-Level Workstation
AMBER Certified
High-End Workstation

Ideal for Graduate Students

Ideal for Researchers

Maximum Performance

• 1x Intel Core i7-4930K CPU
• 1 or 2 x NVIDIA GTX 1080 or 1070 GPUs
• 32 GB system memory
• AMBER16 preinstalled, tested and optimized
• CentOS 6 or 7
• 3 year warranty

• 2x Intel Xeon E5-2620 v4 CPUs
• 2 to 4 x NVIDIA GTX 1080, Titan-X [Pascal] or K40/K80/M40/M60/P40
• 64 GB system memory
• AMBER16 preinstalled, tested and optimized
• CentOS 6 or 7
• 3 year warranty

• 2x Intel Xeon E5-2640 v4 CPUs
• 4x NVIDIA GTX 1080, Titan-X [Pascal] or K40/K80/M40/M60/P40
• 64 GB system memory
• AMBER16 preinstalled, tested and optimized
• CentOS 6 or 7
• 3 year warranty

~ $3999 ~$5999 ~$7999
Example Spec Example Spec Example Spec

Exxact AMBER Certified Rack Mount Nodes

2U x 4 GPU AMBER Certified
Rack Mount Node
(NEW) 4U x 8 GPU AMBER Certified Rack Mount Node

High Density - 4 GPUs per node - supports peer to peer between pairs of GPUs, run 4 x 1 GPU or 2 x 2 GPU per node.

Cost Effective - Supports 8 GPUs per node - with new PLX switched peer to peer between pairs of GPUs - you can run 8 x 1 GPU, 4 x 2 GPU or any combination in between.

• 2x Intel Xeon E5-2600 v4 Broadwell CPUs
• Up to 4 x NVIDIA GTX1080 / Titan-X [Pascal] or Tesla K40/K80/M40/M60/P40 GPUs
• Up to 1TB system memory
• AMBER16 preinstalled, tested and optimized
Onboard QDR or FDR infiniband available as option.
• CentOS 6 or 7
• 3 year warranty

• 2x Intel Xeon E5-2600 v4 Broadwell CPUs
• Up to 8 x NVIDIA GTX980 / Titan-X [Pascal] or K40/K80/M40/M60/P40 GPUs
• Up to 768GB system memory
• AMBER16 preinstalled, tested and optimized
Onboard QDR or FDR infiniband available as option
• CentOS 6 or 7
• 3 year warranty

Contact for Price Contact for Price
Example Spec Example Spec

These machines can be customized to fit a specific budget, please contact myself (ross _at_ rosswalker.co.uk) or Mike Chen at Exxact (mchen@exxactcorp.com) for help and advice. Single socket GTX1070 solutions are also available for prices starting around $2000. Whisper quiet (24db) dual GTX1070 and GTX1080 desktop machines are also available.

Exxact, is family owned, has been in business for over 30 years, and is GSA Compliant. As the only supplier of AMBER Certified GPU Workstations and Clusters they can provide sole source justification statements as needed as well support in providing quotes and hardware details for proposals. These desktops and clusters have been sold to numerous universities, pharmaceutical companies, biotech companies and national labs world wide. By purchasing from Exxact you are also helping to support future AMBER development through the lab of Prof. Ross Walker. If you would like contact details for existing customers please contact ross@rosswalker.co.uk.

Exxact AMBER Certified Mid-Level workstation performance. GPU AMBER can be run in two modes, either using GPUs in parallel to run a single MD calculation (A) or using each GPU in serial to run independent MD calculations (B). The latter is what separates AMBER from other codes such as NAMD and Gromacs which rely on both the CPU and GPU for computation and thus do not support mode (B) efficiently.

Clusters

Clusters tend to be custom designed and should be configured for the specific user's needs and budget. Exxact staff have been trained by AMBER developers on the best way to design custom clusters for running AMBER (and other life sciences applications), both GPU and CPU. Many configurations are possible including 2U and 4U nodes with either 1 to 8 GTX-1080/Titan-X [Pascal] or 1 to 8 K20(X)/K40/K80/M40/M60/P40 GPUs. Both the 2U and 4U configurations have been fully certified and tested with Tesla and GeForce GPUs and carry full warranties with next business day onsite being available if needed. The following page provides a typical cluster example:


Exxact 12U -  5 node x 16 GTX-Titan-X cluster
(Also available as a 5 node - 10U or 20U configuration)

For more details or to obtain a custom quote please contact Mike Chen (mchen@exxactcorp.com) at Exxact Corp or Ross Walker (ross@rosswalker.co.uk) at SDSC.

Exxact Custom 8 and 10 GPU Peer to Peer Solutions

Through an engineering collaboration with Exxact we are pleased to be able to offer unique peer to peer optimized systems as AMBER Certified Peer to Peer GPU MD solutions. These systems can be configured with up to 10 GPUs in a single system image with peer to peer communication supported across all GPUs. These systems were used to obtain the 4 GPU timings provided on the benchmark page (and the 8 GPU Cellulose GB timings). While they have a price premium over the regular systems they provide the extra flexibility of being able to run individual calculations across 1, 2 or 4 GPUs if desired.

^

Free Test Drives

If you are interested in measuring the performance of AMBER running your own simulation on the machines shown above then we encourage you to take advantage of the free test drive program we have put together with Exxact. Test accounts on these machines are available free of charge in blocks of 24 hours for you to try things out for yourself. Please see the following page for AMBER MD Workstation Test Drive signup details.

^

Building your own System

If you are happy putting together your own machines from individual components then you can build unbelievably fast AMBER GPU machines for very little money. Your main considerations are a suitable motherboard, a processor with at least 1 core per GPU and a power supply beefy enough to run everything. Simple 2 or 3 GPU systems can be built for around $3500 INCLUDING THE GPUS! Here's a recommended shopping list of parts for building reliable high performing AMBER GPU machines. This machine runs the DHFR NVE HMR 4fs benchmark at over 400ns/day using just one of the GPUs! The system as specced can support up to 3 GPUs, with a 1600W power supply (you can actually fit 4 in but I have seen issues with overheating with all 4 GPUs in use, and there is limited clearance for the 4th GPU). With 3 GPUs you can run three calculations all at the same time (one on each GPU) without impacting performance. For ideal multi-GPU performance you should configure it with 2 GPUs which will, due to the PCI-E switch on the motherboard, supports peer to peer 2xGPU runs.

Amazon
Prices current as of May 31st 2016
(Hover mouse over links for current prices)

1 x Nanoxia Deep Silence 3 Mid Tower Case ~ $89.99

1 x EVGA Supernova P2 80 Plus Platinum Rated 1600-Watt Modular ATX Power Supply ~ $347.30
(1200W version if fine if you plan to only have 2 GPUs)

1 x ASUS ATX DDR4 3000 (o.c.) LGA 2011 Motherboards X99-E WS ~ $520.62

1 x Samsung 850 EVO 1 TB 2.5-Inch SATA III Internal SSD ~ $317.04

2 x Crucial 16GB Kit (8GBx2) DDR4 2133 MT/s (PC4-17000) CT2K8G4DFD8213  $54.94 each

1 x Cooler Master Hyper 212 EVO - CPU Cooler with 120mm PWM Fan (RR-212E-20PK-R2) Cooler Master Headsink $29.49

1 x Intel Core i7-5930K Haswell-E 6-Core 3.5GHz LGA 2011-v3 140W BX80648I75930K $599.99

3 x MSI GeForce GTX 1080 Founders Edition ~ $699.99 each

Total Price: ~ $4114.28 for 1 machine [3 GPUs] (as of May 2016)

^

Disclosure: Exxact contribute to funding AMBER GPU development and research.