Architecture & Technical Details

Architecture

The Vikram-100 has 97 compute nodes, each with two Intel Xeon E5-2670v3 12-core Intel Haswell CPUs at 2.30 GHz, 256 GB RAM and 500 GB of local scratch storage. 20 of these nodes also have two Nvidia Tesla K40 GPU cards each card capable of 1.66 Tflops (double precision) . The HPC has a global high performance parallel filesystem based 300 TB storage that is shared across all nodes.


Vikram-100 has a internal primary 100% non-blocking FAT Tree Topology FDR (56 Gbits/Sec) Infiniband Network used for inter process communication between compute nodes and for global shared storage. In addition, it has a internal secondary Gigabit Ethernet network for management and backup purposes. It has two High Availability (HA) master nodes where users login to compile, debug and submit jobs. These two nodes constantly 'ping' each other, and if any of them goes down, the other one takes over to provide uninterrupted service to PRL users. Users data is regularly backed up onto latest generation LTO6 tapes on a 4 drive, 48 slot tape library.



Master Nodes

The HPC has two master nodes configured in HA (High Availability) mode. It means that in an unlikely case of failure of one of the master node, the other one will automatically and seamlessly take over without making the user aware of such failure.
Numbers
2
Model
IBM nx360 M5
Processor
2 x Intel Xeon E5-2670 v3
RAM
256 GB
Hard Disk
2 x 900GB SAS at 10000 RPM


CPU Compute Node

Numbers
77
Model
IBM nx360 M5
Processor
2 x Intel Xeon E5-2670 v3
RAM
256 GB
Hard Disk
2 x 900GB SAS at 10000 RPM

GPU Compute Node

Numbers
20
Model
IBM nx360 M5
Processor
2 x Intel Xeon E5-2670 v3
GPU
2 x NVIDIA Tesla K40m
RAM
256 GB
Hard Disk
2 x 900GB SAS at 10000 RPM

Storage

Storage Server

Numbers
2
Model
IBM nx360 M4
Processor
2 x Intel Xeon E5-2609v2
RAM
64GB ECC DDR3 at 1866MHz
Hard Disk
2 x 600GB SAS at 10000 RPM

Storage System

Model
IBM Storwize V7000 Storage
Cache
64 GB
Hard Disk
24 x 1TB 2.5" NLSAS HDDs and V7000 expansion arrays with 108 x 4TB NLSAS HDDs.
Usable Storage Space
300TB
File System
IBM General Parallel File System

Backup

Backup Server

Model
IBM x3650 M4
Processor
2 x Intel Xeon 2609v2
RAM
32 GB ECC DDR3 at 1866 MHz
Hard Disk
2 x 900GB at 10000 RPM
Software
IBM Tivoli

Tape Library

Model
IBM TS3200
Tape Type
LTO-6
Slots
48 x cartridge slots and 3 x mail slots
Drives
4 x Half-high Ultrium LTO-6 Drives