This Monday, Cerebras Systems has presented what is already its most ambitious artificial intelligence supercomputer: Andromeda. Composed of 13.5 million cores and capable of processing more than 1 exaflop of AI computational power. Or what is the same, 1 quintillion operations per second, with an average precision of 16 bits.
Andromeda is actually a set of 16 connected Cerebras CS-2 computers. Each of the Cerebras CS-2 computers outperforms all GPUs, beating the record for largest AI model trained on a single device and embedding over 850,000 cores. Additionally, it enables some 20 billion parameters on a single chip, saving time and cost in training thousands of GPUs, hardware, and scaling requirements.
Each CS-2 contains a Wafer Scale Engine chip, also called a WSE-2which is currently the largest silicon chip ever made at approximately 8.5 inches square and packed with 2.6 trillion transistors organized into 850,000 cores.
This milestone, which marks a before and after in the field of supercomputers, has been built in a data center located in the Californian region of Santa Ana and has had a cost of 35 million dollars. Prior to its introduction, Andromeda has been tested in commercial and academic work, “offering near-perfect scaling through simple data parallelism in large GPT-class language models, including GPT-3, GPT-J, and GPT-NeoX.” , according to Cerebras sources.
A milestone in the field of supercomputers
And why does it offer near-perfect scaling? As Cerebras adds more CS-2 computer units to build the Andromeda supercomputer, neural network training time is reduced by “almost perfect proportion,” Cerebras admits. Note that to scale a deep learning model it is important to add more computing power using GPU-based systems.
Andromeda’s power is such that even can perform tasks that GPU-based systems. For example, it achieved near perfect scaling in GPT-J to 2.5 billion and 25 billion parameters with long sequence lengths. Also MSL of 10,240. While users who tried to do the same job on Polaris, a 2,000 Nvidia A100 cluster, the GPUs couldn’t get the job done due to GPU memory and memory bandwidth limitations.
But where does Andromeda stand compared to other supercomputers? Frontier is currently the fastest in the world, is at Oak Ridge National Labs, and can run at 1,103 exaflops with 64-bit double precision.
An important fact: Andromeda is available for use by multiple users remotely. In fact, it is already being used by commercial writing assistant Jasper AI, Argonne National Laboratory, Cambridge University for research purposes.
