Third Technology 5nm Wafer Scale Engine (WSE-3) Powers Trade’s Most Scalable AI Supercomputers, Up To 256 exaFLOPs by way of 2048 Nodes

Cerebras Techniques, the pioneer in accelerating generative AI, has doubled down on its current world document of quickest AI chip with the introduction of the Wafer Scale Engine 3. The WSE-3 delivers twice the efficiency of the earlier record-holder, the Cerebras WSE-2, on the similar energy draw and for a similar value. Function constructed for coaching the trade’s largest AI fashions, the 5nm-based, 4 trillion transistor WSE-3 powers the Cerebras CS-3 AI supercomputer, delivering 125 petaflops of peak AI efficiency by 900,000 AI optimized compute cores.

Key Specs:

• 4 trillion transistors
• 900,000 AI cores
• 125 petaflops of peak AI efficiency
• 44GB on-chip SRAM
• 5nm TSMC course of
• Exterior reminiscence: 1.5TB, 12TB, or 1.2PB
• Trains AI fashions as much as 24 trillion parameters
• Cluster dimension of as much as 2048 CS-3 techniques

With an enormous reminiscence system of as much as 1.2 petabytes, the CS-3 is designed to coach subsequent technology frontier fashions 10x bigger than GPT-4 and Gemini. 24 trillion parameter fashions could be saved in a single logical reminiscence house with out partitioning or refactoring, dramatically simplifying coaching workflow and accelerating developer productiveness. Coaching a one-trillion parameter mannequin on the CS-3 is as easy as coaching a one billion parameter mannequin on GPUs.

The CS-3 is constructed for each enterprise and hyperscale wants. Compact 4 system configurations can fantastic tune 70B fashions in a day whereas at full scale utilizing 2048 techniques, Llama 70B could be skilled from scratch in a single day – an unprecedented feat for generative AI.

The newest Cerebras Software program Framework gives native help for PyTorch 2.0 and the newest AI fashions and strategies similar to multi-modal fashions, imaginative and prescient transformers, combination of specialists, and diffusion. Cerebras stays the one platform that gives native {hardware} acceleration for dynamic and unstructured sparsity, rushing up coaching by as much as 8x.

“After we began on this journey eight years in the past, everybody stated wafer-scale processors had been a pipe dream. We couldn’t be extra proud to be introducing the third-generation of our groundbreaking water scale AI chip,” stated Andrew Feldman, CEO and co-founder of Cerebras.

“WSE-3 is the quickest AI chip on the planet, purpose-built for the newest cutting-edge AI work, from combination of specialists to 24 trillion parameter fashions. We’re thrilled for carry WSE-3 and CS-3 to market to assist remedy right this moment’s largest AI challenges.”

Superior Energy Effectivity and Software program Simplicity

With each part optimized for AI work, CS-3 delivers extra compute efficiency at much less house and fewer energy than every other system. Whereas GPUs energy consumption is doubling technology to technology, the CS-3 doubles efficiency however stays inside the similar energy envelope. the CS-3 affords superior ease of use, requiring 97% much less code than GPUs for LLMs and the flexibility to coach fashions starting from 1B to 24T parameters in purely knowledge parallel mode. A regular implementation of a GPT-3 sized mannequin required simply 565 strains of code on Cerebras – an trade document.

Trade Partnerships and Buyer Momentum

Cerebras already has a sizeable backlog of orders for CS-3 throughout enterprise, authorities and worldwide clouds.

“As a long-time companion of Cerebras, we’re excited to see what’s attainable with the evolution of wafer-scale engineering. CS-3 and the supercomputers primarily based on this structure are powering novel scale techniques that permit us to discover the boundaries of frontier AI and science,” stated Rick Stevens, Argonne Nationwide Laboratory Affiliate Laboratory Director for Computing, Atmosphere and Life Sciences.  “The audacity of what Cerebras is doing matches our ambition, and it matches how we take into consideration the longer term.”

“As a part of our multi-year strategic collaboration with Cerebras to develop AI fashions that enhance affected person outcomes and diagnoses, we’re excited to see developments being made on the know-how capabilities to boost our efforts,” stated Dr. Matthew Callstrom, M.D., Mayo Clinic’s medical director for technique and chair of radiology.

The CS-3 can even play an essential function within the pioneering strategic partnership between Cerebras and G42. The Cerebras and G42 partnership has already delivered 8 exaFLOPs of AI supercomputer efficiency by way of Condor Galaxy 1 (CG-1) and Condor Galaxy 2 (CG-2). Each CG-1 and CG-2, deployed in California, are among the many largest AI supercomputers on the planet.

As we speak, Cerebras and G42 introduced that Condor Galaxy 3 is underneath building. Condor Galaxy 3 shall be constructed with 64 CS-3 techniques, producing 8 exaFLOPs of AI compute, one of many largest AI supercomputers on the planet. Condor Galaxy 3 is the third set up within the Condor Galaxy community. The Cerebras G42 strategic partnership is about to ship tens of exaFLOPs of AI compute. Condor Galaxy has skilled a number of the trade’s main open-source fashions, together with Jais-30B, Med42, Crystal-Coder-7B and BTLM-3B-8K.

“Our strategic partnership with Cerebras has been instrumental in propelling innovation at G42, and can contribute to the acceleration of the AI revolution on a world scale,” stated Kiril Evtimov, Group CTO of G42.  “Condor Galaxy 3, our subsequent AI supercomputer boasting 8 exaFLOPs, is at the moment underneath building and can quickly carry our system’s whole manufacturing of AI compute to 16 exaFLOPs.”