The most popular and comprehensive Open Source ECM platform
Let us know more
AI Chips: AI-Targeted Massive Wafers Speed the Creation of Large Language Models
‘Wafer-scale engine‘ (WSE) – it is what Cerebras, a semiconductor manufacturer, calls the architecture of their massive GPU engine on a single ‘chip’. It is 56 times the size of the largest ‘standard’ GPU chip and packs 3000 times more chip memory and more than 10000 times the memory bandwidth — 2.6 trillion transistors and over 850,000 cores. The Cerebras WSE is a massive silicon chip, the largest ever built, that is about the size of an Apple iPad, and it was purpose-built to target AI.
The Cerebras website says that “A single CS-2 typically delivers the wall-clock compute performance of many tens to hundreds of graphics processing units (GPU), or more. In one system less than one rack in size, the CS-2 delivers answers in minutes or hours that would take days, weeks, or longer on large multi-rack clusters of legacy, general purpose processors.”
The target of the Cerebras WSE technology is the creation of large natural language learning models, like GPT-3.
Andrew Feldman, Cerebras CEO, said that “what’s hard about training LLMs isn’t the machine learning. It is the distributed compute necessitated by the GPU, the fact that it’s a small compute engine. You have to break up this big problem and spread it out over lots of little engines. That work, distributed parallel computation, is obscure and it’s rare and only a few organizations in the world are good at it.”
Rick Stevens, associate director of the Argonne National Laboratory, said that “Cerebras’ inventions, which will provide a 100x increase in parameter capacity, may have the potential to transform the industry. For the first time, we will be able to explore brain-sized models, opening up vast new avenues of research and insight.”
