Cerebras-GPT

Cerebras-GPT is a family of open, compute-efficient, large language models designed for researchers and developers. It consists of seven models ranging from 111 million to 13 billion parameters, all trained using the Chinchilla formula to achieve state-of-the-art training efficiency. The models are designed to be complimentary to Pythia and cover a wide range of model sizes using the same public Pile dataset.
Cerebras-GPT works by utilizing the Cerebras Wafer-Scale Cluster, which enables easy scale-out and push-button scaling. The models were trained using standard data parallelism on 16 CS-2 systems, allowing for faster training times and lower training costs. The Cerebras-GPT family of models achieves the lowest loss per unit of compute across all model sizes, making it an efficient solution for large language model development.
Researchers and developers working on natural language processing tasks get the most value from Cerebras-GPT. The models are particularly useful for tasks such as sentence completion and question-and-answer, and they preserve state-of-the-art training efficiency for most common downstream tasks. With Cerebras-GPT, researchers can focus on the design of the ML model instead of the distributed system, enabling them to advance the large generative AI industry more efficiently.