As always, we couldn't have gotten here without your help. Special thanks to our team, our community, and the open source movement.

Psyche’s initial training infrastructure is just the beginning of our journey. We plan to integrate full post training stages - supervised finetuning and reinforcement learning workloads, inference, and other parallelizable workloads in the creation and serving of AI going forward.

Looking ahead, we will draw model ideas from the community via our forum and Discord. By enabling highly parallel and scalable experimentation, we’re betting that the next innovation in model creation and design will come from the open source community

The resulting model will be small enough to train on a single H/DGX and run on a 3090, but will be powerful enough to serve as the basis for strong reasoning models and creative pursuits. The model will be trained continually without a final annealing step, resulting in a true unaltered base model.

We are launching testnet with the pre-training of a 40B parameter LLM:

- MLA Architecture
- Dataset consisting of FineWeb (14T) + FineWeb-2 minus some less common languages (4T), and The Stack v2 (1T)

If you have 64+ H100 GPUs, Contact [email protected] to apply to provide hardware to the network’s training pool.

While compute on the network needs to be trusted and approved at this time, we plan to support trustless, community-owned compute resources. For now, open source enthusiasts can contribute via our mining pool, and we will be onboarding more nodes over the next weeks.

Psyche uses the Solana blockchain to decentralize parts of the core infrastructure for coordination and stores attestations for the nodes operating within the network. This design takes meaningful steps towards decentralization while ensuring training does not become too costly or redundant.

Training used to have a bandwidth constraint that kept the process centralized. In 2024, Nous's DisTrO optimizers broke through that constraint. With Psyche, we have created a custom peer-to-peer networking stack to coordinate globally distributed GPUs running DisTrO.

All the links!

Blog: nousresearch.com/nous-psyche/
Training dashboard: psyche.network
Code: github.com/PsycheFounda...
Docs: docs.psyche.network
Forum: forum.psyche.network
HuggingFace: huggingface.co/PsycheFounda...
Discord: discord.com/invite/jqVph...

This run represents the largest pre-training run conducted over the internet to date, surpassing previous iterations that trained smaller models on much fewer data tokens.

We are launching our testnet today with the pre-training of a 40B parameter LLM, a model powerful enough to serve as a foundation for future pursuits in open science.

Psyche is a decentralized training network that makes it possible to bring the world’s compute together to train powerful AI, giving individuals and small communities access to the resources required to create new, interesting, and unique large scale models.

To ensure a smooth rollout, we made a waitlist: portal.nousresearch.com
- Access will be granted on a first-come, first-served basis
- Once granted access, you can create API keys and purchase credits
- OpenAI-compatible API
- Right now all accounts start off with $5.00 in free credits.

Apply to attend Nous Research x Solana Foundation this Thursday in NYC: lu.ma/nousxsolana

You can now experiment with Psyche’s DisTrO-enabled training code on our GitHub, and the larger open-sourced distributed training stack will be released alongside testnet. github.com/PsycheFounda...

Read more in our blog post: nousresearch.com/nous-psyche/

Run Hermes on phones, laptops, and CPUs without sacrificing speed, and may also be a great pairing with 70B for speculative decoding!

Learn more about Hermes, see our technical report, and chat with it now: nousresearch.com/hermes