GPT-Burn 🔥: This repository showcases an innovative implementation of the GPT architecture using Rust 🦀, complemented by Burn 🔥 for machine learning applications.

This project re-imagines the GPT-2 model with an emphasis on simplicity and clarity, encapsulated in under 300 lines of code found in src/model.rs. Ideal for those keen on learning, this implementation serves as a practical demonstration of the powerful capabilities of Rust and Burn, particularly in the realm of machine learning where deployment becomes remarkably straightforward as it compiles to a single binary.

Key Features:

• Character-Level Tokenization: Currently, only a character-level tokenizer is supported, preventing the use of official weights requiring BPE tokenizer, yet enabling experimentation with the model.

• User-Friendly CLI: Includes an intuitive command-line interface for both training and inference:

  Usage: gpt-burn <COMMAND>
  
  Commands:
    run    Generate text using a pre-trained model
    train  Train a new model
  

• Training and Inference: You can easily train a new model or generate text based on a pre-trained one using simple CLI commands. For example, to run the generated model:

  gpt-burn run ./model_83M
  

  This outputs generated text, showcasing the model's capabilities:

  So wurden bis 1977 679 nachhaltige Wörgler Torbauten vorgeworfen, die Einwohnerzahl Sirkes bestand 2015 bis 1998.
  

• Tokenizer Compatibility: The model supports different tokenizers through the Tokenizer trait, including both character-level and simple-vowel tokenizers:

  Character-level example:

  Tokens: ["A", "l", "b", "e", ...]
  

References for deeper exploration include the original GPT-2 paper and implementations by OpenAI and Huggingface.

Whether you’re searching for a project to deepen your understanding of Rust and machine learning or looking to experiment with GPT architecture, gpt-burn is your go-to resource!