minGPT - A PyTorch Re-implementation of GPT

minGPT is a PyTorch re-implementation of GPT, designed to be small, clean, interpretable, and educational. It covers both training and inference.

Visit Website
minGPT - A PyTorch Re-implementation of GPT

Introduction

What is minGPT?

minGPT is a PyTorch re-implementation of GPT, designed to be small, clean, interpretable, and educational. It focuses on being minimal and educational, making GPT more accessible and easier to understand.

Key Features

  • Minimal Implementation: Around 300 lines of code, focusing on the core aspects of GPT.
  • Educational: Designed to be interpretable and easy to learn from.
  • Training and Inference: Covers both aspects for a comprehensive understanding.
  • Library Installation: Simple installation and integration into projects.
  • Usage Examples: Includes demos and projects to illustrate usage.

Installation

To install minGPT:

git clone https://github.com/karpathy/minGPT.git
cd minGPT
pip install -e .

Usage

Instantiate a GPT-2 (124M param version):

from mingpt.model import GPT
model_config = GPT.get_default_config()
model_config.model_type = 'gpt2'
model_config.vocab_size = 50257
model_config.block_size = 1024
model = GPT(model_config)

Train the model:

train_dataset = YourDataset()

from mingpt.trainer import Trainer
train_config = Trainer.get_default_config()
train_config.learning_rate = 5e-4
train_config.max_iters = 1000
train_config.batch_size = 32
trainer = Trainer(train_config, model, train_dataset)
trainer.run()

Additional Information

  • Unit Tests: Run python -m unittest discover tests for testing.
  • Todo List: Includes planned features and improvements.
  • References: Links to related code repositories and papers.

Contributors

License

MIT License.