github.com · Training & Fine tuning
NVIDIA's open source codebase for training transformers at scale: training scripts plus the Megatron Core library with tensor, pipeline, data, expert and context parallelism.
Megatron-LM is one of 9 training and fine tuning tools on Axiomi. See all training and fine tuning tools or how we check facts.
// guides
The best LLM pretraining dataset depends on your goal. FineWeb-Edu for small models, DCLM-Baseline and Nemotron-CC for scale, Common Pile for licensed text.
LLM distillation vs quantization: distillation trains a smaller student model, quantization stores the same weights in fewer bits. When to use each, or both.
Train a small LLM from scratch with nanochat, and learn when NVIDIA Megatron-LM fits: model sizes, budgets, datasets, compute math and multi GPU parallelism.
// related
DeepSpeeddeepspeed.aiLibrary for large scale distributed training and inference, known for the ZeRO memory optimisation family.Training & Fine tuningOpen source★ 43.3knanoGPTgithub.comAndrej Karpathy's minimal repository for training and fine tuning medium sized GPTs, a classic learning resource.Training & Fine tuningOpen source★ 64.2k
Unslothunsloth.aiFine tuning toolkit that speeds up LoRA and full training while cutting GPU memory use, with free notebooks.Training & Fine tuningOpen source★ 78.7k
PEFThuggingface.coParameter efficient fine tuning methods such as LoRA, letting large models adapt on modest hardware.Training & Fine tuningOpen source★ 21.9kTRLhuggingface.coHugging Face library for post training models with supervised fine tuning, DPO, GRPO and other preference optimisation methods.Training & Fine tuningOpen source★ 19.6k
Axolotlaxolotl.aiConfig driven tool for fine tuning open models across many architectures and training methods.Training & Fine tuningOpen source★ 12.6k