Dolma
github.com · Datasets
AI2's open pretraining corpus and toolkit used to train OLMo, with tools for curating large text datasets.
Dolma is one of 5 dataset tools on Axiomi. See all dataset tools or how we check facts.
Open source
// guides
Guides that cover Dolma
Best LLM Pretraining Dataset in 2026: Top Picks
The best LLM pretraining dataset depends on your goal. FineWeb-Edu for small models, DCLM-Baseline and Nemotron-CC for scale, Common Pile for licensed text.
Open Weight vs Open Source AI: The Difference
Open weight vs open source AI explained: what the OSI definition requires, which popular models are which, and a licence table checked against model cards.
// related
Dolma alternatives and related tools
Common Crawlcommoncrawl.orgNonprofit that maintains a free, open archive of web crawl data used in most large scale pretraining corpora.DatasetsFreeHN 1
FineWebhuggingface.coLarge, carefully filtered and deduplicated English web dataset for LLM pretraining, documented with ablations.DatasetsOpen sourceHugging Face Datasetshuggingface.coHub of hundreds of thousands of datasets plus a library for loading and streaming them efficiently.DatasetsFree★ 22.1k
LAIONlaion.aiNonprofit producing open multimodal datasets, models and tools for democratising machine learning research.DatasetsFreeHN 2Megatron-LMgithub.comNVIDIA's open source codebase for training transformers at scale: training scripts plus the Megatron Core library with tensor, pipeline, data, expert and context parallelism.Training & Fine tuningOpen source★ 18.3knanoGPTgithub.comAndrej Karpathy's minimal repository for training and fine tuning medium sized GPTs, a classic learning resource.Training & Fine tuningOpen source★ 64.2k