hiyouga/LlamaFactory

Maintainer? Show your league in your README

Card: your league's emblem, from Copper to Legendary

[![bestofgithub](https://bestofgithub.dev/badge/hiyouga/LlamaFactory.svg?style=card)](https://bestofgithub.dev/repo/hiyouga/LlamaFactory)

Flat: fits beside your other badges

[![bestofgithub](https://bestofgithub.dev/badge/hiyouga/LlamaFactory.svg)](https://bestofgithub.dev/repo/hiyouga/LlamaFactory)

Loading README…

Support Date	Model Name
Day 0	Qwen3 / Qwen2.5-VL / Gemma 3 / GLM-4.1V / InternLM 3 / MiniCPM-o-2.6
Day 1	Llama 3 / GLM-4 / Mistral Small / PaliGemma2 / Llama 4

Model	Model size	Template
BLOOM/BLOOMZ	560M/1.1B/1.7B/3B/7.1B/176B	-
DeepSeek (LLM/Code/MoE)	7B/16B/67B/236B	deepseek
DeepSeek 3-3.2	236B/671B	deepseek3
DeepSeek R1 (Distill)	1.5B/7B/8B/14B/32B/70B/671B	deepseekr1
ERNIE-4.5	0.3B/21B/300B	ernie_nothink
Falcon/Falcon H1	0.5B/1.5B/3B/7B/11B/34B/40B/180B	falcon/falcon_h1
Gemma/Gemma 2/CodeGemma	2B/7B/9B/27B	gemma/gemma2
Gemma 3/Gemma 3n	270M/1B/4B/6B/8B/12B/27B	gemma3/gemma3n
GLM-4/GLM-4-0414/GLM-Z1	9B/32B	glm4/glmz1
GLM-4.5/GLM-4.5(6)V	9B/106B/355B	glm4_moe/glm4_5v
GPT-2	0.1B/0.4B/0.8B/1.5B	-
GPT-OSS	20B/120B	gpt_oss
Granite 3-4	1B/2B/3B/7B/8B	granite3/granite4
Hunyuan/Hunyuan1.5 (MT)	0.5B/1.8B/4B/7B/13B	hunyuan/hunyuan_small
InternLM 2-3	7B/8B/20B	intern2
InternVL 2.5-3.5	1B/2B/4B/8B/14B/30B/38B/78B/241B	intern_vl
Intern-S1-mini	8B	intern_s1
Kimi-VL	16B	kimi_vl
Ling 2.0 (mini/flash)	16B/100B	bailing_v2
LFM 2.5 (VL)	1.2B/1.6B	lfm2/lfm2_vl
Llama	7B/13B/33B/65B	-
Llama 2	7B/13B/70B	llama2
Llama 3-3.3	1B/3B/8B/70B	llama3
Llama 4	109B/402B	llama4
Llama 3.2 Vision	11B/90B	mllama
LLaVA-1.5	7B/13B	llava
LLaVA-NeXT	7B/8B/13B/34B/72B/110B	llava_next
LLaVA-NeXT-Video	7B/34B	llava_next_video
MiMo	7B/309B	mimo/mimo_v2
MiniCPM 4/5	0.5B/1B/8B	cpm4/minicpm5
MiniCPM-o/MiniCPM-V 4.5	8B/9B	minicpm_o/minicpm_v
MiniCPM-V 4.6	3B/8B	minicpm_v_4_6
MiniMax-M1/MiniMax-M2	229B/456B	minimax1/minimax2
Ministral 3	3B/8B/14B	ministral3
Mistral/Mixtral	7B/8x7B/8x22B	mistral
PaliGemma/PaliGemma2	3B/10B/28B	paligemma
Phi-3/Phi-3.5	4B/14B	phi
Phi-3-small	7B	phi_small
Phi-4-mini/Phi-4	3.8B/14B	phi4_mini/phi4
Pixtral	12B	pixtral
Qwen2 (Code/Math/MoE/QwQ)	0.5B/1.5B/3B/7B/14B/32B/72B/110B	qwen
Qwen3 (MoE/Instruct/Thinking/Next)	0.6B/1.7B/4B/8B/14B/32B/80B/235B	qwen3/qwen3_nothink
Qwen3.5	0.8B/2B/4B/9B/27B/35B/122B/397B	qwen3_5/qwen3_5_nothink
Qwen3.6	27B/35B	qwen3_6
Qwen2-Audio	7B	qwen2_audio
Qwen2.5-Omni	3B/7B	qwen2_omni
Qwen3-Omni	30B	qwen3_omni
Qwen2-VL/Qwen2.5-VL/QVQ	2B/3B/7B/32B/72B	qwen2_vl
Qwen3-VL	2B/4B/8B/30B/32B/235B	qwen3_vl
Seed (OSS/Coder)	8B/36B	seed_oss/seed_coder
StarCoder 2	3B/7B/15B	-
TeleChat 2-2.5	3B/7B/35B/115B	telechat2
Xing4.0-29B-A4B	29B-A4B	xing4_0
Yuan 2	2B/51B/102B	yuan

Approach	Full-tuning	Freeze-tuning	LoRA	QLoRA	OFT	QOFT
Pre-Training	✅	✅	✅	✅	✅	✅
Supervised Fine-Tuning	✅	✅	✅	✅	✅	✅
Reward Modeling	✅	✅	✅	✅	✅	✅
PPO Training	✅	✅	✅	✅	✅	✅
DPO Training	✅	✅	✅	✅	✅	✅
KTO Training	✅	✅	✅	✅	✅	✅
ORPO Training	✅	✅	✅	✅	✅	✅
SimPO Training	✅	✅	✅	✅	✅	✅

Mandatory	Minimum	Recommend
python	3.11	>=3.11
torch	2.0.0	2.6.0
torchvision	0.15.0	0.21.0
transformers	4.49.0	4.50.0
datasets	2.16.0	3.2.0
accelerate	0.34.0	1.2.1
peft	0.14.0	0.15.1
trl	0.8.6	0.9.6

Optional	Minimum	Recommend
CUDA	11.6	12.2
deepspeed	0.10.0	0.16.4
bitsandbytes	0.39.0	0.43.1
vllm	0.4.3	0.8.2
flash-attn	2.5.6	2.7.2

hiyouga/LlamaFactory

Used by Amazon, NVIDIA, Aliyun, etc.

Check our new open-source project —
🐧 PenguinHarness: Local-first multi-agent app development platform - create, fine-tune, deploy AI apps automatically!

Supporters ❤️

Easily fine-tune 100+ large language models with zero-code CLI and Web UI

Table of Contents

Features

Day-N Support for Fine-Tuning Cutting-Edge Models

Blogs

Changelog

Supported Models

Supported Training Approaches

Provided Datasets

Requirement

Hardware Requirement

Getting Started

Installation

Install from Source

Install from Docker Image

Install PyTorch

Install BitsAndBytes

Install Flash Attention-2

Install BitsAndBytes

Data Preparation

Quickstart

Fine-Tuning with LLaMA Board GUI (powered by Gradio)

Build Docker

Deploy with OpenAI-style API and vLLM

Download from ModelScope Hub

Download from Modelers Hub

Use W&B Logger

Use SwanLab Logger

Projects using LlamaFactory

License

Citation

Acknowledgement

Method	Bits	7B	14B	30B	70B	`x`B
Full (`bf16` or `fp16`)	32	120GB	240GB	600GB	1200GB	`18x`GB
Full (`pure_bf16`)	16	60GB	120GB	300GB	600GB	`8x`GB
Freeze/LoRA/GaLore/APOLLO/BAdam/OFT	16	16GB	32GB	64GB	160GB	`2x`GB
QLoRA / QOFT	8	10GB	20GB	40GB	80GB	`x`GB
QLoRA / QOFT	4	6GB	12GB	24GB	48GB	`x/2`GB
QLoRA / QOFT	2	4GB	8GB	16GB	24GB	`x/4`GB

Used by Amazon, NVIDIA, Aliyun, etc.

Check our new open-source project —🐧 PenguinHarness: Local-first multi-agent app development platform - create, fine-tune, deploy AI apps automatically!

Supporters ❤️

Easily fine-tune 100+ large language models with zero-code CLI and Web UI

Table of Contents

Features

Day-N Support for Fine-Tuning Cutting-Edge Models

Blogs

Changelog

Supported Models

Supported Training Approaches

Provided Datasets

Requirement

Hardware Requirement

Getting Started

Installation

Install from Source

Install from Docker Image

Install PyTorch

Install BitsAndBytes

Install Flash Attention-2

Install BitsAndBytes

Data Preparation

Quickstart

Fine-Tuning with LLaMA Board GUI (powered by Gradio)

Build Docker

Deploy with OpenAI-style API and vLLM

Download from ModelScope Hub

Download from Modelers Hub

Use W&B Logger

Use SwanLab Logger

Projects using LlamaFactory

License

Citation

Acknowledgement

Check our new open-source project —
🐧 PenguinHarness: Local-first multi-agent app development platform - create, fine-tune, deploy AI apps automatically!