google-research/bert

Maintainer? Show your league in your README

Card: your league's emblem, from Copper to Legendary

[![bestofgithub](https://bestofgithub.dev/badge/google-research/bert.svg?style=card)](https://bestofgithub.dev/repo/google-research/bert)

Flat: fits beside your other badges

[![bestofgithub](https://bestofgithub.dev/badge/google-research/bert.svg)](https://bestofgithub.dev/repo/google-research/bert)

Loading README…

	H=128	H=256	H=512	H=768
L=2	2/128 (BERT-Tiny)	2/256	2/512	2/768
L=4	4/128	4/256 (BERT-Mini)	4/512 (BERT-Small)	4/768
L=6	6/128	6/256	6/512	6/768
L=8	8/128	8/256	8/512 (BERT-Medium)	8/768
L=10	10/128	10/256	10/512	10/768
L=12	12/128	12/256	12/512	12/768 (BERT-Base)

Model	Score	CoLA	SST-2	MRPC	STS-B	QQP	MNLI-m	MNLI-mm	QNLI(v2)	RTE	WNLI	AX
BERT-Tiny	64.2	0.0	83.2	81.1/71.1	74.3/73.6	62.2/83.4	70.2	70.3	81.5	57.2	62.3	21.0
BERT-Mini	65.8	0.0	85.9	81.1/71.8	75.4/73.3	66.4/86.2	74.8	74.3	84.1	57.9	62.3	26.1
BERT-Small	71.2	27.8	89.7	83.4/76.2	78.8/77.0	68.1/87.0	77.6	77.0	86.4	61.8	62.3	28.6
BERT-Medium	73.5	38.0	89.6	86.6/81.6	80.4/78.4	69.6/87.9	80.0	79.1	87.7	62.2	62.3	30.5

Model	SQUAD 1.1 F1/EM	Multi NLI Accuracy
BERT-Large, Uncased (Original)	91.0/84.3	86.05
BERT-Large, Uncased (Whole Word Masking)	92.8/86.7	87.07
BERT-Large, Cased (Original)	91.5/84.8	86.09
BERT-Large, Cased (Whole Word Masking)	92.9/86.7	86.46

SQuAD v1.1 Leaderboard (Oct 8th 2018)	Test EM	Test F1
1st Place Ensemble - BERT	87.4	93.2
2nd Place Ensemble - nlnet	86.0	91.7
1st Place Single Model - BERT	85.1	91.8
2nd Place Single Model - nlnet	83.5	90.1

System	MultiNLI	Question NLI	SWAG
BERT	86.7	91.1	86.3
OpenAI GPT (Prev. SOTA)	82.2	88.1	75.0

google-research/bert

BERT

Introduction

What is BERT?

What has been released in this repository?

Pre-trained models

Fine-tuning with BERT

Fine-tuning with Cloud TPUs

Sentence (and sentence-pair) classification tasks

Prediction from classifier

SQuAD 1.1

SQuAD 2.0

Out-of-memory issues

Using BERT to extract fixed feature vectors (like ELMo)

Tokenization

Pre-training with BERT

Pre-training tips and caveats

Pre-training data

Learning a new WordPiece vocabulary

Using BERT in Colab

FAQ

Is this code compatible with Cloud TPUs? What about GPUs?

I am getting out-of-memory errors, what is wrong?

Is there a PyTorch version available?

Is there a Chainer version available?

Will models in other languages be released?

Will models larger than `BERT-Large` be released?

What license is this library released under?

How do I cite BERT?

Disclaimer

Contact information

System	Seq Length	Max Batch Size
`BERT-Base`	64	64
...	128	32
...	256	16
...	320	14
...	384	12
...	512	6
`BERT-Large`	64	12
...	128	6
...	256	2
...	320	1
...	384	0
...	512	0

BERT

Introduction

What is BERT?

What has been released in this repository?

Pre-trained models

Fine-tuning with BERT

Fine-tuning with Cloud TPUs

Sentence (and sentence-pair) classification tasks

Prediction from classifier

SQuAD 1.1

SQuAD 2.0

Out-of-memory issues

Using BERT to extract fixed feature vectors (like ELMo)

Tokenization

Pre-training with BERT

Pre-training tips and caveats

Pre-training data

Learning a new WordPiece vocabulary

Using BERT in Colab

FAQ

Is this code compatible with Cloud TPUs? What about GPUs?

I am getting out-of-memory errors, what is wrong?

Is there a PyTorch version available?

Is there a Chainer version available?

Will models in other languages be released?

Will models larger than BERT-Large be released?

What license is this library released under?

How do I cite BERT?

Disclaimer

Contact information

Will models larger than `BERT-Large` be released?