bestofgithub.dev

← Back to the directory

huggingface/tokenizers

View on GitHub

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

huggingface/tokenizers is #3,919 on the bestofgithub Elo ladder, in Gold III, with a rating of 1,578 (+41 this week). It has 11,247 stars, 17 gained in the last week.

Gold III

1,578 Elo

  • #3,919 on the ladder
  • +3 today · +41 this week
  • Peak 1,588
2 Elo to Gold II

Stars

11,247

  • +3 today
  • +17 this week
  • – this month

Repository

Forks
1.2k
Language
Rust
Last push
Category
AI & Agents

Achievements

bestofgithub badgeMaintainer? Show your league in your README

Card: your league's emblem, from Copper to Legendary

[![bestofgithub](https://bestofgithub.dev/badge/huggingface/tokenizers.svg?style=card)](https://bestofgithub.dev/repo/huggingface/tokenizers)

Flat: fits beside your other badges

[![bestofgithub](https://bestofgithub.dev/badge/huggingface/tokenizers.svg)](https://bestofgithub.dev/repo/huggingface/tokenizers)

Install

git
git clone https://github.com/huggingface/tokenizers.git

Similar repositories

google-research/bert

Open on GitHubPython

TensorFlow code and pre-trained models for BERT

  • 1,624−8 Elo, change today
  • 40k stars
  • 9.6k forks
  • archived
git
git clone https://github.com/google-research/bert.git
  • google
  • natural-language-processing
  • natural-language-understanding
  • nlp

codertimo/BERT-pytorch

Open on GitHubPython

Google AI 2018 BERT pytorch implementation

  • 1,337−1 Elo, change today
  • 6.5k stars
  • 1.3k forks
git
git clone https://github.com/codertimo/BERT-pytorch.git
  • bert
  • language-model
  • nlp
  • pytorch

EleutherAI/gpt-neo

Open on GitHubPython

An implementation of model parallel GPT-2 and GPT-3-style models using the mesh-tensorflow library.

  • 1,372+2 Elo, change today
  • 8.2k stars
  • 955 forks
  • archived
git
git clone https://github.com/EleutherAI/gpt-neo.git
  • gpt
  • gpt-2
  • gpt-3
  • language-model

jessevig/bertviz

Open on GitHubPython

BertViz: Visualize Attention in Transformer Models

  • 1,386−4 Elo, change today
  • 8.2k stars
  • 887 forks
git
git clone https://github.com/jessevig/bertviz.git
  • bert
  • gpt2
  • machine-learning
  • natural-language-processing

NVIDIA/FasterTransformer

Open on GitHubC++

Transformer related optimization, including BERT, GPT

  • 1,334−2 Elo, change today
  • 6.4k stars
  • 933 forks
git
git clone https://github.com/NVIDIA/FasterTransformer.git
  • bert
  • gpt
  • pytorch
  • transformer

graykode/nlp-tutorial

Open on GitHubJupyter Notebook

Natural Language Processing Tutorial for Deep Learning Researchers

  • 1,512−6 Elo, change today
  • 14.9k stars
  • 3.9k forks
git
git clone https://github.com/graykode/nlp-tutorial.git
  • attention
  • bert
  • natural-language-processing
  • nlp

README

Loading README…