bestofgithub.devRepositories with the GitHub topic “inference”: 37 repositories with 998k stars between them, mostly written in Python, C++ and Go. The highest-rated is ggml-org/llama.cpp, #54 on the bestofgithub Elo ladder.
The open LLM inference engine for AWS Trainium: NKI megakernels, 1M-token context, OpenAI-compatible API. By RunAnywhere.
git clone https://github.com/RunanywhereAI/trnrt.gitA framework for efficient model inference with omni-modality models
pipx install vllm-omniAn open, programmable decision layer for models and compute.
git clone https://github.com/vllm-project/semantic-router.gitInfrastructure for continually self‑improving agents
git clone https://github.com/Human-Agent-Society/reef.gitRun models larger than your RAM or VRAM by streaming MoE experts from SSD. Built on llama.cpp.
git clone https://github.com/wici-ai/inferssd.gitLLM inference in plain C/C++: quantized models on CPUs, Apple Silicon and GPUs with almost no setup.
brew install llama.cppLLM inference in plain C/C++: quantized models on CPUs, Apple Silicon and GPUs with almost no setup.
Run Llama, Gemma, Qwen and other open models locally behind a simple CLI and REST API.
High-throughput LLM serving engine with PagedAttention and an OpenAI-compatible server.
SGLang is a high-performance serving framework for large language models and multimodal models.
Port of OpenAI's Whisper model in C/C++
FreeToken brings datacenter-scale model serving to your desktop. Run massive models locally, fast and efficiently.
Faster Whisper transcription with CTranslate2
Cross-platform, customizable ML solutions for live and streaming media.
Nano vLLM
A framework for efficient model inference with omni-modality models
Ultralytics YOLOv5 in PyTorch for object detection, instance segmentation, classification, training, and export.
DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and…
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
ncnn is a high-performance neural network inference framework optimized for the mobile platform
OpenVINO™ is an open source toolkit for optimizing and deploying AI inference
Machine Learning Engineering Open Book
Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI.
Making large AI models cheaper, faster and more accessible
NVIDIA® TensorRT™ is an SDK for high-performance deep learning inference on NVIDIA GPUs. This repository contains the…
An open, programmable decision layer for models and compute.
Example 📓 Jupyter notebooks that demonstrate how to build, train, and deploy machine learning models using 🧠 Amazon…
Infrastructure for continually self‑improving agents
The Triton Inference Server provides an optimized cloud and edge inferencing solution.
PyTorch implementation of YOLOv3, YOLOv3-SPP, and YOLOv3-tiny for real-time object detection with training, validation…
Open-source inference platform for language, vision, embedding, reranking, image, audio, and video models. Deploy with…
Hello AI World guide to deploying deep-learning inference networks and deep vision primitives with TensorRT and NVIDIA…
Native MLX runtime for Laya typed decision models — 7–14 ms short decisions on M3 Max. No text generation, PyTorch, or…
Production ready toolkit to run AI locally
Large Language Model Text Generation Inference
Easily fine-tune (SFT/RL), evaluate, and deploy Qwen, Gemma, or any open agentic LLM/VLM!
🎨 The exhaustive Pattern Matching library for TypeScript, with smart type inference.
Adversarial Robustness Toolbox (ART) - Python Library for Machine Learning Security - Evasion, Poisoning, Extraction…
💎1MB lightweight face detection model (1MB轻量级人脸检测模型)
On-device Speech AI for Apple Silicon
Runtime type system for IO decoding/encoding
Run models larger than your RAM or VRAM by streaming MoE experts from SSD. Built on llama.cpp.
The open LLM inference engine for AWS Trainium: NKI megakernels, 1M-token context, OpenAI-compatible API. By…