jwuthri-cloud/Tracely-ai

Maintainer? Show your league in your README

Card: your league's emblem, from Copper to Legendary

[![bestofgithub](https://bestofgithub.dev/badge/jwuthri-cloud/Tracely-ai.svg?style=card)](https://bestofgithub.dev/repo/jwuthri-cloud/Tracely-ai)

Flat: fits beside your other badges

[![bestofgithub](https://bestofgithub.dev/badge/jwuthri-cloud/Tracely-ai.svg)](https://bestofgithub.dev/repo/jwuthri-cloud/Tracely-ai)

Loading README…

	Dataset-first tools	Tracely
Where tests come from	You write them	Promoted from real failing traces
Fidelity to production	A guess	The exact failing run, byte for byte
Cost to replay in CI	Live model calls	$0 — recorded tool/LLM fixtures
What happens on regression	A dashboard number moves	The PR is blocked
How you find out	You go and look	It comes to you — Slack, email, your own webhook

	What it does	Where
Ingest	OTLP/HTTP from any language; blob-first durability (S3 before the queue); agent semantics as indexed columns; three message conventions normalized	`backend/`
Evaluators	Columns on the trace table. Structural checks (no model) + LLM judges at conversation / run / span level, multi-output (score, boolean, number, text, JSON schema), basic or `@VARIABLE` advanced prompts with live preview, batch or sequential, per-agent/env targeting, deterministic sampling, advisory verdicts	Docs
Failure clusters	Structural signature + semantic embedding clustering, taxonomy, suggested evaluators, promote-to-case	Docs
Regression cases	Hermetic fixture bundles, fail-to-pass contracts, assertions, reference trajectory, replay history	Docs
Scenarios	Multi-turn conversations Tracely drives at your agent's own endpoint, or an adversarial goal a red-team model pursues (goal achieved = FAIL)	Docs
CI gates	`tracely simulate` / `replay` / `gate` + a GitHub Action: commit status, PR comment, non-zero exit	Docs
Alerts	Trigger + visual flow: conditions, Slack, email, webhooks with headers, an LLM step, Python expressions. Event triggers fire inline; thresholds poll a window. Test runs show what each step actually sent	Docs
Trends & analysis	Daily traces/failures/gate pass-rate, latency + cost, per-agent cross-metric meta-analysis	Docs
Judge calibration	Label judge verdicts against human review; per-evaluator agreement, missed failures vs over-flagging	Docs
Conversation intelligence	Rolling per-turn summary (backs the judge's `@HISTORY`), declared agent catalog, state deltas, Replay + Fleet views, shareable read-only links	Docs
In-app assistant	⌘J — an agent over your workspace: reads traces, creates evaluators, scenarios, cases and alerts, as you. ⌘K jumps anywhere	Docs
MCP server	The API doubles as an MCP endpoint (`/mcp`) so a coding agent drives the workspace with an ordinary ingest key	Docs
Auth & teams	`dev` (open) · `local` (email/password + invites) · `clerk`. Organizations, workspaces, seats, API keys, per-workspace OpenRouter key	Docs
Ops	Beat self-check (queue depth, worker liveness, ingest freshness) + `/health/queue`, optional Sentry, optional trace quota + Stripe billing	`guides/DEPLOY.md`

Trigger	Fires when
`CI gate failed`	a gate run finished `FAIL` — or `NO_COVERAGE`, the suite that could not run, which is the quietly-green failure nobody notices
`Conversation failed`	a live turn failed a non-advisory evaluator. Filter by evaluator, or by the text of the judge's own reason
`New failure mode`	a failure signature no trace has produced before — the alert you cannot get from logs
`Evaluator FAIL rate` / `Average score` / `Overall failure rate`	a rate or average crosses a line over a sliding window

Layer	Tech	Where
Backend (API + domain)	FastAPI + Pydantic v2	`backend/`
Workers	Celery + Redis	`workers/`
Traces + scores (OLAP)	ClickHouse (`ReplacingMergeTree`)	`backend/tracely/infrastructure/clickhouse/ddl`
Registry (OLTP)	Postgres + pgvector + SQLAlchemy 2.0 + Alembic	`backend/migrations`
Queue / Blobs	Redis / MinIO·S3 (blob-first, source of truth)	—
Frontend	Next.js 15 (App Router) + Tailwind + React Flow	`frontend/`
SDK + CI gate CLI	`tracelyai-sdk` (OTel wrapper + `tracely` CLI)	`sdk/`
Tooling	uv workspace (Python) · pnpm (web)	—

Folder	What's inside
`backend/`	The `tracely` package: FastAPI API + shared domain (OTLP mapping, ClickHouse/Postgres/S3, registry, evaluators, failure intelligence, regression, gate, alert flows, auth, Celery tasks).
`workers/`	The deployable Celery worker runtime.
`frontend/`	The Next.js web app — trace explorer, clusters, cases, gates, trends, alerts builder, settings, auth.
`sdk/`	The Python SDK (instrument agents over OTLP, hermetic record-replay) + the `tracely` CI gate CLI.
`docs/`	The published docs site (Nextra) — SDK reference and the per-screen product guide. `make docs` → :3002.
`skills/`	The Tracely agent skill — `npx skills add https://github.com/jwuthri-cloud/Tracely-ai --skill tracely`.
`scripts/`	Dev/demo helpers (raw-OTLP sender, one-command `seed_demo.py`, gate shim).
`design/`	The full design dossier — reverse-engineered Langfuse + every Tracely design decision.

jwuthri-cloud/Tracely-ai

Tracely

Production failures become regression tests.

Why another agent-observability tool?

The loop

1 · Observe — every run, hierarchically

2 · Detect & triage — failures group into issues

3 · Test — freeze the failure

4 · Ship — block the PR

5 · Tell me — alerts as a flow

Plus — trends, cross-metric analysis, judge calibration

Everything in the box

Quickstart

Everything in Docker — the whole product, populated

Local dev (hot reload)

Deploy your own

Send your first trace

Gate your PRs

Get told, don't go looking

Drive it from your editor

Teach your coding agent Tracely

Architecture

Repo map — each folder has its own detailed README

Key environment variables

Contributing

License

Variable	Default	Purpose
`AUTH_MODE`	`dev`	`dev` (open, no login) · `local` (email/password, self-host) · `clerk` (hosted SaaS).
`SESSION_SECRET`	—	Required when `AUTH_MODE=local`: HS256 signing key for JWTs (≥32 chars).
`SECRETS_ENCRYPTION_KEY`	—	Encrypts a workspace's own OpenRouter key at rest (≥32 chars).
`CLERK_ISSUER`	—	Required when `AUTH_MODE=clerk`.
`OPENROUTER_API_KEY`	—	Server-wide LLM access for judges, failure intelligence, meta-analysis. Workspaces can bring their own key instead.
`RESEND_API_KEY`	—	Transactional email: invites, password resets, and the alert flow's email step.
`ALLOW_PRIVATE_URLS`	unset	Whether outbound calls (agent endpoints, alert webhooks) may target private addresses. Unset = allowed everywhere except prod.
`TRACELY_BACKEND_PORT`	`8000`	Backend host port (Docker compose override).
`TRACELY_WEB_PORT`	`3001`	Frontend host port (Docker compose override).