Check an OpenAI-compatible endpoint
before you trust it

A single-file Python CLI. No dependencies, no SDK, no config. It lists what models an endpoint exposes, runs a chat smoke test, and reports latency and tokens/sec — so you can tell whether that gateway actually works.

Get bench.py See usage

Usage

Point it at any OpenAI-compatible endpoint — a provider, a gateway, or your own proxy:

python bench.py --base-url https://keheai.com/v1 \
  --api-key sk-your-key \
  --model deepseek-chat

Compare two endpoints head-to-head (official provider vs. a gateway, for example):

python bench.py --base-url https://api.openai.com/v1 --api-key sk-openai \
  --model gpt-4o-mini \
  --compare-to https://keheai.com/v1 --compare-key sk-your-key

It prints: how many models the endpoint exposes, whether a chat request returns 200, latency, and output tokens/sec.

Why we wrote it

An OpenAI-compatible endpoint is only useful if it behaves like one. Plenty of gateways advertise compatibility and then fail on model names, streaming, or error codes. Verifying that takes 30 seconds with a script and a lot longer by guessing in your app.

Download

Single file, MIT licensed. Save it and run:

curl -O https://keheai.com/bench.py
python bench.py --base-url https://keheai.com/v1 --api-key sk-your-key --model deepseek-chat

Or open https://keheai.com/bench.py and copy it.

Try it against a live endpoint

keheai.com is an OpenAI-compatible gateway with self-serve signup and a free trial quota (no card). Register, create a key in your dashboard, and point this tool at it:

python bench.py --base-url https://keheai.com/v1 --api-key sk-your-key --model deepseek-chat

The same script works unchanged against OpenAI, DeepSeek, OpenRouter, or your own proxy — that's the point.