RUN LOCALLY / 本地运行

From checkout
to local inference.

Start with the locked primary 9B text model, then connect through your preferred interface.

GET STARTED

Your next inference runs here.

Apple Silicon · Python 3.12 / 3.13 · uv. Start the bundled local UI and API with the locked 9B model.

TERMINAL / MAIN BRANCH

git clone --branch main --single-branch https://github.com/ZEVIN17/deepinfer-mac.git
cd deepinfer-mac
uv sync --frozen --extra mlx
uv run deepinfer-mac doctor
uv run deepinfer-mac models --download qwen3.5-9b-8bit --target-only
uv run deepinfer-mac serve

Check your Mac

Doctor reports Metal, Python, available memory and disk.

Open the local workspace

Visit http://127.0.0.1:8765 and select qwen3.5-9b-8bit. The default path is baseline generation.

Build on the same engine

Use CLI, streamed API or the bundled browser UI. Explore the explicit adaptive entry in the architecture guide.

Run guide & API example ↗

MODEL CAPABILITIES / 模型能力

One framework.
Explicit capabilities.

The 9B model is our primary demonstration. Baseline compatibility and optimized execution are validated separately, across dense and hybrid state structures.

Model / path	Status	Scope
Qwen3.5-9B-8bit	Verified CLI / API / UI	Hybrid · explicit adaptive
Qwen3.5-9B-4bit	Verified baseline / native controls	Hybrid · quality / memory control
Qwen3-14B-4bit	Verified baseline / native controls	Dense · engineering control
Qwen3-4B	Verified regression path	Dense · fixed / adaptive
SmolLM2-135M-Instruct-8bit	Verified baseline / cache / CLI / API / SSE	Dense · configurable model adapter
Other custom models	Per-model acceptance required	Capability-gated model path / repo integration
Joint verification service	35/35 real-model service checks	360M target + 135M draft · dense research pair

Verified describes the listed path. Public 9B UI/API use baseline; hybrid adaptive decoding is explicit opt-in. Bounded multi-request execution and joint verification have separate explicit opt-in entries. The small-model joint path validates service behavior.

Compatibility and execution modes ↗

THE SAME MODEL, THREE INTERFACES

Start with one request.

Terminal

uv run deepinfer-mac run \
  --profile qwen3.5-9b-8bit \
  --mode baseline 'Explain KV caching in two sentences.'

Chat API

curl http://127.0.0.1:8765/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"qwen3.5-9b-8bit","messages":[{"role":"user","content":"Hello"}],"max_tokens":128,"temperature":0,"stream":true}'

Local browser workspace

Open http://127.0.0.1:8765 and choose the 9B8bit profile. The wheel includes the UI assets; the local application runs with Python.

Complete interface and model reference ↗

Explore the adaptive path.

Cost-aware hybrid decoding uses an explicit engine configuration. The architecture guide links its model identity, state handling and controls.

Architecture →

From checkoutto local inference.