SDK quickstart

Wrap the model client you already use. SmallerModels can serve an eligible request with a cheaper model; otherwise the original client executes the original call with your cloud configuration and credits.

API preview The SDK is not publicly distributed yet. This documents the interface being prepared for early access.

JavaScript

Create your provider client normally, then wrap it once during application initialization.

JavaScript

import OpenAI from "openai";
import { smallerModels } from "@smallermodels/sdk";

const openai = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
});

const client = smallerModels.wrap(openai, {
  project: "support-production",
  operation: "support-drafting",
  mode: "shadow",
});

Existing calls stay unchanged.

JavaScript

const response = await client.responses.create({
  model: "gpt-4.1",
  input: messages,
});

The model, timeout, metadata, and payload still describe the fallback call. SmallerModels does not replace that configuration.

Python

The Python SDK follows the same client-wrapping model.

Python

import os

from openai import OpenAI
from smallermodels import smaller_models

openai = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

client = smaller_models.wrap(
    openai,
    project="support-production",
    operation="support-drafting",
    mode="shadow",
)

response = client.responses.create(
    model="gpt-4.1",
    input=messages,
)

No decorator is required. Calls can remain inline, conditional, or nested wherever they already live.

Fallback behavior

Every wrapped call follows the same sequence.

1
Capture the call
The adapter records the supported method and its original arguments.
2
Evaluate the policy
The active policy decides whether a smaller inference path is safe.
3
Execute one path
Accepted requests use the smaller model. All others invoke the untouched provider method.

A runtime failure can fall back only before a response is committed. The SDK never combines output from two providers into one response.

Operations

An operation identifies one stable application behavior, such as support drafting or text-to-JSON extraction. Discovery produces a separate policy for each operation.

Option	Purpose
`project`	Separates environments, traffic, and policy history.
`operation`	Selects the workload-specific routing policy.
`mode`	Chooses observation-only or active routing.
`policy`	Optionally pins an explicit frozen policy version.

Multiple providers

Wrap each client independently. Calls retain the account, endpoint, model, region, and retry behavior of the client that made them.

JavaScript

const support = smallerModels.wrap(openai, {
  operation: "support-drafting",
});

const research = smallerModels.wrap(anthropic, {
  operation: "research-synthesis",
});

const internal = smallerModels.wrap(bedrock, {
  operation: "internal-extraction",
});

Applications can keep choosing providers dynamically. SmallerModels handles only the call made through the selected wrapped client.

Streaming

A provider adapter returns the same stream shape expected by the application. Consumption code does not change.

JavaScript

const stream = await client.responses.create({
  model: "gpt-4.1",
  input: messages,
  stream: true,
});

for await (const event of stream) {
  handleEvent(event);
}

Before exposing the first event, the adapter establishes which execution path owns the response. After streaming begins, it does not switch providers.

Shadow and active mode

mode: "shadow"

The original cloud call always serves the user. SmallerModels observes traffic and evaluates candidate paths without changing production behavior.

mode: "active"

Requests accepted by a frozen policy use the smaller path. All others execute through the original provider client.

JavaScript

const client = smallerModels.wrap(openai, {
  project: "support-production",
  operation: "support-drafting",
  mode: "active",
  policy: "support-drafting-2026-08-07",
});

Security

Provider API keys are never forwarded to SmallerModels.
The original client performs fallback requests from the customer process.
Telemetry identifies the operation and route without including provider credentials.
Request retention and redaction are configured per project.

Provider adapters must preserve response types, streaming events, tool calls, usage fields, retries, and native provider errors.