OpenAI Chat Completions in September 2024: Snapshots, Strict Tool Calls, Structured Outputs, Streaming and Batch

On this page

Most applications built on OpenAI’s models call one endpoint: Chat Completions. In the summer of 2024 that endpoint changed more than its stable shape suggests. A cheaper GPT-4o snapshot arrived with Structured Outputs, tool calls gained a strict mode, streamed responses could finally report token usage, and the Batch API halved the price of work that can wait a day. Code written against the spring 2024 API still runs, but it pays more and parses less reliably than it needs to.

This article covers the Chat Completions API as of September 2024: which model snapshot to pin, how to set up the Python client and hold a conversation, how to send images, how to run tool calls and structured outputs safely, how to stream with usage, and when to use the Batch API. The Assistants API gets a short section on where it fits. For how the GPT-3.5 and GPT-4 generations compared at the start of the year, see Choosing an OpenAI Model in January 2024.

Models and snapshots

OpenAI model snapshots in September 2024 with list prices, and the request features to use: strict tool calls, Structured Outputs, streaming with usage and the Batch API
Pin a dated snapshot and switch on the features released in 2024. The alias and the snapshot are not the same model, or the same price.

OpenAI released gpt-4o-2024-08-06 on 6 August 2024 alongside Structured Outputs, priced at $2.50 per million input tokens and $10.00 per million output tokens, against $5.00 and $15.00 for gpt-4o-2024-05-13. The newer snapshot also allows up to 16,384 output tokens, up from 4,096. The plain gpt-4o alias, however, still points to the May snapshot: OpenAI has said it will move the alias to gpt-4o-2024-08-06 on 2 October 2024.

ModelInput / output per 1M tokensUse it for
gpt-4o-2024-08-06$2.50 / $10.00New work; the snapshot with Structured Outputs in response_format
gpt-4o-mini-2024-07-18$0.15 / $0.60High-volume classification, extraction and routing
gpt-4o (alias)$5.00 / $15.00 until the alias movesAvoid in production code; it changes underneath you
gpt-4-turbo, gpt-4$10 / $30 and $30 / $60Existing prompts already tested on them
OpenAI GPT-4 Model Family Comparison
The GPT-4 model family in September 2024: capabilities, context windows and list prices. The GPT-4o prices shown are those of the gpt-4o alias, which still points to gpt-4o-2024-05-13.
⚠️
PIN THE SNAPSHOT, NOT THE ALIAS

An alias that moves on a fixed date changes cost, output limits and behaviour on that date without a deployment. Put the dated snapshot in configuration, run your evaluation set against the next snapshot, and switch deliberately.

The client

The openai Python package (1.44.1 at the time of writing) reads OPENAI_API_KEY from the environment. Two defaults deserve a decision rather than acceptance: requests time out after 600 seconds, and the client retries twice, with backoff, on connection errors and on HTTP 408, 409, 429 and 5xx responses. A ten-minute timeout ties up a web worker long after the user has gone; set one that matches the caller, and raise retries a little for background jobs. Retries already honour the retry-after header, so a hand-written retry decorator around the SDK mostly adds a second, uncoordinated layer.

First calls and conversations

Install a pinned version of the SDK (pip install openai==1.44.1) rather than whatever is latest on the day of deployment; the package releases often and new parameters appear with new versions. A first call sends a list of messages, a system message that sets behaviour and a user message, and gets back the answer, a finish reason and token usage. Check the finish reason: length means the answer was cut off by max_tokens.

The API keeps no memory between calls. A conversation is the history your application sends each time, so every turn costs the tokens of all the turns before it. Cap the history you resend, keep the system message, and summarise or retrieve older context when a conversation needs it. The sample’s Conversation class does the capping, and image_part builds image input for the next section:

Python88 lines
"""First calls, multi-turn conversations and image input with Chat Completions.

Tested with openai==1.44.1.
"""
from __future__ import annotations

import base64
import mimetypes
from dataclasses import dataclass, field
from pathlib import Path
from typing import Literal, Optional

from openai import OpenAI

from chat_api import MODEL


@dataclass
class Answer:
    text: Optional[str]
    finish_reason: str
    prompt_tokens: int
    completion_tokens: int


def ask(client: OpenAI, question: str, system: str = "You are a concise technical assistant.",
        model: str = MODEL, max_tokens: int = 500) -> Answer:
    resp = client.chat.completions.create(
        model=model,
        messages=[{"role": "system", "content": system}, {"role": "user", "content": question}],
        max_tokens=max_tokens,
    )
    choice = resp.choices[0]
    return Answer(choice.message.content, choice.finish_reason,
                  resp.usage.prompt_tokens, resp.usage.completion_tokens)


@dataclass
class Conversation:
    """The API is stateless: the application sends the history on every call.

    max_turns caps how many user/assistant pairs are resent, so cost and
    latency do not grow without limit. The system prompt is always kept.
    """
    client: OpenAI
    system: str
    model: str = MODEL
    max_turns: int = 10
    history: list[dict] = field(default_factory=list)

    def send(self, text: str) -> str:
        self.history.append({"role": "user", "content": text})
        recent = self.history[-(2 * self.max_turns - 1):]
        resp = self.client.chat.completions.create(
            model=self.model, messages=[{"role": "system", "content": self.system}, *recent]
        )
        reply = resp.choices[0].message.content or ""
        self.history.append({"role": "assistant", "content": reply})
        return reply


def image_part(source: str, detail: Literal["low", "high", "auto"] = "auto") -> dict:
    """An image content part from an https URL or a local file.

    Local files are sent as a data URL with the file's real MIME type.
    detail="low" costs a fixed, small number of tokens; "high" costs more for large images.
    """
    if source.startswith("https://"):
        url = source
    else:
        path = Path(source)
        mime = mimetypes.guess_type(path.name)[0]
        if mime not in {"image/png", "image/jpeg", "image/gif", "image/webp"}:
            raise ValueError(f"unsupported image type: {path.suffix}")
        url = f"data:{mime};base64,{base64.b64encode(path.read_bytes()).decode()}"
    return {"type": "image_url", "image_url": {"url": url, "detail": detail}}


def describe_image(client: OpenAI, source: str, prompt: str, model: str = MODEL,
                   detail: Literal["low", "high", "auto"] = "auto", max_tokens: int = 500) -> Answer:
    resp = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": [{"type": "text", "text": prompt}, image_part(source, detail)]}],
        max_tokens=max_tokens,
    )
    choice = resp.choices[0]
    return Answer(choice.message.content, choice.finish_reason,
                  resp.usage.prompt_tokens, resp.usage.completion_tokens)

Tool calls with strict schemas

Tool calling lets the model ask your code to run a function and use the result. Since August 2024, setting strict: true on a function definition makes the generated arguments match its JSON schema, which must then set additionalProperties to false and list every property as required. OpenAI’s announcement notes one limitation: Structured Outputs is not compatible with parallel function calls, so set parallel_tool_calls to false when you rely on strict schemas. The SDK’s openai.pydantic_function_tool builds a strict definition from a Pydantic model.

Strict schemas guarantee shape, not sense. The model can still pick the wrong tool, pass an order number that does not exist or keep calling tools in a loop. The loop below validates arguments again, sends errors back to the model as tool results instead of raising, and stops after a fixed number of model calls. Building LLM Agents with Tools covers agent loops in more depth.

OpenAI Function Calling Flow
Function calling flow: the model asks for a tool, your code runs it, and a second call turns the result into an answer.
%%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#FFF3E0', 'primaryTextColor': '#232F3E', 'primaryBorderColor': '#FF9900', 'lineColor': '#232F3E', 'clusterBkg': '#F8FAFC', 'clusterBorder': '#CBD5E1', 'edgeLabelBackground': '#FFFFFF', 'nodeTextColor': '#232F3E'}}}%%
flowchart TD
    A[Send messages and strict tool schemas] --> B{Model returned tool calls?}
    B -- No --> C[Return the answer]
    B -- Yes --> D[Validate arguments with the Pydantic model]
    D -- Invalid --> E[Send the error back as the tool result]
    D -- Valid --> F[Run the tool]
    F -- Raises --> E
    F -- Returns --> G[Send the result back as the tool result]
    E --> H{Step limit reached?}
    G --> H
    H -- No --> A
    H -- Yes --> I[Stop and report step_limit]
    style C fill:#F0FFF4,stroke:#1D8102,stroke-width:2px,color:#145202
    style I fill:#FEF2F2,stroke:#D13212,stroke-width:2px,color:#991B1B
    style D fill:#EBF8FF,stroke:#0073BB,stroke-width:2px,color:#004C74
💡
DO NOT LEAK TOOL ERRORS TO THE MODEL

Return a short error type to the model and keep the stack trace in your logs. Exception text can contain connection strings, internal hostnames or other customers’ data, and anything in a tool result can end up in the answer.

Structured outputs

For extraction and classification, where the answer itself is data, use Structured Outputs through response_format with a JSON schema. It is available on gpt-4o-2024-08-06 and gpt-4o-mini, not on the May snapshot. In Python, client.beta.chat.completions.parse takes a Pydantic model and returns a parsed object. Two outcomes still need handling: the model can decline, which arrives as a refusal string instead of the schema, and an answer cut off by max_tokens is incomplete JSON, which the SDK reports by raising LengthFinishReasonError. For JSON mode and Pydantic parsing on models without this feature, see LLM Output Formatting.

The sample module covers the client, the tool loop, structured extraction, streaming and batch:

Python215 lines
"""Calling the OpenAI Chat Completions API from Python, as of September 2024.

Tested with openai==1.44.1. Covers:
  * a client with explicit timeout and retries
  * a pinned model snapshot
  * a tool-calling loop with strict schemas, argument validation and a step limit
  * structured output into a Pydantic model, with refusals and truncation reported
  * streaming with token usage
  * Batch API request lines and result parsing
"""
from __future__ import annotations

import json
import logging
from dataclasses import dataclass, field
from typing import Any, Callable, Iterator, Literal, Optional, Type

import openai
from openai import OpenAI
from pydantic import BaseModel, ValidationError

log = logging.getLogger("chat_api")

# Pin a dated snapshot. On 10 Sep 2024 the "gpt-4o" alias still pointed to
# gpt-4o-2024-05-13; the 2024-08-06 snapshot is cheaper and adds Structured Outputs.
MODEL = "gpt-4o-2024-08-06"


def make_client(timeout: float = 60.0, max_retries: int = 3, **kwargs: Any) -> OpenAI:
    """Client with a deliberate timeout (the SDK default is 10 minutes) and retries.

    The SDK retries connection errors, 408, 409, 429 and 5xx with backoff and
    honours retry-after. The API key comes from OPENAI_API_KEY unless passed.
    """
    return OpenAI(timeout=timeout, max_retries=max_retries, **kwargs)


# ---------------------------------------------------------------- tools


@dataclass
class Tool:
    name: str
    args_model: Type[BaseModel]
    func: Callable[[BaseModel], Any]
    description: str = ""

    def schema(self) -> dict:
        return openai.pydantic_function_tool(
            self.args_model, name=self.name, description=self.description or None
        )


@dataclass
class ToolRun:
    answer: Optional[str]
    steps: int
    calls: list[tuple[str, str]] = field(default_factory=list)  # (tool, outcome)
    stopped: Literal["answered", "step_limit"] = "answered"


def run_with_tools(
    client: OpenAI,
    messages: list[dict],
    tools: list[Tool],
    model: str = MODEL,
    max_steps: int = 5,
) -> ToolRun:
    """Loop until the model answers or max_steps model calls have been made.

    Tool errors and invalid arguments go back to the model as tool results
    instead of crashing the request, so it can correct itself.
    """
    by_name = {t.name: t for t in tools}
    messages = list(messages)
    calls: list[tuple[str, str]] = []
    for step in range(1, max_steps + 1):
        resp = client.chat.completions.create(
            model=model,
            messages=messages,
            tools=[t.schema() for t in tools],
            parallel_tool_calls=False,  # strict schemas need this in Sep 2024
            temperature=0,
        )
        msg = resp.choices[0].message
        if not msg.tool_calls:
            return ToolRun(msg.content, step, calls)
        messages.append(msg.model_dump(exclude_none=True))
        for tc in msg.tool_calls:
            content, outcome = _call_tool(by_name, tc.function.name, tc.function.arguments)
            calls.append((tc.function.name, outcome))
            log.info("tool=%s outcome=%s", tc.function.name, outcome)
            messages.append({"role": "tool", "tool_call_id": tc.id, "content": content})
    return ToolRun(None, max_steps, calls, stopped="step_limit")


def _call_tool(by_name: dict[str, Tool], name: str, raw_args: str) -> tuple[str, str]:
    tool = by_name.get(name)
    if tool is None:
        return json.dumps({"error": f"unknown tool {name}"}), "unknown_tool"
    try:
        args = tool.args_model.model_validate_json(raw_args)
    except ValidationError as e:
        return json.dumps({"error": "invalid arguments", "detail": e.errors(include_url=False)}, default=str), "invalid_args"
    try:
        result = tool.func(args)
    except Exception as e:  # report to the model, keep the details in our logs
        log.exception("tool %s failed", name)
        return json.dumps({"error": f"{type(e).__name__}"}), "tool_error"
    return json.dumps(result, default=str), "ok"


# ---------------------------------------------------------------- structured output


@dataclass
class Parsed:
    status: Literal["ok", "refused", "truncated", "filtered"]
    value: Optional[BaseModel] = None
    detail: Optional[str] = None


def extract(client: OpenAI, schema: Type[BaseModel], system: str, text: str,
            model: str = MODEL, max_tokens: int = 1000) -> Parsed:
    try:
        resp = client.beta.chat.completions.parse(
            model=model,
            messages=[{"role": "system", "content": system}, {"role": "user", "content": text}],
            response_format=schema,
            temperature=0,
            max_tokens=max_tokens,
        )
    except openai.LengthFinishReasonError:
        return Parsed("truncated", detail=f"hit max_tokens={max_tokens}")
    except openai.ContentFilterFinishReasonError:
        return Parsed("filtered")
    msg = resp.choices[0].message
    if msg.refusal:
        return Parsed("refused", detail=msg.refusal)
    return Parsed("ok", value=msg.parsed)


# ---------------------------------------------------------------- streaming


@dataclass
class StreamResult:
    text: str = ""
    finish_reason: Optional[str] = None
    prompt_tokens: Optional[int] = None
    completion_tokens: Optional[int] = None


def stream_text(client: OpenAI, messages: list[dict], on_delta: Callable[[str], None],
                model: str = MODEL) -> StreamResult:
    """Stream text to on_delta and return the full text with token usage.

    With include_usage the final chunk has empty choices and carries usage.
    """
    out = StreamResult()
    parts: list[str] = []
    stream = client.chat.completions.create(
        model=model, messages=messages, stream=True, stream_options={"include_usage": True}
    )
    for chunk in stream:
        if chunk.usage:
            out.prompt_tokens = chunk.usage.prompt_tokens
            out.completion_tokens = chunk.usage.completion_tokens
        for choice in chunk.choices:
            if choice.delta.content:
                parts.append(choice.delta.content)
                on_delta(choice.delta.content)
            if choice.finish_reason:
                out.finish_reason = choice.finish_reason
    out.text = "".join(parts)
    return out


# ---------------------------------------------------------------- batch


def batch_lines(items: dict[str, list[dict]], model: str = MODEL, **body: Any) -> Iterator[str]:
    """JSONL request lines for the Batch API, one per custom_id."""
    for custom_id, messages in items.items():
        yield json.dumps({
            "custom_id": custom_id,
            "method": "POST",
            "url": "/v1/chat/completions",
            "body": {"model": model, "messages": messages, **body},
        })


def read_batch_results(lines: Iterator[str]) -> dict[str, dict]:
    """Map custom_id to {"ok": bool, "content" or "error"}. Output order is not guaranteed."""
    results: dict[str, dict] = {}
    for line in lines:
        if not line.strip():
            continue
        rec = json.loads(line)
        resp, err = rec.get("response"), rec.get("error")
        if err or not resp or resp.get("status_code") != 200:
            results[rec["custom_id"]] = {"ok": False, "error": err or (resp or {}).get("body")}
        else:
            results[rec["custom_id"]] = {"ok": True,
                                         "content": resp["body"]["choices"][0]["message"]["content"]}
    return results


def submit_batch(client: OpenAI, jsonl_path: str) -> str:
    """Upload a JSONL file and start a batch; returns the batch id to poll."""
    with open(jsonl_path, "rb") as f:
        upload = client.files.create(file=f, purpose="batch")
    batch = client.batches.create(input_file_id=upload.id, endpoint="/v1/chat/completions",
                                  completion_window="24h")
    return batch.id

Images as input

GPT-4o and GPT-4o mini accept images in the same Chat Completions call. A user message’s content becomes a list of parts: text parts and image_url parts, where the URL is either an https address the API can fetch or a base64 data URL for a local file. Send the file’s real MIME type in the data URL, and choose the detail level deliberately: low sends a small, fixed-cost version of the image, while high lets the model see fine print at a higher token cost. Image tokens count as input tokens, so a screenshot-heavy workload can cost more than its text suggests; check usage.prompt_tokens on a few real requests before estimating.

⚠️
IMAGES CARRY DATA TOO

Screenshots and photos often contain names, addresses, faces or patient details that nobody typed. Apply the same data rules to images as to text, and crop or redact before sending where you can.

Streaming with token usage

Streaming sends text as it is generated, which matters for chat interfaces. Until May 2024 a streamed response carried no token counts, so cost tracking for streamed traffic relied on estimates. Setting stream_options={"include_usage": True} adds a final chunk with the usage for the whole request. That chunk has an empty choices list, so code that reads chunk.choices[0] on every chunk will fail on it; iterate over choices instead, as the sample does.

The Batch API

Work that can wait, such as nightly classification, evaluation runs and backfills, belongs in the Batch API. You upload a JSONL file of requests, each with a custom_id, and results come back within 24 hours at half the synchronous price, with a separate and much larger rate limit. Results are matched by custom_id, not by line order, and a failed line does not fail the batch, so check each result’s status.

Chat Completions, streaming, Batch API and Assistants API compared by response time, cost, conversation state, best use and what to watch for
Choose the call mode by how soon the answer is needed and who should hold the conversation state.

Where the Assistants API fits

The Assistants API keeps conversation threads on OpenAI’s side and runs hosted tools such as file search and code interpreter. It suits applications that need those hosted tools and would otherwise build them. The trade-offs are that state and uploaded files live with OpenAI until you delete them, runs have to be polled or streamed, and hosted tools carry their own fees. For most features, Chat Completions with your own history and retrieval keeps data, cost and behaviour under your control.

⚖️
DECIDE WHAT YOU LOG

Prompts and completions often contain personal data. Log request IDs, model snapshot, token counts, tool names and outcomes by default, and log content only where you have a lawful basis, a retention period and access controls for it.

The tests mock the HTTP layer and check what the client actually sends, as well as how each outcome is handled:

Python141 lines
import json

import httpx
import pytest
from pydantic import BaseModel

import chat_api


def client_with(handler, seen, **kw):
    def wrapped(request: httpx.Request) -> httpx.Response:
        seen.append(request)
        return handler(request)
    return chat_api.make_client(api_key="sk-test", max_retries=kw.pop("max_retries", 0),
                                http_client=httpx.Client(transport=httpx.MockTransport(wrapped)), **kw)


def completion(content=None, finish="stop", refusal=None, tool_calls=None):
    msg = {"role": "assistant", "content": content, "refusal": refusal}
    if tool_calls:
        msg["tool_calls"] = tool_calls
    return httpx.Response(200, json={
        "id": "c1", "object": "chat.completion", "created": 1725958800, "model": chat_api.MODEL,
        "choices": [{"index": 0, "finish_reason": finish, "message": msg}],
        "usage": {"prompt_tokens": 120, "completion_tokens": 30, "total_tokens": 150}})


def tool_call(name, args, id_="call_1"):
    return [{"id": id_, "type": "function", "function": {"name": name, "arguments": args}}]


class OrderLookup(BaseModel):
    order_id: str


def lookup(args: OrderLookup):
    if args.order_id == "boom":
        raise RuntimeError("database down")
    return {"order_id": args.order_id, "status": "shipped"}


TOOLS = [chat_api.Tool("get_order_status", OrderLookup, lookup, "Look up an order's shipping status")]


def test_tool_loop_sends_strict_schema_runs_tool_and_returns_answer():
    seen, replies = [], iter([
        completion(None, finish="tool_calls", tool_calls=tool_call("get_order_status", '{"order_id":"A-17"}')),
        completion("Order A-17 has shipped."),
    ])
    c = client_with(lambda r: next(replies), seen)
    run = chat_api.run_with_tools(c, [{"role": "user", "content": "Where is A-17?"}], TOOLS)
    assert run.answer == "Order A-17 has shipped." and run.steps == 2 and run.calls == [("get_order_status", "ok")]
    first = json.loads(seen[0].content)
    assert first["model"] == "gpt-4o-2024-08-06" and first["parallel_tool_calls"] is False
    fn = first["tools"][0]["function"]
    assert fn["strict"] is True and fn["parameters"]["additionalProperties"] is False
    second = json.loads(seen[1].content)["messages"]
    assert second[1]["tool_calls"][0]["id"] == "call_1"
    assert second[2] == {"role": "tool", "tool_call_id": "call_1",
                         "content": json.dumps({"order_id": "A-17", "status": "shipped"})}


@pytest.mark.parametrize("name,args,outcome", [
    ("get_order_status", '{"order":"A-17"}', "invalid_args"),
    ("delete_order", '{"order_id":"A-17"}', "unknown_tool"),
    ("get_order_status", '{"order_id":"boom"}', "tool_error"),
])
def test_tool_problems_go_back_to_the_model(name, args, outcome):
    seen, replies = [], iter([completion(None, "tool_calls", tool_calls=tool_call(name, args)), completion("Sorry.")])
    run = chat_api.run_with_tools(client_with(lambda r: next(replies), seen), [{"role": "user", "content": "x"}], TOOLS)
    assert run.calls == [(name, outcome)]
    sent = json.loads(json.loads(seen[1].content)["messages"][-1]["content"])
    assert "error" in sent and "database down" not in json.dumps(sent)


def test_step_limit_stops_a_looping_model():
    c = client_with(lambda r: completion(None, "tool_calls", tool_calls=tool_call("get_order_status", '{"order_id":"A"}')), [])
    run = chat_api.run_with_tools(c, [{"role": "user", "content": "x"}], TOOLS, max_steps=3)
    assert run.stopped == "step_limit" and run.answer is None and len(run.calls) == 3


class Ticket(BaseModel):
    category: str
    priority: int


def test_extract_ok_refused_truncated():
    ok = chat_api.extract(client_with(lambda r: completion('{"category":"billing","priority":2}'), seen := []),
                          Ticket, "Classify the ticket.", "I was charged twice")
    assert ok.status == "ok" and ok.value == Ticket(category="billing", priority=2)
    body = json.loads(seen[0].content)
    assert body["response_format"]["type"] == "json_schema" and body["response_format"]["json_schema"]["strict"] is True
    refused = chat_api.extract(client_with(lambda r: completion(None, refusal="I can't help with that."), []), Ticket, "s", "t")
    assert refused.status == "refused"
    cut = chat_api.extract(client_with(lambda r: completion('{"category":"bil', finish="length"), []), Ticket, "s", "t")
    assert cut.status == "truncated"


def sse(*events):
    body = "".join(f"data: {json.dumps(e)}\n\n" for e in events) + "data: [DONE]\n\n"
    return httpx.Response(200, headers={"content-type": "text/event-stream"}, content=body.encode())


def chunk(delta=None, finish=None, usage=None, choices=True):
    return {"id": "s1", "object": "chat.completion.chunk", "created": 1725958800, "model": chat_api.MODEL,
            "choices": [{"index": 0, "delta": delta or {}, "finish_reason": finish}] if choices else [],
            "usage": usage}


def test_stream_collects_text_and_usage():
    seen, got = [], []
    c = client_with(lambda r: sse(chunk({"role": "assistant", "content": ""}), chunk({"content": "Hel"}),
                                  chunk({"content": "lo"}), chunk(finish="stop"),
                                  chunk(choices=False, usage={"prompt_tokens": 9, "completion_tokens": 2,
                                                              "total_tokens": 11})), seen)
    out = chat_api.stream_text(c, [{"role": "user", "content": "hi"}], got.append)
    assert out.text == "Hello" and got == ["Hel", "lo"] and out.finish_reason == "stop"
    assert (out.prompt_tokens, out.completion_tokens) == (9, 2)
    assert json.loads(seen[0].content)["stream_options"] == {"include_usage": True}


def test_client_retries_429_then_succeeds():
    replies = iter([httpx.Response(429, headers={"retry-after-ms": "10"}, json={"error": {"message": "slow down"}}),
                    completion("ok")])
    seen = []
    c = client_with(lambda r: next(replies), seen, max_retries=2)
    assert c.chat.completions.create(model=chat_api.MODEL, messages=[{"role": "user", "content": "x"}]).choices[0].message.content == "ok"
    assert len(seen) == 2 and c.timeout == 60.0


def test_batch_lines_and_results():
    lines = list(chat_api.batch_lines({"t-1": [{"role": "user", "content": "a"}]}, temperature=0))
    rec = json.loads(lines[0])
    assert rec["custom_id"] == "t-1" and rec["url"] == "/v1/chat/completions" and rec["body"]["temperature"] == 0
    out = [
        json.dumps({"id": "r2", "custom_id": "t-2", "response": {"status_code": 400, "body": {"error": {"message": "bad"}}}, "error": None}),
        json.dumps({"id": "r1", "custom_id": "t-1", "response": {"status_code": 200, "body": {
            "choices": [{"message": {"role": "assistant", "content": "done"}}]}}, "error": None}),
    ]
    res = chat_api.read_batch_results(iter(out))
    assert res["t-1"] == {"ok": True, "content": "done"} and res["t-2"]["ok"] is False
📝
HOW THIS WAS CHECKED

chat_api.py was tested with Python 3.12, openai==1.44.1, httpx==0.27.2, pydantic==2.9.1 and pytest==8.3.2: 13 tests against a mocked endpoint cover the first call, conversation history capping, image content parts, the pinned model, strict tool schema and parallel_tool_calls, invalid arguments, unknown tools, tool exceptions, the step limit, structured output with refusal and truncation, streamed usage, retry on 429 and batch result parsing. No OpenAI API call was made. Retry and timeout defaults were read from the 1.44.1 SDK source. Prices, snapshot dates and feature limits are from OpenAI’s announcements linked below.

Key takeaways

  • Pin gpt-4o-2024-08-06 or gpt-4o-mini-2024-07-18; the gpt-4o alias still points to the older, more expensive snapshot until 2 October 2024.
  • Set a timeout and retry count on the client instead of wrapping it in your own retry logic.
  • Use strict tool schemas with parallel_tool_calls off, and still validate arguments, return errors to the model and cap the number of steps.
  • Use Structured Outputs for data extraction and handle refusals and truncation as separate outcomes.
  • Request usage when streaming, and move work that can wait a day to the Batch API at half the price.

Which of these changes made the biggest difference to your OpenAI bill or error rate? Share it in the comments.

Glossary

Snapshot
A dated, fixed version of a model, such as gpt-4o-2024-08-06.
Alias
A model name such as gpt-4o that OpenAI points at a snapshot and can move.
Tool call
A request from the model for your code to run a named function with given arguments.
Strict mode
Constraining generated tool arguments or responses to a supplied JSON schema.
Refusal
A message in which the model declines to produce the requested output.
Batch API
An asynchronous endpoint that processes JSONL files of requests within 24 hours at a discount.
custom_id
Your identifier on each batch request, used to match results to requests.

References & Further Reading


Discover more from C4: Container, Code, Cloud & Context

Subscribe to get the latest posts sent to your email.

Written by

Nithin Mohan TK

Hello Namaste! 👋🏻 I'm an accomplished Enterprise Solution Architect & Engineering Leader with 20+ years of progressive experience architecting, building, and scaling transformative enterprise solutions. ⚜️ Solution Architect / AI & Cloud 📌 Galway, Ireland

Leave a comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.