On this page
Most applications built on OpenAI’s models call one endpoint: Chat Completions. In the summer of 2024 that endpoint changed more than its stable shape suggests. A cheaper GPT-4o snapshot arrived with Structured Outputs, tool calls gained a strict mode, streamed responses could finally report token usage, and the Batch API halved the price of work that can wait a day. Code written against the spring 2024 API still runs, but it pays more and parses less reliably than it needs to.
This article covers the Chat Completions API as of September 2024: which model snapshot to pin, how to set up the Python client and hold a conversation, how to send images, how to run tool calls and structured outputs safely, how to stream with usage, and when to use the Batch API. The Assistants API gets a short section on where it fits. For how the GPT-3.5 and GPT-4 generations compared at the start of the year, see Choosing an OpenAI Model in January 2024.
Models and snapshots

OpenAI released gpt-4o-2024-08-06 on 6 August 2024 alongside Structured Outputs, priced at $2.50 per million input tokens and $10.00 per million output tokens, against $5.00 and $15.00 for gpt-4o-2024-05-13. The newer snapshot also allows up to 16,384 output tokens, up from 4,096. The plain gpt-4o alias, however, still points to the May snapshot: OpenAI has said it will move the alias to gpt-4o-2024-08-06 on 2 October 2024.
| Model | Input / output per 1M tokens | Use it for |
|---|---|---|
gpt-4o-2024-08-06 | $2.50 / $10.00 | New work; the snapshot with Structured Outputs in response_format |
gpt-4o-mini-2024-07-18 | $0.15 / $0.60 | High-volume classification, extraction and routing |
gpt-4o (alias) | $5.00 / $15.00 until the alias moves | Avoid in production code; it changes underneath you |
gpt-4-turbo, gpt-4 | $10 / $30 and $30 / $60 | Existing prompts already tested on them |
An alias that moves on a fixed date changes cost, output limits and behaviour on that date without a deployment. Put the dated snapshot in configuration, run your evaluation set against the next snapshot, and switch deliberately.
The client
The openai Python package (1.44.1 at the time of writing) reads OPENAI_API_KEY from the environment. Two defaults deserve a decision rather than acceptance: requests time out after 600 seconds, and the client retries twice, with backoff, on connection errors and on HTTP 408, 409, 429 and 5xx responses. A ten-minute timeout ties up a web worker long after the user has gone; set one that matches the caller, and raise retries a little for background jobs. Retries already honour the retry-after header, so a hand-written retry decorator around the SDK mostly adds a second, uncoordinated layer.
First calls and conversations
Install a pinned version of the SDK (pip install openai==1.44.1) rather than whatever is latest on the day of deployment; the package releases often and new parameters appear with new versions. A first call sends a list of messages, a system message that sets behaviour and a user message, and gets back the answer, a finish reason and token usage. Check the finish reason: length means the answer was cut off by max_tokens.
The API keeps no memory between calls. A conversation is the history your application sends each time, so every turn costs the tokens of all the turns before it. Cap the history you resend, keep the system message, and summarise or retrieve older context when a conversation needs it. The sample’s Conversation class does the capping, and image_part builds image input for the next section:
"""First calls, multi-turn conversations and image input with Chat Completions.
Tested with openai==1.44.1.
"""
from __future__ import annotations
import base64
import mimetypes
from dataclasses import dataclass, field
from pathlib import Path
from typing import Literal, Optional
from openai import OpenAI
from chat_api import MODEL
@dataclass
class Answer:
text: Optional[str]
finish_reason: str
prompt_tokens: int
completion_tokens: int
def ask(client: OpenAI, question: str, system: str = "You are a concise technical assistant.",
model: str = MODEL, max_tokens: int = 500) -> Answer:
resp = client.chat.completions.create(
model=model,
messages=[{"role": "system", "content": system}, {"role": "user", "content": question}],
max_tokens=max_tokens,
)
choice = resp.choices[0]
return Answer(choice.message.content, choice.finish_reason,
resp.usage.prompt_tokens, resp.usage.completion_tokens)
@dataclass
class Conversation:
"""The API is stateless: the application sends the history on every call.
max_turns caps how many user/assistant pairs are resent, so cost and
latency do not grow without limit. The system prompt is always kept.
"""
client: OpenAI
system: str
model: str = MODEL
max_turns: int = 10
history: list[dict] = field(default_factory=list)
def send(self, text: str) -> str:
self.history.append({"role": "user", "content": text})
recent = self.history[-(2 * self.max_turns - 1):]
resp = self.client.chat.completions.create(
model=self.model, messages=[{"role": "system", "content": self.system}, *recent]
)
reply = resp.choices[0].message.content or ""
self.history.append({"role": "assistant", "content": reply})
return reply
def image_part(source: str, detail: Literal["low", "high", "auto"] = "auto") -> dict:
"""An image content part from an https URL or a local file.
Local files are sent as a data URL with the file's real MIME type.
detail="low" costs a fixed, small number of tokens; "high" costs more for large images.
"""
if source.startswith("https://"):
url = source
else:
path = Path(source)
mime = mimetypes.guess_type(path.name)[0]
if mime not in {"image/png", "image/jpeg", "image/gif", "image/webp"}:
raise ValueError(f"unsupported image type: {path.suffix}")
url = f"data:{mime};base64,{base64.b64encode(path.read_bytes()).decode()}"
return {"type": "image_url", "image_url": {"url": url, "detail": detail}}
def describe_image(client: OpenAI, source: str, prompt: str, model: str = MODEL,
detail: Literal["low", "high", "auto"] = "auto", max_tokens: int = 500) -> Answer:
resp = client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": [{"type": "text", "text": prompt}, image_part(source, detail)]}],
max_tokens=max_tokens,
)
choice = resp.choices[0]
return Answer(choice.message.content, choice.finish_reason,
resp.usage.prompt_tokens, resp.usage.completion_tokens)Tool calls with strict schemas
Tool calling lets the model ask your code to run a function and use the result. Since August 2024, setting strict: true on a function definition makes the generated arguments match its JSON schema, which must then set additionalProperties to false and list every property as required. OpenAI’s announcement notes one limitation: Structured Outputs is not compatible with parallel function calls, so set parallel_tool_calls to false when you rely on strict schemas. The SDK’s openai.pydantic_function_tool builds a strict definition from a Pydantic model.
Strict schemas guarantee shape, not sense. The model can still pick the wrong tool, pass an order number that does not exist or keep calling tools in a loop. The loop below validates arguments again, sends errors back to the model as tool results instead of raising, and stops after a fixed number of model calls. Building LLM Agents with Tools covers agent loops in more depth.
%%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#FFF3E0', 'primaryTextColor': '#232F3E', 'primaryBorderColor': '#FF9900', 'lineColor': '#232F3E', 'clusterBkg': '#F8FAFC', 'clusterBorder': '#CBD5E1', 'edgeLabelBackground': '#FFFFFF', 'nodeTextColor': '#232F3E'}}}%%
flowchart TD
A[Send messages and strict tool schemas] --> B{Model returned tool calls?}
B -- No --> C[Return the answer]
B -- Yes --> D[Validate arguments with the Pydantic model]
D -- Invalid --> E[Send the error back as the tool result]
D -- Valid --> F[Run the tool]
F -- Raises --> E
F -- Returns --> G[Send the result back as the tool result]
E --> H{Step limit reached?}
G --> H
H -- No --> A
H -- Yes --> I[Stop and report step_limit]
style C fill:#F0FFF4,stroke:#1D8102,stroke-width:2px,color:#145202
style I fill:#FEF2F2,stroke:#D13212,stroke-width:2px,color:#991B1B
style D fill:#EBF8FF,stroke:#0073BB,stroke-width:2px,color:#004C74Return a short error type to the model and keep the stack trace in your logs. Exception text can contain connection strings, internal hostnames or other customers’ data, and anything in a tool result can end up in the answer.
Structured outputs
For extraction and classification, where the answer itself is data, use Structured Outputs through response_format with a JSON schema. It is available on gpt-4o-2024-08-06 and gpt-4o-mini, not on the May snapshot. In Python, client.beta.chat.completions.parse takes a Pydantic model and returns a parsed object. Two outcomes still need handling: the model can decline, which arrives as a refusal string instead of the schema, and an answer cut off by max_tokens is incomplete JSON, which the SDK reports by raising LengthFinishReasonError. For JSON mode and Pydantic parsing on models without this feature, see LLM Output Formatting.
The sample module covers the client, the tool loop, structured extraction, streaming and batch:
"""Calling the OpenAI Chat Completions API from Python, as of September 2024.
Tested with openai==1.44.1. Covers:
* a client with explicit timeout and retries
* a pinned model snapshot
* a tool-calling loop with strict schemas, argument validation and a step limit
* structured output into a Pydantic model, with refusals and truncation reported
* streaming with token usage
* Batch API request lines and result parsing
"""
from __future__ import annotations
import json
import logging
from dataclasses import dataclass, field
from typing import Any, Callable, Iterator, Literal, Optional, Type
import openai
from openai import OpenAI
from pydantic import BaseModel, ValidationError
log = logging.getLogger("chat_api")
# Pin a dated snapshot. On 10 Sep 2024 the "gpt-4o" alias still pointed to
# gpt-4o-2024-05-13; the 2024-08-06 snapshot is cheaper and adds Structured Outputs.
MODEL = "gpt-4o-2024-08-06"
def make_client(timeout: float = 60.0, max_retries: int = 3, **kwargs: Any) -> OpenAI:
"""Client with a deliberate timeout (the SDK default is 10 minutes) and retries.
The SDK retries connection errors, 408, 409, 429 and 5xx with backoff and
honours retry-after. The API key comes from OPENAI_API_KEY unless passed.
"""
return OpenAI(timeout=timeout, max_retries=max_retries, **kwargs)
# ---------------------------------------------------------------- tools
@dataclass
class Tool:
name: str
args_model: Type[BaseModel]
func: Callable[[BaseModel], Any]
description: str = ""
def schema(self) -> dict:
return openai.pydantic_function_tool(
self.args_model, name=self.name, description=self.description or None
)
@dataclass
class ToolRun:
answer: Optional[str]
steps: int
calls: list[tuple[str, str]] = field(default_factory=list) # (tool, outcome)
stopped: Literal["answered", "step_limit"] = "answered"
def run_with_tools(
client: OpenAI,
messages: list[dict],
tools: list[Tool],
model: str = MODEL,
max_steps: int = 5,
) -> ToolRun:
"""Loop until the model answers or max_steps model calls have been made.
Tool errors and invalid arguments go back to the model as tool results
instead of crashing the request, so it can correct itself.
"""
by_name = {t.name: t for t in tools}
messages = list(messages)
calls: list[tuple[str, str]] = []
for step in range(1, max_steps + 1):
resp = client.chat.completions.create(
model=model,
messages=messages,
tools=[t.schema() for t in tools],
parallel_tool_calls=False, # strict schemas need this in Sep 2024
temperature=0,
)
msg = resp.choices[0].message
if not msg.tool_calls:
return ToolRun(msg.content, step, calls)
messages.append(msg.model_dump(exclude_none=True))
for tc in msg.tool_calls:
content, outcome = _call_tool(by_name, tc.function.name, tc.function.arguments)
calls.append((tc.function.name, outcome))
log.info("tool=%s outcome=%s", tc.function.name, outcome)
messages.append({"role": "tool", "tool_call_id": tc.id, "content": content})
return ToolRun(None, max_steps, calls, stopped="step_limit")
def _call_tool(by_name: dict[str, Tool], name: str, raw_args: str) -> tuple[str, str]:
tool = by_name.get(name)
if tool is None:
return json.dumps({"error": f"unknown tool {name}"}), "unknown_tool"
try:
args = tool.args_model.model_validate_json(raw_args)
except ValidationError as e:
return json.dumps({"error": "invalid arguments", "detail": e.errors(include_url=False)}, default=str), "invalid_args"
try:
result = tool.func(args)
except Exception as e: # report to the model, keep the details in our logs
log.exception("tool %s failed", name)
return json.dumps({"error": f"{type(e).__name__}"}), "tool_error"
return json.dumps(result, default=str), "ok"
# ---------------------------------------------------------------- structured output
@dataclass
class Parsed:
status: Literal["ok", "refused", "truncated", "filtered"]
value: Optional[BaseModel] = None
detail: Optional[str] = None
def extract(client: OpenAI, schema: Type[BaseModel], system: str, text: str,
model: str = MODEL, max_tokens: int = 1000) -> Parsed:
try:
resp = client.beta.chat.completions.parse(
model=model,
messages=[{"role": "system", "content": system}, {"role": "user", "content": text}],
response_format=schema,
temperature=0,
max_tokens=max_tokens,
)
except openai.LengthFinishReasonError:
return Parsed("truncated", detail=f"hit max_tokens={max_tokens}")
except openai.ContentFilterFinishReasonError:
return Parsed("filtered")
msg = resp.choices[0].message
if msg.refusal:
return Parsed("refused", detail=msg.refusal)
return Parsed("ok", value=msg.parsed)
# ---------------------------------------------------------------- streaming
@dataclass
class StreamResult:
text: str = ""
finish_reason: Optional[str] = None
prompt_tokens: Optional[int] = None
completion_tokens: Optional[int] = None
def stream_text(client: OpenAI, messages: list[dict], on_delta: Callable[[str], None],
model: str = MODEL) -> StreamResult:
"""Stream text to on_delta and return the full text with token usage.
With include_usage the final chunk has empty choices and carries usage.
"""
out = StreamResult()
parts: list[str] = []
stream = client.chat.completions.create(
model=model, messages=messages, stream=True, stream_options={"include_usage": True}
)
for chunk in stream:
if chunk.usage:
out.prompt_tokens = chunk.usage.prompt_tokens
out.completion_tokens = chunk.usage.completion_tokens
for choice in chunk.choices:
if choice.delta.content:
parts.append(choice.delta.content)
on_delta(choice.delta.content)
if choice.finish_reason:
out.finish_reason = choice.finish_reason
out.text = "".join(parts)
return out
# ---------------------------------------------------------------- batch
def batch_lines(items: dict[str, list[dict]], model: str = MODEL, **body: Any) -> Iterator[str]:
"""JSONL request lines for the Batch API, one per custom_id."""
for custom_id, messages in items.items():
yield json.dumps({
"custom_id": custom_id,
"method": "POST",
"url": "/v1/chat/completions",
"body": {"model": model, "messages": messages, **body},
})
def read_batch_results(lines: Iterator[str]) -> dict[str, dict]:
"""Map custom_id to {"ok": bool, "content" or "error"}. Output order is not guaranteed."""
results: dict[str, dict] = {}
for line in lines:
if not line.strip():
continue
rec = json.loads(line)
resp, err = rec.get("response"), rec.get("error")
if err or not resp or resp.get("status_code") != 200:
results[rec["custom_id"]] = {"ok": False, "error": err or (resp or {}).get("body")}
else:
results[rec["custom_id"]] = {"ok": True,
"content": resp["body"]["choices"][0]["message"]["content"]}
return results
def submit_batch(client: OpenAI, jsonl_path: str) -> str:
"""Upload a JSONL file and start a batch; returns the batch id to poll."""
with open(jsonl_path, "rb") as f:
upload = client.files.create(file=f, purpose="batch")
batch = client.batches.create(input_file_id=upload.id, endpoint="/v1/chat/completions",
completion_window="24h")
return batch.idImages as input
GPT-4o and GPT-4o mini accept images in the same Chat Completions call. A user message’s content becomes a list of parts: text parts and image_url parts, where the URL is either an https address the API can fetch or a base64 data URL for a local file. Send the file’s real MIME type in the data URL, and choose the detail level deliberately: low sends a small, fixed-cost version of the image, while high lets the model see fine print at a higher token cost. Image tokens count as input tokens, so a screenshot-heavy workload can cost more than its text suggests; check usage.prompt_tokens on a few real requests before estimating.
Screenshots and photos often contain names, addresses, faces or patient details that nobody typed. Apply the same data rules to images as to text, and crop or redact before sending where you can.
Streaming with token usage
Streaming sends text as it is generated, which matters for chat interfaces. Until May 2024 a streamed response carried no token counts, so cost tracking for streamed traffic relied on estimates. Setting stream_options={"include_usage": True} adds a final chunk with the usage for the whole request. That chunk has an empty choices list, so code that reads chunk.choices[0] on every chunk will fail on it; iterate over choices instead, as the sample does.
The Batch API
Work that can wait, such as nightly classification, evaluation runs and backfills, belongs in the Batch API. You upload a JSONL file of requests, each with a custom_id, and results come back within 24 hours at half the synchronous price, with a separate and much larger rate limit. Results are matched by custom_id, not by line order, and a failed line does not fail the batch, so check each result’s status.

Where the Assistants API fits
The Assistants API keeps conversation threads on OpenAI’s side and runs hosted tools such as file search and code interpreter. It suits applications that need those hosted tools and would otherwise build them. The trade-offs are that state and uploaded files live with OpenAI until you delete them, runs have to be polled or streamed, and hosted tools carry their own fees. For most features, Chat Completions with your own history and retrieval keeps data, cost and behaviour under your control.
Prompts and completions often contain personal data. Log request IDs, model snapshot, token counts, tool names and outcomes by default, and log content only where you have a lawful basis, a retention period and access controls for it.
The tests mock the HTTP layer and check what the client actually sends, as well as how each outcome is handled:
import json
import httpx
import pytest
from pydantic import BaseModel
import chat_api
def client_with(handler, seen, **kw):
def wrapped(request: httpx.Request) -> httpx.Response:
seen.append(request)
return handler(request)
return chat_api.make_client(api_key="sk-test", max_retries=kw.pop("max_retries", 0),
http_client=httpx.Client(transport=httpx.MockTransport(wrapped)), **kw)
def completion(content=None, finish="stop", refusal=None, tool_calls=None):
msg = {"role": "assistant", "content": content, "refusal": refusal}
if tool_calls:
msg["tool_calls"] = tool_calls
return httpx.Response(200, json={
"id": "c1", "object": "chat.completion", "created": 1725958800, "model": chat_api.MODEL,
"choices": [{"index": 0, "finish_reason": finish, "message": msg}],
"usage": {"prompt_tokens": 120, "completion_tokens": 30, "total_tokens": 150}})
def tool_call(name, args, id_="call_1"):
return [{"id": id_, "type": "function", "function": {"name": name, "arguments": args}}]
class OrderLookup(BaseModel):
order_id: str
def lookup(args: OrderLookup):
if args.order_id == "boom":
raise RuntimeError("database down")
return {"order_id": args.order_id, "status": "shipped"}
TOOLS = [chat_api.Tool("get_order_status", OrderLookup, lookup, "Look up an order's shipping status")]
def test_tool_loop_sends_strict_schema_runs_tool_and_returns_answer():
seen, replies = [], iter([
completion(None, finish="tool_calls", tool_calls=tool_call("get_order_status", '{"order_id":"A-17"}')),
completion("Order A-17 has shipped."),
])
c = client_with(lambda r: next(replies), seen)
run = chat_api.run_with_tools(c, [{"role": "user", "content": "Where is A-17?"}], TOOLS)
assert run.answer == "Order A-17 has shipped." and run.steps == 2 and run.calls == [("get_order_status", "ok")]
first = json.loads(seen[0].content)
assert first["model"] == "gpt-4o-2024-08-06" and first["parallel_tool_calls"] is False
fn = first["tools"][0]["function"]
assert fn["strict"] is True and fn["parameters"]["additionalProperties"] is False
second = json.loads(seen[1].content)["messages"]
assert second[1]["tool_calls"][0]["id"] == "call_1"
assert second[2] == {"role": "tool", "tool_call_id": "call_1",
"content": json.dumps({"order_id": "A-17", "status": "shipped"})}
@pytest.mark.parametrize("name,args,outcome", [
("get_order_status", '{"order":"A-17"}', "invalid_args"),
("delete_order", '{"order_id":"A-17"}', "unknown_tool"),
("get_order_status", '{"order_id":"boom"}', "tool_error"),
])
def test_tool_problems_go_back_to_the_model(name, args, outcome):
seen, replies = [], iter([completion(None, "tool_calls", tool_calls=tool_call(name, args)), completion("Sorry.")])
run = chat_api.run_with_tools(client_with(lambda r: next(replies), seen), [{"role": "user", "content": "x"}], TOOLS)
assert run.calls == [(name, outcome)]
sent = json.loads(json.loads(seen[1].content)["messages"][-1]["content"])
assert "error" in sent and "database down" not in json.dumps(sent)
def test_step_limit_stops_a_looping_model():
c = client_with(lambda r: completion(None, "tool_calls", tool_calls=tool_call("get_order_status", '{"order_id":"A"}')), [])
run = chat_api.run_with_tools(c, [{"role": "user", "content": "x"}], TOOLS, max_steps=3)
assert run.stopped == "step_limit" and run.answer is None and len(run.calls) == 3
class Ticket(BaseModel):
category: str
priority: int
def test_extract_ok_refused_truncated():
ok = chat_api.extract(client_with(lambda r: completion('{"category":"billing","priority":2}'), seen := []),
Ticket, "Classify the ticket.", "I was charged twice")
assert ok.status == "ok" and ok.value == Ticket(category="billing", priority=2)
body = json.loads(seen[0].content)
assert body["response_format"]["type"] == "json_schema" and body["response_format"]["json_schema"]["strict"] is True
refused = chat_api.extract(client_with(lambda r: completion(None, refusal="I can't help with that."), []), Ticket, "s", "t")
assert refused.status == "refused"
cut = chat_api.extract(client_with(lambda r: completion('{"category":"bil', finish="length"), []), Ticket, "s", "t")
assert cut.status == "truncated"
def sse(*events):
body = "".join(f"data: {json.dumps(e)}\n\n" for e in events) + "data: [DONE]\n\n"
return httpx.Response(200, headers={"content-type": "text/event-stream"}, content=body.encode())
def chunk(delta=None, finish=None, usage=None, choices=True):
return {"id": "s1", "object": "chat.completion.chunk", "created": 1725958800, "model": chat_api.MODEL,
"choices": [{"index": 0, "delta": delta or {}, "finish_reason": finish}] if choices else [],
"usage": usage}
def test_stream_collects_text_and_usage():
seen, got = [], []
c = client_with(lambda r: sse(chunk({"role": "assistant", "content": ""}), chunk({"content": "Hel"}),
chunk({"content": "lo"}), chunk(finish="stop"),
chunk(choices=False, usage={"prompt_tokens": 9, "completion_tokens": 2,
"total_tokens": 11})), seen)
out = chat_api.stream_text(c, [{"role": "user", "content": "hi"}], got.append)
assert out.text == "Hello" and got == ["Hel", "lo"] and out.finish_reason == "stop"
assert (out.prompt_tokens, out.completion_tokens) == (9, 2)
assert json.loads(seen[0].content)["stream_options"] == {"include_usage": True}
def test_client_retries_429_then_succeeds():
replies = iter([httpx.Response(429, headers={"retry-after-ms": "10"}, json={"error": {"message": "slow down"}}),
completion("ok")])
seen = []
c = client_with(lambda r: next(replies), seen, max_retries=2)
assert c.chat.completions.create(model=chat_api.MODEL, messages=[{"role": "user", "content": "x"}]).choices[0].message.content == "ok"
assert len(seen) == 2 and c.timeout == 60.0
def test_batch_lines_and_results():
lines = list(chat_api.batch_lines({"t-1": [{"role": "user", "content": "a"}]}, temperature=0))
rec = json.loads(lines[0])
assert rec["custom_id"] == "t-1" and rec["url"] == "/v1/chat/completions" and rec["body"]["temperature"] == 0
out = [
json.dumps({"id": "r2", "custom_id": "t-2", "response": {"status_code": 400, "body": {"error": {"message": "bad"}}}, "error": None}),
json.dumps({"id": "r1", "custom_id": "t-1", "response": {"status_code": 200, "body": {
"choices": [{"message": {"role": "assistant", "content": "done"}}]}}, "error": None}),
]
res = chat_api.read_batch_results(iter(out))
assert res["t-1"] == {"ok": True, "content": "done"} and res["t-2"]["ok"] is Falsechat_api.py was tested with Python 3.12, openai==1.44.1, httpx==0.27.2, pydantic==2.9.1 and pytest==8.3.2: 13 tests against a mocked endpoint cover the first call, conversation history capping, image content parts, the pinned model, strict tool schema and parallel_tool_calls, invalid arguments, unknown tools, tool exceptions, the step limit, structured output with refusal and truncation, streamed usage, retry on 429 and batch result parsing. No OpenAI API call was made. Retry and timeout defaults were read from the 1.44.1 SDK source. Prices, snapshot dates and feature limits are from OpenAI’s announcements linked below.
Key takeaways
- Pin
gpt-4o-2024-08-06orgpt-4o-mini-2024-07-18; thegpt-4oalias still points to the older, more expensive snapshot until 2 October 2024. - Set a timeout and retry count on the client instead of wrapping it in your own retry logic.
- Use strict tool schemas with
parallel_tool_callsoff, and still validate arguments, return errors to the model and cap the number of steps. - Use Structured Outputs for data extraction and handle refusals and truncation as separate outcomes.
- Request usage when streaming, and move work that can wait a day to the Batch API at half the price.
Which of these changes made the biggest difference to your OpenAI bill or error rate? Share it in the comments.
The complete sample for this article is on GitHub: nithinmohantk/blog-samples-working-repo/openai-chat-production. Clone it with git clone https://github.com/nithinmohantk/blog-samples-working-repo.git and open the openai-chat-production folder. Its README says how to run it and what has been verified.
Glossary
- Snapshot
- A dated, fixed version of a model, such as gpt-4o-2024-08-06.
- Alias
- A model name such as gpt-4o that OpenAI points at a snapshot and can move.
- Tool call
- A request from the model for your code to run a named function with given arguments.
- Strict mode
- Constraining generated tool arguments or responses to a supplied JSON schema.
- Refusal
- A message in which the model declines to produce the requested output.
- Batch API
- An asynchronous endpoint that processes JSONL files of requests within 24 hours at a discount.
- custom_id
- Your identifier on each batch request, used to match results to requests.
References & Further Reading
- → Introducing Structured Outputs in the API— gpt-4o-2024-08-06 pricing, strict mode, parallel tool call limitation
- → Reminder: GPT-4o default model will be updated on October 2nd, 2024— When the gpt-4o alias moves
- → Usage stats now available when streaming— stream_options include_usage and the final chunk
- → Batch API is now available— 50% discount, higher limits, 24-hour turnaround
- → Sample code for this article— Client, tool loop, structured outputs, streaming and batch
Discover more from C4: Container, Code, Cloud & Context
Subscribe to get the latest posts sent to your email.