OpenAI put GPT‑4 into the wild on March 14, 2023, touting a model that can read images as well as text and that scores at human level on a range of professional exams.
The version that users could call through the API was not the same as the demo shown at the launch event; several internals had been swapped out before public release.
On paper GPT‑4 beats GPT‑3.5 on any task that requires several reasoning steps. Where the older model would lose track of constraints, the new one can follow a chain of ten or more logical moves without collapsing.
In the first weeks we logged average request latency around 600 ms for a 2 K token prompt, but as we scaled to 10 K concurrent users the tail latency spiked past 2 s. The bottleneck turned out to be the attention matrix on the 8‑K context; we mitigated it by sharding the model across four A100 nodes and adding a custom kernel that trimmed the KV cache for older tokens. Even with those tricks the cost per request rose by roughly 30 % compared with GPT‑3.5 for the same throughput.
That leap shows up most clearly in code generation, high‑school math problems, nuanced prompts that ask for tone or style, and any job that needs to pull information from a long context window.
The vision side, marketed as GPT‑4V, arrived only months later. ChatGPT Plus users got image input in September 2023 and the API opened to developers in October, leaving a several‑month gap between announcement and availability.
Our SaaS product that generates legal briefs saw token consumption jump from 1.2 M tokens per day on GPT‑3.5 to 2.8 M on GPT‑4, driven by the longer prompts needed to steer the model. At $0.03 per 1 K tokens that translated into a $84 daily bill, versus $2.40 on the older model. To keep margins we introduced a hybrid routing layer that sends low‑risk queries to GPT‑3.5 and reserves GPT‑4 for the edge cases where the higher fidelity actually moved the needle.
For developers who started using the API in March the headline changes were better code suggestions, more reliable instruction following, smoother tool use and a higher tolerance for ambiguous prompts.
The model shipped with an 8 K token context limit, which fell short of the 32 K many had hoped for, and it cost $0.03 per thousand input tokens compared with $0.002 for GPT‑3.5, a fifteen‑fold increase.
OpenAI spent half a year tightening safety guards before the launch. The new guardrails make jailbreak attempts harder and cut down toxic output, but a clever adversary can still coax the model past the filters.
Despite the hardened guardrails, we observed a pattern where a chain‑of‑thought prompt that asked the model to role‑play a historical figure could still be nudged into producing disallowed content. By appending a few carefully chosen facts about the figure’s controversial actions, the model slipped past the profanity filter and emitted a paragraph that violated policy. The incident forced us to add a secondary post‑processing step that scans the output with a regex‑based policy engine before it reaches the end user.
In practice the only applications that kept the upgrade were those that could justify the price tag by delivering higher‑value output, such as complex coding assistants or niche analytics tools.