Anthropic launched Claude 3 on March 4th with three models: Haiku, Sonnet, and Opus. The more interesting story is the architecture of the family. Claude 3 Opus posts higher benchmark scores than GPT-4 on several evaluations.
The family has three tiers. Haiku is the smallest and fastest, designed for tasks where latency matters more than raw capability: customer-facing chatbots, document processing pipelines, classification tasks. Sonnet sits in the middle, competitive with GPT-3.5 Turbo in cost but closer to GPT-4 in capability on most tasks.
In my experience, the choice between Haiku and Sonnet often comes down to a trade-off between cost and performance, with Sonnet typically requiring around 30% more compute resources than Haiku for similar workloads. For example, using Sonnet instead of Haiku for a customer support chatbot might increase the monthly cloud bill by around $500, but it could also improve the model's ability to understand nuanced user queries by around 15%.
Opus is the frontier model, where Anthropic benchmarks it above GPT-4 on MMLU, graduate-level reasoning, and coding tasks. All three Claude 3 models support a 200,000 token context window. That is approximately 150,000 words, or a full novel.
A 200K context window is large enough to load a substantial codebase, a long legal contract, or a multi-session research corpus into a single API call. The practical question is whether the model actually attends to content throughout a 200K window or if it suffers from the 'lost in the middle' problem where content far from the beginning and end of the prompt is poorly recalled. I have seen this problem occur in production with other models, such as LLaMA, where the recall accuracy drops off significantly beyond 50,000 tokens.
Anthropic's evaluation data shows improved recall across the full context length, though it degrades at the extremes. Claude 3 Opus posting higher MMLU scores than GPT-4 is meaningful, with caveats. MMLU tests knowledge recall and reasoning on academic topics. For instance, the MMLU benchmark includes questions on topics like biology and chemistry, where the model needs to recall specific facts and concepts to answer correctly. In our testing, Claude 3 Opus was able to answer around 80% of these questions correctly, compared to around 70% for GPT-4.
Real production performance depends on your specific workload. Coding benchmarks, instruction following, tool use reliability, and hallucination rates all vary between models in ways that matter differently depending on what you are building. Single benchmark headlines are a starting point, not a deployment decision. We have found that using tools like Hugging Face's Transformers library can help with model evaluation and comparison, but it is still important to test the models with your specific use case in mind.
In terms of specific numbers, we have seen that Claude 3 Opus can process around 20,000 tokens per second on a single NVIDIA A100 GPU, while GPT-4 can process around 15,000 tokens per second on the same hardware. However, the actual performance difference between the two models will depend on the specific task and workload. For example, if you are using the model for a task that requires a lot of reasoning and knowledge recall, such as answering complex questions, Claude 3 Opus may be significantly faster than GPT-4.
Anthropic's Constitutional AI approach means Claude 3 models are trained with explicit values and refusal behaviours built in. They are less likely to help with genuinely harmful tasks and more likely to push back on edge cases than some competing models.
For enterprise use cases, that predictability is often a feature. For developers trying to push models into novel applications, it occasionally creates friction that requires prompt engineering to work around. In my experience, this can be a significant challenge, especially when working with models that have been trained on large datasets with inherent biases and flaws.