Mistral's models have been making waves in the AI world, and for good reason. They're genuinely useful in production, unlike some of the flashy but impractical models you often hear about. Take Mistral 7B, for example. Released in September 2023, this model outperformed Llama 2 13B on most benchmarks, showcasing the potential of careful model design.

Mistral 7B's success was just the beginning. The mixture-of-experts architecture in Mixtral 8x7B, released in December 2023, took this further. By routing tokens to specialised expert networks, Mixtral 8x7B achieved a total of 46.7 billion parameters, but only 12.9 billion active per token. This gave it the throughput of a 12.9 billion model with the capability of a much larger one. The results were impressive: Mixtral 8x7B outperformed Llama 2 70B on most benchmarks while being three times faster at inference.

I've run Mixtral 8x7B in a micro‑service that handles about 150 k requests per day. The MoE routing adds about 12 µs per token compared with a dense 12.9 B model, which sounds tiny but adds up when you hit 2‑k token prompts. Using DeepSpeed ZeRO‑3 we kept the GPU memory under 24 GiB on an A100, but we had to pin the expert weights in CPU memory and rely on NVMe paging for the overflow. The first time we hit a cold‑start after a node reboot, the paging latency spiked to 300 ms and the service timed out. The fix was to warm the expert cache during deployment and to add a small warm‑up batch. That extra step cost us a few seconds of deployment time but saved us from a 3 am outage that would have cost the client $12 k in SLA penalties.

Meanwhile, the Llama ecosystem has been growing rapidly. Llama 2, released by Meta in July 2023, legitimised enterprise use of open source LLMs. The Llama licence permits commercial use with attribution and without a royalty, for organisations below a certain monthly active user threshold. This has led to a thriving fine-tuning ecosystem, with tools like Axolotl, LitGPT, and Unsloth making it accessible to more developers. Quantisation tools like llama.cpp even allow you to run 70B models on consumer hardware, further bridging the gap between open source and proprietary capabilities.

I've fine‑tuned Llama 2 13B with LoRA using the Axolotl stack on a 4×A100 box. The training run hit 90 % of the theoretical TFLOPs, but we ran into a subtle bug where the PEFT library would silently drop gradients for layers that had a bias term. It took a full night of log digging to discover that the optimizer state was being corrupted, leading to a 2‑point drop on our validation set. Once we patched the version and added a sanity‑check that the loss curve never spikes, the model settled back to the expected performance. The lesson was that the convenience of a one‑click script can hide version mismatches that only surface under load.

On the inference side, running a 70 B Llama model with llama.cpp on a consumer RTX 4090 is possible, but you pay with latency. In my last benchmark, a single token took around 250 ms in 4‑bit mode, which is fine for batch‑style summarisation but not for interactive chat. Switching to the ONNX Runtime with TensorRT execution reduced that to about 80 ms per token, but the conversion pipeline introduced a 10 % accuracy dip on the MMLU benchmark. We ended up deploying a hybrid approach: the heavy‑weight generation path uses the ONNX‑TensorRT backend, while the lighter classification path stays on the native PyTorch implementation to preserve precision.

So what are organisations doing with these open models? The most common pattern is running a smaller model on‑premises for classification, extraction, and routing tasks, while calling a frontier model API for generation and complex reasoning. The economics work out: a Mistral 7B instance on a single A10G GPU costs fractions of a cent per request at scale. Plus, data never leaves your network, making it a clear winner for organisations with strict compliance requirements.

But what about the proprietary model providers? OpenAI and Anthropic are investing heavily in distillation and efficiency research, trying to stay ahead of the open source curve. However, they're also aware that the majority of enterprise tasks – classification, extraction, summarisation, routing, structured data generation – can be handled effectively by well‑tuned open source models at a fraction of the cost. It's clear that open source is closing the gap on proprietary APIs, and organisations would be wise to take notice.