When reviewing cloud architectures, I consistently see the same design mistakes that can cause problems at scale. These patterns are often ignored or downplayed by architects until it's too late, leading to painful fixes and expensive rework.
Synchronous coupling at scale creates a perfect storm of latency and brittleness. A 200ms call to A, which then calls B, C, and D, results in a minimum 600ms response time. If any of these downstream calls fail, the entire chain fails. The question is, does this operation actually need a synchronous response, or can it be asynchronous with a polling or notification mechanism?
For example, I've seen systems where a simple login operation took over a second due to a chain of synchronous calls. By converting these calls to asynchronous requests with a message queue, such as Apache Kafka or Amazon SQS, we reduced the response time to under 100ms. This not only improved user experience but also reduced the load on the downstream services.
Designs that ignore failure modes are incomplete. For every external dependency, you need to consider: what happens when it's slow? What happens when it fails? What's the blast radius of a failure? Circuit breakers, timeouts, retries with backoff, bulkheads, and fallbacks are not optional; they're essential design elements. Systems without explicit failure handling fail implicitly. In a system I worked on, we used Netflix's Hystrix to implement circuit breakers and retries, which reduced our error rate by over 30% during a period of high traffic.
Trade-offs are also important to consider, such as the cost of implementing retries versus the cost of handling failures. In one case, we found that implementing retries with a 50ms backoff period reduced our failure rate by 20%, but increased our latency by 10%. We had to weigh the benefits of increased reliability against the cost of increased latency, and adjust our design accordingly. Tools like AWS CloudWatch and New Relic can help monitor and analyze the performance of your system, making it easier to make informed decisions about these trade-offs.
Services that share a domain object without explicit contract management create invisible coupling. When the upstream service changes the format of a shared object, all consumers break simultaneously. The mitigation is to use explicit contracts (Avro schemas, Protobuf definitions, OpenAPI specs), consumer-driven contract tests (Pact), and semantic versioning for published contracts. Sharing a common DTO library across service boundaries is the anti-pattern that makes this worse. For instance, using a tool like Swagger to define and manage API contracts can help prevent these issues and ensure that all services are working with the same understanding of the data.
Architecture reviews that focus on the happy path miss the operational reality. The questions that surface operational complexity are: how does this get deployed? How does a new engineer debug this in production at 3am? How does this scale under 10x load? What's the blast radius of a misconfiguration? Designs that can't answer these questions are not production-ready, regardless of their functional correctness.