I encountered a microservices system that needed a testing strategy overhaul. The testing pyramid principle still applies but with different test types at each level compared to monolithic apps.
Consumer-driven contract testing, using the Pact framework, allows services to be tested independently. It verifies that interactions between a consumer and a provider match agreed contracts. The consumer writes tests defining what it expects from the provider; these are converted to a Pact contract file. The provider then verifies that it satisfies all consumer contracts.
This approach enables independent deployment. If the provider satisfies all contracts, consumers can safely upgrade without worrying about integration issues.
For example, in our system, we have around 20 services, each having 5 to 10 contracts with other services. Using Pact, we are able to test these contracts in isolation, which has reduced our testing time by around 30% and has also improved the quality of our tests. We have also integrated Pact with our CI/CD pipeline, so that any changes to the contracts are automatically tested and verified.
For integration tests, I used Testcontainers, which launches real Docker containers for databases, message queues, and caches. This provides realistic test environments without mocking. A test that needs PostgreSQL gets a real PostgreSQL container spun up before the test and torn down after. The Testcontainers library is available for .NET, Java, Go, and Node.js.
The cost of using Testcontainers is test time due to container startup. However, the benefit is tests that catch integration bugs that mocks would not. This approach has been a game-changer for our team's testing efficiency. We have seen a reduction of around 25% in the number of integration bugs that make it to production, and this number is still decreasing as we continue to improve our tests.
In terms of performance, using Testcontainers has added around 10-15 seconds to our test suite, but this is a trade-off we are willing to make for the increased reliability of our tests. We have also implemented a number of optimizations, such as reusing containers across tests and using smaller container images, to reduce the overhead of using Testcontainers.
End-to-end tests that spin up the full microservices stack are slow, flaky, and expensive to maintain. Instead, I focus on component tests, which test a single service with all its external dependencies replaced by test doubles or Testcontainers instances. This provides similar coverage at a lower cost and higher reliability.
Our testing portfolio now consists of many unit tests, many component tests, few contract tests, and minimal end-to-end tests covering only the most critical user journeys. This approach has significantly improved our testing efficiency and reliability. We have seen a reduction of around 40% in the time it takes to run our full test suite, and this has enabled us to deploy more frequently and with greater confidence.
Test data management in a microservices system is complex. Each service owns its data, and data seeding for integration tests must respect the ownership boundaries. We use test-specific seed data loaded via service APIs, event replay for eventual consistency tests, and database cleanup after each test using transaction rollback or truncation scripts.
We have implemented a number of strategies to manage test data, including using a separate database for testing and implementing a data seeding framework that allows us to easily seed data for our tests. We have also implemented a number of checks to ensure that our tests are properly cleaning up after themselves, to prevent data leakage between tests.
Shared test data across tests creates order dependencies and flakiness. To avoid this, we ensure that each test has its own isolated data setup and teardown. This approach has been successful in reducing the number of flaky tests and has improved the overall reliability of our test suite. We have seen a reduction of around 20% in the number of flaky tests, and this number is still decreasing as we continue to improve our tests.
Overall, our approach to testing has been shaped by our experience with microservices and the challenges that come with it. We have learned to prioritize reliability and efficiency in our tests, and to use tools and strategies that support these goals. By doing so, we have been able to improve the quality of our software and reduce the time it takes to deploy new features.