Each week we're asked 100's of questions on our Slack channel, from "my test is failing, please help" to "I need help convincing my team". Exhibit A:

Convince me!

We've already created a list of reasons to convince you that contract testing is a good idea, but today I'd like to take the opportunity to answer a question that comes up often - what is contract testing, and why should you consider adding it to your microservices testing approach?

What is contract testing?

https://medium.com/better-practices/consumer-driven-contract-testing-using-postman-f3580dba5370

Contract testing is a methodology for ensuring that two separate systems (such as two microservices) are compatible with one other. It captures the interactions that are exchanged between each service, storing them in a contract, which can then be used to verify that both parties adhere to it.

What sets this form of testing apart from other approaches that aim to achieve the same thing, is that each system is able to be tested independently from the other and that the contract is generated by the code itself, meaning the contract is always kept up to date with reality.

There are many other important properties that flow on from this, which we'll discuss further below.

What is consumer-driven contract testing?

The "consumer-driven" prefix simply states an additional philosophical position that advocates for better internal microservices design by putting the consumers of such APIs at the heart of the design process. Provider-driven APIs tend to be biased towards the data that is being exposed and the system that is exposing it.

Pact and Spring Cloud Contracts are examples of frameworks that default to this kind of implementation, but this isn't a requirement to get the key benefits of contract-testing and both can be used to achieve the aims of contract testing.

What are the benefits of contract testing?

It's helpful first to consider where contract testing sits within the context of a broader automation testing approach. When coming up with a test automation strategy, a good rule of thumb in how you should expend your effort is the approach advocated in Mike Cohn's "Test Pyramid":

Stick to the pyramid shape to come up with a healthy, fast and maintainable test suite: Write lots of small and fast unit tests. Write some more coarse-grained tests and very few high-level tests that test your application from end to end [3]
From the Practical Test Pyramid

Contract tests fit in the "Service Tests" layer, as they execute quickly and don't need to integrate to external systems to run. Their job is to give you confidence that the systems you integrate with are compatible with your code before you release.

NOTE: The "UI tests" at the top of the pyramid here are often also referred to interchangeably with "integrated end-to-end (e2e) tests".

The problem with e2e integrated tests

Whilst tests at the top of the pyramid more closely represent what the customer would experience, they have several painful drawbacks. They:

  • are slow; because they traverse multiple systems and generally must be run serially, each test may take several seconds to several minutes to complete, especially if pre-requisite setup (such as data preparation) must be performed.
  • are hard to maintain; end-to-end tests require all systems to be in the correct state before they are run, including the correct version and data.
  • can be unreliable or flakey: because of the complexity in orchestrating a test environment, they can often fail causing false-positives, becoming distractions to the team. In many cases, they fail due to a configuration issue unrelated to any code change.
  • are hard to fix: when an end-to-end test fails, debugging the issue is usually difficult, because of the distributed and remote nature of the problem.
  • scale badly; as more teams' code gets tested, things get more entangled, test suites run exponentially slower and releases get clogged in automation pipelines.
  • find bugs too late in the process: because of the complexity of running such testing suites, in many situations these tests are only run on CI after code has been committed - in many cases, by a separate testing team days afterwards. This delay in feedback is extremely costly to modern, agile delivery teams.

Because of these properties, it is advised to keep them to a minimum number e.g. to ensure key business transactions or features are covered.

The value of contract tests

Contract tests generally have the opposite properties to e2e integrated tests:

  • They run fast, because they don't need to talk to multiple systems.
  • They are are easier to maintain: you don't need to understand the entire ecosystem to write your tests.
  • They are easy to debug and fix, because the problem is only ever in the component your testing - so you generally get a line number or a specific API endpoint that is failing.
  • They are repeatable:
  • They scale: because each component can be independently tested, build pipelines don't increase linearly / exponentially in time
  • They uncover bugs locally, on developer machines: contract tests can and should run on developer machines prior to pushing code.

From a business point of view, it is well known that the later in a project lifecycle that a bug is found, the more costlier it is to fix.

From Beth's excellent talk on the topic

By running most of your automation tests as fast unit or local integration tests, you can keep build speeds to a minimum, and prevent queues building up in teams.

It turns out that contract tests also have other secondary, yet positive side-effects:

Other benefits

  • The ability to develop the consumer (eg. a React Web App) before the API
  • The ability to drive out the requirements for your provider first, meaning you implement exactly and only what you need in the provider.
  • You get a set of well documented use cases ("Given ... a request for ... will return ...") that show exactly how a provider is being used.
  • The ability to see exactly which fields each consumer is interested in, allowing unused fields to be removed, and new fields to be added in the provider API without impacting a consumer.
  • The ability to immediately see which consumers will be broken if a change is made to the provider API.

I hope this brief introduction to the topic showed you how you can use contract testing to replace many (or in some cases, all) of your end-to-end integrated tests to speed up your CI pipeline and increase your teams' velocity.

Contract testing is not a silver bullet and no two teams are the same: so think about your test strategy, how it applies to you and what you need to get from it.

Next

Stay tuned for the next article in this Q&A series on "Getting started with contract testing", where we help you answer key questions on navigating the challenges of make contract testing a success in your organisation.

Resources & Further Reading