Contract testing vs functional testing

Understanding the Difference Between Different Types of Tests and Establishing an Appropriate Testing Strategy

In today's software development landscape, the adoption of distributed architectures is prevalent, where each service handles specific functionality and communicates with others through network message exchanges, typically via REST requests. In this context, the need arises to ensure compatibility between services, ensuring they interact as expected and that changes in one do not negatively affect others. This is where Contract Testing, particularly the Pact tool, plays a crucial role.

But what sets contract testing apart from integration and functional testing? That's what we'll explore in this article.

Why It Matters

Understanding the difference between different types of tests allows you to establish an appropriate testing strategy that optimizes the software development process.

Being able to differentiate the objective and, above all, the scope of each type of test will make your testing suite much more reliable and faster. This is because it will be easier to detect errors in early stages, be more precise in identifying the source of the problem, avoid blockages between teams, and improve communication among them.

Digging Deeper

When it comes to testing APIs, let's now examine the fundamental differences between the types of tests:

Now that we've seen the differences, and it's clear that contract testing is based on compatibility between services, let's see in which scenarios it's interesting to apply it and when it's not.

✅ Validating that the message body is appropriate

This is one of the main advantages of using contract testing because, through the specification of concrete examples ("expectations") and the mocking of network requests, we can automatically verify that both producer and consumer exchange information as agreed upon.

This is not simply checking the data schema; the main differences are:

• Only the fields that actually affect the consumer are taken into account, potentially ignoring the rest of the response data.

• It allows the use of "wildcards" or regular expressions to validate the expected data type (email, date, number, string, etc.). This avoids fragility in checks since it is not tied to the exact value.

• It's not necessary to have the producer part deployed; validations are done based on the defined contract and through mocked applications that generate real network requests.

✅ Validating that the client will be able to process the agreed response

To avoid getting into functional testing territory, we must understand that the scope of this check is solely to validate that the consumer is able to process the provider's response properly. We're not trying to check the side effects that occur afterward.

For example, if our client application requests data of a student to calculate their average grade, the test on the client side would be aimed at ensuring that we indeed receive the student's data and that we're capable of processing it properly. In practice, in most cases, it will consist of checking that we have a class that maps this response correctly.

Here's the resulting code:

//Expectation

@Pact(consumer = "student-consumer", provider = "student-provider")

public V4Pact getStudentWithId1(PactDslWithProvider builder) {

    return builder.given("student 1 exists")

            .uponReceiving("get an existing student with ID 1")

            .path("/students/1")

            .method("GET")

            .willRespondWith()

            .status(200)

            .headers(Map.of("Content-Type", "application/json"))

            .body(newJsonBody(object -> {

                object.stringType("id", "1");

                object.stringType("name", "Fake name");

                object.stringType("email", "some.email@sngular.com");

                object.numberType("studentNumber", 23);

            }).build())

            .toPact().asV4Pact().get();

}

//Validation

@Test

@PactTestFor(pactMethod =

 "getStudentWithId1")

void getStudentWhenStudentExist(MockServer mockServer) {

    Student expected = Student.builder()

            .id("1")

            .name("Fake name")

            .email("some.email@sngular.com")

            .studentNumber(23).build();


    RestTemplate restTemplate = new RestTemplateBuilder().rootUri(mockServer.getUrl()).build();

    //Check that our service can perform and process the response properly

    Student student = new StudentService(restTemplate).getStudent("1");


    assertEquals(expected, student);

}

✅ Validating that the endpoint routes are correct

Following the previous code example, we can see how we're specifying the route and specific parameters with which the request should be made ("students/1"). This will enforce both parties to comply with what's agreed upon regarding the definition of endpoints.

✅ Checking that the response codes are appropriate

Also, in the code example, we can see how the expected response code is specified (200). Similar to the previous case, we're ensuring that the provider will send the agreed-upon response code.

✅ Validating that request headers and parameters are appropriate

In the code example, we can also see how the expected request headers are specified. Therefore, if either party fails to comply, the test will fail.

❌ Validating different request scenarios

Suppose we want to validate that when making "POST" requests, the provider returns "400 Bad Request" under certain cases.

In this case, as mentioned earlier, through contract testing, we should check that there is indeed the possibility for the provider to return the 400 code and that the client can process such a response type.

However, it wouldn't be appropriate to use contract testing to check all possible scenarios where the service might return "Bad Request." Doing so would make our tests too fragile because they would largely depend on the specific examples we're using to generate them. We must consider that validation or business rules may change, and this shouldn't affect the communication between services.

For example, this wouldn't be a good contract definition because if the maximum allowed length is changed, the tests would fail even if both systems are compatible:

When "creating a user with a blank username"

  POST /users { "username": "", email: "...", ... }

Then

  Expected Response is 400 Bad Request

  Expected Response body is { "error": "username cannot be blank" }

When "creating a user with a username with 21 characters"

  POST /users { "username": "thisisalooongusername", email: "...", ... }

Then

  Expected Response is 400 Bad Request

  Expected Response body is { "error": "username cannot be more than 20 characters" }

When "creating a user with a username containing numbers"  

POST /users { "username": "us3rn4me", email: "...", ... }

Then  Expected Response is 400 Bad Request  

Expected Response body is { "error": "username can only contain letters" }

In this case, a single example where a generic "Bad Request" is specified would be sufficient. This way, we should also be more generic when specifying the expected return message, avoiding specifying the exact text expected.

Finally, it could have a case like this:

When "creating a user with an invalid username"

  POST /users { "username": "bad_username_that_breaks_some_rule_that_you_are_fairly_confident_will_not_change", ... }

Then

  Response is 400 Bad Request

  Response body is { "error": "<any string>" }

❌ Validating side effects

By side effects of a test, we can understand system changes such as database changes, calls to third-party services, file modifications, notification sending, etc.

Although technically, these types of situations can be verified using Pact, it's not the purpose of contract tests. The main disadvantages are that it makes these tests extremely fragile and dependent on other systems.

Therefore, while possible, it would be counterproductive since we would lose the main advantages that Pact offers, such as execution speed, test independence, and avoiding blockages.

Hence, these types of validations should be postponed to later stages: integration or end-to-end.

❌ Validating workflows (sequence of steps)

Pact allows specifying an initial system state before the test execution. This "setup" is essential in any testing framework. However, we shouldn't confuse this with executing a series of actions sequentially and checking the final state.

That is, preparing the system to have a certain user on which we request data is not the same as modifying a field on the same user and then checking the consequences of that change.

For example, through contract testing, it wouldn't be appropriate to propose a test like this:

Given a VIP user with ID 1 

When

  PUT /users/1 { VIP: "false" ... }

Then

  Expected Response is 200 OK

And

  GET /discounts?user=1

Then

  Expected Response is 200 OK

  Expected Response body {"No discounts for user 1"}

In Summary

In a world where microservices architecture and distributed applications are the norm, the importance of a solid and efficient testing strategy cannot be underestimated. In this article, we've explored the key distinctions between Contract Testing, Integration Testing, and End-to-End Testing, underscoring their relevance and applicability in different software development scenarios.

Contract Testing, facilitated by tools like Pact, focuses on ensuring that services interact correctly according to defined contracts, offering an efficient way to detect communication problems early in the development cycle. Integration Testing verifies cooperation between modules or services, identifying issues that only arise when these components are deployed and interact with each other. Finally, End-to-End Testing evaluates the entire system in an environment that simulates end user interaction, ensuring that all parts of the system work together smoothly.

By understanding these differences and applying the appropriate type of test at the right time, teams can significantly improve the efficiency of their development workflow, detect errors in early stages, and eliminate blockages, all while ensuring that each component works as intended.

Incorporating these practices not only optimizes the development process but also enhances collaboration between teams, ensuring that changes in one service do not affect others. Ultimately, choosing the right testing strategy is crucial for high-quality software development.