Ramona Schwering on web.dev

Four common types of code coverage

2023-09-06T00:00:00Z

Have you heard the phrase "code coverage"? In this post, we will explore what code coverage in tests is and four common ways to measure it.

What is code coverage? #

Code coverage is a metric that measures the percentage of source code your tests execute. It helps you identify areas that may lack proper testing.

Often, recording these metrics looks like this:

File	% Statements	% Branch	% Functions	% Lines	Uncovered lines
file.js	90%	100%	90%	80%	89,256
coffee.js	55.55%	80%	50%	62.5%	10-11, 18

As you add new features and tests, increasing code coverage percentages can give you more confidence that your application has been thoroughly tested. However, there is more to discover.

Four common types of code coverage #

There are four common ways to collect and calculate code coverage: function, line, branch, and statement coverage.

To see how each type of code coverage calculates its percentage, consider the following code example for calculating coffee ingredients:

/* coffee.js */

export function calcCoffeeIngredient(coffeeName, cup = 1) {
  let espresso, water;

  if (coffeeName === 'espresso') {
    espresso = 30 * cup;
    return { espresso };
  }

  if (coffeeName === 'americano') {
    espresso = 30 * cup; water = 70 * cup;
    return { espresso, water };
  }

  return {};
}

export function isValidCoffee(name) {
  return ['espresso', 'americano', 'mocha'].includes(name);
}

The tests that verify the calcCoffeeIngredient function are:

/* coffee.test.js */

import { describe, expect, assert, it } from 'vitest';
import { calcCoffeeIngredient } from '../src/coffee-incomplete';

describe('Coffee', () => {
  it('should have espresso', () => {
    const result = calcCoffeeIngredient('espresso', 2);
    expect(result).to.deep.equal({ espresso: 60 });
  });

  it('should have nothing', () => {
    const result = calcCoffeeIngredient('unknown');
    expect(result).to.deep.equal({});
  });
});

You can run the code and tests on this live demo or check out the repository.

Function coverage #

Code coverage: 50%

/* coffee.js */

export function calcCoffeeIngredient(coffeeName, cup = 1) {
  // ...
}

function isValidCoffee(name) {
  // ...
}

Function coverage is a straightforward metric. It captures the percentage of functions in your code that your tests call.

In the code example, there are two functions: calcCoffeeIngredient and isValidCoffee. The tests only call the calcCoffeeIngredient function, so the function coverage is 50%.

Line coverage #

Code coverage: 62.5%

/* coffee.js */

export function calcCoffeeIngredient(coffeeName, cup = 1) {
  let espresso, water;

  if (coffeeName === 'espresso') {
    espresso = 30 * cup;
    return { espresso };
  }

  if (coffeeName === 'americano') {
    espresso = 30 * cup; water = 70 * cup;
    return { espresso, water };
  }

  return {};
}

export function isValidCoffee(name) {
  return ['espresso', 'americano', 'mocha'].includes(name);
}

Line coverage measures the percentage of executable code lines that your test suite executed. If a line of code remains unexecuted, it means that some part of the code hasn't been tested.

The code example has eight lines of executable code (highlighted in red and green) but the tests don’t execute the americano condition (two lines) and the isValidCoffee function (one line). This results in a line coverage of 62.5%.

Note that line coverage doesn’t take into account declaration statements, such as function isValidCoffee(name) and let espresso, water;, because they are not executable.

Branch coverage #

Code coverage: 80%

/* coffee.js */

export function calcCoffeeIngredient(coffeeName, cup = 1) {
  // ...

  if (coffeeName === 'espresso') {
    // ...
    return { espresso };
  }

  if (coffeeName === 'americano') {
    // ...
    return { espresso, water };
  }

  return {};
}
…

Branch coverage measures the percentage of executed branches or decision points in the code, such as if statements or loops. It determines whether tests examine both the true and false branches of conditional statements.

There are five branches in the code example:

Calling calcCoffeeIngredient with just coffeeName
Calling calcCoffeeIngredient with coffeeName and cup
Coffee is Espresso
Coffee is Americano
Other coffee

The tests cover all branches except the Coffee is Americano condition. So branch coverage is 80%.

Statement coverage #

Code coverage: 55.55%

/* coffee.js */

export function calcCoffeeIngredient(coffeeName, cup = 1) {
  let espresso, water;

  if (coffeeName === 'espresso') {
    espresso = 30 * cup;
    return { espresso };
  }

  if (coffeeName === 'americano') {
    espresso = 30 * cup; water = 70 * cup;
    return { espresso, water };
  }

  return {};
}

export function isValidCoffee(name) {
  return ['espresso', 'americano', 'mocha'].includes(name);
}

Statement coverage measures the percentage of statements in your code that your tests execute. At first glance, you might wonder, “isn’t this the same as line coverage?” Indeed, statement coverage is similar to line coverage but takes into account single lines of code that contain multiple statements.

In the code example, there are eight lines of executable code, but there are nine statements. Can you spot the line containing two statements?

Check your answer

It's the following line: `espresso = 30 * cup; water = 70 * cup;`

The tests cover only five of the nine statements, therefore the statement coverage is 55.55%.

If you always write one statement per line, your line coverage will be similar to your statement coverage.

What type of code coverage should you choose? #

Most code coverage tools include these four types of common code coverage. Choosing which code coverage metric to prioritize depends on specific project requirements, development practices, and testing goals.

In general, statement coverage is a good starting point because it is a simple and easy-to-understand metric. Unlike statement coverage, branch coverage and function coverage measure whether tests call a condition (branch) or a function. Therefore, they are a natural progression after statement coverage.

Once you have achieved high statement coverage, you can then move on to branch coverage and function coverage.

Is test coverage the same as code coverage? #

No. Test coverage and code coverage are often confused but they are different:

Test coverage: A qualitative metric that measures how well the test suite covers the features of the software. It helps determine the level of risk involved.
Code coverage: A quantitative metric that measures the proportion of code executed during testing. It is about how much code the tests cover.

Here is a simplified analogy: imagine a web application as a house.

Test coverage measures how well the tests cover the rooms in the house.
Code coverage measures how much of the house the tests have walked through.

100% code coverage doesn’t mean no bugs #

While it is certainly desirable to achieve high code coverage in testing, 100% code coverage doesn’t guarantee the absence of bugs or flaws in your code.

A meaningless way to achieve 100% code coverage #

Consider the following test:

/* coffee.test.js */

// ...
describe('Warning: Do not do this', () => {
  it('is meaningless', () => { 
    calcCoffeeIngredient('espresso', 2);
    calcCoffeeIngredient('americano');
    calcCoffeeIngredient('unknown');
    isValidCoffee('mocha');
    expect(true).toBe(true); // not meaningful assertion
  });
});

This test achieves 100% function, line, branch, and statement coverage, but it doesn’t make sense because it doesn’t actually test the code. The expect(true).toBe(true) assertion will always pass regardless of whether the code works correctly.

A bad metric is worse than no metric #

A bad metric can give you a false sense of security, which is worse than having no metric at all. For example, if you have a test suite that achieves 100% code coverage but the tests are all meaningless, then you may get a false sense of security that your code is well tested. If you accidentally delete or break a part of the application code, the tests will still pass, even though the application no longer works correctly.

To avoid this scenario:

Test review. Write and review tests to make sure they are meaningful and test the code in a variety of different scenarios.
Use code coverage as a guideline, not as the only measure of test effectiveness or code quality.

Using code coverage in different types of testing #

Let’s take a closer look at how you can use code coverage with the three common types of test:

Unit tests. They are the best test type for gathering code coverage because they are designed to cover multiple small scenarios and testing paths.
Integration tests. They can help collect code coverage for integration tests, but use them with caution. In this case, you calculate the coverage of a larger portion of the source code, and it can be difficult to determine which tests actually cover which parts of the code. Nonetheless, calculating code coverage of integration tests may be useful for legacy systems that don’t have well-isolated units.
End-to-end (E2E) tests. Measuring code coverage for E2E tests is difficult and challenging due to the intricate nature of these tests. Instead of using code coverage, requirement coverage might be the better way to go. This is because the focus of E2E tests is to cover the requirements of your test, not to focus on the source code.

Conclusion #

Code coverage can be a useful metric for measuring the effectiveness of your tests. It can help you to improve the quality of your application by ensuring that the crucial logic in your code is well tested.

However, remember that code coverage is just one metric. Make sure to also consider other factors, such as the quality of your tests and your application requirements.

Aiming for 100% code coverage is not the goal. Instead, you should use code coverage along with a well-rounded testing plan that incorporates a variety of testing methods, including unit tests, integration tests, end-to-end tests, and manual tests.

See the full code example and tests with good code coverage. You can also run the code and tests with this live demo.

/* coffee.js - a complete example */

export function calcCoffeeIngredient(coffeeName, cup = 1) {
  if (!isValidCoffee(coffeeName)) return {};

  let espresso, water;

  if (coffeeName === 'espresso') {
    espresso = 30 * cup;
    return { espresso };
  }

  if (coffeeName === 'americano') {
    espresso = 30 * cup; water = 70 * cup;
    return { espresso, water };
  }

  throw new Error (`${coffeeName} not found`);
}

function isValidCoffee(name) {
  return ['espresso', 'americano', 'mocha'].includes(name);
}

/* coffee.test.js - a complete test suite */

import { describe, expect, it } from 'vitest';
import { calcCoffeeIngredient } from '../src/coffee-complete';

describe('Coffee', () => {
  it('should have espresso', () => {
    const result = calcCoffeeIngredient('espresso', 2);
    expect(result).to.deep.equal({ espresso: 60 });
  });

  it('should have americano', () => {
    const result = calcCoffeeIngredient('americano');
    expect(result.espresso).to.equal(30);
    expect(result.water).to.equal(70);
  });

  it('should throw error', () => {
    const func = () => calcCoffeeIngredient('mocha');
    expect(func).toThrowError(new Error('mocha not found'));
  });

  it('should have nothing', () => {
    const result = calcCoffeeIngredient('unknown')
    expect(result).to.deep.equal({});
  });
});

To test or not to test, a technical perspective

2023-08-23T00:00:00Z

The previous article covered the basics of test cases and what they should contain. This article delves deeper into the creation of test cases from a technical perspective, detailing what should be included in each test and what to avoid. Essentially, you'll learn the answer to the age-old questions of "What to test" or "What not to test".

General guidelines and patterns #

It's worth noting that specific patterns and points are crucial, regardless of whether you're conducting unit, integration, or end-to-end tests. These principles can and should be applied to both types of testing, so they are a good place to start.

Keep it simple #

When it comes to writing tests, one of the most important things to remember is to keep it simple. It's important to consider the brain's capacity. The main production code takes up significant space, leaving little room for additional complexity. This is especially true for testing.

If there's less headspace available, you may become more relaxed in your testing efforts. That's why it's crucial to prioritize simplicity in testing. In fact, Yoni Goldberg's JavaScript testing best practices emphasize the importance of the Golden Rule—your test should feel like an assistant and not like a complex mathematical formula. In other words, you should be able to understand your test's intent at first glance.

You should aim for simplicity in all types of tests, regardless of their complexity. In fact, the more complex a test is, the more critical it is to simplify it. One way to achieve this is through a flat test design, where tests are kept as simple as possible, and to only test what is necessary. This means each test should contain only one test case, and the test case should be focused on testing a single, specific functionality or feature.

Think about it from this perspective: it should be easy to identify what went wrong when reading a failing test. This is why keeping tests simple and easy to understand is important. Doing so lets you quickly identify and fix issues when they arise.

Test what's worth it #

The flat test design also encourages focus and helps ensure your tests are meaningful. Remember, you don't want to create tests just for the sake of coverage—they should always have a purpose.

Don't test implementation details #

One common problem in testing is that tests are often designed to test implementation details, such as using selectors in components or end-to-end tests. Implementation details refer to things that users of your code will not typically use, see, or even know about. This can lead to two major problems in tests: false negatives and false positives.

False negatives occur when a test fails, even though the tested code is correct. This can happen when the implementation details change due to a refactoring of the application code. On the other hand, false positives occur when a test passes, even though the code being tested is incorrect.

One solution to this problem is to consider the different types of users you have. End users and developers can differ in their approach, and they may interact with the code differently. When planning tests, it is essential to consider what users will see or interact with, and make the tests dependent on those things instead of the implementation details.

For example, choosing selectors that are less prone to change can make tests more reliable: data-attributes instead of CSS selectors. For more details, refer to Kent C. Dodds' article on this topic, or stay tuned—an article on this topic is coming later.

Mocking: Don't lose control #

Mocking is a broad concept used in unit testing and sometimes in integration testing. It involves creating fake data or components to simulate dependencies that have complete control over the application. This allows for isolated testing.

Using mocks in your tests can improve predictability, separation of concerns, and performance. And, if you need to conduct a test that requires human involvement (such as passport verification), you'll have to conceal it using a mock. For all these reasons, mocks are a valuable tool to consider.

At the same time, mocking may affect the accuracy of the test because they are mocks, not the real user experiences. So you need to be mindful when using mocks and stubs.

Should you mock in end-to-end tests? #

In general, no. However, mocking can be a lifesaver sometimes—so let's not rule it out completely.

Imagine this scenario: you're writing a test for a feature involving a third-party payment provider service. You're in a sandbox environment that they have provided, meaning no real transactions are taking place. Unfortunately, the sandbox is malfunctioning, thereby causing your tests to fail. The fix needs to be done by the payment provider. All you can do is wait for the issue to be resolved by the provider.

In this case, it might be more beneficial to lessen the dependency on services you cannot control. It's still advisable to use mocking carefully in integration or end-to-end tests as it decreases the confidence level of your tests.

Test specifics: Dos and don'ts #

So, all in all, what does a test contain? And are there differences between the testing types? Let's take a closer look at some specific aspects tailored to the main testing types.

What belongs to a good unit test? #

An ideal and effective unit test should:

Concentrate on specific aspects.
Operate independently.
Encompass small-scale scenarios.
Use descriptive names.
Follow the AAA pattern if applicable.
Guarantee comprehensive test coverage.

Do ✅	Don't ❌
Keep the tests as small as possible. Test one thing per test case.	Write tests over large units.
Always keep tests isolated and mock the things you need which are outside your unit.	Include other components or services.
Keep tests independent.	Rely on previous tests or share test data.
Cover different scenarios and paths.	Limit yourself to the happy path or negative tests at maximum.
Use descriptive test titles, so you can immediately see what your test is about.	Test by function name only, not being descriptive enough as a result: `testBuildFoo()` or `testGetId()`.
Aim for good code coverage or a broader range of test cases, especially at this stage.	Test from every class down to database (I/O) level.

What belongs to a good integration test? #

An ideal integration test shares some criteria with unit tests, too. However, there are a couple of additional points that you need to consider. A great integration test should:

Simulate interactions between components.
Cover real-world scenarios, and use mocks or stubs.
Consider performance.

Do ✅	Don't ❌
Test the integration points: verify that each unit works together gracefully when integrated with each other.	Test each unit in isolation—that's what unit tests are for.
Test real-world scenarios: use test data derived from real-world data.	Use repetitive auto-generated test data or other data which doesn't reflect real-world use cases.
Use mocks and stubs for external dependencies to maintain control of your complete test.	Create dependencies on third-party services, for example, network requests to outside services.
Use a clean-up routine before and after each test.	Forget to use clean-up measures inside your tests, otherwise this can lead to test failures or false positives, due to lack of proper test isolation.

What belongs to a good end-to-end test? #

A comprehensive end-to-end test should:

Replicate user interactions.
Encompass vital scenarios.
Span multiple layers.
Manage asynchronous operations.
Verify results.
Account for performance.

Do ✅	Don't ❌
Use API-driven shortcuts. Learn more.	Use UI interactions for every step, including the `beforeEach` hook.
Use a clean-up routine before each test. Take even more care of test isolation than you do in unit and integration tests because there's a higher risk of side effects here.	Forget to clean up after each test. If you don't clean up the leftover state, data or side effects, they will affect other tests executed later.
Regard end-to-end tests as system tests. This means you need to test the whole application stack.	Test each unit in isolation—that's what unit tests are for.
Use minimal or no mocking inside the test. Consider carefully if you want to mock external dependencies.	Rely heavily on mocks.
Consider performance and workload by, for example, not over-testing large scenarios in the same test.	Cover large workflows without using shortcuts.

Defining test cases and priorities

2023-08-09T00:00:00Z

In the previous post, you learned about testing strategies, the number of tests needed to test an application, and how to find the best fit to gain the most confidence in the results while bearing in mind your resources. However, this only gives us an idea of how much to test. You still need to determine exactly what to test. The following three criteria can be helpful in understanding what to expect in a test and to see what testing type and level of detail might fit best:

Take care of your happy path. This is the most generic or primary user story of your application, where your user will notice an error very quickly.
Decide carefully on the level of detail. Get into more detail if your use case is vulnerable or where an error would cause high damage.
Prioritize lower-level tests, such as unit and integration tests, over higher-level end-to-end tests whenever possible.

The rest of this article explores these criteria, and how they apply as you define test cases.

What is a test case? #

In software development, a test case is a sequence of actions or circumstances that are devised to confirm the effectiveness of a software program or application. A test case aims to ensure that the software operates as planned and that all its features and functions perform correctly. Software testers or developers typically create these test cases to guarantee that the software meets the specified requirements and specifications.

When a test case is run, the software performs a series of checks to ensure it produces the desired results. While doing that, a test fulfills the following tasks:

Verification. The process of thoroughly checking software to ensure it functions without errors. Determining whether the created product meets all the necessary non-functional requirements and successfully achieves its intended purpose is crucial. The question it answers is: “Are we building the product right?”
Validation. The process of ensuring that the software product meets the necessary standards or high-level requirements. It involves checking whether the actual product aligns with the expected product. Essentially, we’re answering the question: “Are we building the right product for the user’s requirements?”

Suppose the program fails to deliver the expected outcome. In that case, the test case will be the messenger—thus reporting an unsuccessful result, and the developer or tester will need to investigate the issue and find a solution. Think of the checks and actions as paths the computer follows, regardless of the testing type. Groups of input data or conditions used for checking are called "equivalence classes". They should get similar behavior or results from the system under test. The specific paths executed inside a test may vary but should match the activities and assertions done in your test.

Test paths: Typical kinds of test cases #

In software development, a test case is a code execution scenario from which a certain behavior is expected and tested. Typically, there are three scenarios to form test cases from.

The first one is the most well known, which you are probably already using. It’s the happy path, also known as the “happy day scenario” or “golden path”. It defines the most common use case of your feature, application, or change—the way it should work out for the customer.

The second most crucial test path to cover in your test cases is often left out as it’s uncomfortable—as its name may imply. It’s the “scary path” or, in other words, the negative test. This path targets scenarios that cause the code to misbehave or enter an error state. Testing these scenarios is especially important if you have highly vulnerable use cases imposing a high risk on the stakeholders or users.

There are some other paths you might want to know about and consider using, but typically they are less commonly used. The following table summarizes them:

Test path	Explanation
Angry path	This leads to an error, but an expected one; for example, if you want to ensure error handling works correctly.
Delinquent path	This path takes care of any security-related scenarios your application needs to fulfill.
Desolate path	The path testing the scenario in your application doesn’t get enough data to function, for example, testing null values.
Forgetful path	Testing the behavior of your application with insufficient resources, for example, triggering a data loss.
Indecisive path	Testing with a user who is trying to do actions or following user stories in your application but hasn’t completed those workflows. This could be the case, for example, when the user interrupts their workflow.
Greedy path	Trying to test using vast amounts of inputs or data.
Stressful path	Trying to put a load on your application by any means necessary until it no longer functions (similar to a load test).

There are several methods to categorize those paths. Two common approaches are:

Equivalence partitioning. A testing method that categorizes test cases into groups or partitions and, as a result, helps create equivalence classes. This is based on the idea that if one test case in a partition uncovers a defect, other test cases in the same partition will likely reveal similar defects. As all inputs within a specific equivalence class should exhibit identical behavior, you can decrease the number of test cases. Learn more about equivalence partitioning.
Limit analysis. A testing method, also known as boundary-value analysis, that examines the limits or extremes of input values to find any potential issues or errors that might arise at the system's limits of capabilities or constraints.

Best practice: Writing test cases #

A classical test case written by a tester will contain specific data to help you grasp the content of the test you want to conduct and, of course, execute the test. A typical tester would document their testing efforts in a table. There are two patterns we can use at this stage, helping us to structure our test cases and later, our tests themselves, too:

Arrange, act, assert pattern. The "arrange, act, assert" (also known as the "AAA" or "Triple A") testing pattern is a way of organizing tests into three distinct steps: arranging the test, then performing the test, and last but not least, drawing conclusions.
Given, when, then pattern. This pattern is similar to the AAA pattern but has some roots in behavior-driven development.

Future articles will go into more details on these patterns, as soon as we cover the structure of a test itself. If you want to go deeper into these patterns at this stage, check out these two articles: Arrange-Act-Assert: A pattern for writing good tests and Given-When-Then.

According to all the learnings from this article, the following table summarizes a classic example:

Information	Explanation
Prerequisites	Everything which needs to be done before writing the test.
Object under test	What needs to be verified?
Input data	Variables and their values.
Steps to be executed	All the things you will do to bring your test to life: all actions or interactions you do in your tests.
Expected result	What should happen and which expectations are to be fulfilled.
Actual result	What actually happens.

In automated testing, you don’t need to document all these test cases in the way that a tester needs to, even though it’s undoubtedly helpful to do so. You can find all this information in your test if you pay attention. So let’s translate this classical test case into an automated test.

Information	Translation into test automation
Prerequisites	All the things you need, arranging the test, and thinking about what is given to make your test’s scenario happen.
Object under test	This “object” can be various things: an application, flow, unit, or component under test.
Input data	Parameter values.
Steps to be executed	All the actions and commands executed inside your test, the things you act upon, and finding out what happens when you do certain things.
Expected result	The assertion you use to validate your application, the things you assert upon in your application.
Actual result	The result of your automated test.

Pyramid or Crab? Find a testing strategy that fits

2023-07-26T00:00:00Z

Welcome back! The last article laid down lots of groundwork about how to approach the different testing types and what they contain, and clarified the testing type definitions. Remember this little meme image? You might have wondered how all those testing types you learned about could work together.

Next, you will learn exactly that. This article gives an introduction on how to combine these testing types into reasonable strategies and choose one that matches your project.

You can compare the strategies to a number of shapes to better grasp their meaning. Here's a list of strategies with respective sizes and development scopes.

Application size	Team composition	Reliance on manual testing	Testing strategy
Small	Developers only	High	Testing Ice Cone Testing Crab
Small	Developers & QA engineers	High	Testing Ice Cone Testing Crab
Small	Developers only	Low	Test Pyramid
Large	Developers only	High	Testing Trophy Testing Diamond
Large	Developers & QA engineers	High	Testing Trophy Testing Crab
Large	Developers only	Low	Testing Trophy Testing Honeycomb

Let's take a closer look at the strategies and learn the meaning behind their names.

Determine testing goals: What do you want to achieve with these tests? #

Before you can start building a good strategy, figure out your testing goal. When do you consider that your application has been sufficiently tested?

Achieving high test coverage is often viewed as the ultimate goal for developers when it comes to testing. But is it always the best approach? There might be another critical factor to consider when deciding on a testing strategy—serving your users' needs.

As a developer, you also use many other applications and devices. In this respect, you are the user who relies on all these systems to “just work”. In turn, you rely on countless developers to do their best to make their applications and devices work. To turn this back around, as a developer, you also strive to live up to this trust. So your first goal should always be to ship working software and serve your users. This extends to the tests you write to ensure application quality. Kent C. Dodds sums it up very well in his Static vs Unit vs Integration vs E2E Testing for Frontend Apps post:

The more your tests resemble the way your software is used, the more confidence they can give you.
by Kent C. Dodds

Kent describes it as gaining confidence in tests. The closer you get to the users by choosing a testing type that fits, the more you can trust your tests to have valid results. In other words, the higher up you climb the pyramid, the more confident you get. But wait, what is the pyramid?

Determining test strategies: How to choose a testing strategy #

As a first step, determine which parts of the requirements you need to check to make sure they are met. Find out what test types to use and at what level of detail you can achieve the most confidence while maintaining an efficient cost structure. Many developers approach this topic by using analogies. Here are the most common ones, starting with the well-known classic.

The classic: The test pyramid #

As soon as you start looking for test strategies, you will probably come across the test automation pyramid as the first analogy. Mike Cohn introduced this concept in his book "Succeeding with Agile". Later, Martin Fowler expanded upon the concept in his Practical Test Pyramid article. You can represent the pyramid visually as the following:

As shown in this drawing, the test pyramid consists of three layers:

Unit. You find these tests at the base layer of the pyramid because they are fast to execute and simple to maintain. They are isolated and target the most minor test units. For example, see a typical unit test for a very small product.
Integration. These tests are in the middle of the pyramid, because they have an acceptable execution speed but bring you closer to the user than the unit tests can. An example of an integration test is an API test. You can also classify component tests as this type.
E2E tests (also called UI tests). These tests simulate a genuine user and their interaction. Such tests need more time to execute and thus are more expensive. They are at at the top of the pyramid.

Confidence versus resources #

As briefly covered before, the order of the layers is no coincidence. They show the priorities and the corresponding costs. This gives you a clear picture of how many tests you should write for each layer. You have already seen this in the definition of the testing types.

Because E2E tests are closest to your users, they give you the most confidence that you application is working as intended. However, they require a complete application stack and a simulated user, therefore, they are also potentially the most expensive. So the confidence is in direct competition with the resources you need to execute the tests.

The pyramid tries to solve this problem by making you focus more on unit tests and strictly prioritize the cases covered by E2E tests. For example, your most crucial user journeys or the places most vulnerable to defects. As Martin Fowler emphasizes, the two most essential points in Cohn's pyramid are as follows:

Write tests with different granularity.
The more high level you get, the fewer tests you should have.

Pyramid evolved! Adaptations of the test pyramids #

For several years, discussions have revolved around the pyramid. The pyramid seems to oversimplify testing strategies, leaves out a lot of testing types, and no longer fits all the real-world projects. Therefore, it may be misleading. Has the pyramid fallen out of shape? Guillermo Rauch has an opinion about it:

Write tests. Not too many. Mostly integration.
by Guillermo Rauch

It's one of the most commonly cited quotes on this subject, so let's break it down:

"Write tests". Not only because it builds trust, but also because it saves time in maintenance.
"Not too many". 100% coverage is not always good because then your testing isn't prioritized and there will be a lot of maintenance.
"Mostly integration". Here again the emphasis is on integration tests: they have the most business value by giving you a daily high confidence level while maintaining a reasonable execution time.

This makes you think again about the testing pyramid and shift your focus to integration testing. Over the last few years, many adaptations have been proposed, so let's look at the most common ones.

Test diamond #

The first adaptation removes the overemphasis on unit testing, as seen in the test pyramid. Imagine that you have reached 100% coverage on unit tests. However, the next time you refactor, you will have to update many of these unit tests and you might be tempted to skip them. So they erode.

As a result, and together with the higher focus on integration testing, the following shape may arise:

A pyramid evolves into a diamond. You can see the previous three layers, but with a different size, and the unit layer has been cut:

Unit. Write unit tests the way you defined them before. However, because they tend to erode, prioritize and cover only the most critical cases.
Integration. The integration tests you know, testing the combination of single units.
E2E. This layer handles the UI tests similar to the test pyramid. Take care to only write E2E tests for the most critical test cases.

Testing honeycomb #

There is another adaptation, introduced by Spotify, that is similar to the test diamond but further specialized for microservices-based software systems. The testing honeycomb is another visual analogy for the granularity, scope, and number of tests to write for a microservices-based software system. Due to their small size, the most considerable complexity in a microservice is not within the service itself, but in how it interacts with others. So a testing strategy for a microservice should primarily focus on integration tests.

This shape reminds us of a honeycomb, thus the name. It has the following layers:

Integrated tests. The article by Spotify uses a quote from J. B. Rainsberger to define this layer: “A test that will pass or fail based on the correctness of another system.” Such tests have external dependencies that you need to consider, and on the contrary, your system might be a dependency that breaks other systems. Similar to E2E tests in other analogies, use these tests carefully, only for the most essential cases.
Integration tests. Similar to other adaptations, you should focus on this layer. It contains tests that verify the correctness of your service in a more isolated fashion, but still in combination with other services. That means the tests will include some other systems too and focus on the interaction points, for example, via API tests.
Tests on implementation details. These tests resemble unit tests—tests that focus on parts of the code that are naturally isolated and thus have their own internal complexity.

If you want to find out more about this testing strategy, see the post that compares the test pyramid to the honeycomb by Martin Fowler and the original article from Spotify.

Testing trophy #

You can already see a repeating focus on integration tests. However, another type you came across in the previous article is not testing in theory but is still an important aspect you should consider in a testing strategy. Static analysis is missing from the test pyramid and in most of the adaptations you have seen so far. There's the testing trophy adaptation that takes static analysis into account while maintaining the focus on integration tests. The testing trophy originated from the earlier quote by Guillermo Rauch and was developed by Kent C. Dodds:

The testing trophy is an analogy depicting the granularity of tests in a slightly different way. It has four layers:

Static analysis. It plays a vital role in this analogy and lets you catch typos, style mistakes, and other bugs by merely running the debugging steps already outlined.
Unit tests. They ensure that your smallest unit is appropriately tested, but the testing trophy won't emphasize them to the same extent as the test pyramid.
Integration. This is the main focus as it balances the cost and the higher confidence in the best way, as with other adaptations.
UI tests. Including E2E and visual tests, they are at the top of the testing trophy, similar to their role in the test pyramid.

To read more about the testing trophy, see the blog post by Kent C. Dodds on this subject.

Some more UI-focused approaches #

That's all well and good but no matter how you call your strategy, a “pyramid”, “honeycomb,” or “diamond”, there's still something missing. While test automation is valuable, it's important to remember that manual testing is still essential. Automated testing should alleviate routine tasks and free the quality assurance engineers to focus on crucial areas. Instead of replacing manual testing, automation should complement it. Is there a way to integrate manual testing with automation for optimal results?

Testing ice cone and testing crab #

There are indeed two adaptations of the testing pyramid that focus more on these UI-focused ways of testing. Both have the advantage of high confidence, but are naturally more costly due to slower test execution.

The first one, the test ice cone, looks like the pyramid in reverse. Without the manual testing step, it is also known as the testing pizza.

The ice cone has bigger focus on manual or UI testing and the least focus on unit testing. It often takes shape in projects where developers started work with only a few thoughts on the testing strategy. The ice code is considered an anti-pattern and rightfully so. It is costly in terms of resources and manual work.

The test crab is similar to the test ice cone, but with more emphasis on E2E and visual testing:

This testing strategy includes one more aspect: it verifies that your application functions and looks good. The testing crab highlights the importance of visual testing, defined in the previous article. Integration testing, divided into component and API testing, moves further into the background, and unit testing plays an even more secondary role here. You can find further details on this testing strategy in this article on the testing crab.

While being more costly, these two testing strategies have their place: for example, in smaller projects where fewer tests are needed, or less complexity needs to be covered. In this case, a full-blown testing strategy focusing on integration testing might be over-engineered.

Although these two testing strategies are more costly, they have their place, for example, in smaller projects that require fewer tests and don't need to cover a lot of complexity. In this case, a full scale testing strategy focused on integration testing may be unnecessarily complex.

Practical advice: Let's strategize! #

You have now learned about the most common testing strategies. You started with the classic—the test pyramid—and got to know its many adaptations. Now you need to evaluate them for your product and decide which is be the best for your project. The answer to this question should start with everyone's favorite "It depends". That doesn't make it any less accurate though.

The choice of the most appropriate testing strategy from those described—and even the ones left out—depends on your application. It should fir your architecture, your requirements, and last but not least, your users and their requirements. All this might differ from application to application. That's completely normal. Remember that your most important goal is to serve your users, not a textbook definition.

More often than not, real-world tests are difficult to separate and define individually. Even Martin Fowler himself emphasizes the positive aspect of differing definitions, such as in the case of unit tests. As Justin Searls states correctly in his tweet:

[…] write expressive tests that establish clear boundaries, run quickly & reliably, and only fail for useful reasons.
by Justin Searls

Focus on the tests that report actual errors that the users might encounter, and don't get distracted from your goal. Tests should be designed to benefit the user, not just provide 100% coverage or to debate which percentage of which testing type to write.

Focus on tests that report real-life errors that your users might encounter and don't get distracted from your goal. Tests should be designed to benefit the user, not just provide 100% coverage or spark debates on what percentage of a particular testing type you should write.

Three common types of test automation

2023-07-12T00:00:00Z

We've all been there: what is a recurring coding meme that happens all too often in real life?

This meme sums it up quite nicely: each drawer works perfectly well individually, but in combination with the other drawer, they block each other and fail to function. You want both drawers to work well with each other and be operable at the same time.

Apply this to web development: you wrote some tests, maybe even achieved 100% test coverage, but your application still needs to work once other parts fall into place. The units may work well on their own but not in relation to each other. Writing some tests is crucial but it's only one part of the ideal test setup for your project. As a very first step, you need to determine what parts of the application quality you need to ensure and how you can achieve that.

Simply put, you need a plan before you start writing the actual test code. To approach the topic of how to test practically, let's start with a clean slate and answer two basic questions:

How do you want to test?
What do you want to test?

This article focuses on the general things you need to know to answer the first question. To start from a common ground, let's first learn what testing modes exist and then focus on the common types of testing. In later articles, we will answer the second question, combine the answers, and find the testing strategy that works best for your project. Let's go! 🙌

Start with the basics: General testing modes #

When answering the question of how to test, the first point to clarify is very abstract. Should you test manually or let a computer take over? It's important, however, not to fall into binary thinking here.

Manual testing versus automated testing #

If you ask quality assurance engineers to define testing, they will probably break it down into two "modes" first:

Manual testing. This is a typical testing method conducted by actual people. A quality assurance engineer click through the application, check if it works and, at the same time, try to break it. The most common way is exploratory testing, where the engineer investigates the application using their knowledge of the application against a predefined path or checklist.
Automated testing. This is a type of testing conducted by a computer. Quality assurance engineers implement it to automate away repetitive and monotonous tests.

This series of guides will mostly focus on automated testing. However, you shouldn't focus on only one way of testing. Even if automation saves a lot of time and effort, humans and manual testing will always play a vital role. Rather, test automation should free up people to focus on exploratory testing and creative problem-solving. For example, ensuring the quality of user experiences or protecting the high-risk business logic. In other words, automation has your back. ❤️

Opaque box versus clear box #

So, you have defined the general modes of testing. However, that's not enough yet. To plan the testing strategy, there is one more question to answer: should you know how your application works under the hood or is it better to test without this knowledge? Depending on the answer, there are two procedures to choose from for deriving and selecting test cases:

Opaque box testing (or black box testing). It is based on analyzing a component or system's functional or non-functional requirements (specifications) without considering its internal structure.
Clear box testing (or white box testing) is a procedure that takes into account the internal structure of said box. In other words, how your application works under the hood.

Both procedures can be applied to manual and automated testing. However, some aspects of general testing modes may focus more on one of the two—we will cover that later. For now, let's further break down test automation into types.

Test automation types: How do you want to test? #

As you get closer to answering the "how" question, you have already decided to do some manual testing. However, choosing and applying test automation types is a bit more challenging. The types of automation testing are closely related to the metrics you want to create in your projects. So let's take a closer look at the most important ones.

As illustrated in the meme mentioned earlier, you have already come across two types: unit testing and integration testing. End-to-end testing is the third important one to consider. But that isn't all of them still. Let's take a closer look.

Unit testing #

Unit testing is a testing type in which minor testable parts or units of an application are individually and independently tested for proper operation. These units can vary in scope from functions, classes, or interfaces, to services or complete components. Their primary attributes are execution speed, isolation, and comfortable maintainability. If you want to dive deeper into unit testing, head over to this guide on unit testing.

Integration testing #

Integration testing focuses on interactions between components or systems. In other words, on how well they work together. Typical examples of integration tests are API or component tests.

End-to-end testing #

These tests are often called UI tests and this name explains their function even better. These tests interact with your application's UI, including the complete application stack, and test your application from one end to the other.

They resemble a system test if you refer to the theory of quality assurance. These tests simulate a genuine user and their interactions. End-to-end tests take more runtime because they involve the whole system and more runtime requires more computing power. As a result, this additional effort results in higher maintenance costs.

Visual UI testing #

An interesting subcategory of UI tests is visual tests. These tests are extended end-to-end tests that provide a means to verify the visible output of an application. Such a test takes a screenshot after a change and another screenshot containing the “status quo” (or golden file), then provides those results to a human reviewer to inspect and check. In other words, it helps find “visual bugs” in the appearance of a page, beyond purely functional bugs and not explicitly written down into assertions.

Static analysis #

There's one more thing to introduce here: static analysis. It isn't a testing type in the textbook sense. However, it will be an essential aspect in quality assurance strategies later on. You can imagine it working like a spell check function: it scans your code for more significant defects and syntax errors without running the program, thus detecting code style issues. This simple measure can prevent many bugs. This is a good point to learn about Static Analysis if you want to get to know it in more detail.

Testing in all shapes: How does this all work together? #

While searching for answers to all these questions, you might find a possible solution in some analogies. In the web and testing communities specifically, developers tend to use these analogies to give you an idea of how many tests you should use of which type.

The following five strategies depicted in this image are the most common ones:

Test Pyramid
Test Diamond
Test Ice Cone (also known as Test Pizza)
Test Honeycomb
Test Trophy

This is truly a lot of information to process. How should you decide on a matching test strategy based on all this? Don't worry, we've got you covered. In the next article, we will discuss these different strategies in more detail and explain how to choose the best fit for your project. Stay tuned! 🔥