What is End to End Testing? - American Technology Consulting

Software

What is End to End Testing?

Kelsey Meyer

Published April 12, 2019

Subscribe to the blog

What is end to end testing? One of the most effective ways to ensure end-user application performance.

In a microservices-based, Agile-driven world, software development and delivery has become quite complex.

Today’s applications are built on layers of code, networks of subsystems, and third-party integrations that are so intertwined, if one fails component fails, so does the entire product.

This makes the stability of each component - both on its own and as part of the entire application system - vital to the success of an enterprise’s software products. And it highlights the clear need to test entire applications from start to finish, from software modules and APIs through networking systems and end-user interfaces.

That’s where End to End testing comes in.

But what does E2E testing entail?

That’s what we’ll cover in today’s article.

What is End to End Testing?

End to end testing is a methodology used to verify the functionality and performance of an application flow under production product-like circumstances and data.

This not only includes individual software modules, but the internal and external APIs, network and database infrastructure the application relies on.

Hence the name “end to end.”

The goal is to simulate a real-world user scenario to ensure the application performs as intended on the whole - not just individual components (which is covered by other types functional testing) or your system isolated from 3rd parties or real end-user scenarios (which is covered by system testing).

Benefits of E2E Testing

As a result of complicated modern applications build on internal and external sub-systems, modern software products need to be validated on the whole, not just in part; while subsystems might pass tests individually, there may be failures and faults in the way they interact as a complete application.

Conducting end to end testing helps ensure your software is production ready, layer by layer.

Here are a few reasons why it’s key to your success.

E2E Testing Confirms Your Application’s Health

End to end testing validates your software functions at every level, front to back, on multiple systems, and as a whole. At the same time, it provides perspective on application performance across different end-user environments as well to ensure the end user experience works as intended.

E2E Testing Expands Test Coverage

By accounting for the interaction between services and subsystems, end to end testing also expands test coverage beyond what’s normally considered in more isolated testing practices to further ensure quality and mitigate risk.

E2E Testing Catches More Bugs and Increases Application Productivity

End to end testing performed in a modern Agile or DevOps development environment is usually done every iteration or every few iterations, meaning errors are caught and corrected faster. This reduces the chances of bugs getting buried deep into a product where they become hard to find and hard to fix.

E2E Testing Reduces Effort and Costs

With comprehensive testing at each iteration as well as fewer bugs and breakdowns in released products, end to end testing decreases the need to repeat tests or redo work - ultimately reducing costs and times associated with testing.

How End to End Testing Works 

how e2e testing works diagram

Because end to end testing covers entire applications, it’s important to properly plan and iteratively deploy a new e2e testing program.

Failing to conduct end to end testing risks a catastrophic application collapse in the event a subsystem fails.

For example, imagine your purchasing a software product like Evernote. From your perspective as an end user, the process might look like this:

  • Go to Evernote’s website.
  • Navigate to the “compare plans” page to learn about the costs, features, and specifications.
  • Choose the premium plan and click “Free Trial.”
  • Proceed to create a username and password and click “continue.”
  • Enter your billing information and click “start free trial.”
  • Land on Evernote’s web app home page.
  • Navigating from the home page to the “compare plans” page.
  • Clicking the “free trial” button on the premium plan.
  • Filling in and submitting the username and password fields to create an account.
  • Enter and submit billing information.

You might then also receive an email confirming your registration, then a follow-up email with resources on how to use Evernote.

From start to finish, you’d expect this process would flow seamlessly from the website loading through being able to navigate the registration process and receiving the emails.

As an end user you wouldn’t think about the back-end systems that are relied on at each step, but for Evernote’s Quality Assurance and Control teams, the number of steps in this user flow alone can make testing their platform complex and time-consuming. Including other services outside of their core app, like their payment processor and emailing systems, testing the full workflow can be a nightmare.

Verifying the network infrastructure needed to keep the site live and responsive is working, that the user interface looks and functions as expected, that the private financial information you shared is encrypted in transit and at rest, that the emails are properly triggered and sent…

The number of moving parts can be quite overwhelming and opens up a whole realm of failure points and possibilities.

End to end testing not only ensures a complex workflow like this functions from the user’s perspective across various interfaces, it also ensures the flow of information between Evernote’s application, its network appliances, as well as it's (likely third party) email and credit card processing platforms also work as intended.

The E2E Testing Process

Because of its broad scope, end to end testing can consist of many intricate and unique parts, but the basic process is as follows:

  1. Requirements Planning: Analyze your existing systems and subsystems and define minimum as well as recommended software and hardware requirements.
  2. E2E System Design: Outline the components, tools, workflows, methodologies, and metrics for your end to end testing system, as well as the individual roles and responsibilities of all team members involved.
  3. Testing Component Design: Design your individual Test Cases for each subsystem.
  4. Test Infrastructure Development: Code your test cases and set up the hardware and software infrastructure needed by your test environment.
  5. Test Execution & Measurement: Test your systems, recording all input and output data and analyze these against your target metrics to identify flaws and failures.

End to End Testing Methods

E2E testing is performed in two main ways - horizontally and vertically.

Horizontal E2E Testing

Horizontal end to end testing operates primarily works from the user’s perspective, stepping through each phase of an application’s business logic from start to finish to ensure each interrelated process executes correctly.

In our Evernote example, their application needed to not only work from a UI standpoint, but also properly integrate with their payment processor and emailing system, as well as their network and database infrastructure.

Since horizontal testing occurs at the end of a release cycle (when changes to systems and subsystems are complete), it’s essential that your test environment is completely set up in by your testing team in advance of execution.

Vertical E2E Testing

Vertical end to end testing involves testing each layer of an application’s architecture from top to bottom, meaning each subsystem is tested independently, then testing is continued through network infrastructure, API integration, and UI layers.

So in our Evernote example, their core system would be tested granularly with unit tests, then their payment processing system, then their emailing system, before they’d move on to testing their networks, API integrations, and finally the UI.

Due to its granular focus, vertical testing is often more integrated into the development cycle using strategies such as behavior-driven development, test driven development, or continuous testing.

While horizontal testing can be set up and performed by a relatively independent team, vertical testing’s close integration with your existing SDLC means each major stakeholder - developers, testers, product owners - all need to buy in and assist with the testing process.

Advantages

Both types of end to end testing - horizontal and vertical - can help you accomplish the same goal (ensuring your entire application functions efficiently and effectively), but each methodology has unique benefits and drawbacks that make them better for implementing with certain teams.

Benefits of Horizontal Testing

  • Focuses on the end-user perspective to ensure the final application works as intended.
  • Ensures coverage of business logic requirements.
  • Can be set up and executed independently
  • Higher coverage of core code through focused testing.
  • Faster test execution and defect resolution
  • Can be implemented in stages (doesn’t require a full application test environment to be set up first).

Drawbacks of Horizontal Testing

  • Slows time to test execution and defect fixing (must wait until all release changes are completed)
  • Requires buy-in and coordination of all major SDLC stakeholders

Benefits of Vertical Testing

  • Higher coverage of core code through focused testing.
  • Faster test execution and defect resolution
  • Can be implemented in stages (doesn’t require a full application test environment to be set up first).

Drawbacks of Vertical Testing

  • Requires buy-in and coordination of all major SDLC stakeholders

Creating an End to End Testing System

E2e testing systems are built in three major stages - User Functions, Test Conditions, and Test Cases.

1. Build User Functions

The first step is to build functions that simulate real user behavior and use of your application.

So in our Evernote example, we’d need to model functions for:

Building User Functions involves:

  1. Listing the features of your application and their relationships within your core system and interconnected subsystems.
  2. List the actions as well as input and output data for each function.
  3. Determine whether functions are independent or are duplicate (so one coded function can be reused across test cases).

2. Build Test Conditions

Once User Functions are mapped and built it’s time to create conditions to assess whether a test is passed or failed.

This consists of setting parameters around timing, data, and sequence for each user function.

In our Evernote example, conditions might include:

  • Complete loading of the “compare plans” page within x seconds of clicking the navigation item on the home page.
  • Complete loading of the “create account” page within x seconds of clicking the free trial button on the “compare plans” page.
  • Checking whether the username is already in use.
  • Checking the strength of the submitted password.
  • Checking whether payment info is correctly formatted
  • Checking whether the payment method is valid/active
  • Checking whether the automated email response is triggered upon payment info submission
  • Checking whether the automated email is sent once triggered.

3. Build Test Cases

With user functions built and conditions set it’s time to build Test Cases that execute functions according to a specific workflow scenario and measure performance against the test conditions.

Multiple test cases should be built to test each user scenario as well as each individual condition.

Metrics to Measure in E2E Testing

As with any testing process, it’s important to ensure your teams’ time and efforts aren’t wasted by tracking a few key metrics for success.

For end to end testing, these include:

  • Test Case Preparation Status: Compare how many test cases have been prepared and are being prepared against the total number of planned cases.
  • Weekly Testing Progress: Measure the number of tests completed week by week as a percentage of total tests planned, as well as the number of passed and failed tests.
  • Defect Details and Status: Track the percentage of defects opened and closed each week, as well as the distribution of defects based on severity and priority.
  • Test Environment Availability: Monitor the amount of time allotted for running tests in your test environment, as well as the amount of time actually used.

How to Implement E2E Testing

Now that you have a better understanding of what end to end testing is, its benefits, and how it works, perhaps you’re ready to start implementing your own system?

But how do you do that in your organization? How do you get from “this is a good idea” to actually executing tests and fixing defects?

Once you’re committed to end to end testing, here’s the basic implementation process we recommend:

1. Identify Your Most Important Tests

Especially for large enterprise applications, end to end tests can potentially become massive in scope. Instead of attempting to test every possible test case across every one of your software products, we recommend beginning with one most important application (either most critical to your business or most prone to defects) and outlining a few of the most important cases to test within that application to maximize ROI early on.

2. Choose a Testing Framework

With your most important tests defined, evaluate various E2E testing frameworks to find one that best suits your business. The main criteria, as with most business solutions, are feature set (do they meet your needs), cost (do they fit your budget?) and ease of implementation (can you get them set up quickly?).

A few of the more popular end to end testing frameworks include:

3. Integrate E2E Testing with your Existing SDLC

With your essential tests Identified and end to end testing framework chosen, the next step is to determine how end to end testing can be integrated into your existing software development lifecycle.

The main options here are:

  1. Running e2e tests with every code push.
  2. Running these tests periodically (perhaps nightly, weekly, or monthly).

Vertical end to end testing lends itself well to an “every push” testing cycle as it’s much easier to integrate it’s granular tests into your development process without adding a large testing overhead at the end of the development process, though if your organization has the resources horizontal testing can work as well.

Running e2e tests periodically is a much easier strategy as it doesn’t require as much testing and allows software to be delivered more quickly. However, if your existing process has quality assurance challenges, this can ultimately lead to more rework/user experience flaws as more defective code gets pushed to production and needs to be caught, fixed, and redeployed.

4. Create a Prototype Testing Environment and Process

The last step is to create a proof of concept testing environment, process, and test cases to validate your new testing system without interrupting normal workflows or potentially disrupt your existing codebase.

Create an isolated repository of your code and run your tests in a staging environment then, as you perfect your testing process, begin to fully integrate it with your existing SDLC and infrastructure.

Or, Call in the Experts

User expectations and high-quality competition demand modern enterprises improve their testing coverage and defect fixing speed to keep up.

But even for enterprises with a vast array of resources and expertise, implementing a new testing program can be a daunting task.

That’s why we offer our Quality Assurance services.

With experience and mix of full-cycle testing tools, process improvement strategies, and ongoing quality governance, we can help you:

  • Reduce development costs, optimize software quality, and enhance business value through fully managed testing services.
  • Develop detailed, exhaustive, realistic plans to help you manage change and establish new processes and policies that maintain quality across your organization and projects.
  • Build and manage a center for quality assurance and implement the tools, frameworks, and methodologies needed to seamlessly blend QA into your development lifecycle.

Reach out today to for help with implementing your end to end testing program and ensure quality across your product portfolio.

More from our blog

The ATC Guide to Building SaaS Products

Nick Reddin

May 5, 2020 | 3 min read
How to Accelerate Digital Transformation

Nick Reddin

December 23, 2019 | 33 min read
A Complete Breakdown of CI/CD: Differences, Benefits, and Tools

Kelsey Meyer

July 12, 2019 | 7 min read

Let's talk about your project.

Contact Us