What is End to End Testing?

What is end to end testing? One of the most effective ways to ensure end-user application performance.

In a microservices-based, Agile-driven world, software development and delivery have become quite complex.

Today’s applications are built on layers of code, networks of subsystems, and third-party integrations that are so intertwined, if one component fails, so does the entire product.

This makes the stability of each component - both on its own and as part of the entire application system - vital to the success of an enterprise’s software products. And it highlights the clear need to test entire applications from start to finish, from software modules and APIs through networking systems and end-user interfaces. That’s where End to End testing comes in.

But what does E2E testing entail?

That’s what we’ll cover in today’s article.

What is End to End Testing?

End-to-end (E2E) testing ensures that applications meet expectations by behaving as they were designed to. This also checks if the data flow across processes is consistent. The ultimate goal of this type of testing is simulating a real-world situation from the perspective of the end user.

This not only includes individual software modules, but the internal and external APIs, network and database infrastructure the application relies on.

Hence the name “end to end.” The application should be able to perform as intended on the whole - not just as individual components (which is covered by other types of functional testing) or real end-user scenarios (which is covered by system testing).

Benefits of E2E Testing

As a result of complicated modern applications built on internal and external sub-systems, modern software products need to be validated on the whole, not just in part; while subsystems might pass tests individually, there may be failures and faults in the way they interact as a complete application.

Conducting end-to-end testing helps ensure your software is production ready, layer by layer.

The following are a few reasons why it’s key to your success.

E2E Testing Confirms Your Application’s Health

End to end testing validates your software functions at every level, front to back, on multiple systems, and as a whole. At the same time, it provides perspective on application performance across different end-user environments as well to ensure the end user experience works as intended.

E2E Testing Expands Test Coverage

By accounting for the interaction between services and subsystems, end to end testing also expands test coverage beyond what’s normally considered in more isolated testing practices to further ensure quality and mitigate risk.

E2E Testing Catches More Bugs and Increases Application Productivity

End to end testing performed in a modern Agile or DevOps environment is usually done for every iteration or every few iterations, meaning errors are caught and corrected faster. This reduces the chances of bugs getting buried deep into a product where they become hard to find and hard to fix.

E2E Testing Reduces Effort and Costs

With comprehensive testing at each iteration as well as fewer bugs and breakdowns in released products, end to end testing decreases the need to repeat tests or redo work - ultimately reducing costs and times associated with testing.

How End to End Testing Works 

Because end to end testing covers entire applications, it’s important to properly plan and iteratively deploy a new e2e testing program.

Failing to conduct end to end testing risks a catastrophic application collapse in the event a subsystem fails.

For example, imagine you are purchasing Evernote. From your perspective as an end user, the process might look like this:

• Go to Evernote’s website.
• Navigate to the “compare plans” page to learn about the costs, features, and specifications.
• Choose the premium plan and click “Free Trial.”
• Proceed to create a username and password and click “continue.”
• Enter your billing information and click “start free trial.”
• Land on Evernote’s web app home page.
• Navigate from the home page to the “compare plans” page.
• Click the “free trial” button on the premium plan.
• Fill in and submit the username and password fields to create an account.
• Enter and submit billing information.

You might then also receive an email confirming your registration, then a follow-up email with resources on how to use Evernote.

From start to finish, you’d expect this process would flow seamlessly from the website loading through navigating the registration process and receiving the emails.

As an end user, you wouldn’t think about the back-end systems that are relied on at each step, but for Evernote’s Quality Assurance and Control teams, the number of steps in this user flow alone can make testing their platform complex and time-consuming. Including other services outside of their core app, like their payment processor and emailing systems, testing the full workflow can be a nightmare.

Verifying the network infrastructure needed to keep the site live and responsive is working, that the user interface looks and functions as expected, that the private financial information you shared is encrypted in transit and at rest, that the emails are properly triggered and sent.

The number of moving parts can be quite overwhelming and opens up a whole realm of failure points and possibilities.

End-to-end testing not only ensures a complex workflow like the functions from the user’s perspective across various interfaces, it also ensures the flow of information between Evernote’s application, its network appliances, as well as its (likely third party) email, and credit card processing platforms.

The E2E Testing Process

Because of its broad scope, end to end testing can consist of many intricate and unique parts, but the basic process is as follows:

1. Requirements Planning: Analyze your existing systems and subsystems and define minimum as well as recommended software and hardware requirements.
2. E2E System Design: Outline the components, tools, workflows, methodologies, and metrics for your end to end testing system, as well as the individual roles and responsibilities of all team members involved.
3. Testing Component Design: Design your individual Test Cases for each subsystem.
4. Test Infrastructure Development: Code your test cases and set up the hardware and software infrastructure needed by your test environment.
5. Test Execution & Measurement: Test your systems, recording all input and output data and analyze these against your target metrics to identify flaws and failures.

End to End Testing Methods

E2E testing is performed in two main ways - horizontally and vertically.

Horizontal E2E Testing

Horizontal end to end testing operates primarily from the user’s perspective, stepping through each phase of an application’s business logic from start to finish to ensure each interrelated process executes correctly.

In our Evernote example, their application needed to not only work from a UI standpoint, but also properly integrate with their payment processor and emailing system, as well as their network and database infrastructure.

Since horizontal testing occurs at the end of a release cycle (when changes to systems and subsystems are complete), it’s essential that your test environment is completely set up by your testing team in advance of execution.

Vertical E2E Testing

Vertical end to end testing involves testing each layer of an application’s architecture from top to bottom, meaning each subsystem is tested independently, then testing is continued through network infrastructure, API integration, and UI layers.

So in our Evernote example, their core system would be tested granularly with unit tests, then their payment processing system, then their emailing system, before they’d move on to testing their networks, API integrations, and finally the UI.

Due to its granular focus, vertical testing is often more integrated into the development cycle using strategies such as behavior-driven development, test driven development, or continuous testing.

While horizontal testing can be set up and performed by a relatively independent team, vertical testing’s close integration with your existing SDLC means each major stakeholder - developers, testers, product owners - all need to buy in and assist with the testing process.

Advantages

Both types of end to end testing - horizontal and vertical - can help you accomplish the same goal (ensuring your entire application functions efficiently and effectively), but each methodology has unique benefits and drawbacks that make them better for implementing with certain teams.

Benefits of Horizontal Testing

• Focus on the end-user perspective to ensure the final application works as intended.
• Ensuring coverage of business logic requirements.
• It can be set up and executed independently
• Higher coverage of core code through focused testing.
• Faster test execution and defect resolution
• It can be implemented in stages (doesn’t require a full application test environment to be set up first).

Drawbacks of Horizontal Testing are:

• It slows time to test execution and defect fixing (must wait until all release changes are completed)
• It requires buy-in and coordination of all major SDLC stakeholders

Benefits of Vertical Testing

• Higher coverage of core code through focused testing.
• Faster test execution and defect resolution
• Can be implemented in stages (doesn’t require a full application test environment to be set up first).

The drawbacks of Vertical Testing

• Requires buy-in and coordination of all major SDLC stakeholders.

We shall discuss the drawbacks in detail below.

Disadvantages

To have a holistic understanding of E2E testing, it is important to grasp its disadvantages as well. These are as follows:

• It is time-intensive: Since you need to consider every facet of an application or software before testing it, it can take a while before you are ready to test it. Rather than testing each user workflow, companies only use E2E testing for the high-priority paths, while testing the rest with integration tests and snapshot tests.

• Real-world simulation: The biggest advantage as well as disadvantage with E2E testing is real-world simulation. This does allow for comprehensive testing but at the same time requires a lot of consideration and deliberation to ensure an accurate simulation of the real world, making this more complicated than other tests. 

• User goal dilemma: You can put all the world's features into your app and it still wouldn't solve problems unless you understand user needs. End-to-end testing always focuses on this - user expectations. As dev teams may not necessarily have a clear understanding here, research into user experiences becomes a critical factor in E2E testing, which can be expensive and sometimes inconclusive.

Creating an End to End Testing System

E2E testing systems are built in three major stages - User Functions, Test Conditions, and Test Cases.

1. Build User Functions

The first step is to build functions that simulate real user behavior and use of your application.

Building User Functions involves:

1. Listing the features of your application and their relationships within your core system and interconnected subsystems.
2. Listing the actions as well as input and output data for each function.
3. Determining whether functions are independent or are duplicate (so one coded function can be reused across test cases).

2. Build Test Conditions

Once user functions are mapped and built it’s time to create conditions to assess whether a test is passed or failed.

This consists of setting parameters around timing, data, and sequence for each user function.

In our Evernote example, conditions might include:

• Complete loading of the “compare plans” page within x seconds of clicking the navigation item on the home page.
• Complete loading of the “create account” page within x seconds of clicking the free trial button on the “compare plans” page.
• Checking whether the username is already in use.
• Checking the strength of the submitted password.
• Checking whether payment info is correctly formatted
• Checking whether the payment method is valid/active
• Checking whether the automated email response is triggered upon payment info submission
• Checking whether the automated email is sent once triggered.

3. Build Test Cases

With user functions built and conditions set it’s time to build Test Cases that execute functions according to a specific workflow scenario and measure performance against the test conditions.

Multiple test cases should be built to test each user scenario as well as each individual condition.

Metrics to Measure in E2E Testing

As with any testing process, it’s important to ensure your teams’ time and efforts are worthwhile.

Ensure:

• Test Case Preparation Status: Compare how many test cases have been prepared and are being prepared against the total number of planned cases.
• Weekly Testing Progress: Measure the number of tests completed week by week as a percentage of total tests planned, as well as the number of passed and failed tests.
• Defect Details and Status: Track the percentage of defects opened and closed each week, as well as the distribution of defects based on severity and priority.
• Test Environment Availability: Monitor the amount of time allotted for running tests in your test environment, as well as the amount of time actually used.

How to Implement E2E Testing

Now that you have a better understanding of what end to end testing is, its benefits, and how it works, perhaps you’re ready to start implementing your own system?

But how do you do that in your organization? How do you get from “this is a good idea” to actually executing tests and fixing defects?

Once you’re committed to end to end testing, here’s the basic implementation process we recommend:

1. Identify Your Most Important Tests

Especially for large enterprise applications, end to end tests can potentially become massive in scope. Instead of attempting to test every possible test case across every one of your software products, we recommend beginning with one most important application (either most critical to your business or most prone to defects) and outlining a few of the most important cases to test within that application to maximize ROI early on.

2. Choose a Testing Framework

With your most important tests defined, evaluate various E2E testing frameworks to find one that best suits your business. The main criteria, as with most business solutions, are feature set (do they meet your needs), cost (do they fit your budget?) and ease of implementation (can you get them set up quickly?).

A few of the more popular end to end testing frameworks include:

• Protractor
• Nightwatch
• CasperJS
• Testcafe

3. Integrate E2E Testing with your Existing SDLC

With your essential tests Identified and end to end testing framework chosen, the next step is to determine how end to end testing can be integrated into your existing software development lifecycle.

The main options here are:

1. Running e2e tests with every code push.
2. Running these tests periodically (perhaps nightly, weekly, or monthly).

Vertical end to end testing lends itself well to an “every push” testing cycle as it’s much easier to integrate it’s granular tests into your development process without adding a large testing overhead at the end of the development process, though if your organization has the resources horizontal testing can work as well.

Running E2E tests periodically is a much easier strategy as it doesn’t require as much testing and allows software to be delivered more quickly. However, if your existing process has quality assurance challenges, this can ultimately lead to more rework/user experience flaws as more defective code gets pushed to production and needs to be caught, fixed, and redeployed.

4. Create a Prototype Testing Environment and Process

The last step is to create a proof of concept testing environment, process, and test cases to validate your new testing system without interrupting normal workflows or potentially disrupting your existing codebase.

Create an isolated repository of your code and run your tests in a staging environment then, while you perfect your testing process, begin to fully integrate it with your existing SDLC and infrastructure.

Or, Call in the Experts

User expectations and high-quality competition demand modern enterprises improve their testing coverage and fix detects faster to keep up.

But even for enterprises with a vast array of resources and expertise, implementing a new testing program can be a daunting task.

That’s why we offer our Quality Assurance services.

With experience and mix of full-cycle testing tools, process improvement strategies, and ongoing quality governance, we can help you:

• Reduce development costs, optimize software quality, and enhance business value through fully managed testing services.
• Develop detailed, exhaustive, realistic plans to help you manage change and establish new processes and policies that maintain quality across your organization and projects.
• Build and manage a center for quality assurance and implement the tools, frameworks, and methodologies needed to seamlessly blend QA into your development lifecycle.

Reach out today to for help with implementing your end to end testing program and ensure quality across your product portfolio.