DevOps Test – Test types in the deployment pipeline

Test types Introduction

There are many test types. The question is when is which test type to be executed. This articles gives some advice.

Test types – Definitions

Chain test

The chain test is used to make sure that the applications in a chain are working correct together. Sometimes this type of test is referred to as integration test. But to differentiate the chain test from the integration test of application modules the name chain test is advised.

Component Test (CT)

Larger applications are build based on components or modules. Each component consists in that case of several S-CI’s. This test is used to test these components.

Functional Acceptance Test (FAT)

The functional acceptance test is used to verify that the happy path of the application works in accordance with the functional requirements. The alternate path and error path should also be tested too.

Integration Test (IT)

This test checks whether the components / modules of one applications works correctly together. There is a little overlap with the component test. However, this test type is meant to be focused on more than one component. The component test is focused on one component.

Performance Stress Test (PST)

The performance stress test is used to verify whether the system is capable of handling the request timely given a known number of users, transactions types and background workload.

Production Acceptance Test (PAT)

The production acceptance test is used to verify the application works fine in the target environment.

System Test (ST)

The system test is meant to check whether the complete application fulfils the defined requirements.

Test Driver

A test driver is a dummy piece of software that is used to call an isolated function, component or complete application. In this way test cases can call the test driver and thus execute unit test cases, component test cases, integration test cases or system test cases.

Test Stub

A test stub is a dummy piece of software that is used to simulate an interaction between functions, components or complete applications.

Unit Test (UT)

A unit test is a test case that test only one small object for example one S-CI.

User Acceptance Test (UAT)

The user acceptance test is used to determine whether the user is able to use the application in terms of usability, user friendliness, effort to register information etc.

Test types Concepts

Master test plan (MST)

A master test plan is used to define the test strategy. The test strategy is formulated to determine which test types are tested where and when and by whom in such a way that with the minimal effort the most important functions are tested to in accordance with the requirements.

Test types Best Practices

Since decades test management best practices are defined in TMap NEXT, BiSL, ASL and ITIL. Therefor, this article focusses on the relationship between the deployment pipeline and test management best practices. The deployment pipeline is used to automate the test cases as much as possible. To be able to plot the test types on the deployment pipeline they should be more clarified. This is done based on the onion layer of test types. Next this article describes the position in the deployment pipeline.

The Onion Test Type layer

In Figure 1 one the test types are related to each other in the form of an onion. The layers indicate the order in which the test cases have to be performed.

Deployment Pipeline

The following table shows the distribution of the test types through the deployment pipeline.

Commit Phase	Acceptance Phase	Capacity Phase	Manual Testing	Production Testing
UT	CT	PST	UAT	PAT
Pre CT	IT
	ST
	FAT
	NFR
	Pre PST
	Pre UAT

Table 1‑1, Test types per deployment pipeline phase.

Commit phase

The commit phase must be very fast. Normally the norm is 5 minutes for a local auto build. For unit test cases, it is not allowed to require the connection to other applications of infrastructure services. Normally it is just one or more test cases that are run with the use of a test driver and/or test stub. During the commit phase some component test cases may be executed to check the correct working of the interaction between components. This decreases the chance that the check-in on the central trunk will break the build.

Acceptance phase

The accept phase is used to perform the longer running test cases.

Capacity phase

A PST may take a day or more to run. Running this test in the acceptance environment would disrupt the deployment pipeline. So therefore, there is a separate environment for the PST.

Manual testing

Continuous Deployment means no manual testing. This is possible for content publications where no user acceptance is requested. In case of Continuous Delivery there is still a manual go/no go in the pipeline. The test cases that can be performed are:

• Smoke test (short test cases to see if at least there is an E2E connection)
• Usability tests (test cases that indicates the user friendliness of the applications user interface)
• Exploratory tests (this test approach is about learning during testing. The test cases are not predefined but rather created during the test run)
• Showcases (show cases are demonstrations to the user of the new product)

Production testing

At least a smoke test in the production environment should be performed before the go-live is approved. But also test cases in a production like environment may take place. In the old days where a manual configuration was normal, you might expect configuration checks to be part of the PAT. Modern environment management prevent these kinds of errors. Though smoke tests are still valuable.

Discuss with us about this article on LinkedIn.

More information

Related Books:

DevOps Best Practices, ISBN: 9789492618078
Agile Service Management with Scrum, ISBN: 9789071501807

Related training sessions:

• DevOps Foundation
• DevOps in practice
• DevOps Master
• Masterclass DevOps Operations
• Masterclass DevOps Application Development
• Masterclass DevOps Architecture

Related Article:

Service Management