The Importance of Software Testing in DevOps

Software testing is the process of identifying the correctness and quality of a software program. In other words, testing is executing a system or application in order to find software bugs, defects, errors or unexpected behavior.

Software testing is necessary because we all make mistakes. Some of those mistakes are minor, but others can be expensive or dangerous. Especially while practicing continuous integration, continuous delivery, or continuous deployment, we need to test anything and everything we produce, because things can always go wrong.

Testing is mainly classified into two types, Functional Testing and Non-functional Testing.

Types of Functional and Non-functional testing

Functional testing processes test against the specifications, or functional requirements for a system and its components (what the system should do), while non-functional testing processes test the operation of that system (how the system should be).

Common types of Functional Testing

Unit Testing

In Unit Testing, the scope of testing is limited to a particular unit or component—for example, a specific API, database, or GUI.


  • API: Have separate unit tests for individual web services
  • Database: Create tests to add, edit or remove data
  • GUI: Focus on field-level validation
  • Use branch coverage or statement coverage.
  • Add some negative tests
  • As a best practice, it is more reliable to use ‘mock’ service(s) to write unit tests, in order to avoid interdependence between components
  • Think about data re-usability, and clearing any test data from the database at end of your test (if applicable)

Tools / Frameworks:

The Junit and TestNG unit testing frameworks are popular for a Java-based development environment. For Ruby, RSpec is widely used.

For the front-end, the Karma test runner is commonly used in development environments using AngularJS or Vue.js to run JavaScript-based unit tests. Protractor and the BDD-based Jasmine framework are also widely used. Those using React might prefer Jest, while Ember.js has an integrated unit testing provision. Mocha is yet another JavaScript alternative for front-end unit testing.

Sanity Testing

Sanity testing runs before any detailed functional or regression tests and tests basic, integral functionalities of an application. Sanity testing is required to run every commit to make sure critical business functionalities (such as the launching of the application) are not affected by code changes.

System Integration Testing

System Integration Testing (SIT) will touch multiple units/components and is a black box testing method. Performing this testing should not require knowledge of the internal workings of the system.

Normally within a sprint, manual testing will be the first phase of the SIT. Later on, this will be converted to an automated test script. If there is a dedicated automation testing resource available in the Agile team, then an automation test script can be developed in parallel to development activities. The automated test script will fast-track regression testing prior to each release or every commit in the continuous integration / continuous deployment model.

Tools / Frameworks:

These tools and frameworks are used by the Europe PMC development team.

  • Defect tracking tool – JIRA
  • Automation Testing tool – Selenium
  • Automation Framework – Serenity BDD
  • Programming Language – Java
  • Unit Testing Framework for Test Automation – JUnit
  • Build Tool – Maven
  • Schedule and Execution – Jenkins

Automation scripts can be developed in any language, and it is not mandatory for the testing team to use the same technology that the development team uses.

Regression Testing

Regression testing ensures that software performs the same functionality in the same way after it is updated or changed. Regression testing should be performed after any code change to a previously developed and tested program, to ensure that new defects have not been introduced.

User Acceptance Testing (UAT)

UAT is normally conducted by the actual users of a particular system. There will be end-to-end scenarios for the UAT users to execute to verify system functionalities. In the waterfall model, UAT will be conducted only at the end of development, whereas in the Agile model, UAT can be performed at the end of every sprint and feedback given to the team.

Common types of Non-functional Testing

Performance Testing

Software performance testing is used to determine the speed or effectiveness of a computer, network, software program, or device. This process can involve quantitative tests done in a lab, such as measuring the response time or the number of MIPS (millions of instructions per second) at which a system functions. Performance testing (“How fast is the system?”) can also include load testing (“How much volume can the system process?”).

Tools / Frameworks:

Gatling or JMeter

Sample ‘Gatling’ Performance Test Results:

Sample 'Gatling' Performance Test Results:

Security Testing

Security testing is a technique to determine if an information system protects data and maintains functionality as intended. Its aims should be verifying three basic principles—confidentiality, integrity, and authentication.

Compatibility Testing

Compatibility testing involves verifying a service’s functionality is the same across various browsers, different browser versions, mobile devices, operating systems, etc.

Usability Testing

Usability testing is used in user-centered interaction design to evaluate a product by testing it on users. This is an irreplaceable testing practice, since it gives direct input on how real users use the system.

Automating your Testing

Test automation is the process of converting manual test scripts to automated test scripts, using testing tools like Selenium, Appium, SoapUI, REST-Assured, Serenity, Cucumber, etc. Test automation will fast-track regression testing, and it avoids repeated and time-consuming manual testing efforts. It is also convenient to use automated testing for compatibility testing.

Sample Automation / Cucumber-BDD-Serenity Test Report:

Sample Automation / Cucumber-BDD-Serenity Test Report

Benefits of Automated Testing

  • Faster feedback
  • Earlier Detection of Defects
  • Accelerated Results
  • Running tests 24/7
  • Fewer human resources
  • Reusability
  • Reliability
  • Simultaneity
  • Better test coverage with multiple data combinations

Test Flows, Test Management & Execution Frequency

Sample test flow

Example of execution frequency of different types of testing

Sample Build Pipeline model along with Software Testing steps

Sample Build Pipeline model along with Software Testing steps

In the above build pipeline model, there are three pre-production servers, i.e. Development (DEV), Testing (TEST), and Staging. Normally the DEV server will be used for all development activities, with the TEST server for functional testing by the quality assurance team, and the Staging server for performance testing and user acceptance testing. On some occasions, there will only be DEV, TEST and PROD (production) servers available. In such cases, internal testing and UAT will happen on the TEST server as well.

Proposed Tools & Framework Summary

#DescriptionPreferred ‑ 1Paid?Preferred ‑ 2Paid?Remarks
1Automation FrameworkSerenity-BDD with Cucumber - JunitNoPython with BehaveNo
2Front-end Automation Selenium, ProtractorNoHP-UFTYes
3Mobile AutomationAppiumNoSeeTestYes
4Performance TestingGatling, JMeterNoLoadRunnerYes
5SOAP Web ServiceSOAP-UINoReadyAPIYesJAX-WS/JAXB concept for Test Automation
6REST Web ServiceSOAP-UINoPostmanNoRest-Assured for Test Automation and Frisby with JavaScript
7Continuous IntegrationJenkinsNoBambooYes
8Service VirtualizationStub-o-matic, MockitoNoHP Service TestYes
9Static Code QualitySonarQubeNoPMDNo
10Code CoverageJACOCONoCoberturaNoConsidering the primary language is Java
11Version ControlGitHub/GitLabYesSVNYes
12Defect ManagementJIRAYesHP ALMYes
13Agile Project ManagementJIRA-AgileYesTrelloNoTrello free version is available with some limitations
14Security TestingVegaNoZED Attach ProxyNo
15Penetration TestingMetasploitNoWiresharkYes
16Build ArtifactsjFrog ArtifactoryNoNexusNo
17CD ToolOpenShiftYesPuppetYes
18Cloud-based Non-Mobile executionAmazon - AWSYesMS AzureYesFor Test Execution and deployment
19Cloud-based Mobile executionSauceLabsYesAmazon Device FarmYes


Considering the multiple iterations over code in the Agile/DevOps software development model, testing is one of the most important phases of software delivery. To fast-track the testing process, automated functional & non-functional test-scripts are recommended. In the ideal scenario, each Agile team may have at least one QA resource who performs the testing activities for the current sprint, as well as ensuring the preservation of existing functionalities through regression tests.