The Importance of Software Testing in DevOps

Software testing is the process of identifying the correctness and quality of a software program. In other words, testing is executing a system or application in order to find software bugs, defects, errors or unexpected behavior.

Software testing is necessary because we all make mistakes. Some of those mistakes are minor, but others can be expensive or dangerous. Especially while practicing continuous integration, continuous delivery, or continuous deployment, we need to test anything and everything we produce, because things can always go wrong.

Testing is mainly classified into two types, Functional Testing and Non-functional Testing.

Functional testing processes test against the specifications, or functional requirements for a system and its components (what the system should do), while non-functional testing processes test the operation of that system (how the system should be).

Common types of Functional Testing

Unit Testing

In Unit Testing, the scope of testing is limited to a particular unit or component—for example, a specific API, database, or GUI.

Tips:

• API: Have separate unit tests for individual web services
• Database: Create tests to add, edit or remove data
• GUI: Focus on field-level validation
• Use branch coverage or statement coverage.
• Add some negative tests
• As a best practice, it is more reliable to use ‘mock’ service(s) to write unit tests, in order to avoid interdependence between components
• Think about data re-usability, and clearing any test data from the database at end of your test (if applicable)

Tools / Frameworks:

The Junit and TestNG unit testing frameworks are popular for a Java-based development environment. For Ruby, RSpec is widely used.

For the front-end, the Karma test runner is commonly used in development environments using AngularJS or Vue.js to run JavaScript-based unit tests. Protractor and the BDD-based Jasmine framework are also widely used. Those using React might prefer Jest, while Ember.js has an integrated unit testing provision. Mocha is yet another JavaScript alternative for front-end unit testing.

Sanity Testing

Sanity testing runs before any detailed functional or regression tests and tests basic, integral functionalities of an application. Sanity testing is required to run every commit to make sure critical business functionalities (such as the launching of the application) are not affected by code changes.

System Integration Testing

System Integration Testing (SIT) will touch multiple units/components and is a black box testing method. Performing this testing should not require knowledge of the internal workings of the system.

Normally within a sprint, manual testing will be the first phase of the SIT. Later on, this will be converted to an automated test script. If there is a dedicated automation testing resource available in the Agile team, then an automation test script can be developed in parallel to development activities. The automated test script will fast-track regression testing prior to each release or every commit in the continuous integration / continuous deployment model.

Tools / Frameworks:

These tools and frameworks are used by the Europe PMC development team.

• Defect tracking tool – JIRA
• Automation Testing tool – Selenium
• Automation Framework – Serenity BDD
• Programming Language – Java
• Unit Testing Framework for Test Automation – JUnit
• Build Tool – Maven
• Schedule and Execution – Jenkins

Automation scripts can be developed in any language, and it is not mandatory for the testing team to use the same technology that the development team uses.

Regression Testing

Regression testing ensures that software performs the same functionality in the same way after it is updated or changed. Regression testing should be performed after any code change to a previously developed and tested program, to ensure that new defects have not been introduced.

User Acceptance Testing (UAT)

UAT is normally conducted by the actual users of a particular system. There will be end-to-end scenarios for the UAT users to execute to verify system functionalities. In the waterfall model, UAT will be conducted only at the end of development, whereas in the Agile model, UAT can be performed at the end of every sprint and feedback given to the team.

Common types of Non-functional Testing

Performance Testing

Software performance testing is used to determine the speed or effectiveness of a computer, network, software program, or device. This process can involve quantitative tests done in a lab, such as measuring the response time or the number of MIPS (millions of instructions per second) at which a system functions. Performance testing (“How fast is the system?”) can also include load testing (“How much volume can the system process?”).

Tools / Frameworks:

Gatling or JMeter

Sample ‘Gatling’ Performance Test Results:

Security Testing

Security testing is a technique to determine if an information system protects data and maintains functionality as intended. Its aims should be verifying three basic principles—confidentiality, integrity, and authentication.

Compatibility Testing

Compatibility testing involves verifying a service’s functionality is the same across various browsers, different browser versions, mobile devices, operating systems, etc.

Usability Testing

Usability testing is used in user-centered interaction design to evaluate a product by testing it on users. This is an irreplaceable testing practice, since it gives direct input on how real users use the system.

Automating your Testing

Test automation is the process of converting manual test scripts to automated test scripts, using testing tools like Selenium, Appium, SoapUI, REST-Assured, Serenity, Cucumber, etc. Test automation will fast-track regression testing, and it avoids repeated and time-consuming manual testing efforts. It is also convenient to use automated testing for compatibility testing.

Sample Automation / Cucumber-BDD-Serenity Test Report:

Benefits of Automated Testing

• Faster feedback
• Earlier Detection of Defects
• Accelerated Results
• Running tests 24/7
• Fewer human resources
• Reusability
• Reliability
• Simultaneity
• Better test coverage with multiple data combinations

Test Flows, Test Management & Execution Frequency

Sample Build Pipeline model along with Software Testing steps

In the above build pipeline model, there are three pre-production servers, i.e. Development (DEV), Testing (TEST), and Staging. Normally the DEV server will be used for all development activities, with the TEST server for functional testing by the quality assurance team, and the Staging server for performance testing and user acceptance testing. On some occasions, there will only be DEV, TEST and PROD (production) servers available. In such cases, internal testing and UAT will happen on the TEST server as well.

Proposed Tools & Framework Summary

#	Description	Preferred ‑ 1	Paid?	Preferred ‑ 2	Paid?	Remarks
1	Automation Framework	Serenity-BDD with Cucumber - Junit	No	Python with Behave	No
2	Front-end Automation	Selenium, Protractor	No	HP-UFT	Yes
3	Mobile Automation	Appium	No	SeeTest	Yes
4	Performance Testing	Gatling, JMeter	No	LoadRunner	Yes
5	SOAP Web Service	SOAP-UI	No	ReadyAPI	Yes	JAX-WS/JAXB concept for Test Automation
6	REST Web Service	SOAP-UI	No	Postman	No	Rest-Assured for Test Automation and Frisby with JavaScript
7	Continuous Integration	Jenkins	No	Bamboo	Yes
8	Service Virtualization	Stub-o-matic, Mockito	No	HP Service Test	Yes
9	Static Code Quality	SonarQube	No	PMD	No
10	Code Coverage	JACOCO	No	Cobertura	No	Considering the primary language is Java
11	Version Control	GitHub/GitLab	Yes	SVN	Yes
12	Defect Management	JIRA	Yes	HP ALM	Yes
13	Agile Project Management	JIRA-Agile	Yes	Trello	No	Trello free version is available with some limitations
14	Security Testing	Vega	No	ZED Attach Proxy	No
15	Penetration Testing	Metasploit	No	Wireshark	Yes
16	Build Artifacts	jFrog Artifactory	No	Nexus	No
17	CD Tool	OpenShift	Yes	Puppet	Yes
18	Cloud-based Non-Mobile execution	Amazon - AWS	Yes	MS Azure	Yes	For Test Execution and deployment
19	Cloud-based Mobile execution	SauceLabs	Yes	Amazon Device Farm	Yes

Conclusion

Considering the multiple iterations over code in the Agile/DevOps software development model, testing is one of the most important phases of software delivery. To fast-track the testing process, automated functional & non-functional test-scripts are recommended. In the ideal scenario, each Agile team may have at least one QA resource who performs the testing activities for the current sprint, as well as ensuring the preservation of existing functionalities through regression tests.

Useful links

• BDD Serenity

• Gatling

Written on 8 March 2018, by Rakesh Nambiar, in Testing