The Importance of Software Testing in DevOps
Software testing is the process of identifying the correctness and quality of a software program. In other words, testing is executing a system or application in order to find software bugs, defects, errors or unexpected behavior.
Software testing is necessary because we all make mistakes. Some of those mistakes are minor, but others can be expensive or dangerous. Especially while practicing continuous integration, continuous delivery, or continuous deployment, we need to test anything and everything we produce, because things can always go wrong.
Testing is mainly classified into two types, Functional Testing and Non-functional Testing.
Functional testing processes test against the specifications, or functional requirements for a system and its components (what the system should do), while non-functional testing processes test the operation of that system (how the system should be).
Common types of Functional Testing
- API: Have separate unit tests for individual web services
- Database: Create tests to add, edit or remove data
- GUI: Focus on field-level validation
- Use branch coverage or statement coverage.
- Add some negative tests
- As a best practice, it is more reliable to use ‘mock’ service(s) to write unit tests, in order to avoid interdependence between components
- Think about data re-usability, and clearing any test data from the database at end of your test (if applicable)
Tools / Frameworks:
Sanity testing runs before any detailed functional or regression tests and tests basic, integral functionalities of an application. Sanity testing is required to run every commit to make sure critical business functionalities (such as the launching of the application) are not affected by code changes.
System Integration Testing
System Integration Testing (SIT) will touch multiple units/components and is a black box testing method. Performing this testing should not require knowledge of the internal workings of the system.
Normally within a sprint, manual testing will be the first phase of the SIT. Later on, this will be converted to an automated test script. If there is a dedicated automation testing resource available in the Agile team, then an automation test script can be developed in parallel to development activities. The automated test script will fast-track regression testing prior to each release or every commit in the continuous integration / continuous deployment model.
Tools / Frameworks:
These tools and frameworks are used by the Europe PMC development team.
- Defect tracking tool – JIRA
- Automation Testing tool – Selenium
- Automation Framework – Serenity BDD
- Programming Language – Java
- Unit Testing Framework for Test Automation – JUnit
- Build Tool – Maven
- Schedule and Execution – Jenkins
Automation scripts can be developed in any language, and it is not mandatory for the testing team to use the same technology that the development team uses.
Regression testing ensures that software performs the same functionality in the same way after it is updated or changed. Regression testing should be performed after any code change to a previously developed and tested program, to ensure that new defects have not been introduced.
User Acceptance Testing (UAT)
UAT is normally conducted by the actual users of a particular system. There will be end-to-end scenarios for the UAT users to execute to verify system functionalities. In the waterfall model, UAT will be conducted only at the end of development, whereas in the Agile model, UAT can be performed at the end of every sprint and feedback given to the team.
Common types of Non-functional Testing
Software performance testing is used to determine the speed or effectiveness of a computer, network, software program, or device. This process can involve quantitative tests done in a lab, such as measuring the response time or the number of MIPS (millions of instructions per second) at which a system functions. Performance testing (“How fast is the system?”) can also include load testing (“How much volume can the system process?”).
Tools / Frameworks:
Sample ‘Gatling’ Performance Test Results:
Security testing is a technique to determine if an information system protects data and maintains functionality as intended. Its aims should be verifying three basic principles—confidentiality, integrity, and authentication.
Compatibility testing involves verifying a service’s functionality is the same across various browsers, different browser versions, mobile devices, operating systems, etc.
Usability testing is used in user-centered interaction design to evaluate a product by testing it on users. This is an irreplaceable testing practice, since it gives direct input on how real users use the system.
Automating your Testing
Test automation is the process of converting manual test scripts to automated test scripts, using testing tools like Selenium, Appium, SoapUI, REST-Assured, Serenity, Cucumber, etc. Test automation will fast-track regression testing, and it avoids repeated and time-consuming manual testing efforts. It is also convenient to use automated testing for compatibility testing.
Sample Automation / Cucumber-BDD-Serenity Test Report:
Benefits of Automated Testing
- Faster feedback
- Earlier Detection of Defects
- Accelerated Results
- Running tests 24/7
- Fewer human resources
- Better test coverage with multiple data combinations
Test Flows, Test Management & Execution Frequency
Sample Build Pipeline model along with Software Testing steps
In the above build pipeline model, there are three pre-production servers, i.e. Development (DEV), Testing (TEST), and Staging. Normally the DEV server will be used for all development activities, with the TEST server for functional testing by the quality assurance team, and the Staging server for performance testing and user acceptance testing. On some occasions, there will only be DEV, TEST and PROD (production) servers available. In such cases, internal testing and UAT will happen on the TEST server as well.
Proposed Tools & Framework Summary
|#||Description||Preferred ‑ 1||Paid?||Preferred ‑ 2||Paid?||Remarks|
|1||Automation Framework||Serenity-BDD with Cucumber - Junit||No||Python with Behave||No|
|2||Front-end Automation||Selenium, Protractor||No||HP-UFT||Yes|
|4||Performance Testing||Gatling, JMeter||No||LoadRunner||Yes|
|5||SOAP Web Service||SOAP-UI||No||ReadyAPI||Yes||JAX-WS/JAXB concept for Test Automation|
|8||Service Virtualization||Stub-o-matic, Mockito||No||HP Service Test||Yes|
|9||Static Code Quality||SonarQube||No||PMD||No|
|10||Code Coverage||JACOCO||No||Cobertura||No||Considering the primary language is Java|
|12||Defect Management||JIRA||Yes||HP ALM||Yes|
|13||Agile Project Management||JIRA-Agile||Yes||Trello||No||Trello free version is available with some limitations|
|14||Security Testing||Vega||No||ZED Attach Proxy||No|
|16||Build Artifacts||jFrog Artifactory||No||Nexus||No|
|18||Cloud-based Non-Mobile execution||Amazon - AWS||Yes||MS Azure||Yes||For Test Execution and deployment|
|19||Cloud-based Mobile execution||SauceLabs||Yes||Amazon Device Farm||Yes|
Considering the multiple iterations over code in the Agile/DevOps software development model, testing is one of the most important phases of software delivery. To fast-track the testing process, automated functional & non-functional test-scripts are recommended. In the ideal scenario, each Agile team may have at least one QA resource who performs the testing activities for the current sprint, as well as ensuring the preservation of existing functionalities through regression tests.