In part 1 of How to optimise test automation on Agile projects, I shared some key points about automation test objectives and selecting suitable automation tools.
In part 2, I move on to cover the key things we consider while creating and running automation tests at Triad Group Plc for our customers across the private and public sector.
Create test automation sub-tasks
We always advise clients to create test automation sub-tasks under the user stories, so that the automation test scripts can be implemented at the end of the current iteration, or at the beginning of the next. In addition, this will help in tracking the progress of automation tests and the test coverage.
Building automation tests
Here are a few things to consider when creating automation test scripts:
Reusable and maintainable test scripts
Reusability is one of the key reasons for considering test automation. So, consider this aspect in your automation test strategy, as not only does the reuse of test scripts speed up the automation process it is also a major contributor to making the process cost-effective.
Ideally, you will create a reusable framework that can be optimised for testing across other device platforms or for other projects, giving due consideration to concerns like maintainability and execution time.
The automation test pyramid
As part of a test automation strategy for an Agile project, we always try to implement an automation test pyramid, as it sets clear expectations and helps increase the test coverage.
Automated Unit Testing – 70%
- Unit tests are written by developers for any new feature that is developed. Test automation starts at the unit level. These tests will test each testable unit separately.
- Unit tests can be run on the developers’ local environment as well as the CI/CD environment.
- Unit testing provides the greatest ROI for the team as they are very quick to run, easy to maintain and modify, and give quick feedback to the developer about the quality of the code.
Automated API / integration testing – 20%
- Integration testing forms the next level up from unit tests to test the functionality of each service (API) independently. These tests are executed at the API layer independently of the User Interface (UI).
- Mocks and stubs will be used to factor out the dependence on other services or 3rd party systems.
- These tests are executed once Unit tests have been run and passed.
- Integration tests can be created by both Developers and Automation Testers.
- These tests are fast to execute as they do not depend on UI and can be run in local environments and in the CI/CD environment.
User Interface (UI) automation testing – 10%
- The UI automation tests cover typical user flows, journeys, and end-to-end scenarios.
- These automation tests simulate a user’s interaction with the application through the UI which provides good and meaningful scenarios, however, it is prone to issues such as: a dependency on html locators, slow page load-times, limited testing, etc. So, it is a good strategy to keep the number of these tests to a minimum.
Version control
Always try to employ the same version control tools as the developers. It keeps the automation testing in line with the project tech stack and the synchronisation helps avoid mistakes.
All unit tests and integration tests can, and ideally should, live in the main repository with the application. UI automation tests may be better held in a separate remote repository as these tests depend on third-party tools.
Running automation tests in CI/CD pipelines
A CI/CD pipeline helps to automate steps in the software delivery process, such as initiating code builds, running automated tests and deploying to different environments such as integration, staging and production.
A pipeline procedure is triggered when code is committed to a repository hosted somewhere like GitHub or Bitbucket. This pipeline can build the application, execute unit tests and report the results.
Once unit tests have run successfully, it is deployed to an integration (system test) environment where automated integration (API) tests and UI smoke tests can be executed. Once all tests have passed, then a build will be triggered to deploy it to the staging (pre-production) environment where a full set of regression tests are run.
Reporting results
Even with good test scripts, it is difficult to find bugs through automation if there is no adequate reporting available. Clear and comprehensive reporting helps us to reach a conclusion after script execution has completed.
The reporting format might be different for each tool, but should be easily understood. It also helps to include screenshots in test reports. There are many third party reporting tools available in the market.
Conclusion
The main challenge for the QAs on any Agile project is the amount of regression testing that needs to be performed before each release. This can be significantly reduced and other benefits secured using automation testing as described above.
You can magnify these benefits by always trying to integrate your test automation strategy into your Software Development Life Cycle (SDLC), including different levels of testing such as unit, integration and UI.
Ensure you think about the test automation pyramid while creating your test scripts as creating more UI tests will make your automation slow, it will become out dated quickly, and be difficult to maintain in a long run.
Test automation is a long term investment but opportunities exist throughout the process and its different facets. For example, you can get a quick ROI when you run your test scripts in CI/CD pipelines as part of the release process.
Selecting tools that fit into the project tech stack and the way the team works will help you realise those benefits much faster; while thinking about reusability and maintainability whilst developing your strategy through to your test scripts is key. Finally, without clear and comprehensive reporting against objectives agreed on by the whole team, it’s hard to convince others of the scale of benefits automated testing can bring to Agile projects, which are immense when implemented correctly.
Venu Botla
Senior consultant, Triad Group Plc
