When you are in a need to introduce test automation process in your organization or build a test automation framework from scratch there might be many thoughts, ideas and questions running in your mind. Having the right strategy in place together with a supporting solid sustainable framework is not an easy job. You may need to go through all the life cycle phases of product development right from deciding the requirements. Most of the cases test automation fails due to unrealistic goals set by team. Automation is not just QA team’s responsibility. Until and unless we set goals for the entire product team and show value addition, automation is not considered as successful. Today there is no shortage of readily available tools, frameworks and utilities for plug and play. We hardly need to code everything from scratch. Coding/implementation is the easiest part of the automation process, but which may not result what you actually want if your goal/requirement understanding/strategy is not right from the beginning.
To introduce test automation in your organization can be consists of the following process as represented in the below image.
Any automation strategy should start with a predefined goal and time lines. Solid understanding of System under test along with environment, test levels, test types and test data needs helps to come up with automation strategy.
In this article I will be explaining about different automation testing goals which can be set before starting any automation project.
What is the goal behind starting automation? Why it is important to set the goal/outcome expected from automation with timelines? How it will helps in coming up with an automation strategy? How that helps in coming up with automation areas to be targeted? Why automating smoke or regression test cases are just not enough? Below figure have listed down some probable goals which can help you to achieve maximum from your automation process.
Quick feedback - Release readiness is an important aspects of software development. Automation can play crucial role here. This is valuable for the entire team not just the management to take decision. As the name suggested the tests belongs to this category should be small, atomic and fast. Unit/integration testing with mocks, integrated static code checkers and feature based smoke test can be the right fit here. And last but not least everything should be version controlled and integrated with CI/CD environment. So what we need to achieve this goal? Automated unit testing should be part of the process and accessible to all. Same applicable for automated smoke tests. And everything should be version controlled and easily accessible for execution locally in dev/test machines and in appropriate containers/virtual machines dedicated for different build execution. Tests should be packaged so that can be integrated with CI/CD. Workflows should be defined like unit tests execution, smoke tests execution and sharing the reports with everyone in the team. Enough care should be taken for selecting/creating relevant tests which are reliable and faster in execution. Both developers/testers can help each other here to make the added tests valuable. This will surely increase the confidence of the team on each release irrespective of the frequency.
Regression - One of the most common goal of test automation is to support regression testing. Selecting regression tests to be automated is the key here. We should cover the key user journeys here which can be combination of navigation and functionality validation. If we are testing a web application selenium can be ideal tool here but may not be enough if we have 3rd party integrations also to be validated as part of the journey. Combination of selenium with api testing frameworks can be considered for end to end coverage. Another selection criterion for picking regression tests is based on priority of the requirement under tests. Priority can be from customer and/or product owner and we can include those tests in the regression suite need basis for execution. Next criteria can be tests selection based on complexity of the feature under development. For example, if some feature is interacting with so many internal as well as external components the related tests should be considered for regression test suite. We should also not miss tests belongs to features on which more defects are already reported.
Data centric - One of the major advantages of using automation testing is to validate data which takes more time or some time impossible for validate manually. Database testing, ETL testing, migration testing where we need to deal with large volume of data it’s better to add automated checks instead of doing it manually. Let’s take an example of eCommerce application where product catalogs, price books, inventory lists which vary based on location, country and currency. Now the requirements to be tested are data migration from source to eCom backend and then validate the same in the front end. This may need multiple layers of automated checks like source and destination data comparison, location/country-based data validation which cannot be achieve with manually. So, this can be a perfect scenario to be considered for automation. Here single tool or framework may not help, but carefully chosen approach of heterogeneous framework having multiple capabilities will help. Search/Filter results with large combinations is another area to be targeted for automation. Similarly reports with different formats and with dynamic data contents can be another one.
Configuration - Why do we need to consider configuration items under automation. Real world applications are always not developed for single platform. So, we need to consider this aspect as well while planning for automation. For example, testing web application browsers/platform to be supported is a major requirement like windows with chrome, Firefox, IE, Edge, Linux with chrome and Firefox, mac with safari and chrome etc. We may need to support for device platforms like Android with Chrome or iOS with Safari. When devices come into picture, we may need to consider the version and varieties as well like android 5, 6 and 7 or iOS 8 and 9 (iPad/iPhone). So, it will helpful if we prepare a browser/platform/version matrix in the beginning. If its only desktop browsers need to be supported Selenium might me enough but if device browsers are also needed, we need to consider Appium in this case. Another important configuration item that comes under automation is backup and restore of data items. This is needed for deployment and replicating environments. Similarly, installation, uninstallation and upgration via automation should also be considered. Test data needed by the entire team should be prepared in an automated way so that entire team can use consistent data for development and testing from one source.
Non-functional - Non-functional areas like performance, security, load, stability comes under automation umbrella. Those tests are always done with tools/frameworks for the kind of results accuracy in results we expect and volume of test actions/data needs to be performed. If non-functional tests are your automation goal, make sure you have right kind of tools and frameworks setup before starting.
Localization - Another test area which can be considered for automation is localization. When we test an application for multiple languages, the common challenges can be wrong translation, font issues, text overlapping, layout corruption due to size of text etc. Manually testing all those is time consuming which can be solved using tools. Another area here testing multiple versions of the application based on location/country which also can be solved using automation.
I would like to know what is your short/long time automation goal and success stories. Please share your experience and comments on the same line.