QA plays a crucial part in product quality and product delivery. In agile development methodologies like scrum, QA needs to work in highly compressed test execution cycles. One of the challenges the QA engineer's face is that the code that worked in previous sprints often gets churned by the new features and bug fixes in each subsequent sprint, increasing the risk of regression. Without test automation it might be hard to achieve sufficient test coverage and provide rapid feedback.
So is test automation a bullet?
Before we jump to the conclusion...we need to understand the following:
WHEN TO AUTOMATE?
Check if your application is automation ready? Below are the questions you need to ask yourself before you rush to automate the tests.
1. Is your application ready for automation?
We need to careful when automating GUI. Never start automating at the start of the project or you might end up re-writing the automation scripts that you have written.
To start automating, you need to have a set of stable and functional features ready. It is wise to start automating when you are sure that these features and the GUI will not change. If you proceed and automate the features that are undergoing continuous changes in GUI then the cost of maintaining the script will be too high. Predicting what will change and what will remain stable is tough to determine when working in an agile environment as we need to embrace and welcome the changes.
Another important thing is not to overlook is the maintenance that is required to keep all your scripts up to date and working. Even after spending hours on programming the automation scripts, your automation scripts might fail when you get a new version of the product. You need to design the scripts in such a way that it will require the smallest amount of maintenance in the future. We should have a plan of how we are going to accomplish that in our 'automation strategy'.
2. Do you have skilled testers in your team?
Your QA team need testers with the skillet to utilize the capabilities of the tool. No tool can help you unless you have skilled resources to implement automation. Your test team should capable of handling challenges that might arise in automating the test suite. Most of the tools demand testers to code. Most of the automation tool vendors provide training and support to the resources else you should consider bringing in an expert from outside. Test Automation is a long term investment. It will take time and expertise in developing and maintaining test automation frameworks and automated scripts.
3. Have you selected the right tool?
Choosing the right tool is crucial to succeed in test automation. There are several factors that need to be considered when selecting a test automation tool. Some of the tools are free, some are expensive.
Several times after few days of initial excitement and after automating few tests the tool often becomes "shelf-ware" and testing returns to Manual.
Here are some factors that you might need to consider before selecting the tool:
- Does the tool fit into current process and infrastructure?
- Does the tool vendor support training and provide good services,
- Do you have the budget for the tool?
- Product features: How does the tool identify the objects? Image based recognition? Object-based recognition?
- Which programming languages does it support? Does the QA team have the knowledge on these languages?
- If using an open source tool: Does the tool have good community support and enough documentation?
- The list goes on…
There are risks of the vendor going out of stock or the tool being acquired or retired. It's a good practice to explore the tool on a small scale with a pilot project and determine if the tool can accomplish what is needed.
We should keep in mind that no tool is perfect, be it an open-source or commercial.we need to consider the above factors and select the tool that serves our purpose and provides good ROI.
WHAT TO AUTOMATE??
Did you say "Everything"?
Then it is an unrealistic and impractical expectation!!
1. AUTOMATE REPETITIVE STUFF!!
Testing software involves a lot of repetitive work which is tedious to do manually. People become bored and make mistakes when doing the same task over and over. People tend to do the same task differently when they are repeating it again and again. A tool will exactly reproduce what it did before, so each time it is run the result is consistent. The regression suite is the ideal candidate for automation.
Repetitive work is excellent job for a computer to do.
2. AUTOMATE THE TESTS THAT ARE HARD TO BE PERFORMED MANUALLY.
Assume you are testing an app which has a feature called ‘People’ that sync's all your contacts from various social media and email accounts. You are testing if the ‘contact count’ displayed is accurate. The test account has 5000+ contacts. It is not feasible count the contacts manually and check if the count displayed is accurate.
Assume your testing an email app and the app crashes frequently when you 'archive' around 30-35 emails in succession(one after another). When developers ask you to replicate this issue, think how tiresome it would be to do it manual. Let's be positive and think that you replicated the issue after few attempts..and the developer fixes it. But that's not the end of the story.... you need to confirmation test if the issue is really fixed..Good luck!
Automation tool eases the life of a QA engineer in such situations and can save a lot of time and effort by automating these annoying tests.
3. AUTOMATE BVT/SMOKE TESTS
BVT(Build Verification tests) are cursory tests that are helpful to determine the stability and testability of the build. These tests need to be run every time a build is deployed. The BVT’s are ideally quick tests that concentrate on core functionality.
Automating BVT's and integrating them to the continuous integration server will ensure that the changes made have not led to regressions. One thing to we need to keep in mind while designing these tests is that the test should be short and should the test should execute in few minutes. There may be multiple commits a day it does not make sense if the automated BVT's run for hours.
Also consider the following while automating:
- Automation is a great option when we have to validate the same feature with large set of different inputs and data (Data-driven testing).
- Load and performance testing should be ideally automated where we test the system’s ability to handle load by creating virtual users.
WHAT NOT TO AUTOMATE?
Do not try to automate the tests that would require human intelligence and intuition.
- Tests which are not executed frequently or would be too cumbersome to automate.
- Tests like exploratory testing which based on intuition and expertise of the tester.
- Tests that need human intervention. An automated test should be able to run and verify the outcome entirely unattended.
- Do not automate the volatile features which will have frequent changes.
Prioritize your automation using the 20-80-20 rule. Automate 20 percent of the scripts that take 80 percent of the execution time and 20 percent of the resource skills.
CAN TEST AUTOMATION REPLACE HUMAN TESTERS?
One of the developer who watched me automate was fascinated and said “This is cool stuff! Now you need not have to do anything, the computer will do everything for you. I feel that computers will replace the testers soon”.
I replied “The day when testers will be replaced by automated tester's is the day when developer's will be replaced by auto-developer or a CEO will be replaced by an auto-CEO ...”
QA experts do not use the word 'test automation'. They rather call it ‘automation checks’ as they feel testing cannot be automated, but checking can be completely automated. Humans have the ability to do things, notice things, and analyze things that computers cannot. Our humanity as testers plays a big role in identifying defects while testing. A computer cannot experience or cannot express emotions.
A computer cannot 'determine if the game is fun or not?' ..... 'if an app has good user experience or a pleasant interface'..... Or to 'take decisions using heuristics' etc. Automated test can only find defects that you already thought of or predicted might happen.
You may ask if the test automation can run the same test more quickly, more accurately and with same consistency every time, is it not a better than a human tester?
No! Because running a test quickly with more precision and consistency does not produce better software but it would definitely reduce the time required to complete testing. Test Automation is probably one of the most useful weapon in tester's inventory. It has its own place in the inventory. The weapon is useless without its wielder. Automated tests require scripts to be written and maintained, this in turn requires a HUMAN.
Automation tools are here to assist the testers not to replace them.
A right mix of manual tests and automated tests are essential to test a software effectively. Automated testing requires a higher initial investment but can yield a higher ROI. Test Automation increases the test team’s efficiency by allowing releases to get to production on schedule. Planning, training the resources, selecting the right tool, understanding the limitations and setting realistic expectations is important for the success of test automation.
Test Automation is certainly not a "Silver bullet".
Please share your thoughts on this!
