Augmenting ROI en route QA Automation

In addition to higher efficiency and accuracy of the testing process, QA automation has often been associated with increased ROI via better speed, superior quality, reduced cost and optimal utilization of resources. However, automation cannot be applied indiscriminately. Being a strategic decision, it requires a fair amount of understanding as to what it can or cannot offer to your project. Project teams and QA decision makers must evaluate the potential ROI before making a final commitment to invest in test automation.

Once the groundwork is done, and you have set your foot on the automation playground, the initial step is to get acquainted with the factors that can help you reap the desirable ROI from the process.

Identify What to Automate: To identify this, you need to study and analyze factors like the application under test, requisite test environment, hardware dependencies and so on. Tasks that are recurring and mandatory in nature, or functions that remain stable for the most part of the development lifecycle, or tests that have complex business logic constitute some of the ideal test cases for mechanization.

Moreover, the important question is whether, in an attempt to accrue the benefits of automation, can we automate each and every layer of the app and its related components, and be done away with the human element? The answer is a crisp ‘No’. Why?

Here are a few good reasons :

  • Even Automation Test scripts need to be written by testers/humans
  • Scripts cannot provide inputs on how a UI looks and feels
  • Automated tests can only notice the errors they are programmed to find
  • Automation can prove to be a costly affair for smaller projects
  • Only skilled, wise testers can choose apt automation tools or the effort can be a sheer waste

Hence, both manual and automated testing are crucial for a successful app to be developed in a given time and budget, but yes, automation can increase the benefits manifold when applied aptly. Hence, choose wisely what you really need to automate.

Task Prioritization: Once the test cases for automation have been identified, it is imperative to start prioritizing and dividing tasks as per the expected timelines. It is equally important to consider team skill-set and accordingly finalize the automation tasks.

Prepare Test Cases & Identify Scenarios Ahead: This step assists in identifying the estimated execution time and possible defects well in advance, in addition to identifying other scenarios and application workflows that can also be automated but could have been otherwise missed.

Identify the Test Process Beforehand, and Create Tests That Do Not Affect UI Changes: Identifying Test Process before going for automation ensures optimal allocation of resources and early risk analysis. This assists in the determination of best possible methods to perform the task efficiently. The automation scripts must, however,  be easy to maintain and update. Finally, tests should be so written as to provide smooth and reliable results even when the UI changes.

Ensure Well-designed Test Data: Test data that is fresh/updated, valid, and reliable results in successful automation endeavors. Hence, make sure that you have quality data – optimal size data with information and components relevant to app testing environment.

Identify Apt Test Tools, Frameworks and Techniques: To obtain the desired output from the test automation project, you need to choose the right test tool- emulators or real devices, make the buying decision- buy the device or use the Cloud services, and select the right automation framework or technique for a particular app type or test scenario.

Periodic Test Review: Periodic review of your test data and cases is essential for eliminating unnecessary and outdated tests resulting in reduced costs and maintenance efforts.

Hence, automation, when implemented aptly, changes the entire test environment from design to deployment for good. Speedy and efficient delivery, better visibility into your app performance, detection of more bugs, and cost effectiveness are some of the hallmarks of automation that can be expected when you tread the automation journey in sync with the above guidelines. So automate, but automate wisely!

Emerging Trends in QA & Testing

The software testing space has undergone a significant transformation with technology advancing at a rapid pace. Big Data, Virtualization, and Cloud-based applications are evolving speedily along with the hyper-connected devices being our future. Besides, trends like mobile app testing, crowdsourced testing, and context-driven testing have completely reframed the testing and development landscape.

On top of this, tough competition has created pressure on the testing teams to manage faster product releases without losing focus on superior quality. This has led traditional testing methods to take a backseat and latest QA & testing trends to rise up to the challenge.

Let’s observe the latest QA trends that are currently influencing the market.

Increased Automation Levels

In response to the deployment velocity requirements and need to ensure wider coverage, testing teams are adopting automation, wherever possible. This trend is set to rise. The test execution turnaround time and bug detection time will require a more robust and innovative test automation strategy to survive future need for speed and quality.

Agility & DevOps Will be the Norm

Agility as a concept has been there for quite a while now, however, in terms of its application, agility in testing is still not highly evolved, and so is DevOps. Nevertheless, with delivery cycles getting shorter, traditional models are taking a backseat, and businesses with an application of Agility and DevOps in their true sense will be the future frontrunners, with CI and CD becoming central components in the application lifecycle management process.

TCoE’s Will Grow in Numbers

The burgeoning business requirements for speed, quality and cost-effectiveness have led testing companies to set up CoE’s, which are likely to rise in number in near future. Their aim is to establish highly standardized QA and testing practices that deliver near zero-defect apps to the clients and contribute towards a positive shift in the organizational culture. This is must in order to provide the requisite competitive edge to the companies in the current market scenario.

Security Will be a Big Concern

With the growth in cloud computing, mobility and IoT developments, focus on end to end security solutions will be paramount. Sensitive and confidential online data is highly vulnerable to cyber-attacks, necessitating the companies to dig deeper and avoid any leaks, code errors, and holes. Open source security tools will be in demand and security testing may evolve as a separate specialization dealing with the continuously varying nature and severity of security attacks.

Context-Driven Testing Will Rise

The trend is emerging slowly but is likely to rise. Higher levels of diversity and device integration (which is likely to increase further) is making it a complex task for QA teams to set up a single testing strategy, thus demanding context changes to be accounted for in addition to ensuring wider test coverage from varied angles, necessitating the context-driven testing.

Crowdsourced Testing Will Witness a Surge

Sophisticated software has its own development and testing expenses. And in the current complex scenario, companies may not be well-equipped with all the requisite testing resources and may lack budget required to test the software in varied environments. This leads to the demand for crowdsourced testing which can help companies manage their costs, and ensure testing quality. This trend is gaining momentum and is likely to rise in future with multifaceted testing requirements.

Manual Testing Will Always Remain in Demand

Though Automation will be critical to faster product releases, manual testing will remain an integral part of software testing. The wisdom, judgment, and experience of testers can never be replaced by automation. Nevertheless, testers will be required to learn additional technical skills to remain competent.

Concluding Thoughts

These emerging trends will help you get prepared for upcoming testing challenges. With your readiness to learn, grow and adopt these trends; you may plan, strategize and navigate your future testing processes in an efficient manner. And these trends are likely to accelerate with Virtualization, Predictive Analytics, and Machine Learning. So keep a watch on the latest developments to remain competitive. Happy testing!

Avoiding the Pesticide Paradox in Software Testing

Boris Beizer defined the Pesticide Paradox as: “Every method you use to prevent or find bugs leaves a residue of subtler bugs against which those methods are ineffectual.”

This simply means that as the same test suite is run multiple times, it become ineffective in catching bugs. And moreover, these test sets will also fail to catch parts of new bugs introduced into the system with recurring enhancements and fixes.

And with agility gaining momentum- speed to market is becoming the decisive factor for getting a competitive edge, making the implications of this paradox all the more relevant. As we add automated testing into our mix of testing methods, we start relying on these tests for eternity. We keep running them frequently and review sparingly. If you are a tester, you certainly know what it means to get attached to the tests you have added to the test suite and fall into the illogicality of complete reliance on the same set of tests over time. This simply means the invisible bugs will be left unattended, only to be caught later in the SDLC or will get passed into the release- a faux pas leading to loss of credibility and revenue.                  
                                                                Source: IBM

In order to avoid these bugs from getting released or to be caught later, causing big losses; there is a need for constant maintenance and updating of the test suites, whether automated or manual.

But how should one go about making the tests relevant?

Constantly monitor changes

Tester’s ability to make all structural and functional connections to identify new scenarios and update the existing test cases will increase the test coverage and support new functionality, and hence, increasing the chances to find new defects.

Track the bug statistics regularly

This will help you with a clear understanding of how effective your tests have been. If a test has not reported a bug in last few runs, you need to check if the test is worth moving into archives section. This needs responsiveness to regular test feedbacks by continually renewing/reviewing the tests, and a sharp eye on your tests in the test suite to remove useless test cases, that may be piling up.  So revisit, revise and renew often, and change your test data.

Build variance into tests

This can be done in the design phase itself, where models can be designed to create different paths through or to the feature under test. Additional data may be created to add-on to the alternative flows.  This aim is to ensure that the feature is fully exercised in different ways.  It is certainly easy to create the additional set of data when you are in the process rather than reviewing the complete set of tests. So yes, this is a good way to avoid pesticide paradox.

Go for Exploratory Testing

Exploratory tests are not identified in advance. In absence of dependency on the scripted tests, exploratory testing involves the breadth and depth of the tester’s imagination and his knowledge about the product. This, in turn, helps in finding more bugs than normal testing and can cover various scenarios and cases normally ignored. After all, mechanized processes cannot think, but testers can.  The human element should thus be incorporated to enhance the testing effectiveness, and escape from the trap of repeating same automated tests again and again.

Conclusion

To recap, there is no “foolproof testing suite” that can discover all the bugs without the need for any modification. If you rely on anyone for eternity, you will never know how worn out your test suite is- only to result in a miserable product release.

A better way is thus to keep a tab on changes and review the suit regularly by adding more scenarios and cases, as required. Additionally, a hawk-eye on bug statistics will let you know the effectiveness of your test suite. Besides, you can keep adding the extra set of test data with alternative paths within the build phase, to imbibe variance in the tests.  Finally, human element and intelligence should be added to the testing process, as exploratory tests can help find out bugs through cases and scenarios that the system is unable to identify.

Moreover, it does not harm to do some contemplation or take peer reviews, or even start out fresh if there is a major change in the component. This will, in turn, help to control the impact of pesticide paradox, and though, there is no guarantee that all bugs can be caught this way, but definitely, a better and efficient outcome can be expected.