Software testing is used to evaluate the product and understand whether it really matches the declared functionality.
The quality assurance process in software testing provides the product owner with information about its condition and whether all requirements are taken into account during the development process.
In this article, we want to share with you 5 tips and a strategy for improving the optimization of the software testing process.
Automation test planning
Automated testing today plays an essential role in any technology company. Whether it's a mobile or web app or website, code validation is extremely important. Planning correctly when you need automation testing helps to save time and money.
Automated testing is writing code. It compares the expected scenarios with what the user receives, and then indicates the discrepancies. Automated testing plays an important role in heavy, feature-rich applications. Let's look at the pros and cons of both automated and manual testing below.
Pros of manual testing
The tester evaluates the product as an ordinary user and can provide the most detailed and understandable report on the functionality.
The specialist looks at not only the functionality, but also the design, so he can assess the convenience of the site. The program is not.
It is possible to implement atypical scenarios - a person can find a bug that the program cannot find.
Manual testing of atypical scenarios is less expensive than automating them.
The tester can check insignificant changes immediately after their implementation.
Cons of manual testing
There is a human factor - no matter how professional a tester is, he can make a mistake.
There is no way to simulate a high load, which means that it is impossible to objectively assess how the site will behave with a large number of users.
Manual testing takes longer than automated testing.
Pros of automated testing
The human factor is excluded - the program cannot make mistakes due to carelessness and carelessness.
High speed of verification: functionality, on testing which a specialist will spend a lot of time, can be processed by the program in seconds.
The test results report is generated and saved automatically.
While the autotest is in progress, the technician can do other tasks. Also, tests can be performed outside of working hours according to pre-written scripts.
Manual testing of large applications is a long and laborious process, while scripts are written only once.
A test script, written once, can be used in the future with the next project update.
Cons of automated testing
Automated tests always run strictly according to plan, while with manual tests, the tester pays attention to details and may find unexpected errors.
With an illiterate approach, there is a risk that the development of autotests can turn into the process of creating an application for testing applications and be seriously delayed.
Automation requires higher competencies from a specialist than manual testing.
Automation is indispensable in regression testing, when the functionality is re-checked after changes are made (bug fixes). If your goal is to get a quality product at the end, regression testing should be carried out even with the slightest changes in the code.
While the effort required to make small changes is usually minimal, retesting functionality involves a relatively large amount of work. Automation helps out - it allows you to reduce the time for regression testing to a minimum.
How do you know when to use automation testing?
If you try to answer in a completely simplified way - for short-term projects with a small number of releases, it will be more profitable to conduct manual testing several times.
The longer the project is, the more frequent releases are expected, the more profitable the use of automated testing becomes.
There are so-called ROI (Return on Investment) calculators. Based on a number of parameters, they help determine at what point automated testing becomes more profitable than manual testing.
It is important to understand here that the calculations will be very approximate, since it is extremely difficult to predict and take into account everything with high accuracy. In addition, automation only replaces some of the manual checks. At the same time, automation itself is the same product where technical debt arises, which must be repaid, and this should also be taken into account.
5 Test Automation Tips of Successful Test Automation
Practice a quality defect description and work planning.
Competent bug tracking and test management add to the overall value of the software automation testing process by improving the individual skills of each tester, timely overcoming difficulties that can slow down the testing process, and building interaction between development and testing teams.
Describe the software testing process appropriate for your team.
To ensure effective testing as a type of software quality assurance, it is important to create a pre-established process. It should be debugged and approved by experts who know how business processes in the company go so that sudden difficulties do not affect the testing process. Despite the fact that unconditional adherence to the established process is not necessary, it may be useful in the early stages of testing. The description can be enhanced by adding preconditions for performing product tests.
Tune in to the irremissible bug detection.
For testing to be effective, the tester must tune in to a positive result - i.e. the presence of at least a few bugs - before starting work. This will inspire him to carefully search for defects and use the ability to correctly describe them in practice. Believe that the application has a bunch of different bugs and errors, and at the end of testing you will definitely find them all.
Build a proper testing environment.
Make sure that the environment in which the product is tested adequately reflects the environment in which it goes into mass use - the duty of the testing team. This is necessary to ensure that no bugs were missed due to problems in organizing the testing process.
Delve into the core of issues.
To make changes and improvements to the test plan, based on the bugs found by customers, the tester must constantly delve into their details. The tester’s reputation depends on this: he has to try to demonstrate the presence/absence of the stated problem. There is one more reason why experienced testers themselves often study projects for bugs: by doing this, you can better understand the structure of the product, understand who its potential user is, how and why it will use the product, what problem it should solve and how the product should look and work, which in the end will help to test it in more detail and with high quality.
The main types of automated testing
Testing at the level of software modules, classes and libraries (in fact, automatic unit tests);
Graphical user interface testing.
A special program (build an automation testing framework) allows you to generate user events - keystrokes, mouse clicks, and track the program's reaction to these actions - whether it meets the specification.
Automation of testing API (Application Programming Interface).
Program interface of the program. Interfaces are tested that are designed to interact, for example, with other programs or with the user. Here again, as a rule, special frameworks are used.
How to develop an automation testing strategy?
There are not many software automation testing strategies. The pyramid proposed by Mike Cohn can help teams create an automation testing strategy successfully.
The most straightforward automation testing techniques that you can come up with is simply to take test cases created for manual testing and automate them at the user interface (GUI) level using tools like Selenium. At the same time, this is the least effective approach. Automated tests at the UI level are slow, vulnerable to any changes, and difficult to maintain.
Mike Cohn's pyramid of automation perfectly illustrates a more efficient approach. The width of each level of the pyramid shows how many tests should be at each level compared to others.
At the lower level of the pyramid are a component, or unit tests. They should make up most of your tests. For example, to test a class that calculates interest on the amount, a unit test is created that passes the interest rate x and balance y. Expected result: the correct calculation of the amount of interest with the desired accuracy.
The middle level of developing an automation testing strategy is occupied by tests that verify business behavior (but not through the GUI!). Such tests are sometimes called API tests. If you use the behavior-driven development (BDD) methodology, your automated tests will be at that level. You may need a fairly large number of tests at this level, but still, there should be fewer than unit tests. Such tests can affect several components at the same time and test the behavior of the product from a business point of view. Example: after calculating the interest on the deposit, the required amount is added to the balance.
The top level of the pyramid is automatic tests that directly affect the user interface. They should be much smaller than the rest. An example of such a test: after calculating the interest, the correct, new amount is displayed in the bank statement.
And, “cherry on a cake” - manual tests. Some types of testing cannot be automated, for example, research testing or usability testing, but ideally, you should try to minimize the number of manual tests.
Some more important principles related to the pyramid of automation:
Avoid duplicating tests at different levels.
It makes no sense to repeat the same tests at different levels. For example, if the boundary values were checked at the unit test level, do not repeat them at the GUI level - this way we are unlikely to get new information.
In general, where possible, you should
strive to shift the tests to a lower level.
Suppose checking the calculation of percentages from a negative amount is probably possible at the “average level” or even at the unit test level, so why do this at the user interface level? Automating tests at a lower level is more efficient, it allows earlier detection of defects, saves time and money.
Is it worth investing in test automation?
To conclude whether it is advisable to invest in test automation, let's imagine a situation. Let's say there is a certain company X, in which all the specialists have always tested the functionality manually. That is, automation is a kind of experiment, which, according to the idea, should confirm the hypothesis that testing with the help of the program will reduce the time for checking the site and increase the quality of the result at the output. The experiment period is 3 years.
To calculate the return on investment, you need the following data:
How much is an hour for a specialist who will take care of the automation process.
How much does an hour of a specialist who is currently engaged in manual testing cost?
How often regression testing is performed or planned.
For how many test cases testing is started.
How long does it take to switch from testing one project to another.
How long will it take to finalize the documentation for automating a specific test case. It is important to consider possible problems when studying the documentation and diving into the specifics of each project.
How long does it take to file a bug into a bug list.
How long does it take to develop one autotest?
How long does manual testing for 10 cases take?
Let's substitute the conditional values to make the calculation:
Remuneration for the work of a specialist who is engaged in the automation process
Remuneration for the labor of a specialist who is currently engaged in manual testing
Regression Testing Frequency
Every day every 2 hours
The number of test cases for which testing is started
The time it takes to switch a specialist from testing one project to another
Time required to finalize the documentation for automating a specific test case
Time to file a bug into a bug list
Time to develop one autotest
Average duration of manual testing for 10 cases
Now we can compare the costs of automated and manual testing. Please note: the calculations below are based on an initial approximation and do not claim to be as accurate as possible. We show the approximate labor costs for the described work algorithm - in your company, the numbers may be different.
Automated testing costs
We will use the following formula: Ip = Io + Co + Σ (Ce + Ca + Cm)
Ip - test automation costs
Io - initial investment
Co - test development cost
Σ - planned number of testing cycles
Ce - an estimate of the cost of one-time execution of an automated test cycle
Ca - an estimate of the cost of analyzing the results of the completed cycle
Cm - an estimate of the cost of maintaining automated tests in working order and up-to-date
When calculating, we do not take into account the initial investment - they are not needed, because already existing free technologies (IDE, frameworks) are used and there is no need to invest in additional equipment.
The cost of developing automated tests:
10 tests x 2 hours x $17 / h = $340.
Planned number of testing cycles:
3 years x 52 weeks x 2 hours / day = 312 times
The cost estimate for one-time execution of an automated test cycle is zero, because preparation for the testing cycle is not required, and the testing itself does not need additional control from a specialist and takes place completely autonomously.
Estimating the cost of analyzing bugs by a tester:
10 tests x 0.125 hours x $17 / h = $21.25.
Cost of maintaining autotests in working order:
10 tests x 0.3 cycles x 0.5 hours x $17 / h = $25.5.
Let's substitute the data into the formula:
0 + 340 +312 x (0 + 21.25 + 25.5) = $30.481
Therefore, company X needs $30,481 to automate testing.
Manual testing costs
Automated testing costs
Gp = Go + Σ (Gc + Ga + Gm)
Gp - manual testing costs
Go - initial investment
Σ - planned number of testing cycles
Gc - the cost of developing test cases
Ga - estimating the cost of analyzing bugs by a tester
When calculating, we do not take into account the cost of developing a base of test cases for manual testing - it is equal to zero, since a company that has already been involved in testing has this base.
Cost of one manual testing cycle:
0.75 hours x 10 tests x $15 / h = $112.5.
The cost of analyzing bugs by a tester:
10 tests x 0.25 hours x $15 / h = $37.5.
The cost of developing a tester's documentation, taking into account possible risks:
10 tests x 0.5 hours x $15 / h = $75.
Let's substitute the data into the formula:
0 + 112.5 + 312 x (0 + 37.5 + 75) = $47.756
Comparing the results of both calculations ($30,481 for automated testing and $47,756 for manual testing), we can conclude that automation is advisable for company X at this stage.
Test automation is not easy, but a very important process for teams that advocate for the high quality of the products being developed. Minimizing the human factor, reducing financial investments, increasing productivity are related results, but automation alone is not enough to achieve them. It is equally important to build effective relationships within the testing team. Successful development!