Top 5 Test Automation Tips for Your Test Automation Strategy

Automated testing today plays an essential role in any technology company. Whether it's a mobile or web app or website, code validation is extremely important. Planning correctly when you need automation testing helps to save time and money.

When planning tests, using automation tools helps a lot more than just saving time and cash. These tools make it easier to find and fix problems early on, which means the apps or websites work better and are less likely to have issues later. This early catch-and-fix approach that software automation tools bring to the table makes the final product much better for everyone using it. So, in the big picture, using these tools smartly not only cuts down on effort and expenses but also makes sure what's being built is top-notch.

Automated testing is writing code. It compares the expected scenarios with what the user receives, and then indicates the discrepancies. Automated testing plays an important role in heavy, feature-rich applications. Let's look at the pros and cons of both automated and manual testing below.

Pros of manual testing

• The tester evaluates the product as an ordinary user and can provide the most detailed and understandable report on the functionality.
• The specialist looks at not only the functionality, but also the design, so he can assess the convenience of the site. The program is not.
• It is possible to implement atypical scenarios - a person can find a bug that the program cannot find.
• Manual testing of atypical scenarios is less expensive than automating them.
• The tester can check insignificant changes immediately after their implementation.

Cons of manual testing

• There is a human factor - no matter how professional a tester is, he can make a mistake.
• There is no way to simulate a high load, which means that it is impossible to objectively assess how the site will behave with a large number of users.
• Manual testing takes longer than automated testing.

Pros of automated testing

• The human factor is excluded - the program cannot make mistakes due to carelessness and carelessness.
• High speed of verification: functionality, on testing which a specialist will spend a lot of time, can be processed by the program in seconds.
• The test results report is generated and saved automatically.
• While the autotest is in progress, the technician can do other tasks. Also, tests can be performed outside of working hours according to pre-written scripts.
• Manual testing of large applications is a long and laborious process, while scripts are written only once.
• A test script, written once, can be used in the future with the next project update.

Cons of automated testing

• Automated tests always run strictly according to plan, while with manual tests, the tester pays attention to details and may find unexpected errors.
• With an illiterate approach, there is a risk that the development of autotests can turn into the process of creating an application for testing applications and be seriously delayed.
• Automation requires higher competencies from a specialist than manual testing.

Automation is indispensable in regression testing, when the functionality is re-checked after changes are made (bug fixes). If your goal is to get a quality product at the end, regression testing should be carried out even with the slightest changes in the code.

While the effort required to make small changes is usually minimal, retesting functionality involves a relatively large amount of work. Automation helps out - it allows you to reduce the time for regression testing to a minimum.

If you try to answer in a completely simplified way - for short-term projects with a small number of releases, it will be more profitable to conduct manual testing several times.

The longer the project is, the more frequent releases are expected, the more profitable the use of automated testing becomes.

There are so-called ROI (Return on Investment) calculators. Based on a number of parameters, they help determine at what point automated testing becomes more profitable than manual testing.

It is important to understand here that the calculations will be very approximate, since it is extremely difficult to predict and take into account everything with high accuracy. In addition, automation only replaces some of the manual checks. At the same time, automation itself is the same product where technical debt arises, which must be repaid, and this should also be taken into account.

Code-driven testing.

Testing at the level of software modules, classes and libraries (in fact, automatic unit tests);

Graphical user interface testing.

A special program (build an automation testing framework) allows you to generate user events - keystrokes, mouse clicks, and track the program's reaction to these actions - whether it meets the specification.

Automation of testing API (Application Programming Interface).

Program interface of the program. Interfaces are tested that are designed to interact, for example, with other programs or with the user. Here again, as a rule, special frameworks are used.

There are not many software automation testing strategies. The pyramid proposed by Mike Cohn can help teams create an automation testing strategy successfully.

The most straightforward automation testing techniques that you can come up with is simply to take test cases created for manual testing and automate them at the user interface (GUI) level using tools like Selenium. At the same time, this is the least effective approach. Automated tests at the UI level are slow, vulnerable to any changes, and difficult to maintain.

Mike Cohn's pyramid of automation perfectly illustrates a more efficient approach. The width of each level of the pyramid shows how many tests should be at each level compared to others.

At the lower level of the pyramid are a component, or unit tests. They should make up most of your tests. For example, to test a class that calculates interest on the amount, a unit test is created that passes the interest rate x and balance y. Expected result: the correct calculation of the amount of interest with the desired accuracy.

The middle level of developing an automation testing strategy is occupied by tests that verify business behavior (but not through the GUI!). Such tests are sometimes called API tests. If you use the behavior-driven development (BDD) methodology, your automated tests will be at that level. You may need a fairly large number of tests at this level, but still, there should be fewer than unit tests. Such tests can affect several components at the same time and test the behavior of the product from a business point of view. Example: after calculating the interest on the deposit, the required amount is added to the balance.

The top level of the pyramid is automatic tests that directly affect the user interface. They should be much smaller than the rest. An example of such a test: after calculating the interest, the correct, new amount is displayed in the bank statement.

And, “cherry on a cake” - manual tests. Some types of testing cannot be automated, for example, research testing or usability testing, but ideally, you should try to minimize the number of manual tests.

Some more important principles related to the pyramid of automation:

• Avoid duplicating tests at different levels. It makes no sense to repeat the same tests at different levels. For example, if the boundary values ​​were checked at the unit test level, do not repeat them at the GUI level - this way we are unlikely to get new information.
• In general, where possible, you should strive to shift the tests to a lower level. Suppose checking the calculation of percentages from a negative amount is probably possible at the “average level” or even at the unit test level, so why do this at the user interface level? Automating tests at a lower level is more efficient, it allows earlier detection of defects, saves time and money.

To conclude whether it is advisable to invest in test automation, let's imagine a situation. Let's say there is a certain company X, in which all the specialists have always tested the functionality manually. That is, automation is a kind of experiment, which, according to the idea, should confirm the hypothesis that testing with the help of the program will reduce the time for checking the site and increase the quality of the result at the output. The experiment period is 3 years.

To calculate the return on investment, you need the following data:

• How much is an hour for a specialist who will take care of the automation process.
• How much does an hour of a specialist who is currently engaged in manual testing cost?
• How often regression testing is performed or planned.
• For how many test cases testing is started.
• How long does it take to switch from testing one project to another.
• How long will it take to finalize the documentation for automating a specific test case. It is important to consider possible problems when studying the documentation and diving into the specifics of each project.
• How long does it take to file a bug into a bug list.
• How long does it take to develop one autotest?
• How long does manual testing for 10 cases take?

Let's substitute the conditional values to make the calculation:

Now we can compare the costs of automated and manual testing. Please note: the calculations below are based on an initial approximation and do not claim to be as accurate as possible. We show the approximate labor costs for the described work algorithm - in your company, the numbers may be different.

Automated testing costs

We will use the following formula: Ip = Io + Co + Σ (Ce + Ca + Cm)

Ip - test automation costs

Io - initial investment

Co - test development cost

Σ - planned number of testing cycles

Ce - an estimate of the cost of one-time execution of an automated test cycle

Ca - an estimate of the cost of analyzing the results of the completed cycle

Cm - an estimate of the cost of maintaining automated tests in working order and up-to-date

When calculating, we do not take into account the initial investment - they are not needed, because already existing free technologies (IDE, frameworks) are used and there is no need to invest in additional equipment.

Calculation

The cost of developing automated tests:
10 tests x 2 hours x $17 / h = $340.

Planned number of testing cycles:
3 years x 52 weeks x 2 hours / day = 312 times

The cost estimate for one-time execution of an automated test cycle is zero, because preparation for the testing cycle is not required, and the testing itself does not need additional control from a specialist and takes place completely autonomously.

Estimating the cost of analyzing bugs by a tester:
10 tests x 0.125 hours x $17 / h = $21.25.

Cost of maintaining autotests in working order:
10 tests x 0.3 cycles x 0.5 hours x $17 / h = $25.5.

Let's substitute the data into the formula:
0 + 340 +312 x (0 + 21.25 + 25.5) = $30.481

Therefore, company X needs $30,481 to automate testing.

Automated testing costs

Formula:
Gp = Go + Σ (Gc + Ga + Gm)

Gp - manual testing costs

Go - initial investment

Σ - planned number of testing cycles

Gc - the cost of developing test cases

Ga - estimating the cost of analyzing bugs by a tester

When calculating, we do not take into account the cost of developing a base of test cases for manual testing - it is equal to zero, since a company that has already been involved in testing has this base.

Calculation

Cost of one manual testing cycle:
0.75 hours x 10 tests x $15 / h = $112.5.

The cost of analyzing bugs by a tester:
10 tests x 0.25 hours x $15 / h = $37.5.

The cost of developing a tester's documentation, taking into account possible risks:
10 tests x 0.5 hours x $15 / h = $75.

Let's substitute the data into the formula:
0 + 112.5 + 312 x (0 + 37.5 + 75) = $47.756

Comparing the results of both calculations ($30,481 for automated testing and $47,756 for manual testing), we can conclude that automation is advisable for company X at this stage.

Test automation is not easy, but a very important process for teams that advocate for the high quality of the products being developed. Minimizing the human factor, reducing financial investments, increasing productivity are related results, but automation alone is not enough to achieve them. It is equally important to build effective relationships within the testing team. Successful development!