Toughts about… The Purpose of Testing (Part 2)

As described in part 1 class tests are a great help for you as a developer. They document the intended behaviour of your code, being an insurance for changes and enable you to do test driven development. But they only verify that one isolated class does what it should do. Changing the way a class handles a parameter, won't break the test of a class that relies on the way it handled the parameter before. Your software may also rely on a certain environment, like an application server, load balancers, a web server etc. We need some kind of test telling us if the software as a whole system works as it was meant to. A system test.

System Tests

The ISTQB Glossary of Testing Terms defines "System Testing" rather shortly as

The process of testing an integrated system to verify that it meets specified requirements. [Hetzel]

So system tests test a whole system with all its modules and components working together. To do that, the software must be installed on a test system. That system needs to be comparable to the production system (an identical one is ideal of course).

There are various types of system tests:

  • feature tests, to verify that a feature works as specified,
  • performance tests, to test the system's behaviour under a certain work load,
  • security tests, to check for security issues,

There are just as many types of them as there are types of requirements to your software.

A system test should not rely on any knowledge about the inner workings of the software (treating it as a "black box"). This may be inconvenient sometimes but directly interacting with inner components would make the test results less accurate and the tests would depend on the used inner architecture of the system. Real black box tests specify what the system should do but not how and therefore stay valid even if the system is partly or even completely redesigned.

This way system tests are as good for ensuring system level changes in a way like class tests are for class level changes. As class tests are an executable documentation for the class, system tests are an executable specification for the system.

Requirements

To be able to create real system tests there are certain requirements that need to be met. The main requirement to be able to do automated system tests is the test system. It needs to be like the production system. For a feature test a remotely similar system may suffice while a stress test without identical hardware is almost pointless.

After we got our test system, we need to install the software and keep it up to date. This may require some copying of files to remote systems, starting and stopping of services etc. I suggest using a script triggered by your continuous integration to ensure this. It may also be a good idea to use some kind of DevOps technology (like Puppet or Chef) to manage your test systems and ensure its configuration and the installed software. Also, this might help to ensure needed test data by triggering SQL scripts.

Purpose

There are many purposes for a system test depending on what they test and how they test it. The most important purpose, however, is to ensure certain features are working as expected.

Assure Acceptance

First of all, a system test is the only way to know if a complex system of self written code, used frameworks, containers, instrumented tools, hardware, … does implement a desired feature in an acceptable way.

There are ways to be quite confident, but system tests are the only way to know. And we are professionals and so want to know, right?

Enable Global Changes by being an Executable Specification

Another purpose of system tests is to enable developers to do huge refactorings, implement necessary global architectural changes, switching from one framework to another as they feel they need to.

All these changes might have an impact on the whole system and developers typically are afraid of the effort it might take to make such changes even though they know how dearly these changes are needed or at least desired to keep the system secure, maintainable or fast.

With a sufficient suite of system tests developers can just do these changes, run the suite and know if their change broke any feature of the software system. This is a huge return of invest from writing the test suite.

Even a complete rewrite of a system is easy when you got a system test suite telling you when you make a false assumption about the behaviour of the original system.

Deepen Understanding & Trust

One final purpose of system tests is a rather psychological one. Making developers write these tests and telling stakeholders that they do, can improve the relationship between these groups.

Developers will need to fully understand a requested feature to be able to write a test for it, so they will (hopefully) start to ask questions about it. Otherwise they write a wrong test and will need to change both: the wrong implementation and the wrong test.

For the stakeholders knowing that there is a red light flashing if any of their dear features is broken somewhere in the future, is a huge relief and a source of trust in the developers.

If the system tests are well readable (like the example below) – maybe using the Test Actor concept – a stakeholder not suffering from allergic issues toward code may even be able to understand the test and point out misunderstandings or missing test cases.

given:
User user = new User("username", "password")
 
when:
user.login()
 
then:
user.isLoggedIn()

Conclusion

System tests succeed in removing the uncertainty about inter class behaviour, that class tests leave us in. They make sure that the system behaves in a specified way without making any compromises. To some this may seem too expensive in material and time but I am absolutely convinced that system tests are a needed to create long term maintainable and stable software products and will always return the investment for these.

So we got class tests and system tests. This combination can be sufficient, and I suggest to make these two a fixed part in you Definition of Done. However, these two are worlds apart from each other and due to treating the system as a black box, a failing system test may give you little information about the failure's cause.

In the next and final part of this series I present a test type to fill the gap between class and system.