The ability to produce reliable technologies that rapidly follow market trends creates a competitive advantage in the digital world.
Part of being a technology company is about producing reliable technology at a rapid pace. At the same time, we cannot sacrifice code quality just to deliver slightly faster. One of the primary tools for ensuring code quality while maintaining a rapid release schedule is writing good tests. Like any other skill, test writing is best developed through practice and experience. Monitoring development performance and knowing when you have tested enough are very valuable things to consider in any software development project.
Many software developers are surprised when the customer reports an error. We spend countless hours defining requirements, testing code and reviewing the final product. Despite this time investment how is it that mistakes find their way into the deliverable unnoticed?
Assuming that the customer is reporting valid concerns, we can answer the question with one of the following statements:
> The customer has executed part of the application that has never been tested. Incomplete testing could be deliberate due to time or cost constraints.
> The order or process in which a customer has used the software is different from the use anticipated by the development team or, more likely, the testing team. This actual usage was not built into the test suite.
> A combination of inputs were received by the application that were never tested. Software is rarely tested with every possible input value. It is the job of the tester to select a reduced set of typical input conditions that reproduce real-world usage. If the assumptions of the tester are wrong, errors slip through.
> The environment in which the software is being used differs between the develop/test teams and the customer. Typical discrepancies can be a different operating system version or hardware. Perhaps the real-world environment was not available to the test team at all, and it had to be simulated or assumed.
Just Test It
Software is almost never 100% tested. Unfortunately, this even applies to the more rigorously tested safety-critical embedded applications.
Consider a piece of control software which processes up to 30 different input variables.
If we wanted to test all possible input combinations and prove that there were no unwanted interrelations between the inputs, we would have to test for 20 years, even if we could create and run 100 test cases per second.
Various measurements of coverage can be used to set and monitor testing progress and performance to help minimise the occurrence of errors in the field.
Consider a piece of control software which processes up to 30 different input variables.
If we wanted to test all possible input combinations and prove that there were no unwanted interrelations between the inputs, we would have to test for 20 years, even if we could create and run 100 test cases per second.
Various measurements of coverage can be used to set and monitor testing progress and performance to help minimise the occurrence of errors in the field.
Coverage Concepts
Throughout the software industry, many commonly used terms have no concrete definition. The meaning of technical terms fluctuates depending on who you are talking to. Software testing is an essential activity in the software development and maintenance life cycles. It is a practice often used to decide and improve software quality. When it comes to measuring software testing performance and progress, it is therefore essential that everyone has the same understanding of the measurement terms (metrics) used.
‘Coverage’ is a broad umbrella term that encompasses a number of useful numerical measures for developers of robust software systems. These measures, when used effectively, can be used both to define quality goals for your end product and track your progress towards achieving them.
In software testing, there are 3 basic types of items to which coverage measurements can be applied:
> Requirements — various levels of detail defining e.g. functional, safety or non-functional (such as performance or usability) what the software should do, and sometimes what it should not do.
> Code — implementation in software (and sometimes hardware or firmware) to meet the requirements.
Tests — a means to verify that the software does what it should do (and sometimes what it should not do — often called robustness tests).
The 3 different uses of the term ‘coverage’ should not be confused. Requirements coverage measures the proportion of requirements which have been verified by requirements-based tests. Structural Code coverage measures the proportion of the code which has been executed by tests. Test coverage measures the proportion of tests which have been run and passed.
Working to standards
If you are working in a safety-critical industry, it is likely that you will be working toward achieving certification in the relevant international software standard. The standard and integrity level within it that you are working towards, will determine the coverage metrics and target coverage values that you should achieve in your project.
Safety-critical software standards, such as DO-178C and ISO 26262, recommend the use of requirements coverage, structural code coverage and test coverage. Correct use of these concepts can also help developers outside of the safety-critical arena. Appreciation of the terms and their use will help deliver a more reliable and robust application.
Link to Requirements
Well-defined requirements at the start of the project, or at the start of an agile sprint, will make the technical teams’ jobs easier and more measurable. It is unlikely that requirements supplied by the customer will be complete enough to ensure project success. Usually, there will be some degree of interpretation, and an upfront process of adding detail to initial high-level requirements, to help shape the architectural design and implementation of the requirements in code. Situations where requirements are not documented, or where initial requirements are not final, will mandate additional study phases with the team and agreement with the customer.
Monitor Code Coverage
Measurement of structural coverage of code is an objective means of assessing the thoroughness of testing. There are various industry-standard metrics available for measuring structural coverage, these can be gathered easily with support from software tools. Such metrics do not constitute testing techniques, but a measure of the effectiveness of testing techniques.
Different code coverage metrics measure the execution of different syntax constructs within the code. The most common code coverage metrics are:
> Function / Method Entry Points
> Function / Method Calls (and their Returns)
> Lines (of executable code)
> Statements
> Basic Blocks (of sequential Statements)
> Decisions
> Conditions (Boolean operands)
> Relational Operators
> Loops
> MC/DC (Modified Condition / Decision Coverage), both Masking & Unique Cause forms
The fundamental strategic question of how much testing you should do is generally driven by available resources, both time and budget.
Dynamic Test Results
Test coverage measures the proportion of your tests have run successfully (i.e. been executed and pass) on the code. While having tests which verify the requirements can be measured using requirements coverage, and the amount of the code executed by the test can be measured using structural code coverage, whether or not these tests are actually run and pass is the third critical thing to measure for software testing. Requirements coverage and code coverage only have meaning when the status of the tests which those metrics measure is itself measured.
Set Some Goals
Setting project goals around defined metrics, such as coverage, has several benefits to project success.
> Optimise the use of resources
There are never enough resources to do everything, so setting coverage goals can help you to prioritise. By allocating most time and budget to test what is most important you can help focus testing efforts. If you want to better manage your time on testing, a simple solution is to stop doing what doesn’t need to be done.
> Add clarity to project meetings
Knowing what you are trying to achieve means that you can tackle the question: “does this activity get me closer to my goal?” Setting goals enables you to clarify with other developers and testers what you are trying to do, and therefore what they need to do to contribute or support.
> Easier measurement of project status
Setting coverage goals allows you to measure how effectively you are moving towards completion.
An important consideration is knowing when to stop testing. For those working towards standards, the coverage goals will be mandated. For others, an important first step is defining the targets of coverage to aim for.
If you reach the end of your project and have missed or not properly completed one of the three critical components of your coverage criteria, you risk delivering code that:
> Has not been verified to meet the requirements
> Has not passed all its tests
> Has not been exercised by any tests
Take away
Obtaining coverage metrics is not a final task, but rather a constant effort towards a reliable system. The easier coverage is to monitor and report, the more useful it is to development teams. Use of coverage can guide the creation of tests cases, help optimise a set of tests and provide an empirical measure of testing sufficiency. Without measuring how much of the code is tested, verification activity always risks shipping untested code. While there are some hurdles to overcome by using coverage software in an embedded or continuous integration environment, the use of suitable tools and testing framework can solve the problems in almost all instances.
If you found this introductory article useful, QA-Systems have a detailed paper available for download for free here.