Proposed Testability Guidelines (TAB Approved)

This document has no formal standing in OASIS at this point in time. It has been written by the TAB in order to exchange ideas within OASIS.

References to External Work

• QA Framework: Specification Guidelines (W3C, November 2004)

• Test Assertion Guide (W3C Editors' Draft, 2006)

• Test Development FAQ, WG note (W3C, 2005)

• Test Metadata, QA Interest Group note, (W3C, September 2005)

• Test Assertions Guidelines from TAG TC: This TC is producing a Guideline doc. Could it be reused as is? Or should the TAB produce a "digest" of the most relevant features (analogous to a "conformance clause" for specifications)?

• The latest draft of the Test Assertions Guidelines from OASIS TAG: Test Assertions Guidelines (2013)

Testability

PREAMBLE: Not every specification qualifies for the testing of implementations. The notion of testability introduced here concerns implementations (of specifications) the behavior or properties of which can be observed and manipulated - e.g. repeated. Such implementations are typically embodied as devices, software or artifacts that are designed according to the specification. It may be the case that the behavior or properties of such devices are not easily observed or manipulated, or that the specification implementation is a process or set of practices that have to be followed by preexisting entities or humans, and cannot easily be repeated, decomposed or observed while unfolding. In such cases testing may be impossible or may have little value for a high cost. More generally, testing has a cost and this cost has to be measured against the expected value of testing.

NOTE: These guidelines concern specifications that qualify for testing, by the nature of their implementations and by a favorable cost/benefit evaluation.

A specification should be written with at least two kinds of readers in mind:

• (1) developers who will design and implement conforming products,
• (2) test writers who will design test suites for assessing conformance and interoperability.

The best way to address (2) while helping (1) is to add a "testing perspective" to a specification in the form of test assertions. These test assertions (TAs) provide a bridge between the specification narrative and a test suite. They also lead specification writers to pay attention to the "testability" of their specification, i.e. how easy it is to verify that an implementation is conforming to the normative requirements.

"Testability" is a relative notion that depends on a future testing environment and its restrictions. Some feature may be untestable with a test harness that only supports black-box testing, i.e. is only aware of inputs and outputs of the implementation unit, while it becomes testable in a context where it is possible to insert some monitoring component (e.g. a "shim") in the implementation under test. Specification authors may not want to be concerned with these considerations. Test assertion writers will natutally have to assume some form of test environment or will influence test suite writers in choosing one. Even if little is known about a future test environment when writing test assertions, a major benefit of test assertions is to provide a better understanding of what is expected from implementations in order to fulfill the requirements.

The test assertion definition and model below is inspired and abstracted from the work of the Test Assertions Guideline OASIS TC, in particular the first CD produced by this TC: http://docs.oasis-open.org/tag/model/v1.0/cos01/testassertionsmodel-1.0-cos01.pdf

What is a Test Assertion ?

A test assertion (TA) is a testable or measurable expression for evaluating adherence of [part of] an implementation to a normative statement in a specification.

Test assertions sit between the narrative of the specification or of its conformance clauses, and concrete tests parts of a test suite that verifies conformance or otherwise.

What a Test Assertion is not:

• A test assertion is distinct from a conformance clause, in that it does not make any conformance statement on its own. A test assertion is also of smaller granularity as it will generally address a single normative statement in the specification. However, test assertions may be referred to in the wording of the conformance clause to make it clear exactly how conformance is to be evaluated for an implementation.

• A test assertion is distinct from a test case (part of a test suite) in that it does not provide details about how to execute the test, and remains largely independent from the test harness architecture. Instead, test assertions are "blueprints" for the future test cases to be derived from them.

Why write Test Assertions, What does it cost you and How to do it ?

Benefits for specification writers:

When developped concurrently with the specification, test assertions may help to provide for a tighter specification. Any ambiguities, contradictions and unspecified features or behaviors (e.g. error cases, "corner cases") can be noted as they become apparent during test assertion creation. If not developed by the specification authors, test assertions should be reviewed and approved by them. The quality and usability of the specification are improved, and the extra time spent writing these test assertions will pay off in shorter overall time-to-deployment.

Benefits for test suites developers:

Test assertions provide a starting point for writing conformance and interoperability test suites. They simplify the distribution of the test development effort between different groups: often, test suite writers are not expert in the specification to be tested, and need guidance. By interpreting specification statements in terms of test conditions, test assertions improve confidence in the resulting test and provide a basis for coverage analysis (estimating the extent to which the specification is tested).

Benefits for promoting a Standard:

Increasingly, charters for new standard committees include a testing section that lists the writing of test assertions and sometimes of a test suite, as parts of the deliverables of the new TC. The credibility of a standard is better when it is delivered along with a test assertions document or better with a conformance test suite. Users have more confidence in its maturity, and see a shorter path to implementation.

Benefits for Product Vendors:

Product vendors benefit from several angles when a set of test assertions is made available with a standard: they get a better understanding of implementation requirements, can use test assertions in their QA process, and have a conformance indicator that will help them reduce discrepancies with similar products from other vendors and reduce interoperability troubles for their customers. Overall, the time to market is shortened.

Constraints for specification writers:

Writing test assertions is not a negligeable effort. It takes time and commitment to develop a useful set of test assertions, which in turn may delay the completion of a specification. It is recommended to elect early on a "test wizard" in the specification committee, before the first specification draft is complete. The role of the test wizard will be:

• to draft test assertions concurrently with the specification under way.
• to challenge the technical committee with questions on the testability of every specification requirement, and what test conditions can indicate fulfillment or violation.
• to identify features in the main specification that are under-specified or ambiguous from a testing perspective.
• to chair a testing sub-committee that allows a few more members of the committee to get familiar with, contribute to and validate the test assertion work.

An important part of this effort is to establish the local guidelines and practices for developing a consistent set of test assertions:

• What part(s) of an implementation can / cannot be a TA target?
• Do we allow dependencies between test assertions?
• How detailed should be the wording of the TA predicate?
• How much of the future test environment can be assumed?
• How will each TA refer to the normative statement(s) it addresses?
• What ID scheme will be used to identify test assertions?

When to write Test Asserions:

Several options may be considered:

• At the same time the specification is written: This is the best option, as it allows for avoiding most of the schedule pressure at the end of a specification work. It also helps discover gaps and ambiguities early-on, and makes room for early alignment of the specification with testing imperatives.

• Just after the specification draft is complete: This may help to not split the effort of a small team into several concurrent deliverables. But the challenges that a late test assertion design will pose to an already complete specification draft may not be well accepted as they may delay the next steps.

• Just after the specification is finalized: Test assertions written at that time cannot be expected to benefit the specification design process itself. However there is still a benefit in writing test assertions at this time, so that test suite writers will have enough guidance. At that time there may be a better knowledge of what conformance requirements are (the conformance clause(s) are usually written last). There may be more availability in the committee to help on this, and a better knowledge of the future test environment.

• At the time a test suite needs be written: This time is assumed to be after the standard is past the active design work and now a standard or late in the final stages of it. The schedule pressure will come here no longer from the need to roll-out the standard as quickly as possible, but from the need to start and complete the test suite on time. It is strongly recommended that some members of the original specification writing team be involved in the test assertion design, which may be more difficult to do after the specification activity is winding down.

Overview of a Test Assertion

A test assertion (TA) includes:

• Test Assertion Identifier (TA id) : This unique identifier facilitates tools development and the mapping of assertions to specification statements. It is recommended that the identifier be made universally unique.

• Normative Source(s) : These identify the precise requirements or normative statements that the test assertion addresses in the specification.

• Test Assertion Target : Such a target categorizes an implementation or a part of an implementation of the referred specification.

• Predicate : The predicate asserts, in the form of a logical expression, the feature (a behavior or a property of the Target) described in the specification statement(s) referred in the Normative Source part of the TA. If the predicate evaluates to “true” over the TA Target, this means that the target exhibits this feature. “False” means the target does not exhibit this feature.

In addition, a test assertion may optionally include:

• Prerequisite(s) : A test assertion Prerequisite is a logical expression (similar to a Predicate) which further establishes the applicability of the test assertion (TA) to an instance of the Target. It may include a description of a particular state(s) in which the TA applies, as well as references to the outcome of other test assertions. If it evaluates to "false" then the test assertion is considered "not relevant" to this Target instance.

• Prescription Level : A keyword that qualifies how imperative it is that the requirement referred in Normative Source, be met. This element will reflect RFC2119 keywords such as MUST/NOT, SHOULD/NOT, MAY. Three levels are defined: mandatory, preferred, optional.

An Example

Consider the following as a requirement from a specification on “widgets” :

• [requirement 100] “A widget MUST be of cuboid shape”.

Here is a test assertion addressing this requirement:

• TA id: widget-TA100-1

  Target: widget

  Normative Source: “widget specification”, requirement 100

  Predicate: [the widget] has six facets, and each one of the [the widget] facets is of rectangular shape.

  Prescription Level: mandatory

The TA predicate is worded as an assertion, not as a requirement: the 'MUST' keyword of the requirement is absent from the predicate but reflected in the prescription level. (In fact, the predicate would be the same whether the keyword is MUST, SHOULD or MAY). It has a clear Boolean value: either the statement is true, or it is false for a particular target. Additionally, in the above example "testability" depends on some properties of the testing environment, assumed here to only have the ability to detect the shape of surfaces - e.g. by optical scan. The wording of the predicate takes this knowledge into account, by restating the normative requirement in terms of "facets". Here the wording of the predicate may also provide guidance to a test suite writer who is not expert in 3D-geometry. As illustrated in our example, the predicate wording must not however modify the semantics of the referred normative source. A domain expert can ensure that the predicate is a good semantical match to the normative source.

NOTE: it is not always the case that such an equivalent rewording can be done when writing a test assertion predicate. There is still value in keeping the TA predicate "abstract" from test environment conditions.

Assume now that the specification requirement was:

• [requirement 101] “A widget made of plastic MUST be of cuboid shape”.

There is now a pre-condition for the applicability of the previous test assertion. It will translate into an additional Prerequisite element:

• TA id: widget-TA101-1

  Target: widget

  Normative Source: “widget specification”, requirement 101

  Prerequisite: [the widget] is made of plastic.

  Predicate: [the widget] has six facets, and each one of the [the widget] facets is of rectangular shape.

  Prescription Level: mandatory

The prerequisite could refer to another test assertion in case the property "made of plastic" is itself subject to testing, which would be the case if "plastic widget" is not a property known upfront.

Advanced Features

Test assertions may support more advanced usages that are described with more details in the Test Assertions Guidelines [http://docs.oasis-open.org/tag/guidelines/v1.0/cn02/guidelines-v1.0-cn02.pdf]. Among advanced features:

• Grouping of test assertions: Groups and subgroups of test assertions can reflect different levels of conformance or profiles. Grouping may be explicitly defined (as a list) or implicitly by adding "tags" to test assertions that represent properties shared by all TAs of a group. Test assertions groups may be "blueprints" for entire test suites.

• Implicit TA elements: Some TA elements can be made "implicit" depending on the context where the TA appears, e.g. all TAs in a group may have same Target definition defined at group level. An advanced test assertion group definition allows for adding rules that define how to infer implicit elements.

• Use of variables: Variables may be introduced in the definition of a test assertion, for parameterization purpose or for avoiding duplication of values and expressions. When introduced at group level, they may parameterize the entire group and allow for the sharing of values across test assertions, ultimately enabling the reuse of these in a different context.

• Versioning of test assertions: When large specifications are versioned, it is often impractical to rewrite a new set of test assertions. It is easier to version the existing set of TAs, using an appropriate mechanism allowing for describing the update ("diff") compared to the previous version.

• Modeling of Target categories: Targets elements in a test assertion define categories or classes of implementation items. It is useful to explicitly model how these target categories relate to each other, typically using object-oriented relationships such as sub-class, component-of, or related-to. This modeling allows for determining what test assertions apply to which target instances (and their components), and also for using more advanced prerequisites.

• Test assertions at meta-level: Such test assertions make statements on the results of other test assertions. For example, a conformance profile is matched to a set of test assertions. A meta-level test assertion may summarize the combined outcomes of this set of test assertions in a single Predicate statement, thus expressing the general test for addressing this conformance profile.