From Test Anything Protocol

Please see the Implementation section on how to work on this document.

[edit] Current outstanding points

• Work in progress! Please ignore any obvious mistakes, and jump in and fix anything you like.
• Yes, I'm redefining 'test point' in the TAP spec to 'test'. Good idea/bad idea? If not, rename 'testing system' to 'test' and 'test' to 'test point' to go back to the spec definition. I find 'test' a much simpler and easier to define object.
• How are we defining character sets? UTF-8 or left to implementation? CRLF line endings?
• The most recent version of the TAP standard might be at TAP::Parser. It's hard to tell exactly. I'm currently writing this spec "to" TAP 1.00 and Test::More, as they agree on a simple "standard" TAP. We can add the YAML/advanced TAP later.

[edit] Prologue

[edit] Expiration date

• To be inserted (185 days after posting)

[edit] Authors

• Gaurav Vaidya

[edit] Title

Test Anything Protocol draft-vaidya-test-anything-protocol-00

[edit] Feedback and comments

Comments are solicited and should be addressed to the mailing list at tap@ietf.org and/or the author(s).

[edit] Boilerplate

By submitting this Internet-Draft, each author represents that any applicable patent or other IPR claims of which he or she is aware have been or will be disclosed, and any of which he or she becomes aware will be disclosed, in accordance with Section 6 of BCP 79.

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts.

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."

The list of current Internet-Drafts can be accessed at http://www.ietf.org/1id-abstracts.html

The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html

[edit] Abstract

The Test Anything Protocol (TAP) is a data format used by testing systems to report individual test results. It allows such systems to report on success and failure of individual tests, mark tests as unimplemented or to be skipped, and to provide additional information to aid in debugging of failed tests.

It has been in use by the Perl interpreter and its core modules since 1987.

[edit] Table of Contents

Contents 1 Current outstanding points 2 Prologue 2.1 Expiration date 2.2 Authors 2.3 Title 2.4 Feedback and comments 2.5 Boilerplate 3 Abstract 4 Table of Contents 5 Introduction 6 Definitions 7 Overall structure 7.1 Test result lines 7.2 Ignored lines 8 Security considerations 9 IANA considerations 10 Acknowledgements 11 References 11.1 Normative references 11.2 Informative references 12 Author's address 13 IPR disclosure 14 IPR disclaimer 15 Copyright notice and disclaimer 16 Implementation notes

[edit] Introduction

The Test Anything Protocol (TAP) describes a standard format for testing suites to use. This format provides both human and machine-readable information on which tests were run, whether they were successful or failed, as well as other information which might be useful in tracing the cause of the failure. Being machine-readable allows this information to be easily parsed by an automated testing harness. Test results may then be summarized and analysed. Being human-readable allows the results of a testing system to be read manually, useful for quickly determining whether individual tests pass and to aid in debugging the test system itself.

TAP has been in use since 1987, when Perl 1.0 was released. The format is very stable, and capable of providing any amount of data on failed tests to help in debugging. One of the goals of this specification is to standardize some of this data, in order to provide a set of debugging information which every testing system will be guaranteed to return.

[edit] Definitions

• Test: A test is a piece of software which determines if a particular facility or component is functional. A test may be successful or a failure. A test may additionally be reported as having been skipped, or not having being reported on yet.
• Testing system: A testing system consists of a number of tests. The TAP allows a testing system to report on all its tests.
• Testing harness: An automated system, which executes, retrieves and collates the results from multiple testing systems. It usually reports on the status of all testing systems to the tester, but may provide other high-level functionality such as repeating tests, monitoring pass/fail history and other such tasks. This specification places no limitations on any testing harness which reads testing results formatted in TAP.

[edit] Overall structure

The following structures use Augmented Backus-Naur Form (ABNF), as defined by [STD68].

A TAP file or stream can be split into lines, separated by newline characters. Any line beginning with the letters 'ok' or 'not ok' are test results. All lines not explictly test results may be ignored by the TAP parser.

The overall structure of a TAP file is:

<TAP-file> = 0*1<Plan> *((<Test-Result-Line>) / (<Ignored-Line>))

The first line in a TAP file defines the plan the test system intends to carry out. The format of the plan is:

<Number-Of-Tests> = 1*DIGIT ; The number of tests contained in this file.
<Plan> = "1.." <Number-Of-Tests> CRLF

• Question: What about TAP streams, such as STDOUT on Test::More? Do we define that ONLY the first line of a stream can have the number of tests?

[edit] Test result lines

A test result line is in the form:

<Status> = "ok" / "not ok" ; Whether the test succeeded or failed
<Index> = 1*DIGIT ; The index of this test, compared to all the others.
<Description> = 1*VCHAR ; A description of this test.
<Directive> = "SKIP" / "TODO"
<Explanation> = 1*VCHAR ; The reason for this directive
<Test-Result-Line> = <Status> [SP <Index>] [SP <Description>] [SP "#" SP <Directive> [SP <Explanation>]] CRLF

[edit] Ignored lines

Any line not explicitly a test result line should be ignored by the parser.

[edit] Security considerations

• See RFC 3552 for tips on how to write this section.
• Problems with parsing debugging status?

[edit] IANA considerations

• Register application/tap as a Content-type for test results.

[edit] Acknowledgements

This document is based on Andy Armstrong's description of the TAP protocol, version 13, which is based on Andy Lester's description of the TAP protocol, version 1.00. The basis for the TAP format was created by Larry Wall in the original test script for Perl 1. Tim Bunce and Andreas Koenig developed it further with their modifications to the Test::Harness module.

[edit] References

[edit] Normative references

• [STD68]: Crocker, D., and Overel, P. "Augmented BNF for Syntax Specifications: ABNF", RFC5234, January 2008.

[edit] Informative references

[edit] Author's address

• Gaurav Vaidya

[edit] IPR disclosure

By submitting this Internet-Draft, each author represents that any applicable patent or other IPR claims of which he or she is aware have been or will be disclosed, and any of which he or she becomes aware will be disclosed, in accordance with Section 6 of BCP 79.

[edit] IPR disclaimer

The IETF takes no position regarding the validity or scope of any Intellectual Property Rights or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; nor does it represent that it has made any independent effort to identify any such rights. Information on the procedures with respect to rights in RFC documents can be found in BCP 78 and BCP 79.

Copies of IPR disclosures made to the IETF Secretariat and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the IETF on-line IPR repository at http://www.ietf.org/ipr.

The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology that may be required to implement this standard. Please address the information to the IETF at ietf-ipr@ietf.org.

[edit] Copyright notice and disclaimer

Copyright (C) The IETF Trust (2008).

This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights.

This document and the information contained herein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

[edit] Implementation notes

• The IETF strongly recommends that xml2rfc is used to ensure that Internet-Drafts are formatted exactly as required. If it's easier to work on the draft in this Wiki, I'll volunteer to convert it into XML.
• Recommended reading:
  • The Tao of IETF (if you don't have any experience with IETF procedures and/or meetings)
  • Guidelines to Authors of Internet-Drafts
  • Internet-Drafts checklist
  • RFC Editorial Guidelines and Procedures