TAP logging syntax
From Test Anything Protocol
Right now there is no way for a TAP producer to indicate that they would like a hunk of text displayed to the user. Currently you do that by printing to STDERR, a convention we'd like to squash. This proposal allows the producer to suggest to the harness that it display various hunks of text at various levels of severity.
Contents |
[edit] Example
1..2 ok 1 not ok 2 **fail** Failed test in foo.t at line 3. **fail** got: 23 **fail** expected: 42
[edit] Status
This is a proposed TAP extension
[edit] For
- Schwern 23:45, 4 September 2007 (BST)
- AndyArmstrong 00:01, 5 September 2007 (BST)
[edit] Against
[edit] Syntax
A line beginning with two ** characters, followed by an optional severity label, then two more ** is a candidate for display by the parser. If the label is omitted then the severity defaults to "notice".
[edit] Display levels
Stealing liberally from the syslog protocol and Log4perl, we come up with various levels of logging specific to testing. It is suggested that the harness, by default, display only "notice", "warn", "fail" and "fatal". Which levels to display should be configurable by the user.
[edit] fatal
There's an error in the TAP producer and it has to shut down. Test::More's "WHOA THERE!" messages are another.
[edit] fail
Info about a test which failed. This would be all existing failure diagnostics, the "Bail out!" message and end of failing test script messages like "You ran 3 but said you'd run 5".
[edit] warn
There's an ambiguity in the use of the TAP producer. Test::More's messages like "you named a test with a number, confusing".
[edit] notice
"Normal but significant condition" (from syslog). This is some text you want the user to see that isn't associated with a particular test. "Trying to connect to the Internet". "The next test might take a long time".
[edit] pass
Info about a test which passed.
[edit] info
General information about the test that the user probably doesn't need to see.
This is currently done with a "# foo" comment sent to STDOUT.
[edit] debug
Debugging information.
[edit] Use cases
[edit] Colorization
Based on the severity of the message a harness can color the output accordingly. Bright red for failures, flashing for fatals, something dull for notices, etc...
[edit] Compatibility
This proposal is entirely backwards compatible with existing TAP and can be implemented in a TAP harness without any effect on existing producers.
A TAP producer using this syntax instead of printing to STDERR runs the risk of being parsed by an older harness which cannot understand this syntax and thus their diagnostics not being displayed.
[edit] Notes
The need to display free-form failure diagnostics is largely supplanted by the TAP diagnostic syntax propsal.
[edit] FAQ
[edit] Why not just display diagnostics?
While it is not forbidden by TAP, currently no commonly used harness displays diagnostics in the TAP stream. TAP producers safely assume they won't be displayed. If we start displaying them then passing tests will be noisy.
Worse, even though TAP producers can adapt to the new harness behavior of displaying diagnostics, they now have no legal means to put an undisplayed comment in the TAP stream. Their only option is to output a non-TAP junk line and hope it won't conflict with a future TAP extension.
This proposal retains diagnostics as undisplayed comments while offering a rich way for a producer to send messages to the user.
[edit] How does this play with the YAML diagnostics?
How does this play with the idea that diagnostic information is expressed in YAML? YAML documents can span multiple lines. Should the YAML parser barf if all the lines don't have the same number of prefix bangs?
Why is the YAML parser reading these log messages? --Schwern 00:04, 18 March 2007 (UTC)
[edit] Why not just use YAML?
Why not just express the severity in the YAML?
not ok 3
---
diag:
severity: fail
message: Results not ordered
got:
- 1
- 3
- 2
expected:
- 1
- 2
- 3
...
ok 4
ok 5
If your message is associated with a test then yes, it should go in with the test diagnostics in YAML and you can express the severity, although its unnecessary since its implied from context. The test failed therefore the message is of severity "fail".
But not every message is associated with a test and we need a way to express that.
pass("this is a test");
notice("We're about to connect to the Internet.");
ok 1 - this is a test --- message: We're about to connect to the Internet. severity: notice ...
vs
pass("this is a test", { message => "My hovercraft is full of eels." });
ok 1 - this is a test --- message: My hovercraft is full of eels. ...
The first message is not associated with the proceeding test. The second is. The only way the parser can know is by the severity. This seems a thin line and one easily gotten wrong.
OTOH I can see reason to want to output structured information not associated with any particular test, so this ambiguity will likely have to be solved.
The second reason is the human readability. Consider the following hypothetical code:
notice("Attention.");
notice("Attention.");
notice("Testicles.");
notice("That is all.");
Using the proposed logging syntax that would produce a single block of prefixed text.
**** Attention. **** Attention. **** Testicles. **** That is all.
Using YAML diagnostics it would produce four distinct YAML documents.
--- message: Attention. severity: notice ... --- message: Attention. severity: notice ... --- message: Testicles. severity: notice ... --- message: That is all. severity: notice ...
Eww.
Of course you could be careful and put everything together in one notice() call producing one YAML block but that's not always possible and "the user has to be careful" is not a great way to start out a protocol. The **label** syntax makes it unnecessary.
[edit] Criticisms
[edit] Overlap with structured diagnostics
It seems its use cases overlap with the structured YAML diagnostics. Indeed, this is true. Many of the cases where we currently output free-form failure diagnostics would be replaced with structured diagnostics.
[edit] Logging is the duty of the Harness
This logging proposal seems too redundant with meta information. It's the duty of a harness to format the output. TAP need only proved structured data, with the exception of the meta information which is the part designed for unstructured information. For example, one could do:
1..2 ok 1 not ok 2 # **fail** Failed test in foo.t at line 3. # **fail** got: 23 # **fail** expected: 42
And the right harness could show this meta information. But again, that's basically redundant b/c this information should be provided in the YAMLish and a Harness can format it from there.
In the case of non-per-test notices maybe the thing to do is create a "YAMLish" for meta information.
ok 1 - this is a test # --- # message: We're about to connect to the Internet. # severity: notice # ...
