[tap-l] Nested TAP

[Aristotle Pagaltzis]

* Andy Armstrong <andy at hexten.net> [2008-04-15 21:20]:
> I think the semantics for the summary assertion should work
> like this:
> 
> ===
> 
> After a numbered nested TAP block a TAP producer may optionally
> emit a "summary result" with the same number as the block. The
> overall success of the block is (success of nested TAP AND
> success of summary result). A summary result can therefore
> cause an otherwise apparently successful nested block to fail.
> 
> ===

++

> And in fact that's quite tidy because we can say that a result
> is either a block, a simple result or both in which case the
> simple result must follow the block.

++

> A small point - that gives us no less than three sites where
> YAML diagnostics scoped for the entire nested block can be
> situated: […] The first of those three is probably not a
> problem. I guess the parser will have an API that treats the
> interior of the block as a self contained TAP document via
> which the YAML diagnostic can be reached. The other two are
> resolved either by saying they can both appear or by ruling
> that only the third location is valid I think. I vote for the
> latter.

Diagnostics will be useful only in v13 consumers anyway, so I
think there is merit at all in diagnostics trailing the optional
simple result. However, outright forbidding such diagnostics will
only complicate parsers for not much real gain.

So I’d say that if, by implementation detail, it is possible for
the consumer to make them available in some form, that’s fine. If
not, it’s fine to drop them on the floor also. Interoperability
for such diagnostics is therefore uncertain, which in RFC 2119
parlance means that such diagnostics SHOULD NOT be emitted.

So people are free to put diagnostics there if they want, and if
they are certain they have full control over the entire
infrastructure then they can rely on those. However, they are
strongly encouraged to stick to the one widely accepted location
so it will be easy to write code to utilise their data.

As for diagnostics trailing the block vs those that are part of
the interior TAP stream, I agree that the trailing diagnostics
should be made available as the test’s diagnostics in the outer
TAP scope. That way, flipping a test between a single assertion
and a nested block does not cause a sudden change in behaviour.

Regards,
-- 
Aristotle Pagaltzis // <http://plasmasturm.org/>