[tap-l] multi-file TAP

[Michael Peters]

Bringing this over from Perl-QA:

Fergal Daly wrote:
> On 07/04/07, Andy Armstrong <andy at hexten.net> wrote:
>> On 6 Apr 2007, at 23:17, A. Pagaltzis wrote:
>>
>> > * Michael Peters <mpeters at plusthree.com> [2007-04-06 23:15]:
>> >> Maybe I'm missing something (and if so, please point out the
>> >> obvious to me) but I can't find a way for a single TAP file to
>> >> contain results from multiple test files.
>> >
>> > There isn't one. There was recently discussion about this, with
>> > mixed opinions about whether it's needed and what it might look
>> > like. One of the suggestions was that it has overlap with the
>> > grouping/nesting proposals that keep re-appearing.
>>
>> It's definitely a valuable idea and it's entirely consistent with the
>> notion that TAP is a streaming protocol. We should be able to stream
>> a complete test run without any out-of-band communication about where
>> one file stops and another starts.
>>
>> The overlap with the grouping/nesting proposals is that once you can
>> split a TAP stream into top level chunks why not allow nested chunks?
>> I like that idea too. It means that you can always wrap a TAP stream
>> in BEGIN/END [1] even if it's already chunked.
>>
>> [1] Or whatever.
> 
> To address the whatever, BEGIN/END draws you into the XMLish problems
> of waiting for a complete document before acting on the contents and
> even if you avoid this it still makes it hard to interleave streams.
> 
> I'm starting to think that every line should be prefixed by a test
> number, including diags (diags are not currently associated with a
> particular test but could be made so) so something like
> 
> 1.2.3 not ok blah
> 1.2.3 # got: 4
> 1.2.3 # not: 3
> 1.2.3 end

I think this has problems for identifying which test file it came from. For one,
there is no mention of which file is being processed. But the biggest problem is
that in order for a file (or stream) to know which number it is, the harness
needs to let it know (passing some environment var to Test::Builder?). This
means the harness tells the test file which prefix to use just so that the
harness can then group them properly. Seems backwards. The TAP producers
shouldn't care about the bookkeeping that the TAP harness needs.

Now I'm not saying that nested groups wouldn't be nice, but used within a single
file/stream, not mixing between them.

Here are my simple thoughts on the matter (and you can substitute "file" for
"stream"):

+ A single test file can be run and output valid TAP and be considered a full
test run.
+ Multiple test files can be run in whatever sequence the harness wants and
there should be a way for the harness to join those results together into a
single valid TAP file so that the entire test run can be captured.
+ Joining the results of multiple files is the job of the harness, not the
individual test files, so the behavior of the individual files should not be
altered whether they are being run by themselves, or with others.

In my opinion it would be nice if TAP::Harness would act as both a consumer and
producer of TAP with the option of joining the individual TAP streams into a
single file that can then be consumed by something else downstream. If things
get really complicated with multiple streams running at the same time, then it's
up to the harness to do the coordinating. I agree with Andy Armstrong in the
other part of this thread where he said to just give each stream it's own
TAP::Parser and then coordinate the results when they're done.

And with regard to BEGIN/END blocks, I don't think we need an END, just a BEGIN
since we can treat it like Perl's "package" and just assume one file until we
see another.

-- 
Michael Peters
Developer
Plus Three, LP