[tap-l] multi-file TAP

[Andy Armstrong]

On 7 Apr 2007, at 16:10, Michael Peters wrote:
> + A single test file can be run and output valid TAP and be  
> considered a full
> test run.
> + Multiple test files can be run in whatever sequence the harness  
> wants and
> there should be a way for the harness to join those results  
> together into a
> single valid TAP file so that the entire test run can be captured.
> + Joining the results of multiple files is the job of the harness,  
> not the
> individual test files, so the behavior of the individual files  
> should not be
> altered whether they are being run by themselves, or with others.

Yes. Absolutely. I propose

begin <optional chunk description>
<valid TAP>
end

where <valid TAP> includes begin and end. For the purposes of the  
plan a begin, <tap>, end sequence is a single test - so each begin /  
end block also needs a plan.

plan 3
begin "Spigot extraction"
   plan 3
   ok 1 Spigot::Extractor loaded
   ok 2 Spigot visble
   not ok 3 Spigot free
end
begin "Documentation"
   plan 1
   ok 1 Documentation makes sense
end
ok 3 All tests completed

I'm not sure whether begin / end blocks could also carry a test  
number since they're semantically equivalent to single tests from the  
PoV of the containing TAP.

> In my opinion it would be nice if TAP::Harness would act as both a  
> consumer and
> producer of TAP with the option of joining the individual TAP  
> streams into a
> single file that can then be consumed by something else downstream.

TAP::Pipes :)

Yes, I like that.

> If things
> get really complicated with multiple streams running at the same  
> time, then it's
> up to the harness to do the coordinating. I agree with Andy  
> Armstrong in the
> other part of this thread where he said to just give each stream  
> it's own
> TAP::Parser and then coordinate the results when they're done.
>
> And with regard to BEGIN/END blocks, I don't think we need an END,  
> just a BEGIN
> since we can treat it like Perl's "package" and just assume one  
> file until we
> see another.

You need END *if* you're going to allow blocks to nest. Of course we  
haven't decided that they should nest but it seems desirable to me  
that they should. If you run 100,000 tests and just glue them  
together end to end then you can't, for example, build a web based  
interface that lets you drill down to the results you're interested  
in. The more test output people produce with TAP the more they're  
going to appreciate working with a hierarchy of test results rather  
than just a linear list.

Fergal anticipated that nesting would give us the same problems you  
get with XML where you can't process a node until you've seen it all.  
That's true for XML in general because many grammars aren't strict  
about the ordering of elements - so you might have to parse the whole  
document to get to the title which just happens to be at the end. We  
don't have that problem - TAP output is strictly ordered (admittedly  
apart from the trailing plan special case).

-- 
Andy Armstrong, hexten.net