[tap-l] Mad TAP proposal

[Andy Armstrong]

On 1 Nov 2007, at 15:44, A. Pagaltzis wrote:

> * Andy Armstrong <andy at hexten.net> [2007-11-01 15:50]:
>> The config needs to be dynamic at test time - so it might as
>> well be a script that runs and outputs a description of which
>> tests to run, right?
>
> But it only needs to be dynamic in a minority of cases. So it
> seems to me it should be the other way around – rather than run
> a script to emit the config, have a config that has a directive
> to run a script and include its output as part of the config.

Well if it doesn't need to be dynamic it doesn't need to exist at all  
- you just have the tests you want to run.

> That makes it easier for other tools to analyse the test configs.
>
>> So isn't that nearly TSP? :)
>
> In some ways. I just don’t think the harness should look at the
> output of tests to check if it’s TSP rather than TAP. The list of
> tests to run should be determined up front

Well that's desirable certainly - but isn't it also valuable to be  
able to dynamically include include or exclude large numbers of tests  
when you need to? And to do so using a well defined protocol?

> , and the test scripts
> should only ever output TAP.

Yeah, I have sympathy with that. I really don't want to complicate  
things gratuitously.

> Mixing them with scripts that output TSP feels like a confusion
> of concerns to me.

I can see that too. In that sense the existing "Bail Out" directive  
feels like an anomaly too.

> For the configs there are also a number of issues like dealing
> well with multiple possible configurations and resolution of
> relative paths (esp. in compound configs).

Yup - not impossible though. You can use HTML-like semantics -  
relative paths are resolved relative to the config, abs paths relative  
to some well defined project root.

> I think this needs a little bit more design. And since YAML is
> making its way into TAP anyway, maybe it should be YAML-based
> rather than a completely ad-hoc TAPish format?


Oh sure - that'd be fine. The choice of representation is somewhat  
orthogonal to whether it's a good idea in the first place though.

-- 
Andy Armstrong, Hexten