[tap-l] User Supplied YAML Diagnostic Keys: Descriptive Version

[Michael G Schwern]

chromatic wrote:
> On Thursday 17 April 2008 19:10:21 Michael G Schwern wrote:
> 
>> As for why it'll work with TAP, with a few exceptions (exit_status, or
>> whatever we decide to call it, is currently the only one), diagnostic keys
>> do not effect test parsing.  It's not a show stopper.  At worst, a
>> displayer that has been customized to do something special with a user
>> defined key might show some odd output.  The tests still pass and fail as
>> before.
> 
> Then why is there a problem with reserved keys at all?  If TAP v15 adds a new 
> reserved key, anyone who deliberately upgrades may need to modify both the 
> producer and consumer to deal with the collision, if that person even cares.

I don't understand.  There can be no collision.  Official TAP keys all start 
with a lower case letter.  User defined keys don't.  A new, official key 
cannot collide with anything.

Can you provide an example of your scenario?


>> Prefixing brings an additional problem, how do you tell a TAP displayer to
>> do anything with all those prefixed keys?  Let's say Test::Stuff has
>> Perl.Test.Stuff-time.  So you program your displayer to do something with
>> Perl.Test.Stuff-time.  Now the Test::Thing author thinks that's a neat idea
>> and adds Perl.Test.Thing-time.  So you add in Perl.Test.Thing-time to your
>> displayer to do the same thing as Perl.Test.Stuff-time.  Test::Foo picks up
>> on it, and you have to add another special case.  Test::Bar adds it in,
>> another special case in your displayer.  Another module, add another to the
>> list.
>>
>> You're either going to wind up with a big list of prefixes and keys, which
>> is annoying work, or you're going to break down and match on /-time$/ and
>> defeat the point of prefixing.
> 
> Every part of customizable diagnostics has this problem.

Sorry, I don't understand.  Can you provide an example?


>> User key collision is not a show stopper for the user.  As mentioned above,
>> at worst a customized TAP displayer shows some confused output for that
>> key.  So it's not urgent that it be worked out.
> 
> Then why is it urgent to bless a namespace or character set or subset of a 
> character set?  You're arguing that collisions don't matter; someone can just 
> write code to fix it, if it matters to them.

There is a line being walked here.  Your concerns about collision are valid. 
It's a question of how big the collision problem is vs how much the solution 
fucks up readability and extensibility.  Separating official vs user keys 
solves a good chunk of the collision problem, it provides some stability and 
allows us to add new keys without worry.  Upper vs lower case doesn't effect 
readability much and users still get a big wilderness of prime real estate to 
homestead.  Big benefit, minimal drawback.

Prefixing solves a much smaller problem, and adds problems, while screwing up 
readability.  So it's not worth it.


>> The authors of the two colliding producers simply hash it out amongst
>> themselves.
> 
> Or they don't, which they won't.
> 
> Non-executive summary: customizable diagnostic keys aren't that useful, no 
> one's using them, and collisions aren't a problem, so why even bother 
> standardizing them?

Nobody's using them because nothing supports them yet.

Just because they're not a show stopper doesn't mean they're not useful. 
These diagnostics will be extremely useful, probably the biggest thing to 
happen to TAP since "ok".  We have a decent pile of use cases already.

Realize this solves the oft cried for problem of parsable failure diagnostics. 
  There's a huge need for this.  Remember how many times I've chastised people 
on perl-qa for trying to parse out Test::More diagnostics?


> Here's the question I should have asked much earlier in this thread, before 
> people who want these diagnostic keys started arguing both sides of the 
> question: can anyone provide a real, actually encountered use case where 
> collisions are a real, actually encountered problem, or is this just an 
> attempt to predict the future badly based on what people think might be a 
> problem?

Are you talking about collisions between official and user-defined keys?  Or 
are you talking about collisions between different authors' user-defined keys?

For the former, you're right.  We have no information about this and we're 
trying to predict the future.  However, it's a pretty safe assumption.  We've 
seen the problem in Perl 5.  We have some pretty clear use cases and some 
stability is desirable.  The impact of the "lower case reserved" solution on 
readability and expression is minimal.  The implementation and spec is simple. 
  And if it turns out to be wrong, we can lift the restriction.

As to the latter, we weren't arguing about that.


> (Given that even unstructured diagnostics have never actually appeared in TAP 
> documents before, my guess is "No, everyone's arguing out of ignorance", 
> which seems to me to be a poor way to write a specification, and probably at 
> least half of the reason I keep getting different answers when I ask 
> questions like "Is this useful" and "Is this actually a problem".)

I believe you misinterpreted an argument that was carried over from Oslo.  We 
were still arguing using X- vs /^[a-z]/ to denote official keys which has been 
mostly settled.  It was all clear to the folks who were in Oslo but probably 
not so much to everyone else.


-- 
package Outer::Space;  use Test::More tests => 9;