Language independent testing

From Test Anything Protocol

On the TAP mailing list Schwern suggested that we should distribute language-independent acceptance tests for the TAP grammar:

   From: Michael G Schwern
   To: <tap-l@testanything.org>
   Subject: TAP tests
   Date: Mon, 19 Mar 2007 17:44:43 -0700
   
   By which I mean acceptance tests to ensure your TAP consumer is
   consuming properly. We don't have anything official, it sure would
   be nice if we did.
   
   [snip]
   
   A good place to start is the "sample-tests" directory in the
   TAP::Parser distribution. The expected results are in t/020-
   regression.t. I'd suggest the tests be encoded in YAMLish with the
   TAP and expected results in each document. For example...
   
   ---
   name: simple
   TAP: |
     1..1
     ok 1
   results:
     overall:
       has_plan:      1
       planned_tests: 1
       passed:        1
       version:       12
       skipped:       []
       todo:          []
     lines:
        -
          type:          plan
          planned_tests: 1
        -
          type:          test
          result:        ok
          actual_result: ok
          directive:     
          description:   
   ...
   
   These should be posted on the Wiki and probably eventually shipped
   with the TAP module.

It's quite common for test scripts to be heavily data driven anyway. In the case of any protocol or interface that might be implemented in more than one language (or any code that might later be migrated to another language) it'd be great to express the greater part of the test suite declaratively in a language-neutral notation.

While it's quite possible now to represent your test data as YAML (or XML, JSON, whatever) documents the lack of any standard grammar for such data driven test specifications means that you effectively have to implement a small DSL in each test to interpret the data.

I've lost count of the times I've written the "loop over a bunch of test data doing setup, process, compare, teardown for each case" idiom. Instead I'd like to be able to write:

   use Test::More;
   use Test::DataDriven;
   
   use Class::To::Test;
   
   my $setup = sub {
       # init and return an object
   };
   
   my $teardown = sub {
       # Tidy up
   };
   
   Test::DataDriven->test_class_against_spec(
       class       => 'Class::To::Test',
       source      => 'spec.yaml',
       setup       => $setup,
       teardown    => $teardown
   );

A standardised grammar for data driven testing might make that possible.