Bovnar — Conformance Test Tool

Bovnar (BVNR) spec 1.2 documentation · Also available as Markdown and PDF.

Spec version: 1.2 Status: Normative — conformance protocol bvnr-conformance-v1 Scope: The conformance corpus, the IUT protocol, and how to validate a third-party implementation.

Table of Contents

1. Purpose
2. Quick Start
   • 2.1 Self-test (verify the reference implementation)
   • 2.2 Testing an external implementation
   • 2.3 Filtering by group
3. Architecture
   • 3.1 Validation tiers
4. Building
5. Running
   • 5.1 Command-line options
   • 5.2 Examples
   • 5.3 Test groups
6. IUT Protocol
   • 6.1 Communication model
   • 6.2 Success response
   • 6.3 Error response
   • 6.4 Arguments
7. Writing a Compliant IUT Adapter
   • 7.1 Event log format reference
   • 7.2 Field details
   • 7.3 TOKEN_TYPE values for DATA lines
   • 7.4 Effective width and base
   • 7.5 Example traces
8. Test Case Corpus
   • 8.1 Coverage summary
9. Output Format (TAP)
10. Extending the Corpus
11. CMake Integration

• See also

1. Purpose

The Bovnar Conformance Test Tool (bvnr_conformance) verifies that any implementation of the Bovnar serialization format produces correct, spec- compliant behaviour. The reference implementation (this repository) is used both as the test driver and as the oracle against which candidate implementations are judged.

Three use modes are supported:

2. Quick Start

2.1 Self-test (verify the reference implementation)

cd build
cmake --build . --target bvnr_conformance
ctest -R bvnr_conformance_self --output-on-failure

2.2 Testing an external implementation

# Build your adapter (see Section 7)
cc -o my_impl_adapter my_adapter.c -lmy_bovnar

# Run the conformance suite against it
./tests/bvnr_conformance --iut ./my_impl_adapter

2.3 Filtering by group

./tests/bvnr_conformance --filter units
./tests/bvnr_conformance --iut ./my_impl_adapter --filter arrays

3. Architecture

┌───────────────────────────────────────────────────────────────────┐
│                       bvnr_conformance                            │
│                                                                   │
│  Test corpus (470 cases) ──→ for each test case:                │
│                                                                   │
│  Self-test mode:                    IUT mode:                     │
│  ┌─────────────────────┐            ┌──────────────────────────┐  │
│  │ ref_parse()         │            │ ref_parse() → event log  │  │
│  │  (uses libbvnr)     │            │ fork/exec IUT binary     │  │
│  │ verify error codes  │            │ compare IUT stdout to    │  │
│  │ verify key presence │            │ reference event log      │  │
│  └─────────────────────┘            └──────────────────────────┘  │
│                                                                   │
│  Output: TAP v14 stream                                           │
└───────────────────────────────────────────────────────────────────┘

The reference implementation is the single authoritative oracle. An implementation is conformant when its IUT adapter produces output byte-for-byte identical to the reference for every test case.

3.1 Validation tiers

Most cases exercise the streaming reader (bvnr_read): the lexer, validator, and the on_verified event stream the IUT protocol mirrors. A smaller set — the homogeneity group — exercises the materialised-document (DOM) tier (bvn_dom_parse), because the spec-1.0 array-homogeneity (§7.4), struct-shape, and duplicate-key (§8.1) rules are enforced above the lexer and are therefore unreachable through the streaming on_verified callback. These DOM-tier cases run in self-test mode only; under --iut they are reported as # SKIP, since IUT protocol v1 is streaming-only and cannot express a DOM-tier check. An implementation that targets full spec-1.0 conformance must still enforce these rules in its document/tree API; the self-test cases pin the reference behaviour and its frozen error codes (39, 40, 41).

4. Building

The conformance tool is built automatically as part of the standard CMake build. Both the main driver and the reference IUT adapter are compiled:

mkdir build && cd build
cmake ..
cmake --build .

Targets produced:

The conformance tool links against bvnr_static. No external dependencies beyond libc and POSIX are required.

5. Running

5.1 Command-line options

bvnr_conformance [OPTIONS]

Options:
  --iut <binary>   Path to the IUT binary to test
  --filter <group> Run only cases in the specified group
  --feed           Instead of the expectations, check that every case fed
                   to bvnr_read_feed in pieces (cut at every byte, and byte
                   by byte) reads exactly as bvnr_read reads it whole
  --list           List all test case IDs and descriptions
  --verbose        Print additional diagnostic information
  --help, -h       Show this help and exit

5.2 Examples

# Run self-test (reference vs. reference)
./tests/bvnr_conformance

# Run with verbose output
./tests/bvnr_conformance --verbose

# List all test cases
./tests/bvnr_conformance --list

# Run only unit-related tests
./tests/bvnr_conformance --filter units

# Test an external IUT adapter
./tests/bvnr_conformance --iut ./my_adapter

# Test external adapter, units group only
./tests/bvnr_conformance --iut ./my_adapter --filter units

5.3 Test groups

6. IUT Protocol

The IUT (Implementation Under Test) Protocol version 1 (bvnr-conformance-v1) defines the interface between the conformance driver and a candidate implementation's adapter binary.

6.1 Communication model

conformance driver                    IUT adapter
       │                                   │
       │  fork, exec IUT binary            │
       │ ─────────────────────────────────→│
       │                                   │
       │  Bovnar text (stdin)              │
       │ ─────────────────────────────────→│
       │                                   │
       │       closes stdin (EOF)          │
       │ ─────────────────────────────────→│
       │                                   │
       │  event log or error (stdout)     │
       │ ←─────────────────────────────── │
       │                                   │
       │  exit(0) or exit(1)              │
       │ ←─────────────────────────────── │

6.2 Success response

The IUT must:

1. Exit with code 0.
2. Write the conformance event log to stdout with no trailing garbage.

The event log is a sequence of lines, one event per line, in the order the events were received from the on_verified callback. One event has no line: ev_stream_end, which closes every successful parse and so distinguishes nothing — the exit code already says the stream ended cleanly.

6.3 Error response

The IUT must:

1. Exit with code non-zero (typically 1).
2. Write exactly one line to stdout: ERROR <code_name> followed by \n.

Where <code_name> is the value returned by bvn_error_to_string() for the first error encountered, e.g. value_out_of_range.

6.4 Arguments

The Bovnar text arrives on stdin, but a case that overrides a reader limit passes the override on the adapter's command line. Each option is followed by one decimal value, and is present only when the case overrides that limit:

An adapter MUST apply every option it is given and MUST ignore an option it does not know. An adapter that reads stdin and ignores its arguments fails the eight cases that pin a limit (ID-011, STR-015, NUM-015, SYM-006, REF-004, STU-007, ARR-012, ARR-013). No other state is passed: every case runs with recovery off and with the default, lenient version handling.

7. Writing a Compliant IUT Adapter

The file tests/bvnr_conformance_iut.c is the reference IUT adapter. It uses the reference libbvnr and is intended both for self-testing and as a template for third-party implementors.

A minimal conforming adapter must:

1. Read all of stdin into a buffer.
2. Apply the limit options on its command line (§6.4).
3. Parse the buffer using the implementation under test.
4. Collect events from the on_verified callback.
5. Emit the event log to stdout on success.
6. Emit ERROR <code_name>\n to stdout and exit non-zero on failure.

7.1 Event log format reference

Each event maps to one line. Text fields use \xNN escaping for any byte outside the printable ASCII range 0x20–0x7E or for the backslash character 0x5C. The two hex digits are lowercase (\x09, \xc3, \x5c) — the comparison is byte for byte, so \xC3 does not match.

STREAM_START
ASSIGNMENT_START <key>
TYPE_ANN_START <ann>
TYPE_FAMILY <ann>
TYPE_PARAM_WIDTH <N>
TYPE_PARAM_BASE <N>
TYPE_PARAM_Q <N>
TYPE_PARAM_UNIT <unit>
TYPE_ANN_END <ann>
DATA <token_type> <value>
STRUCT_START
STRUCT_END
ARRAY_ROW_START
ARRAY_ROW_END
ARRAY_DIM_START
OCTET_STREAM_START
OCTET_STREAM_END

7.2 Field details

Synthesised annotations. A value written without an annotation gets one synthesised for it (doc/03 §10). Its TYPE_ANN_START, TYPE_FAMILY and TYPE_ANN_END lines carry the bare family keyword, and the lines between them depend on the family:

7.3 TOKEN_TYPE values for DATA lines

For octets token type, the value field is <N> bytes (decimal byte count, then a space, then the literal string bytes), not the raw binary data.

7.4 Effective width and base

• Effective width: if the stored width is 0, emit 64. Use bvn_effective_width(value_type).
• Effective base: if the stored base is 0, emit 10. Use bvn_effective_base(value_type).

These rules ensure that untyped values synthesised to uint:64,_10 produce TYPE_PARAM_WIDTH 64 and TYPE_PARAM_BASE 10, matching the reference output exactly.

7.5 Example traces

Input: .x = 42;

STREAM_START
ASSIGNMENT_START x
TYPE_ANN_START uint
TYPE_FAMILY uint
TYPE_PARAM_WIDTH 64
TYPE_PARAM_BASE 10
TYPE_PARAM_UNIT no_unit
TYPE_ANN_END uint
DATA number 42

Input: .s = "hello";

STREAM_START
ASSIGNMENT_START s
TYPE_ANN_START utf8
TYPE_FAMILY utf8
TYPE_ANN_END utf8
DATA string hello

Input: .v = <float:64,m/s> 9.81;

STREAM_START
ASSIGNMENT_START v
TYPE_ANN_START float:64,m/s
TYPE_FAMILY float:64,m/s
TYPE_PARAM_WIDTH 64
TYPE_PARAM_UNIT m/s
TYPE_ANN_END float:64,m/s
DATA number 9.81

Note what an explicit annotation does differently from the synthesised one above, because this is where an adapter is most likely to diverge: the three TYPE_ANN_*/TYPE_FAMILY lines carry the annotation body verbatim rather than the family keyword, and there is no TYPE_PARAM_BASE line at all — the source named no base, and the event stream mirrors the parameters that are there.

Input: #!bovnar 1.1 + .t = <datetime:64,tai> 1000; (the datetime family is spec-1.1 gated, so the directive is required)

STREAM_START
ASSIGNMENT_START t
TYPE_ANN_START datetime:64,tai
TYPE_FAMILY datetime:64,tai
TYPE_PARAM_WIDTH 64
TYPE_PARAM_UNIT tai
TYPE_ANN_END datetime:64,tai
DATA number 1000

The epoch arrives as TYPE_PARAM_UNIT, not as a parameter of its own.

Input: #!bovnar 1.1 + .t = <datetime:64,unix> 1000; (the unix epoch, written out)

STREAM_START
ASSIGNMENT_START t
TYPE_ANN_START datetime:64,unix
TYPE_FAMILY datetime:64,unix
TYPE_PARAM_WIDTH 64
TYPE_ANN_END datetime:64,unix
DATA number 1000

The unix epoch is the implicit one, so it produces no TYPE_PARAM_UNIT line even when the annotation names it. The annotation body still carries it verbatim; an adapter that emits TYPE_PARAM_UNIT unix here fails the byte comparison.

Input: .x = ok; (symbol)

STREAM_START
ASSIGNMENT_START x
DATA symbol ok

Input: .a = [1, 2];

STREAM_START
ASSIGNMENT_START a
ARRAY_ROW_START
TYPE_ANN_START uint
TYPE_FAMILY uint
TYPE_PARAM_WIDTH 64
TYPE_PARAM_BASE 10
TYPE_PARAM_UNIT no_unit
TYPE_ANN_END uint
DATA array_number 1
DATA array_number 2
ARRAY_ROW_END

Inside an array a synthesised annotation is emitted only when the element's synthesised type differs from the type currently in force — at the first element, and again at each change. It is not repeated per element or per row (the second row of [1, 2]/[3, 4] carries no annotation lines at all, while [1, 2.5, 3] carries three blocks); an explicit element annotation is always emitted.

Input: .x = <uint:8> 999; (error — value out of range)

ERROR value_out_of_range

(exit code 1)

8. Test Case Corpus

The corpus is embedded in tests/bvnr_conformance.c. Each case specifies:

8.1 Coverage summary

9. Output Format (TAP)

The tool emits TAP version 14 (Test Anything Protocol), which is consumed natively by CTest and many CI systems. Each case group is a TAP 14 subtest: a 4-space-indented child stream of the individual cases, a trailing child plan, and a leading # Subtest: comment, rolled up into one parent test point. The parent plan therefore counts the groups (currently 23), not the 470 cases.

TAP version 14
1..23
# Subtest: encoding
    ok 1 - [ENC-001] empty stream
    ok 2 - [ENC-002] UTF-8 BOM at byte 0
    not ok 3 - [ENC-003] UTF-8 BOM after first comment
      ---
      message: expected error unexpected_input_byte but parse succeeded
      ...
    1..14
not ok 1 - encoding
# Subtest: limits
    ...
ok 2 - limits
# Subtest: version
    ok 1 - [VER-001] directive declaring the current spec version
    ...
    1..13
ok 3 - version

The closing summary line (# All 470 conformance tests passed, or the count of failures) goes to standard error, so it is not part of the TAP stream.

A group's parent point is not ok if any of its cases failed. Exit code is 0 when all tests pass, 1 when any test fails. (--filter <group> restricts the run to a single group; the parent plan is then 1..1.)

10. Extending the Corpus

To add a new conformance test, add an entry to the g_cases[] array in tests/bvnr_conformance.c using one of the provided macros:

/* Valid input — no expected error */
VALID("GRP-NNN", "group_name", "description of the test",
      ".input = value;"),

/* Valid input — verify a specific key and value appear */
VALID_KEY("GRP-NNN", "group_name", "description",
          ".key = value;", "key", "value"),

/* Error — no special limits */
ERROR_CASE("GRP-NNN", "group_name", "description",
           ".bad = <uint:8> 999;",
           error_value_out_of_range),

/* Error — with continue_on_error (resync mode) */
ERROR_CONT("GRP-NNN", "group_name", "description",
           ".a = 1; .bad = <uint:8> 999; .b = 2;",
           error_value_out_of_range),

/* Error — with custom limits (max_id, max_str, max_num,
                               max_struct_nesting, max_array_nesting,
                               max_array_items) */
ERROR_LIM("GRP-NNN", "group_name", "description",
          ".long_name = 1;",
          error_identifier_too_long,
          4 /*max_id*/, 0, 0, 0, 0, 0),

/* Materialised-document (DOM) tier — validated via bvn_dom_parse instead of
   the streaming reader; runs in self-test mode only (skipped under --iut).
   Use for the spec-1.0 homogeneity / struct-shape / duplicate-key rules. */
DOM_VALID("GRP-NNN", "group_name", "description",
          ".a = [1, 2.5, 3];"),
DOM_ERROR("GRP-NNN", "group_name", "description",
          ".a = [1, \"two\"];",
          error_array_element_type_mismatch),

After editing, rebuild:

cmake --build build --target bvnr_conformance

11. CMake Integration

Three CTest tests are registered:

All three are in the conformance label group and can be run with:

ctest -L conformance --output-on-failure

The bvnr_conformance_iut_self test validates the IUT adapter itself: it must produce bit-for-bit identical output to the internal reference path for every valid test case.

To run all tests including conformance:

ctest --output-on-failure

See also

• Specification — the behaviour the corpus tests
• EBNF Grammar — the grammar an implementation under test must accept
• Read & Write API — the reference reader and writer the adapter drives
• Unit & Currency Reference — the unit table a conforming implementation needs

End of Bovnar — Conformance Test Tool (Bovnar spec 1.2).