Evaluate a schema registry by changing a real representative contract and observing which producers and consumers still work. A green compatibility result is useful evidence about a defined schema rule. It is not proof that an application understands the business meaning of every field or can replay every retained event.
This guide proposes sandbox exercises for a purchasing team. It does not report a product benchmark or recommend a vendor. Use synthetic events and the serialization formats, client libraries, and deployment sequence your organization actually plans to support.
Start with one producer and two consumer versions
Choose a simple event contract with required information, optional information, and at least one field whose meaning could change. Keep an old consumer and a new consumer available in the test environment. This lets the team observe the mixed-version period that occurs during a real rollout.
Ask the vendor to map schema identifiers, subjects or artifact names, versions, and client caches. The terminology differs across products, so define which identifier appears in a message and what a consumer needs to resolve it. A migration plan becomes difficult if the team confuses a human-readable name with the identifier embedded in stored data.
Write expected outcomes before registering changes
For each proposed change, state which producer and consumer combinations should work. Include both new traffic and older retained events. Ask the vendor to explain a rejected registration as carefully as a successful one; the reason and scope of the rule are part of what you are buying.
A small test contract
Use a synthetic order-status event with an identifier, status, and optional note. Try adding a field, removing a field, and changing a field type in separate exercises. Keep the exact expected result tied to the format and compatibility mode, rather than assuming every format handles those changes identically.
Distinguish backward, forward, and transitive checks
Confluent’s schema-evolution documentation distinguishes compatibility modes and whether checks consider the latest version or the broader version history. Ask the proposed platform to show the configured policy for the actual subject or artifact. A global default does not prove that an individual contract has the same policy.
Request a sequence of at least three versions. A change that is compatible with the immediately preceding version may raise different questions for a much older consumer or retained record. Ask the vendor to demonstrate the exact historical range covered by its check and the reason that range fits your retention and rollout practices.
| Test | Evidence to preserve | Question it answers |
|---|---|---|
| Add an optional field | Registration result and consumer outputs | How does evolution work for this format? |
| Introduce a breaking change | Rejection reason and policy scope | Where is prevention enforced? |
| Replay an old event | Schema resolution and application result | Does retained data remain usable? |
| Disable registry access | Cached and uncached client behavior | What fails during an outage? |
| Export and restore | Identifiers, versions, and references | Can the platform be recovered or replaced? |
Check semantics outside the schema rule
Apache Avro specifies schema resolution, including the roles of writer and reader schemas. Those structural rules do not establish that a value’s unit or business definition remained unchanged. Ask the team to demonstrate a change from one unit or interpretation to another while the field’s technical type stays the same.
For a hypothetical example, a numeric duration field might change from seconds to milliseconds without changing its numeric type. A structural compatibility check may not catch the resulting application error. The buying decision should account for ownership, documentation, review, and any additional data-contract controls needed for such changes.
Our data quality buying checklist covers validation beyond structural shape. The stream-processing evaluation guide helps connect schema changes to the systems consuming the events.
Test policy enforcement and emergency changes
Apicurio’s rule documentation describes rules for artifact validity and compatibility. Use that reference to ask where a proposed registry applies rules and who can override them. Test ordinary developer access separately from administrative access, and preserve the audit record of a policy change.
Ask what happens when a deployment tries to register an incompatible schema automatically. Does the application fail before sending traffic, continue with an older schema, or require a separate release action? The appropriate behavior depends on your design, but it should be demonstrated rather than discovered during a production release.
Also test recovery after a mistaken policy change. Restoring the rule is only one step if incompatible data was already emitted. Ask how operators identify the affected versions and consumers, and which records remain available for investigation. Avoid treating a settings rollback as a complete data repair.
Plan for outages, deletion, and departure
- Test clients with a warm schema cache and with no cached schema.
- Confirm availability and recovery expectations for schema lookup and registration separately.
- Ask what soft deletion, permanent deletion, and reference removal mean in the proposed product.
- Demonstrate export with version history, identifiers, references, and relevant policies.
- Price environments, artifacts, requests, storage, and support according to the actual quote.
- Assign contract ownership and approval responsibilities to named teams.
Before deleting anything in the sandbox, create a retained event that refers to the target schema. Observe whether the old event remains decodable after each supported deletion action. Record the result and its recovery method. Do not assume deletion terminology or identifier preservation is portable between vendors.
Approve the purchase when the team can explain one complete change: the policy applied, the consumers affected, the rollout sequence, and the recovery evidence. A registry is most valuable when it makes those decisions visible and enforceable in the team’s actual release process.
Frequently asked questions
Does schema compatibility guarantee business compatibility?
No. Units, meaning, and application assumptions can change without a structural type change.
Why test three schema versions?
It helps reveal whether the configured policy checks only an adjacent version or a wider history.
Should registry outages be part of the demo?
Yes. Cached and uncached clients can behave differently, and registration may fail differently from lookup.
Can deleting an old schema affect retained events?
It can, depending on identifiers, references, caches, and deletion behavior. Demonstrate the exact case before relying on a policy.
What should the final evidence packet include?
Versioned contracts, consumer results, policy settings, audit records, outage observations, and a tested export or recovery path.
Primary documentation
- Confluent: schema evolution and compatibility
- Apicurio Registry: rule reference
- Apache Avro 1.12.0 specification
Documentation checked September 14, 2026. Verify behavior for the versions and formats included in your proposed deployment.