Schema Migrator

Overview

This tool is designed to migrate schemas from one Schema Registry to another. It has the following capabilities:

• Read schemas via REST or file (including v1 style exports written by the previous Python tool)
• Write schemas to file or direct to a topic
• Sorting schemas into dependency order (by Schema ID)
• Validate that a registry export is self-consistent (all references are available within the export)
• Validate two registry sources to look for inconsistencies / missing elements by performing a diff

Compile and Run

$ go build .

./go-schema-migrator --config config.yaml

Configuration

There are two modes of operation: migrate and validate. Use migrate when copying schemas from a source to a sink. Use validate when comparing two sources to look for inconsistencies.

Migrate

In migrate mode, specify exactly one source and one sink:

action: migrate

source:
  rest:
    ...
  

sink:
  file:
    ...

Validate

action: validate

sourceA:
  rest:
    ...
  

sourceB:
  rest:
    ...

Sources

There are three sources available today:

• REST: for connecting to a Schema Registry instance over HTTP
• File: for reading back an intermediate YAML file
• FileV1: for reading an intermediate file produced by the previous Python tool

The following configuration snippets show these sources in use:

source:
  rest:
    url: https://schema-registry-redacted.redacted.fmc.prd.cloud.redpanda.com:30081
    username: redacted
    password: redacted
    tls:
      enabled: true

source:
  file:
    filename: ./registry-export.yaml

source:
  v1file:
    filename: ./exported.schemas

Sinks

There are three sources available today:

• File: for writing out an intermediate YAML file
• Topic: for writing out messages directly to a _schemas topic
• Debug: for console output

The following configuration snippets show these sinks in use:

sink:
  file:
    filename: ./registry.yaml

sink:
  topic:
    seed: seed-redacted.redacted.fmc.prd.cloud.redpanda.com:9092
    topic: _schemas
    compatibility: BACKWARD
    tls:
      enabled: true
    sasl:
      username: redacted
      password: redacted
      mechanism: SCRAM-SHA-256

sink:
  debug: {}

Use Cases

The following use cases are envisaged:

• Export an existing registry to an intermediate file: export.yaml
• Import an intermediate file to a new registry: import.yaml
• Validate a new registry against an existing registry: validate.yaml
• Convert a V1 export into V2 (for using in a future import): convert_v1.yaml

Customisation

The tool enables an export to be customised once loaded into memory by implementing the Process interface. One example of a process that can be performed is schema metadata removal to increase compatibility.

Adding a custom process

Create a new Go file, with an implementation of the process interface. Once that is added, modify main.go to include your implementation in the list of available processes.

Configuring a custom process

To configure a custom process, simply add the processes section the configuration file:

processes:
  - remove_metadata: {}

A full example can be seen in examples/import_without_metadata.yaml.

Possible Future Work

• REST Sink, allowing migration without writing messages to the _schemas topic
• Topic Source, allowing an existing _schemas topic to be used directly