v0.26.2

Release 0.26.2

What's New

• Transport
  • WS/WSS Identity Support
• Identity
  • Alternate Server Certificate Support
• Edge
  • N/A
• Fabric
  • N/A
• Ziti CLI
  • Improvements to ziti edge list posture-check output
• SDK Golang
  • N/A

Transport

WS/WSS Identity Support

The binding ws and wss in the transport library now use identity for server certificates. Prior to this release
ws and wss would load the server_cert and key field from files only. Both now support an optional field named
identity. If not specified, the root identity field will be used. If specified it will be used for the specified
ws or wss binding. Since this field is processed by the identity library
it supports all the private key and certificate sources that the identity framework supports (file, pem, hsm, etc.).
Additionally it also enables SNI support for ws and wss listeners.

transport:
  ws:
    writeTimeout:      10
    readTimeout:       5
    idleTimeout:       5
    pongTimeout:       60
    pingInterval:      54
    handshakeTimeout:  10
    readBufferSize:    4096
    writeBufferSize:   4096
    enableCompression: false
    identity:
      server_cert:          ./certs/er1.server.cert.pem
      server_key:                  ./certs/key.pem

Example: Relying on in the root server_cert and alt_server_cert field

v: 3

identity:
  cert:                 ./certs/er1.client.cert.pem
  server_cert:          ./certs/er1.server.cert.pem
  key:                  ./certs/er1.key.pem
  ca:                   ./certs/er1.ca-chain.cert.pem
  alt_server_certs:
    - server_cert: ./certs/er1.alt.server.cert.pem
      server_key:  ./certs/er1.alt.server.cert.pem
...

transport:
  ws:
    writeTimeout:      10
    readTimeout:       5
    idleTimeout:       5
    pongTimeout:       60
    pingInterval:      54
    handshakeTimeout:  10
    readBufferSize:    4096
    writeBufferSize:   4096
    enableCompression: false

Identity

Alternate Server Certificate Support

The identity library has been updated to support a new field: alt_server_certs
. This field is an array of objects with server_cert and server_key fields. alt_server_certs is not touched by
higher level Ziti automations to renew certificates and is intended for manual or externally automated use. It allows
additional server certificates to be used for the controller and routers with separate private keys. It is useful in
scenarios where routers or controllers are exposed using certificates signed by public CAs (i.e. Let's Encrypt).

The server_cert and server_key work the same as the root identity properties of the same name. In any single
server_cert source that provides a chain, it assumed that all leaf-certificates are based on the private key in
server_key. If server_key is not defined, the default root server_key will be used. The identity library will use
the certificate chains and private key pairs specified in alt_server_certs when generating a TLS configuration via
ServerTLSConfig(). All identity sources are viable: pem, file, etc.

Go Identity Config Struct Definition:

type Config struct {
	Key            string       `json:"key" yaml:"key" mapstructure:"key"`
	Cert           string       `json:"cert" yaml:"cert" mapstructure:"cert"`
	ServerCert     string       `json:"server_cert,omitempty" yaml:"server_cert,omitempty" mapstructure:"server_cert,omitempty"`
	ServerKey      string       `json:"server_key,omitempty" yaml:"server_key,omitempty" mapstructure:"server_key,omitempty"`
	AltServerCerts []ServerPair `json:"alt_server_certs,omitempty" yaml:"alt_server_certs,omitempty" mapstructure:"alt_server_certs,omitempty"`
	CA             string       `json:"ca,omitempty" yaml:"ca,omitempty" mapstructure:"ca"`
}

JSON Example:

{
  "cert": "./ziti/etc/ca/intermediate/certs/ctrl-client.cert.pem",
  "key": "./ziti/etc/ca/intermediate/private/ctrl.key.pem",
  "server_cert": "./ziti/etc/ca/intermediate/certs/ctrl-server.cert.pem",
  "server_key": "./ziti/etc/ca/intermediate/certs/ctrl-server.key.pem",
  "ca": "./ziti/etc/ca/intermediate/certs/ca-chain.cert.pem",
  "alt_server_certs": [
    {
      "server_cert": "./ziti/etc/ca/intermediate/certs/alt01-ctrl-server.cert.pem",
      "server_key": "./ziti/etc/ca/intermediate/certs/alt01-ctrl-server.key.pem"
    },
    {
      "server_cert": "pem:-----BEGIN CERTIFICATE-----\nIIGBjCCA+6gAwIBAgICEAAwDQYJKoZIhvcNAQELBQAwgZcxCzAJBgNVBAYTAlVT...",
      "server_key": "pem:-----BEGIN CERTIFICATE-----\nMIIEuDCCAqCgAwIBAgICEAAwDQYJKoZIhvcNAQELBQAwgYsxCzAJBgNVBAYTAlVT..."
    }
  ]
}

YAML Example:

cert: "./ziti/etc/ca/intermediate/certs/ctrl-client.cert.pem"
key: "./ziti/etc/ca/intermediate/private/ctrl.key.pem"
server_cert: "./ziti/etc/ca/intermediate/certs/ctrl-server.cert.pem"
server_key: "./ziti/etc/ca/intermediate/certs/ctrl-server.key.pem"
ca: "./ziti/etc/ca/intermediate/certs/ca-chain.cert.pem"
alt_server_certs:
 - server_cert: "./ziti/etc/ca/intermediate/certs/alt01-ctrl-server.cert.pem"
   server_key: "./ziti/etc/ca/intermediate/certs/alt01-ctrl-server.key.pem"
 - server_cert: "pem:-----BEGIN CERTIFICATE-----\nIIGBjCCA+6gAwIBAgICEAAwDQYJKoZIhvcNAQELBQAwgZcxCzAJBgNVBAYTAlVT..."
   server_key: "pem:-----BEGIN CERTIFICATE-----\nMIIEuDCCAqCgAwIBAgICEAAwDQYJKoZIhvcNAQELBQAwgYsxCzAJBgNVBAYTAlVT..."

v0.26.1

Release 0.26.1

There was a missed dependency update for xweb in 0.26.0 that kept SNI from working in HTTP API components. This would
affect SNI support for all REST APIs.

What's New

• Edge
  • Fixes missing identity update in xweb
• Fabric
  • Fixes missing identity update in xweb
  • Bug Fixes
• Ziti CLI
  • N/A
• SDK Golang
  • N/A

Edge

Bug Fixes

• Fix panic on remote resolve connections

Fabric

Bug Fixes

• Logging erroneously indicates conflicting conditions returned from route attempt

v0.26.0

Release 0.26.0

Breaking Changes

• The fabric management terminators API has changed the name of some fields. See below for details.
• The management channel, which was formerly deprecated is now removed
• Support for the old metrics subsystem has been removed.

What's New

• Edge
  • N/A
• Fabric
  • Terminator fields name changes
  • Circuit failed events
  • Additional circuit inspect information gathered
  • Management channel has been removed
  • Old metrics subsystem removed
  • Circuit createdAt
  • Bug Fixes
• Ziti CLI
  • Terminator fields name changes
  • Bug Fixes
• SDK Golang
  • N/A
• Identity
  • All OpenZiti implementations now support multiple certificate chains in the server_cert field to enable SNI scenarios

Fabric

Terminator fields name changes

The following fields have been renamed:

• identity -> instanceId
• identitySecret -> instanceSecret

The use of identity was confusing as identity is also used in the edge. While terminator instanceId
could be an edge identity id or something related to an edge identity, it could also be something
entirely unrelated. To reduce semantic overload, we've renamed it to instanceId, which hopefully is
more descriptive. In general all terminators with the same instance id should end up at the same
hosting process.

Circuit failed events

The fabric can now emit circuit events when a circuit creation failed.

Here is an example event:

{
  "namespace": "fabric.circuits",
  "event_type": "failed",
  "circuit_id": "DtZLURFgP",
  "timestamp": "2022-06-22T14:24:18.389718316-04:00",
  "client_id": "cl4pxcvyl000m5qgd1xwcfg1u",
  "service_id": "dH0lwdc5P",
  "instance_id": "",
  "creation_timespan": 739021,
  "path": "[r/niY.XmLArx]->[l/1UZCUTGhHuJygXld8CxXPs]->[r/YPpTEd8JP]",
  "terminator_local_address": "",
  "link_count": 1,
  "path_cost": 327152,
  "failure_cause": "ROUTER_ERR_CONN_REFUSED"
}

Note the event_type is failed. For events of this type only, the failure_cause will be populated. The current set of failure causes is:

• INVALID_SERVICE
• ID_GENERATION_ERR
• NO_TERMINATORS
• NO_ONLINE_TERMINATORS
• NO_PATH
• PATH_MISSING_LINK
• INVALID_STRATEGY
• STRATEGY_ERR
• ROUTER_ERR_GENERIC
• ROUTER_ERR_INVALID_TERMINATOR
• ROUTER_ERR_MISCONFIGURED_TERMINATOR
• ROUTER_ERR_DIAL_TIMED_OUT
• ROUTER_ERR_CONN_REFUSED

In addition to the failure_cause field, there is also a new instance_id field. This will be populated for all circuit event types and
will have the instance id requested by the dial. This is generally only applicable when using addressable terminators. If no instance id
was specified, the field will be blank.

Circuit Inspect Enhancements

Circuit inspect will now gather more information.

• xgress details now includes the xgress sequence
• The receive buffer now has the following new fields
  • acquiredSafely
  • maxSequence
  • nextPayload
  • payloadCount
  • sequence

Management channel removed

The management channel has been removed. The ziti-fabric cli, which used to use the management channel,
has been absorbed into the ziti CLI, and now used the fabric REST API and/or websockets where appropriate.

The mgmt: stanza in configuration files, which used to be required, will now be ignored.

Old Metrics Subsystem removed

Formerly metrics could be exported to file via the metrics: configuration stanza. This was superceded by
the events subsystem, which contains metrics as well as other events.

This also means that we no longer support pushing metrics directly to InfluxDB. However, we now have a
Prometheus endpoint available, which can also be used to feed information to InfluxDB.

Circuit createdAt

Circuits now have a createdAt field, visible via the REST API.

Bug Fixes

• Fix for issue where smart routing could break a circuit if a router became unavailable while circuits were being updated

Ziti CLI

Terminator Field Name Changes

The ziti fabric create terminator operation now takes a --instance-id flag instead of an --identity flag.

The ziti fabric list terminators operation now shows InstanceId instead of Identity.

Bug Fixes

• Fixed a bug where the controller advertised name was not properly set when the value of EXTERNAL_DNS was set.

v0.25.13

Release 0.25.13

What's New

• Edge
  • Bug fixes
• Fabric
  • N/A
• Ziti CLI
  • N/A
• SDK Golang
  • N/A

Edge

Bug Fixes

• [https://github.com/openziti/edge/issues/1055](Fix for an edge router panic)

v0.25.12

Release 0.25.12

What's New

No functional changes, build process changes only

v0.25.11

Release 0.25.11

What's New

• Edge
  • Management API: Breaking Changes
  • Management API: New Endpoints
  • Management API: JWKS Support
  • Bug fixes
• Fabric
  • Bug fixes
  • Metrics API
• Ziti CLI
  • N/A
• SDK Golang
  • N/A

Edge

Management API Breaking Changes

The following Edge Management REST API Endpoints have breaking changes:

• POST /ext-jwt-signers
  • kid is required if certPem is specified
  • jwtEndpoint or certPem is required
  • issuer is now required
  • audience is now required
• PUT /ext-jwt-signers - kid is required if certPem is specified, issuer is required, audience is required
  • kid is required if certPem is specified
  • jwtEndpoint or certPem is required
  • issuer is now required
  • audience is now required
• PATCH /ext-jwt-signers - kid is required if certPem is specified, issuer is required, audience is required
  • kid is required if certPem is set and kid was not previously set
  • jwtEndpoint or certPem must be defined or previously set of the other is null
  • issuer may not be set to null or ""
  • audience may not be set to null or ""

The above changes will render existing ext-jwt-signers as always failing authentication is issuer and audience
were not previously set.

Management API: New Endpoints

The following new endpoints have been added:

• GET /identities/:id/enrollments - returns a pre-filtered list of enrollments for the identity specified by :id

Management API: JWKS Support

JWKS (JSON Web Key Sets) is defined in rfc7517 and defines the format
and methods that public and private keys may be published via JSON. JWKS support enables Ziti to obtain
public signing keys from identity providers as needed. This enables identity providers to rotate signing keys without
breaking SSO integrations.

To facilitate this, ext-jwt-signers now support jwksEndpoint which is a URL that resolves to a service that returns
a JWKS JSON payload. When specified, the certPem and kid files are no longer required. Additionally, when a JWT iss
fields matches an existing extj-jwt-signers's issuer field and the kid is currently unknown, the jwksEndpoint
will be interrogated for new signing keys. The jwksEndpoint will only be interrogated at most once every five seconds.

Bug Fixes

• openziti/edge#1027
• openziti/edge#1025
• openziti/edge#1035
• openziti/edge#1045
• openziti/edge#1049

Fabric

Bug Fixes

• openziti/fabric#406
• #565 - Moved terminator information to its own field.

Metrics API

The following new endpoint has been added:

• GET /metrics - returns metrics for the controller and all routers in the Prometheus text exposition format. See [https://openziti.github.io/ziti/metrics/prometheus.html] for more information and instructions to set it up.

v0.25.10

Release 0.25.10

What's New

• Edge
  • N/A
• Fabric
  • N/A
• Ziti CLI
  • CLI support for enrollments/authenticators/re-enrollment
  • Fix prox-c download
  • ziti-fabric cleanup
• SDK Golang
  • N/A

Ziti CLI

CLI support for enrollments/authenticators/re-enrollment

The CLI has been augmented to support the following commands:

• ziti edge list authenticators - to generically list existing authenticators
• ziti edge list enrollments - to generically list existing enrollments
• ziti edge delete enrollment <id> - to generically delete existing enrollments
• ziti edge delete authenticator <id> - to generically delete existing authenticator
• ziti edge create enrollment ott ... - to create a new one-time-token enrollment for an existing identity
• ziti edge create enrollment ottca ... - to create a new one-time-token enrollment for an existing identity for a 3rd party CA issued certificate
• ziti edge create enrollment updb ... - to create a new updb (username/password) enrollment for an existing identity

These commands, specifically the enrollment related ones, can be used to re-enroll existing identities. See the 0.25.9 changeFor all arguments and options, please see their CLI related -h.

Also note that the ziti edge delete authenticator updb command has been supplanted by ziti edge delete authenticator <authenticator id>

Fix prox-c download

The prox-c releases on GitHub now include the architecture in the download URL.
ziti install ziti-prox-c has been updated to take this into account.

ziti-fabric cleanup

Ziti CLI install/upgrade/remove commands related to ziti-fabric have been
removed since ziti-fabric was deprecated and is not being published anymore.

v0.25.9

Release 0.25.9

What's New

• Edge
  • Create Identity Enrollments / Allow Identity Re-Enrollment
• Fabric
  • Bug fixes
• Ziti CLI
  • N/A
• SDK Golang
  • N/A

Edge

Create Identity Enrollments / Allow Identity Re-Enrollment

The ability to create identity enrollments, allows new enrollment JWTs to be generated throughout any identity's
lifetime. This allows Ziti to support scenarios where re-enrolling an identity is more convenient than recreating it.

The most common scenario is new device transitions. Previously, the only way to deal with this scenario was to remove
the identity and recreate it. Depending on how the role attributes and policies were configured this may be a trivial or
demanding task. The more policies utilizing direct identity reference, instead of attribute selectors, the
more difficult it is to recreate that identity. Additional, re-enrolling an identity retains MFA TOTP enrollment,
recovery codes, and authentication policy assignments/configuration.

New Endpoints

• POST /enrollments - Create enrollments associated to an identity

POST /enrollments Properties

• method - required - one of ott, ottca, or updb to specify the type of enrollment (this affects other field requirements)
• expiresAt - required - the date and time the enrollment will expire
• identityId - required - the identity the enrollment is tied to
• caId - ottca required, others ignored - the verifying 3rd party CA id for the ottca enrollment
• username - updb required, others ignored - the default username granted to an identity during updb enrollment

Creating Identity Enrollments

Identity enrollments only allow one outstanding enrollment for each type of enrollment supported. For example attempting
to create multiple ott (one-time-token) enrollments will return a 409 Conflict error. Deleting existing enrollments will
resolve the issue.

As noted in the properties' section above, some properties are utilized for different method types. Please be aware
that while setting these values through the API will not be rejected, they are not utilized.

Please note that it is possible for an identity to have multiple authentication types. Authentication policies should
be used to restrict the type of authenticators that are valid, even if enrolment has been completed.

Fabric

Bug Fixes

• openziti/fabric#404
  • Goroutine pool metrics for xgress and link dials not working

v0.25.8

Release 0.25.8

Maintenance

Improved MacOS compatibility with cert handling and ioKit.

Fabric

Bug Fixes

• openziti/fabric#403

v0.25.7

Release 0.25.7

Fabric

Dial Timeout Propagation

Currently each section of the dial logic has its own timeouts. It can easily happen that
an early stage timeout expires while a later one doesn't, causing work to be done whose
results will be ignored. A first pass has been completed at threading timeouts/deadline
through the dial logic, spanning controller and routers, so that we use approximately
the same timeout througout the dial process.

Link Initial Latency

Previously links would start with a latency of 0, which they would keep until the latency
was reported from the routers. Now, latency will be initialized to a default of 65 seconds,
which it will stay at until an actual latency is reported. If a new link is the only
available one for a given path, this won't prevent the link from being used. However, if
there are other paths available, this will bias the network to the existing paths until
it can see what the actual link latency is. Latency should generally be reported
within a minute or so.

This value can be adjusted in the controller config, under the network section.

network:
  initialLinkLatency: 65s

Link Verification changes

In previous releases when a router recieved a link dial from another router, it would verify
that the link was known to the controller and the dialing router was valid. Router validity
was checked by making sure the fingerprints of the certs used to establish the link matched
the fingerprints on record for the router.

From this release forwards we will only verify that the router requesting the link is valid
and won't check that the link is valid. This is because the router has more control over the
links now, and in future, may take over more of link management. As long as we're getting
link dials from a valid router, we don't care if they were controller initiated or router
initiated. For now they are all controller initiated, but this also covers the case where
the controller times out a link, but the router still manages to initiate it. Now the router
can report the link back to the controller and it will be used.

Add Goroutine Pool Metrics

We use goroutine pools which are fed by queues in several places, to ensure that we have
guardrails on the number of concurrent activities. There are now metrics emitted for these
pools.

The pool types on the controller are:

• pool.listener.ctrl
• pool.listener.mgmt

The pool types on router are:

• pool.listener.link
• pool.link.dialer
• pool.route.handler
• pool.listener.xgress_edge (if edge is enabled)

Each pool has metrics for

• Current worker count
• Current queue size
• Current active works
• Work timer, which includes count of work performed, meter for work rate and histogram for work execution time

An example of the metric names for pool.listener.link:

pool.listener.link.busy_workers
pool.listener.link.queue_size
pool.listener.link.work_timer.count
pool.listener.link.work_timer.m15_rate
pool.listener.link.work_timer.m1_rate
pool.listener.link.work_timer.m5_rate
pool.listener.link.work_timer.max
pool.listener.link.work_timer.mean
pool.listener.link.work_timer.mean_rate
pool.listener.link.work_timer.min
pool.listener.link.work_timer.p50
pool.listener.link.work_timer.p75
pool.listener.link.work_timer.p95
pool.listener.link.work_timer.p99
pool.listener.link.work_timer.p999
pool.listener.link.work_timer.p9999
pool.listener.link.work_timer.std_dev
pool.listener.link.work_timer.variance

Add Link Count and Cost to Circuit Events

Link Count will now be available on all circuit events. Circuit cost will be available on
circuit created events. The circuit cost is the full circuit cost (router costs + link costs

• terminator costs).

Example:

{
  "namespace": "fabric.circuits",
  "event_type": "created",
  "circuit_id": "XpSWLZB1P",
  "timestamp": "2022-05-11T13:00:06.976266668-04:00",
  "client_id": "cl31tuu93000iaugd57qv6hjc",
  "service_id": "dstSybunfM",
  "creation_timespan": 969933,
  "path": "[r/h-DqbP927]->[l/1qp6LIzSlWkQM1jSSTJG1j]->[r/Ce1f5dDCey]",
  "link_count": 1,
  "path_cost": 890
}

Remove ziti-fabric CLI command

The previously deprecated ziti-fabric command will no longer be published as part of Ziti releases.
All of ziti-fabric's functionality is available in the ziti CLI under ziti fabric.

Add link delete

If a link gets in a bad state (see bug below for how this could happen), you can now use
ziti fabric delete link <link id>. This will remove the link from the controller as well
as send link faults to associated routers. If the link is not known to the controller, a
link fault will be sent to all connected routers.

Miscellaneous

The ziti-probe tool will no longer be built and published as part of Ziti releases.

Bug Fixes

• openziti/fabric#393
• openziti/fabric#395
• openziti/channel#50
• ziti fabric list circuits was showing the router id instead of the link id in the circuit path