Releases: openziti/ziti
v0.33.0
Release 0.33.0
What's New
- SDK Terminator stability improvements
- Minor feature updates and bug fixes
SDK Terminator stability improvements
This release was focused on creating a chaos test for SDK terminators, running it and fixing any issues found.
The test repeatedly and randomly restarts the controller, routers and tunnelers then verifies that terminators
end up in the correct state.
The following tools were also used/added to aid in diagnosing and fixing issues:
ziti fabric validate router-sdk-terminators
- Compares the controller state with the router state
ziti fabric validate terminators
- Checks each selected terminator to ensure it's still valid on the router and/or sdk
ziti fabric inspect sdk-terminators
- Allows inspecting each routers terminator state
ziti fabric inspect router-messaging
- Allows inspecting what the controller has queued for router state sync and terminator validations
ziti edge validate service-hosting
- Shows how many terminators each identity which can host a service has
Several changes were made to the terminator code to ensure that terminators are properly created and cleaned up.
The routers now use an adaptive rate limiter to control how fast they send terminator related requests to the
controller. For this to work properly, the rate limiting on the controller must be enabled, so it can report
back to the routers when it's got too much work.
Component Updates and Bug Fixes
- github.com/openziti/edge-api: v0.26.10 -> v0.26.12
- github.com/openziti/ziti: v0.32.2 -> v0.33.0
- Issue #1815 - Panic if api session sync failed handler is called twice in the router
- Issue #1794 - Add SDK terminator chaos test and fix any bugs found as part of chaos testing
- Issue #1781 - Improve performance when adding intercepted services
- Issue #1369 - Allow filtering by policy type when listing identities for service or services for identity
- Issue #1791 - route dial isn't checking for network timeouts correctly
- Issue #1204 - ziti cli identity tags related flags misbehaving
- Issue #987 - "ziti create config router edge" doesn't know about --tunnelerMode proxy
- Issue #652 - Update CLI script M1 Support when github actions allows
v0.32.2
NOTE: This release has an incompatibility with the latest c-based tunnelers, which will cause hosting services to fail.
Release 0.32.2
What's New
- Terminator performance improvements
- API Rate Limiter enabled by default
Component Updates and Bug Fixes
-
github.com/openziti/channel/v2: v2.0.117 -> v2.0.119
- Issue #127 - Support some additional types in message headers
-
github.com/openziti/foundation/v2: v2.0.36 -> v2.0.37
-
github.com/openziti/identity: v1.0.69 -> v1.0.70
-
github.com/openziti/metrics: v1.2.43 -> v1.2.45
-
github.com/openziti/runzmd: v1.0.37 -> v1.0.38
-
github.com/openziti/sdk-golang: v0.22.21 -> v0.22.28
- Issue #495 - Refresh edge session if listen attempt fails, to ensure that session is still valid
-
github.com/openziti/storage: v0.2.28 -> v0.2.30
-
github.com/openziti/transport/v2: v2.0.121 -> v2.0.122
-
github.com/openziti/ziti: v0.32.1 -> v0.32.2
- Issue #1741 - Prevent stuck links
- Issue #1736 - controller crashes generating create circuit responses
- Issue #1733 - Improve terminator creation performance
- Issue #1734 - Make API rate limiter enabled by default
- Issue #1726 - Fix some sdk hosting logging
- Issue #1725 - Fix panic in entity event processing
v0.32.1
Do not use, the release contains a deadlock which can be triggered if many SDK terminators are being created at a time.
Release 0.32.1
What's New
- Bugfixes
- New router setting to control startup timeout
Router startup timeout
The router now has a configuration setting to control how long it wait on startup to be able to
connect to a controller, before it gives up and exits.
ctrl:
endpoints:
- tls:localhost:1280
startupTimeout: 5m
Component Updates and Bug Fixes
-
github.com/openziti/channel/v2: v2.0.116 -> v2.0.117
- Issue #125 - Ensure reconnecting channel is marked as connected before calling reconnect callback
-
github.com/openziti/edge-api: v0.26.8 -> v0.26.10
-
github.com/openziti/sdk-golang: v0.22.17 -> v0.22.21
-
github.com/openziti/ziti: v0.32.0 -> v0.32.1
- Issue #1709 - Fix link management race conditions found by chaos testing
- Issue #1715 - Ensure controller raft peers don't end up with duplicate connections
- Issue #1702 - Add link management chaos test
- Issue #1691 multiple er re-enrolls creates multiple enrollments
v0.32.0
NOTE: This release has bugs related to link management and is not recommended for production use!
Release 0.32.0
What's New
- Auth Rate Limiter
- Link Management Fixes
- ziti edge quickstart command deprecates redundant --already-initialized flag. The identical behavior is implied by --home.
Backwards compatibility
This release includes new response types from the REST authentication APIS. They are now able to return
429
(server too busy) responses to auth requests. As this is an API change, the version number is
being bumped to 0.32.
If controller and router are both v0.32 or later, only the router which dialed a link will report it to the controller.
If the controller is older, newer routers will report links from both the dialing and listening side of the link.
Auth Rate Limiter
In order to prevent clients from overwhelming the server with auth requests, an auth rate limiter has been introduced.
The rate limiter is adaptive, in that it will react to auth attempts timing out by shrinking the number of allowed
queued auth attempts. The number will slowly recover over time.
Example configuration:
edge:
# This section allows configurating the rate limiter for auth attempts
authRateLimiter:
# if disabled, no auth rate limiting with be enforced
enabled: true
# the smallest window size for auth attempts
minSize: 5
# the largest allowed window size for auth attempts
maxSize: 250
New metrics:
auth.limiter.queued_count
- current number of queued auth attemptsauth.limiter.window_size
- current size at which new auth attempts will be rejectedauth.limiter.work_timer
- tracks the rate at which api sessions are being created and how long it's taking to create them
Link Management Fixes
With long lived link ids, there was potential for link control message to be ambiguous, as the link id wasn't enough to identify
a specific iteration of that link. An iteration field has been added to links so that messaging is unambiguous.
Links will also only be reported from the dialing router now to reduce ambiguouity and race condition in link control channel
messaging.
Router SSL Handshake Timeout Config
There is a new router config setting which allows setting the SSL handshake timeout for TLS connections, when using ALPN for listeners.
tls:
handshakeTimeout: 15s
Component Updates and Bug Fixes
-
github.com/openziti/channel/v2: v2.0.111 -> v2.0.116
- Issue #123 - Ensure hello messages respect connect timeout
- Issue #120 - Allow handling new underlay instances with function instead of channel
-
github.com/openziti/edge-api: v0.26.6 -> v0.26.8
-
github.com/openziti/foundation/v2: v2.0.35 -> v2.0.36
- Issue #391 - goroutine pool can stall if configured for 0 min workers and with single producer
-
github.com/openziti/identity: v1.0.68 -> v1.0.69
-
github.com/openziti/metrics: v1.2.41 -> v1.2.43
-
github.com/openziti/runzmd: v1.0.36 -> v1.0.37
-
github.com/openziti/sdk-golang: v0.22.0 -> v0.22.17
- Issue #482 - Deprecate ListenOptions.MaxConnections in favor of MaxTerminators
-
github.com/openziti/secretstream: v0.1.14 -> v0.1.16
-
github.com/openziti/storage: v0.2.27 -> v0.2.28
-
github.com/openziti/transport/v2: v2.0.119 -> v2.0.121
- Issue #73 - Allow overriding shared TLS/ALPN listener SSL handshake timeout
-
github.com/openziti/ziti: v0.31.4 -> v0.32.0
- Issue #1692 - Improve link stability with long lived link ids
- Issue #1693 - Make links owned by the dialing router
- Issue #1685 - Race condition where we try to create terminator after client connection is closed
- Issue #1678 - Add link validation utility
- Issue #1673 - xgress dialers not getting passed xgress config
- Issue #1669 - Make sure link accepts are not single threaded
- Issue #1657 - Add api session rate limiter
v0.31.4
Release 0.31.4
What's New
- Bug fix for a data flow stall which is especially likely to happen on circuits with single router paths
Thanks
- @marvkis - for providing high quality debug data which made tracking down a couple of flow control stall issues much easier
Component Updates and Bug Fixes
-
github.com/openziti/metrics: v1.2.40 -> v1.2.41
-
github.com/openziti/sdk-golang: v0.21.2 -> v0.22.0
- Issue #468 - SDK does an unnecessary number of session refreshes
-
github.com/openziti/storage: v0.2.26 -> v0.2.27
-
github.com/openziti/ziti: v0.31.3 -> v0.31.4
- Issue #1645 - Once routers share a link id, we can't use the link id to decide which duplicate link to discard
- Issue #1642 - Revert posture check optimization
- Issue #1586 - If ack is received before payload is processed by link send buffer, a stall can result
v0.31.3
DO NOT USE THIS RELEASE
This release contains a posture check optimization to speed up service list queries. Unfortunately it also broke posture query support for networks with no posture checks.
The 0.31.4 release reverts the optimization. We'll revisit the optimization in a future release.
Release 0.31.3
What's New
- Services Max Idle Time
- Add/Remove Peer and Transfer Leadership via REST
Service Max Idle Time
A max idle time can now be configured on services. The default value of 0 indicates that no maximum will
be enforced. A circuit is considered idle when no traffic is flowing across through the initiating or
terminating router.
ziti edge create service test-service --max-idle-time 5m
Note that the idle time calculation is done on the router, so if max idle time on a service is less
than the configured scan interval on the router, it make take longer than expected for idle circuits
to be removed.
Raft Cluster Management via REST
The controller now allows some Raft cluster management operations to be performed via REST.
NOTE: If your cluster is not bootstrapped yet, the REST API won't be available. These will only work on a bootstrapped cluster!
The following operations are now supported:
- Add member
- Remove member
- Transfer leadership
ziti fabric raft add-member tls:localhost:6363
ziti fabric raft add-member tls:localhost:6464
ziti fabric raft transfer-leadership
ziti fabric raft transfer-leadership ctrl3
ziti fabric raft remove-member ctrl2
ziti fabric raft remove-member ctrl3
Component Updates and Bug Fixes
-
github.com/openziti/edge-api: v0.26.1 -> v0.26.6
-
github.com/openziti/sdk-golang: v0.20.139 -> v0.21.2
- Issue #465 - Allow listen options to specify how many listeners need to be established before returning
- Issue #462 - Allow refreshing a single service
-
github.com/openziti/ziti: v0.31.2 -> v0.31.3
- Issue #1583 - xgress: Potential data stall due when processing acks after checking window size
- Issue #1578 - Send BindSuccess notifications to SDK if supported
- Issue #1544 - Support transfer raft leadership via REST
- Issue #1543 - Support add/remove raft peer via REST
- Issue #1496 - Configurable Timer needed to close idle circuits
- Issue #1402 - Allow router to decomission itself
v0.31.2
Release 0.31.2
What's New
- Go version updated from 1.20 to 1.21
v0.31.1
Release 0.31.1
What's New
- SDK Hosting Improvements
- Terminator validation utility
- Circuit/Link query support
SDK Hosting Improvments
In previous versions of OpenZiti, if many SDK clients were attempting to establish hosting, the controller could get overwhelmed.
In this release, routers will use the rate limiter pool introduced in 0.27.6 when creating terminators on behalf of sdk clients
hosting applications. Additionally, routers now have the ability to verify terminator state with the sdk, if the sdk supports it.
In general, hosting large numbers of services using the sdk should now be less suceptible to thundering herd issues.
Manual Terminator Validation
There is a new CLI command available to validate terminator state. This is primarily a developer tool to validate that terminator
setup logic is correct. However it may also be used to diagnose and resolve issues with production systems, should the need arise.
ziti fabric validate terminators
Circuit/Link Query Support
Previously listing circuit and links always showed the full list. This is because these types are in memory only and are not stored
in the bbolt datastore. There's now basic support for querying in-memory types and circuits and links can now be filtered/paged/sorted
the same as other entity types.
Component Updates and Bug Fixes
-
github.com/openziti/channel/v2: v2.0.105 -> v2.0.111
- Issue #118 - Allowing checking if reconnecting impl is currently connected
-
github.com/openziti/edge-api: v0.26.0 -> v0.26.1
-
github.com/openziti/foundation/v2: v2.0.33 -> v2.0.35
-
github.com/openziti/identity: v1.0.66 -> v1.0.68
-
github.com/openziti/metrics: v1.2.37 -> v1.2.40
-
github.com/openziti/runzmd: v1.0.33 -> v1.0.36
-
github.com/openziti/sdk-golang: v0.20.129 -> v0.20.139
- Issue #457 - Add inspect support
- Issue #450 - Support idempotent terminator creation
-
github.com/openziti/secretstream: v0.1.13 -> v0.1.14
-
github.com/openziti/storage: v0.2.23 -> v0.2.26
- Issue #57 - Support querying collections of in memory objects
-
github.com/openziti/transport/v2: v2.0.113 -> v2.0.119
-
github.com/openziti/ziti: v0.31.0 -> v0.31.1
- Issue #1555 - Consolidate fabric/edge persistence code
- Issue #1547 - Support filtering, sorting and paging circuits and links
- Issue #1446 - Allow for idempotent sdk based terminators
- Issue #1540 - Transit router create fails in HA environment
- Issue #1523 - Bootstrap members not working
- Issue #1525 - Improve cluster list output
- Issue #1519 - Simplify link ack handling
- Issue #1513 - DNS service failure should not cause a router restart
- Issue #1494 - Panic if applying raft log returns nil result
v0.31.0
Release 0.31.0
What's New
- Rate limited for model changes
Rate Limiter for Model Changes
To prevent the controller from being overwhelmed by a flood of changes, a rate limiter
can be enabled in the configuration file. A maximum number of queued changes can also
be configured. The rate limited is disabled by default for now. If not specified the
default number of queued changes is 100.
When the rate limit is hit, an error will be returned. If the request came in from
the REST API, the response will use HTTP status code 429 (too many requests).
The OpenAPI specs have been updated, so if you're using a generated client to make
REST calls, it's recommened that you regenerate your client.
commandRateLimiter:
enabled: true
maxQueued: 100
If the rate limiter is enabled, the following metrics will be produced:
command.limiter.queued_count
- guage of the current number of queued operationscommand.limiter.work_timer
- timer for operations. Includes the following:- A histogram of how long operations take to complete
- A meter showing that rate at which operations are executed
- A count of how many operations have been executed
Component Updates and Bug Fixes
-
github.com/openziti/agent: v1.0.15 -> v1.0.16
-
github.com/openziti/channel/v2: v2.0.101 -> v2.0.105
-
github.com/openziti/edge-api: v0.25.38 -> v0.26.0
- Issue #49 - Add 429 responses to allow indicating that the server is too busy
-
github.com/openziti/identity: v1.0.64 -> v1.0.66
-
github.com/openziti/metrics: v1.2.36 -> v1.2.37
-
github.com/openziti/sdk-golang: v0.20.122 -> v0.20.129
- Issue #443 - Don't send close in reponse to a close on a listener
-
github.com/openziti/secretstream: v0.1.12 -> v0.1.13
-
github.com/openziti/storage: v0.2.20 -> v0.2.23
-
github.com/openziti/transport/v2: v2.0.109 -> v2.0.113
-
github.com/openziti/ziti: v0.30.5 -> v0.31.0
- Issue #1471 - Router links not resilient to controller crash
- Issue #1468 - Quickstart quietly fails if password is < 5 characters long
- Issue #1445 - Add controller update guardrail
- Issue #1442 - Network watchdog not shutting down when controller shuts down
- Issue #1465 - Upgrade functions
getZiti
andperformMigration
were only functional on Mac OS, now they are functional for Linux and Mac OSs. - Issue #1217 - Quickstart was improperly handling special characters in
ZITI_PWD
. Special characters are now supported forZITI_PWD
in quickstart functions.
v0.30.5
Release 0.30.5
What's New
- Initial proxy support in host.v1/host.v2
Proxy Support in host.v1/host.v2
host.v1
and host.v2
configurations may now specify a proxy to use.
Currently only HTTP Connect proxies which don't require authentication are supported.
Example using host.v1
{
"address": "192.168.2.50",
"port": 1234,
"protocol": "tcp",
"proxy": {
"address": "192.168.1.110:3128",
"type": "http"
}
}
Component Updates and Bug Fixes
-
github.com/openziti/channel/v2: v2.0.99 -> v2.0.101
-
github.com/openziti/edge-api: v0.25.37 -> v0.25.38
-
github.com/openziti/foundation/v2: v2.0.32 -> v2.0.33
-
github.com/openziti/identity: v1.0.63 -> v1.0.64
-
github.com/openziti/metrics: v1.2.35 -> v1.2.36
-
github.com/openziti/runzmd: v1.0.32 -> v1.0.33
-
github.com/openziti/sdk-golang: v0.20.116 -> v0.20.122
- Issue #436 - HTTP calls should respect environment proxy settings
-
github.com/openziti/storage: v0.2.18 -> v0.2.20
- Issue #52 - Grammar should expect single valid query followed by EOF
-
github.com/openziti/transport/v2: v2.0.107 -> v2.0.109
-
github.com/openziti/ziti: v0.30.4 -> v0.30.5
- Issue #1336 -
ziti edge quickstart
did
not create the usual edge router/service edge router policy. - Issue #1397 - HTTP Proxy suport for host.v1/host.v2 config types
- Issue #1423 - Controller crashes when edge router reconnects (Client Hello)
- Issue #1414 - Race condition in xgress_edge_tunnel tunneller at start but not seen in pre-compiled binary
- Issue #1406 - Entity change event dispatcher isn't shutting down properly when controller shuts down
- Issue #1382 - service failure costs are not shrinking over time
- Issue #1336 -