From ef569013bf934ec8cc55812a68af8b00f5a7fbfb Mon Sep 17 00:00:00 2001 From: Gabor Retvari Date: Thu, 20 Feb 2025 21:51:47 +0100 Subject: [PATCH] doc: Document premium features --- cmd/stunnerctl/README.md | 26 ++++- docs/AUTH.md | 2 + docs/GATEWAY.md | 29 +++--- docs/PREMIUM.md | 204 +++++++++++++++++++++++++++++++++++++++ docs/SECURITY.md | 10 +- 5 files changed, 253 insertions(+), 18 deletions(-) create mode 100644 docs/PREMIUM.md diff --git a/cmd/stunnerctl/README.md b/cmd/stunnerctl/README.md index 02079af1..12eedf42 100644 --- a/cmd/stunnerctl/README.md +++ b/cmd/stunnerctl/README.md @@ -136,6 +136,20 @@ The `status` sub-command reports the status of the dataplane pods for a gateway, TERMINATING ``` +### License status + +STUNner requires a valid license to unlock premium features. The below will report STUNner's license status: + +```console +stunnerctl license +License status: + Subscription type: member + Enabled features: DaemonSet, STUNServer, ... + Last updated: ... +``` + +This command will connect to your STUNner gateway operator and validate your license. It will also report any errors STUNner may have encountered while validating your license. + ### Authentication The `auth` sub-command can be used to obtain a TURN credential or a full ICE server config for connecting to a specific gateway. The authentication service API is usually served by a separate [STUNner authentication server](https://github.com/l7mp/stunner-auth-service) deployed alongside the gateway operator. The main use of this command is to feed an ICE agent manually with the ICE server config to connect to a specific STUNner gateway. @@ -144,14 +158,22 @@ The `auth` sub-command can be used to obtain a TURN credential or a full ICE ser ``` console stunnerctl -n stunner auth udp-gateway - {"iceServers":[{"credential":"pass-1","urls":["turn:10.104.19.179:3478?transport=udp"],"username":"user-1"}],"iceTransportPolicy":"all"} + {"iceServers":[{"credential":"pass-1","urls":["turn:1A.B.C.D:3478?transport=udp"],"username":"user-1"}],"iceTransportPolicy":"all"} ``` - Request a plain [TURN credential](https://datatracker.ietf.org/doc/html/draft-uberti-behave-turn-rest-00) using the authentication service deployed into the `stunner-system-prod` namespace: ``` console stunnerctl -n stunner auth udp-gateway --auth-turn-credential --auth-service-namespace=stunner-system-prod - {"password":"pass-1","ttl":86400,"uris":["turn:10.104.19.179:3478?transport=udp"],"username":"user-1"} + {"password":"pass-1","ttl":86400,"uris":["turn:A.B.C.D:3478?transport=udp"],"username":"user-1"} +``` + +- Obtain an ICE config for the Gateway `stunner/udp-gateway` and setting the user-id to `my-user` + (useful for ephemeral authentication): + + ``` console + stunnerctl -n stunner auth udp-gateway --username my-user + {"iceServers":[{"credential":"...=","urls":["turn:A.B.C.D:3478?transport=udp"],"username":"1740150333:my-user"}],"iceTransportPolicy":"all"} ``` ### ICE test diff --git a/docs/AUTH.md b/docs/AUTH.md index 329b4e1b..ea659db1 100644 --- a/docs/AUTH.md +++ b/docs/AUTH.md @@ -2,6 +2,8 @@ STUNner uses the IETF STUN/TURN protocol suite to ingest media traffic into a Kubernetes cluster, which, [by design](https://datatracker.ietf.org/doc/html/rfc5766#section-17), provides comprehensive security. In particular, STUNner provides message integrity and, if configured with the TURN-TLS or TURN-DTLS listeners, confidentiality. To complete the CIA triad, this guide shows how to configure user authentication with STUNner. +Note that STUNner can also be deployed as a STUN server without enabling the TURN protocol features (only available in premium tiers), in which case it runs with no authentication (`authType:none`). Refer to the [user guide](PREMIUM.md) for the details. + ## The long-term credential mechanism STUNner relies on the STUN [long-term credential diff --git a/docs/GATEWAY.md b/docs/GATEWAY.md index f92ed8b6..ae1df65c 100644 --- a/docs/GATEWAY.md +++ b/docs/GATEWAY.md @@ -60,18 +60,20 @@ spec: Below is a reference of the most important fields of the GatewayConfig [`spec`](https://kubernetes.io/docs/concepts/overview/working-with-objects/kubernetes-objects) -| Field | Type | Description | Required | -| :--- | :---: | :--- | :---: | -| `dataplane` | `string` | The name of the Dataplane template to use for provisioning `stunnerd` pods. Default: `default`. | No | -| `logLevel` | `string` | Logging level for the dataplane pods. Default: `all:INFO`. | No | -| `realm` | `string` | The STUN/TURN authentication realm to be used for clients to authenticate with STUNner. The realm must consist of lower case alphanumeric characters or `-` and must start and end with an alphanumeric character. Default: `stunner.l7mp.io`. | No | -| `authRef` | `reference` | Reference to a Secret (`namespace` and `name`) that defines the STUN/TURN authentication mechanism and the credentials. | No | -| `authType` | `string` | Type of the STUN/TURN authentication mechanism. Valid only if `authRef` is not set. Default: `static`. | No | -| `userName` | `string` | The username for [`static` authentication](AUTH.md). Valid only if `authRef` is not set. | No | -| `password` | `string` | The password for [`static` authentication](AUTH.md). Valid only if `authRef` is not set. | No | -| `sharedSecret` | `string` | The shared secret for [`ephemeral` authentication](AUTH.md). Valid only if `authRef` is not set. | No | -| `authLifetime` | `int` | The lifetime of [`ephemeral` authentication](AUTH.md) credentials in seconds. Not used by STUNner.| No | -| `loadBalancerServiceAnnotations` | `map[string]string` | A list of annotations that will go into the LoadBalancer services created automatically by STUNner to obtain a public IP address. See more detail [here](https://github.com/l7mp/stunner/issues/32). | No | +| Field | Type | Description | Required | +|:---------------------------------|:-------------------:|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------:| +| `dataplane` | `string` | The name of the Dataplane template to use for provisioning `stunnerd` pods. Default: `default`. | No | +| `logLevel` | `string` | Logging level for the dataplane pods. Default: `all:INFO`. | No | +| `realm` | `string` | The STUN/TURN authentication realm to be used for clients to authenticate with STUNner. The realm must consist of lower case alphanumeric characters or `-` and must start and end with an alphanumeric character. Default: `stunner.l7mp.io`. | No | +| `authRef` | `reference` | Reference to a Secret (`namespace` and `name`) that defines the STUN/TURN authentication mechanism and the credentials. | No | +| `authType` | `string` | Type of the STUN/TURN authentication mechanism. Valid only if `authRef` is not set. Default: `static`. | No | +| `userName` | `string` | The username for [`static` authentication](AUTH.md). Valid only if `authRef` is not set. | No | +| `password` | `string` | The password for [`static` authentication](AUTH.md). Valid only if `authRef` is not set. | No | +| `sharedSecret` | `string` | The shared secret for [`ephemeral` authentication](AUTH.md). Valid only if `authRef` is not set. | No | +| `authLifetime` | `int` | The lifetime of [`ephemeral` authentication](AUTH.md) credentials in seconds. Not used by STUNner. | No | +| `loadBalancerServiceAnnotations` | `map[string]string` | A list of annotations that will go into the LoadBalancer services created automatically by STUNner to obtain a public IP address. See more detail [here](https://github.com/l7mp/stunner/issues/32). | No | +| `userQuota` | `int` | Limit the number of allocations per username (**not supported in the free tier**). Default is 0, which means no quota. | No | +| `stunMode` | `bool` | Toggle STUN-only mode (**not supported in the free tier**). In this mode STUNner responds only to STUN requests; any TURN allocation request is rejected. Default is false. | No | At least a valid username/password pair *must* be supplied for `static` authentication, or a `sharedSecret` for the `ephemeral` mode, either via an external Secret or inline in the GatewayConfig. External authentication settings override inline settings. Missing both is an error. @@ -251,7 +253,7 @@ The below table summarizes the Gateway annotations supported by STUNner. |:----------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------|:--------------:| | `stunner.l7mp.io/service-type: ` | [Type of the Service](https://kubernetes.io/docs/concepts/services-networking/service) per Gateway, either `ClusterIP`, `NodePort`, or `LoadBalancer`. | `LoadBalancer` | | `stunner.l7mp.io/enable-mixed-protocol-lb: ` | [Mixed protocol load balancer service](https://kubernetes.io/docs/concepts/services-networking/service/#load-balancers-with-mixed-protocol-types) support. | False | -| `stunner.l7mp.io/external-traffic-policy: ` | Se the value to `Local` to preserve clients' source IP at the load balancer. | `Cluster` | +| `stunner.l7mp.io/external-traffic-policy: ` | Set the value to `Local` to preserve clients' source IP at the load balancer. | `Cluster` | | `stunner.l7mp.io/disable-managed-dataplane: ` | Switch managed-dataplane support off for a Gateway. | False | | `stunner.l7mp.io/nodeport: ` | Request a specific NodePort for particular listeners. Value is a JSON map of listener-nodeport key-value pairs. | None | | `stunner.l7mp.io/targetport: ` | Request a specific target port for particular listeners. Value is a JSON map of listener-targetport key-value pairs. | None | @@ -411,6 +413,7 @@ The following fields can be set in the Dataplane `spec` to customize the provisi | `hostNetwork` | `bool` | Deploy the [dataplane](/cmd/stunnerd/README.md) into the host network namespace of Kubernetes nodes. Useful for implementing headless TURN services. May require elevated privileges. Default: false. | No | | `disableHealthCheck` | `bool` | Disable health-checking. If true, enable HTTP health-checks on port 8086: liveness probe responder will be exposed on path `/live` and readiness probe on path `/ready`. Default: true. | No | | `enableMetricsEndpoint` | `bool` | Enable Prometheus metrics scraping. If true, a metrics endpoint will be available at `http://0.0.0.0:8080`. Default: false. | No | +| `dataplaneResource` | `string` | Kubernetes resource type to be used for deploying the dataplane, either a Deployment (default) or DaemonSet (**not supported in the free tier**). | No | There can be multiple Dataplane resources defined in a cluster, say, one for the production workload and one for development. Use the `spec.dataplane` field in the GatewayConfig to choose the Dataplane per each STUNner install. diff --git a/docs/PREMIUM.md b/docs/PREMIUM.md new file mode 100644 index 00000000..fa29e7bd --- /dev/null +++ b/docs/PREMIUM.md @@ -0,0 +1,204 @@ +# Premium features + +[STUNner](https://github.com/l7mp/stunner) comes in multiple flavors. The open-source [*STUNner free* tier](/README.md) will suit most simple applications. Certain advanced features are however available only in STUNner's paid premium editions. The *STUNner member* tier includes the premium features targeted for small and medium scale operations, while the fully featured *STUNner enterprise* tier is optimized for large-scale deployments. + +This guide helps you configure STUNner's premium features; see [here](https://l7mp.io/#/products) for up-to-date info on how to purchase a license. + +## Getting started + +Start with a fresh Kubernetes cluster. Remove all previous STUNner installations, otherwise some premium features may not be available. + +Use the below Helm chart to deploy the premium version of STUNner: + +```console +helm repo add stunner https://l7mp.io/stunner +helm repo update +helm install stunner stunner/stunner-premium --create-namespace --namespace=stunner-system +``` + +We recommend you deploy STUNner into the `stunner-system` namespace, this simplifies configuration later. See the [installation guide](INSTALL.md) for more info on customization options for the Helm chart. + +## License validation + +In order to unlock the premium features, STUNner will need a valid customer key. You should have received one during the subscription procedure; if not, [contact us](mailto:info@l7mp.io). STUNner will search for the customer key in the Kubernetes Secret named `stunner-gateway-operator-customer-secret` in the namespace where you deployed STUNner (usually `stunner-system`). + +1. Set your customer key: The first step is to update the default Secret created by the Helm chart with your customer key. The simplest way is the manually edit the Secret: + + ```console + EDITOR=nano kubectl -n stunner-system edit secret stunner-gateway-operator-customer-secret + ``` + + You should see something like the below (with some additional lines that you can safely ignore): + + ```yaml + apiVersion: v1 + kind: Secret + metadata: + name: stunner-gateway-operator-customer-secret + namespace: stunner-system + type: Opaque + data: + CUSTOMER_KEY: X19kZWJ1Z19jdXNfUkZPU2FHMnNJNWtwNms= + ``` + + Rewrite `data.CUSTOMER_KEY` with your customer key. In order to prevent Kubernetes from base64-encoding the key, use the `stringData` field instead of `data`: `stringData.CUSTOMER_KEY`. Eventually you should see something like the below: + + ```yaml + apiVersion: v1 + kind: Secret + metadata: + name: stunner-gateway-operator-customer-secret + namespace: stunner-system + type: Opaque + stringData: + CUSTOMER_KEY: + ``` + + Don't forget to replace the placeholder `` with your own customer key. Save and exit. If all goes well, `kubectl` should report that the secret has been successfully modified: + + ``` + secret/stunner-gateway-operator-customer-secret edited + ``` + + Alternatively, you can use the below to patch the Secret with your customer key: + + ```console + kubectl -n stunner-system patch secret stunner-gateway-operator-customer-secret --type='json' \ + -p='[{"op": "add" ,"path": "/stringData" ,"value": {}}, {"op": "replace" ,"path": "/stringData/CUSTOMER_KEY" ,"value": ""}]' + ``` + +2. Restart the operator: STUNner will read the customer key on startup so every time you update your customer key you have to restart the operator: + + ```console + kubectl -n stunner-system rollout restart deployment stunner-gateway-operator-controller-manager + ``` + + If all goes well, the operator will validate the license associated with your customer key and unlock the premium features available in your tier. + +3. Check your license status: The simplest way to check your license status is via the handy [`stunnerctl`](/cmd/stunnerctl/README.md) command line tool: + + ```console + stunnerctl license + License status: + Subscription type: member + Enabled features: DaemonSet, STUNServer, ... + Last updated: ... + ``` + + This command will connect to your STUNner operator and report the license status. It will also report any errors encountered while validating your license. + + It is also possible to check the license status of STUNners's dataplane pods. The below [`stunnerctl`](/cmd/stunnerctl/README.md) command will connect to each dataplane pod deployed into the `stunner` namespace and report the running licensing status. + + ```console + stunnerctl -n stunner status + stunner/udp-gateway-...: + admin:{id="stunner/udp-gateway",logLevel="all:INFO",health-check="http://:8086",quota=0,license-info={tier=enterprise,unlocked-features=TURNOffload,UserQuota,..,valid-until=...}} + ... + ``` + + Look for the `license-info` field in the above admin status: you should see your subscription tier (e.g., `free`, `member` or `enterprise`) with all the available premium features in `unlocked-features` listed. + + If something goes wrong, check the gateway operator logs for lines like the below that should help you debug the problem: + + ``` + kubectl -n stunner-system logs $(kubectl -n stunner-system get pods -l \ + control-plane=stunner-gateway-operator-controller-manager -o jsonpath='{.items[0].metadata.name}') + ... + license-mgr license manager client created {"server": "https://license.l7mp.io:443", "customer-key-status": "set"} + ... + license-mgr new license status {"subscription-type": "enterprise", "enabled-features": ["DaemonSet", "UserQuota", "STUNServer", "TURNOffload"], "last-updated": "..."} + ... + ``` + +## Premium features + +STUNner's premium features are designed to help medium to large scale enterprises to administer, scale and operate a fleet of STUN and TURN servers. Below is a list of the premium features currently available in STUNner. + +### User quota (feature: `UserQuota`, available in the member and enterprise tiers) + +Once a client has obtained a valid TURN authentication credential, they can open any number of TURN connections via STUNner by reusing the same credential. Since TURN credentials are available in plain text format at clients (this is by WebRTC API design), malicious clients can easily launch a Denial-of-Service (DoS) attack by creating many TURN allocations in quick succession. Unfortunately, even an [`ephemeral` credential](AUTH.md) leaves an open time window for a DoS attack before it expires. + +STUNner's `UserQuota` feature allows to set an upper limit on the number of simultaneous allocations that can be made with the same TURN credential. This feature is available in your tier if `UserQuota` is enabled in the license status (recall, the status can be obtained using `stunnerctl license`). + +Note that STUNner's quotas are per per-user-id. This means if that if you obtain multiple different credentials for the same user-id (e.g., by using `stunnerctl auth --username my-user`) then the credentials map to the same quota: the TURN allocations authenticated with the same credential add up when imposing the quota. Also note that stale TURN allocations also count towards the quota. If a client fails to close unused TURN allocations (which TURN clients routinely do) then these stale allocations will be active until they time out (usually 5 mins). This may prevent clients from re-connecting when an overly restrictive user quota is in effect. + +Configure a user quota for a Gateway by setting the `userQuota` field in the corresponding GatewayConfig: + +```yaml +apiVersion: stunner.l7mp.io/v1 +kind: GatewayConfig +metadata: + name: stunner-gatewayconfig +spec: + authType: ephemeral + sharedSecret: my-shared-secret + userQuota: 10 +``` + +This will set the quota to 10. Setting the quota to zero means no quota (the default setting). + +You can query the configured user quota using [`stunnerctl`](/cmd/stunnerctl/README.md). Suppose you deployed a TURN/UDP gateway called `udp-gateway` in the `stunner` namespace. The current quota can be obtained as follows: + +```console +stunnerctl -n stunner status udp-gateway -o jsonpath='{.admin.quota}' +10 +``` + + + + + + + + +Once the number of allocations created for a user-id reach the configured quota, new connections will be rejected with an `error 486: Allocation Quota Exceeded` error status. + +### STUN server mode (feature: `STUNServer`, available in the member and enterprise tiers) + +By default STUNner is configured to run as a TURN server. As TURN is an extension of the STUN protocol, this setting lets STUNner to serve plain [STUN requests](https://medium.com/l7mp-technologies/deploying-a-scalable-stun-service-in-kubernetes-c7b9726fa41d) as well. Running a TURN server, however, comes at a potentially high cost, typically needing a high-bandwidth network connection and consuming pricey CPU resources. This is suboptimal for the case when STUNner is deployed as a pure STUN service, since malicious clients can consume excess server resources by creating phony TURN allocations. + +In order to prevent this potential DoS attack vector, STUNner's TURN protocol engine can be completely turned off. This makes prohibits clients from making new TURN allocations, but still guarantees that STUNner will serve STUN requests. This feature is available in your tier if `STUNServer` is enabled in the license status (recall, the status can be obtained using `stunnerctl license`). + +To switch a Gateway into STUN server mode, set `STUNServer: true` in the corresponding GatewayConfig: + +```yaml +apiVersion: stunner.l7mp.io/v1 +kind: GatewayConfig +metadata: + name: stunner-gatewayconfig +spec: + STUNMode: true +``` + +This will disable STUNner's authentication engine, prohibiting clients from creating TURN allocations all together: + +```console +bin/stunnerctl -n stunner status udp-gateway -o jsonpath='{.auth.type}' +none +``` + +Set `STUNServer: false` to re-enable the TURN protocol engine. + +### Deploying the dataplane in a DaemonSet (feature: `DaemonSet`, available in the member and enterprise tiers) + +By default, the TURN server pods that run the dataplane for STUNner gateways are deployed into a Kubernetes Deployment. This ensures that a fix number of TURN servers are available per each Gateway. In certain cases, however, it may be desirable to deploy STUNner with a single dataplane pod per each Kubernetes node instead. This is crucial, for instance, when the STUNner dataplane is [deployed in the host-network namespace](https://github.com/l7mp/stunner/blob/main/docs/GATEWAY.md#dataplane) to run a [public TURN service](https://medium.com/l7mp-technologies/running-stunner-as-a-public-turn-server-1a2c61f78e67), or when a Gateway is exposed with the [`service.spec.externalTrafficPolicy: Local`](https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/#preserving-the-client-source-ip) configuration to implement [Direct Server Return](https://en.wikipedia.org/w/index.php?title=Load_balancing_(computing)#Load_balancer_features) for minimizing clients' round-trip-time. + +To configure STUNner to run a single STUNner dataplane pod per each node in the Kubernetes cluster, you can set `spec.dataplaneResource` to `DaemonSet` in the [`Dataplane` resource](https://github.com/l7mp/stunner/blob/main/docs/GATEWAY.md#dataplane) corresponding to your Gateway. This will instruct STUNner to re-deploy the dataplane into a [Kubernetes DaemonSet](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset) instead of a Deployment. This feature is available in your tier if `DaemonSet` is enabled in the license status (recall, the status can be obtained using `stunnerctl license`). + +The below will set the dataplane for all gateways using the `default` Dataplane to use a DaemonSet. The `hostNetwork: true` setting will deploy the TURN server pod in the host-network namespace of each Kubernetes node in a cluster. + +```yaml +apiVersion: stunner.l7mp.io/v1 +kind: Dataplane +metadata: + name: default +spec: + dataplaneResource: DaemonSet + hostNetwork: true +``` + +Set `dataplaneResource: Deployment` to return to the default deployment mode. + +### TURN offload (available only in the enterprise tier) + +To be available soon. diff --git a/docs/SECURITY.md b/docs/SECURITY.md index 2131a6c7..75dbcb65 100644 --- a/docs/SECURITY.md +++ b/docs/SECURITY.md @@ -92,11 +92,11 @@ By default, STUNner uses a single static username/password pair for all clients For production deployments we recommend the `ephemeral` authentication mode, which uses per-client fixed lifetime username/password pairs. This makes it more difficult for attackers to steal and reuse STUNner's TURN credentials. See the [authentication guide](AUTH.md) for configuring STUNner with `ephemeral` authentication. +Note that STUNner can also be deployed as a STUN server without enabling the TURN protocol (only available in the premium tiers), in which case it needs no authentication. Refer to the [user guide](PREMIUM.md) for the details. + ## Access control -STUNner requires the user to explicitly open up external access to internal services by specifying -a proper UDPRoute. For instance, the below UDPRoute allows access *only* to the `media-server` -service in the `media-plane` namespace, and nothing else. +STUNner requires the user to explicitly open up external access to internal services by specifying a proper UDPRoute. For instance, the below UDPRoute allows access *only* to the `media-server` service in the `media-plane` namespace, and nothing else. ```yaml apiVersion: stunner.l7mp.io/v1 @@ -151,3 +151,7 @@ The trick in STUNner is that both the TURN relay transport address and the media The threat model is that, possessing the correct credentials, an attacker can scan the *private* IP address of all STUNner pods and all media server pods. This should pose no major security risk though: remember, none of these private IP addresses can be reached externally. The attack surface can be further reduced to the STUNner pods' private IP addresses by using the [symmetric ICE mode](DEPLOYMENT.md#symmetric-ice-mode). Nevertheless, if worried about information exposure then STUNner may not be the best option at the moment. In later releases, we plan to implement a feature to obscure the relay transport addresses returned by STUNner. Please file an issue if you think this limitation is a blocker for your use case. + +## User quota (only available in the premium version) + +In order to prevent potential Denial-of-Service (DoS) attacks that may be launched by an attacker creating a huge number of parallel TURN allocations to overwhelm STUNner, it is possible to impose a user quota on the number of simultaneous allocations that can be made with the same TURN credential. Refer to the `UserQuota` feature description in the [premium user guide](PREMIUM.md) for the details.