diff --git a/docusaurus/docs/guides/05-troubleshooting/pki-troubleshooting/browser-cert-viewer-invalid.png b/docusaurus/docs/guides/05-troubleshooting/pki-troubleshooting/browser-cert-viewer-invalid.png new file mode 100644 index 000000000..9afc00dcd Binary files /dev/null and b/docusaurus/docs/guides/05-troubleshooting/pki-troubleshooting/browser-cert-viewer-invalid.png differ diff --git a/docusaurus/docs/guides/05-troubleshooting/pki-troubleshooting/invalid-certificate.png b/docusaurus/docs/guides/05-troubleshooting/pki-troubleshooting/invalid-certificate.png new file mode 100644 index 000000000..f7e94defa Binary files /dev/null and b/docusaurus/docs/guides/05-troubleshooting/pki-troubleshooting/invalid-certificate.png differ diff --git a/docusaurus/docs/guides/05-troubleshooting/pki-troubleshooting.md b/docusaurus/docs/guides/05-troubleshooting/pki-troubleshooting/pki-troubleshooting.md similarity index 95% rename from docusaurus/docs/guides/05-troubleshooting/pki-troubleshooting.md rename to docusaurus/docs/guides/05-troubleshooting/pki-troubleshooting/pki-troubleshooting.md index 084731d9d..e88f166ce 100644 --- a/docusaurus/docs/guides/05-troubleshooting/pki-troubleshooting.md +++ b/docusaurus/docs/guides/05-troubleshooting/pki-troubleshooting/pki-troubleshooting.md @@ -1,183 +1,183 @@ ---- -title: PKI Troubleshooting ---- - -Configuring a Ziti Network's PKI can be confusing. Validating a single side of a mutual TLS connection is -straightforward it becomes tedious to ensure all the certificates and cas in use are valid when you have a fully -configured Ziti Network. It's the goal of this page to make diagnosing PKI issues easier. This guide will also use the -paths you will find in the Ziti Edge - Developer Edition. Change paths accordingly. - -### Prerequisites - -The following steps are [bash-based](https://en.wikipedia.org/wiki/Bash_(Unix_shell)) functions and use the -[openssl](https://www.openssl.org/), [jq](https://stedolan.github.io/jq/) and [ruby](https://www.ruby-lang.org/en/) -commands. If you don't have bash, openssl and ruby - this page is not for you! Do your best to follow along with the -scripts and guidance herein or just make sure bash, openssl, ruby, and jq are installed. All of which are widely -available on linux/MacOS/Windows. - -The `ruby` and `jq` commands are not strictly required. They are used to make it easy for you to copy/paste these -commands. The `ruby` command is used to translate yaml into json while the `jq` command is used to pull the specific -values out of the given files. You can certainly do the same manually (without `ruby` and `jq`) if you choose. - -#### Define the verifyCertAgainstPool Function - -In your bash prompt copy and paste these two functions: - - function yaml2json() - { - ruby -ryaml -rjson -e 'puts JSON.pretty_generate(YAML.load(ARGF))' $* - } - - function verifyCertAgainstPool() - { - if [[ "" == "$1" ]] - then - echo "Usage: verifyCertAgainstPool [cert to test] [ca pool to use]" - return 1 - fi - - if [[ "" == "$2" ]] - then - echo "Usage: verifyCertAgainstPool [cert to test] [ca pool to use]" - return 1 - fi - - echo " Verifying that this certificate:" - echo " - $1" - echo " is valid for this ca pool:" - echo " - $2" - echo "" - openssl verify -partial_chain -CAfile "$2" "$1" - if [ $? -eq 0 ]; then - echo "" - echo "============ SUCCESS! ============" - else - echo "" - echo "============ FAILED TO VALIDATE ============" - fi - } - -### Validating the PKI - -Every connection in a Ziti Network is mutually authenticated via X509 certificates. It is easiest to first identify the -two components trying to communicate to isolate and minimize the configuration and files that need to be inspected. -Below you will find sections relevant to each of the pieces of Ziti which connect. - -Each section will refer to a bash variable that is expected to be established before running the command. This variable -is intended to make it easier for you to simply copy/paste the command and determine if the configuration is valid or -not. - -Using the provided bash function above - you will see one of two results: - -#### Success - - verifyCertAgainstPool /path/to/cert-to-test.cert /path/to/capool.pem - Verifying that this certificate: - - /path/to/cert-to-test.cert - is valid for this ca pool: - - /path/to/capool.pem - - /path/to/cert-to-test.cert: OK - - ============ SUCCESS! ============ - -#### Failure - - verifyCertAgainstPool /path/to/cert-to-test.cert /path/to/capool.pem - Verifying that this certificate: - - /path/to/cert-to-test.cert - is valid for this ca pool: - - /path/to/capool.pem - - C = US, ST = NC, L = Charlotte, O = NetFoundry, OU = Ziti, CN = 87f8cee9-b288-49f1-ab90-b664af29e17a - error 20 at 0 depth lookup: unable to get local issuer certificate - error /path/to/cert-to-test.cert: verification failed - - ============ FAILED TO VALIDATE ============ - --------------------------------------------------------------- - -### Ziti Edge Router to Controller - -#### Variables to Establish Manually - -These two variables represent the Ziti Edge Router configuration file and the Controller configuration file. - - controller_config_file=~/.config/ziti/ziti-controller/ziti_controller.yml - edge_router_config_file=~/.config/ziti/ziti-router/ziti_router.yml - -#### Variables - Copy/Paste - -These commands extract the files specified in the configuration and store them into the assigned variables. - - edge_router_cert=$(yaml2json $edge_router_config_file | jq -j .identity.cert) - signing_cert=$(yaml2json $controller_config_file | jq -j .edge.enrollment.signingCert.cert) - controller_cert=$(yaml2json $controller_config_file | jq -j .identity.cert) - edge_router_ca=$(yaml2json $edge_router_config_file | jq -j .identity.ca) - -#### Commands to Verify PKI Configuration - -Both of these commands should report SUCCESS. - - verifyCertAgainstPool $edge_router_cert $signing_cert - verifyCertAgainstPool $controller_cert $edge_router_ca - --------------------------------------------------------------- - -### Ziti Client to Controller - API - -#### Variables to Establish Manually - -These two variables represent the identity file in json for a Ziti client and the Controller configuration file. - - identity_file=/path/to/enrolled-identity.json - controller_config_file=~/.config/ziti/ziti-controller/ziti_controller.yml - -#### Variables - Copy/Paste - -These commands will extract the cert and ca from the enrolled identity file and put it into a file in the /tmp folder - - jq -j .id.cert $identity_file | cut -d ":" -f2 > /tmp/identity.cert - jq -j .id.ca $identity_file | cut -d ":" -f2 > /tmp/identity.ca - -These commands extract the files specified in the configuration and store them into the assigned variables. - - controller_cert=$(yaml2json $controller_config_file | jq -j .identity.cert) - signing_cert=$(yaml2json $controller_config_file | jq -j .edge.enrollment.signingCert.cert) - - controller_api_server_cert=$(yaml2json $controller_config_file | jq -j .edge.api.identity.server_cert) - if [[ "null" == "$controller_api_server_cert" ]]; then controller_api_server_cert=$(yaml2json $controller_config_file | jq -j .identity.server_cert); fi - -#### Commands to Verify PKI Configuration - -Both of these commands should report SUCCESS. - - verifyCertAgainstPool /tmp/identity.cert $signing_cert - verifyCertAgainstPool $controller_api_server_cert /tmp/identity.ca - --------------------------------------------------------------- - -### Ziti Client to Ziti Edge Router - Data - -#### Variables to Establish Manually - -These two variables represent the identity file in json for a Ziti client and the Controller configuration file. - - identity_file=/path/to/enrolled-identity.json - edge_router_config_file=~/.config/ziti/ziti-router/ziti_router.yml - -#### Variables - Copy/Paste - -This command will extract the ca from the enrolled identity file and put it into a file in the /tmp folder - - jq -j .id.ca $identity_file | cut -d ":" -f2 > /tmp/identity.ca - -This command extracts the file specified in the configuration and stores it into the assigned variable. - - edge_router_cert=$(yaml2json $edge_router_config_file | jq -j .identity.cert) - -#### Commands to Verify PKI Configuration - -The following command should report SUCCESS. - - verifyCertAgainstPool $edge_router_cert /tmp/identity.ca +--- +title: PKI Troubleshooting +--- + +Configuring a Ziti Network's PKI can be confusing. Validating a single side of a mutual TLS connection is +straightforward it becomes tedious to ensure all the certificates and cas in use are valid when you have a fully +configured Ziti Network. It's the goal of this page to make diagnosing PKI issues easier. This guide will also use the +paths you will find in the Ziti Edge - Developer Edition. Change paths accordingly. + +### Prerequisites + +The following steps are [bash-based](https://en.wikipedia.org/wiki/Bash_(Unix_shell)) functions and use the +[openssl](https://www.openssl.org/), [jq](https://stedolan.github.io/jq/) and [ruby](https://www.ruby-lang.org/en/) +commands. If you don't have bash, openssl and ruby - this page is not for you! Do your best to follow along with the +scripts and guidance herein or just make sure bash, openssl, ruby, and jq are installed. All of which are widely +available on linux/MacOS/Windows. + +The `ruby` and `jq` commands are not strictly required. They are used to make it easy for you to copy/paste these +commands. The `ruby` command is used to translate yaml into json while the `jq` command is used to pull the specific +values out of the given files. You can certainly do the same manually (without `ruby` and `jq`) if you choose. + +#### Define the verifyCertAgainstPool Function + +In your bash prompt copy and paste these two functions: + + function yaml2json() + { + ruby -ryaml -rjson -e 'puts JSON.pretty_generate(YAML.load(ARGF))' $* + } + + function verifyCertAgainstPool() + { + if [[ "" == "$1" ]] + then + echo "Usage: verifyCertAgainstPool [cert to test] [ca pool to use]" + return 1 + fi + + if [[ "" == "$2" ]] + then + echo "Usage: verifyCertAgainstPool [cert to test] [ca pool to use]" + return 1 + fi + + echo " Verifying that this certificate:" + echo " - $1" + echo " is valid for this ca pool:" + echo " - $2" + echo "" + openssl verify -partial_chain -CAfile "$2" "$1" + if [ $? -eq 0 ]; then + echo "" + echo "============ SUCCESS! ============" + else + echo "" + echo "============ FAILED TO VALIDATE ============" + fi + } + +### Validating the PKI + +Every connection in a Ziti Network is mutually authenticated via X509 certificates. It is easiest to first identify the +two components trying to communicate to isolate and minimize the configuration and files that need to be inspected. +Below you will find sections relevant to each of the pieces of Ziti which connect. + +Each section will refer to a bash variable that is expected to be established before running the command. This variable +is intended to make it easier for you to simply copy/paste the command and determine if the configuration is valid or +not. + +Using the provided bash function above - you will see one of two results: + +#### Success + + verifyCertAgainstPool /path/to/cert-to-test.cert /path/to/capool.pem + Verifying that this certificate: + - /path/to/cert-to-test.cert + is valid for this ca pool: + - /path/to/capool.pem + + /path/to/cert-to-test.cert: OK + + ============ SUCCESS! ============ + +#### Failure + + verifyCertAgainstPool /path/to/cert-to-test.cert /path/to/capool.pem + Verifying that this certificate: + - /path/to/cert-to-test.cert + is valid for this ca pool: + - /path/to/capool.pem + + C = US, ST = NC, L = Charlotte, O = NetFoundry, OU = Ziti, CN = 87f8cee9-b288-49f1-ab90-b664af29e17a + error 20 at 0 depth lookup: unable to get local issuer certificate + error /path/to/cert-to-test.cert: verification failed + + ============ FAILED TO VALIDATE ============ + +-------------------------------------------------------------- + +### Ziti Edge Router to Controller + +#### Variables to Establish Manually + +These two variables represent the Ziti Edge Router configuration file and the Controller configuration file. + + controller_config_file=~/.config/ziti/ziti-controller/ziti_controller.yml + edge_router_config_file=~/.config/ziti/ziti-router/ziti_router.yml + +#### Variables - Copy/Paste + +These commands extract the files specified in the configuration and store them into the assigned variables. + + edge_router_cert=$(yaml2json $edge_router_config_file | jq -j .identity.cert) + signing_cert=$(yaml2json $controller_config_file | jq -j .edge.enrollment.signingCert.cert) + controller_cert=$(yaml2json $controller_config_file | jq -j .identity.cert) + edge_router_ca=$(yaml2json $edge_router_config_file | jq -j .identity.ca) + +#### Commands to Verify PKI Configuration + +Both of these commands should report SUCCESS. + + verifyCertAgainstPool $edge_router_cert $signing_cert + verifyCertAgainstPool $controller_cert $edge_router_ca + +-------------------------------------------------------------- + +### Ziti Client to Controller - API + +#### Variables to Establish Manually + +These two variables represent the identity file in json for a Ziti client and the Controller configuration file. + + identity_file=/path/to/enrolled-identity.json + controller_config_file=~/.config/ziti/ziti-controller/ziti_controller.yml + +#### Variables - Copy/Paste + +These commands will extract the cert and ca from the enrolled identity file and put it into a file in the /tmp folder + + jq -j .id.cert $identity_file | cut -d ":" -f2 > /tmp/identity.cert + jq -j .id.ca $identity_file | cut -d ":" -f2 > /tmp/identity.ca + +These commands extract the files specified in the configuration and store them into the assigned variables. + + controller_cert=$(yaml2json $controller_config_file | jq -j .identity.cert) + signing_cert=$(yaml2json $controller_config_file | jq -j .edge.enrollment.signingCert.cert) + + controller_api_server_cert=$(yaml2json $controller_config_file | jq -j .edge.api.identity.server_cert) + if [[ "null" == "$controller_api_server_cert" ]]; then controller_api_server_cert=$(yaml2json $controller_config_file | jq -j .identity.server_cert); fi + +#### Commands to Verify PKI Configuration + +Both of these commands should report SUCCESS. + + verifyCertAgainstPool /tmp/identity.cert $signing_cert + verifyCertAgainstPool $controller_api_server_cert /tmp/identity.ca + +-------------------------------------------------------------- + +### Ziti Client to Ziti Edge Router - Data + +#### Variables to Establish Manually + +These two variables represent the identity file in json for a Ziti client and the Controller configuration file. + + identity_file=/path/to/enrolled-identity.json + edge_router_config_file=~/.config/ziti/ziti-router/ziti_router.yml + +#### Variables - Copy/Paste + +This command will extract the ca from the enrolled identity file and put it into a file in the /tmp folder + + jq -j .id.ca $identity_file | cut -d ":" -f2 > /tmp/identity.ca + +This command extracts the file specified in the configuration and stores it into the assigned variable. + + edge_router_cert=$(yaml2json $edge_router_config_file | jq -j .identity.cert) + +#### Commands to Verify PKI Configuration + +The following command should report SUCCESS. + + verifyCertAgainstPool $edge_router_cert /tmp/identity.ca diff --git a/docusaurus/docs/guides/05-troubleshooting/pki-troubleshooting/renew-cert.md b/docusaurus/docs/guides/05-troubleshooting/pki-troubleshooting/renew-cert.md new file mode 100644 index 000000000..24c4c49de --- /dev/null +++ b/docusaurus/docs/guides/05-troubleshooting/pki-troubleshooting/renew-cert.md @@ -0,0 +1,90 @@ +--- +title: Renew Client/Server Certs +id: renew-cert +--- + +## Creating New Certs + +If you set up your network following the [quickstart guides](/docs/learn/quickstarts), your PKI will have client and +server certs for the controller's control plane and edge/API plane. + +### Setup Environment Variables +The following environment variables are the minimum required to renew a cert with the `ziti-cli-functions.sh` script. +These can be obtained through the `.env` file for your network. However, it is recommended that the `FILE_NAME_ROOT` value is different than +the existing (expired) cert filenames so a new cert will be created rather than overwriting the existing cert. + +``` +# Path to the network PKI root folder +export ZITI_PKI="/home/ubuntu/.ziti/quickstart/homeassistant2/pki" + +# The DNS entries the PKI should be valid for +export DNS_ALLOW_LIST="localhost,homeassistant2" + +# The IP addresses the PKI should be valid for +export IP_ALLOW_LIST="127.0.0.1,150.136.141.199" +``` + +### Control Plane Certs +The following values are unique to control plane certs as the edge/API plane will have its own CA and certs. Update +these values as they pertain to your network PKI setup. + +``` +export ZITI_CA_NAME="homeassistant2-intermediate" +export FILE_NAME_ROOT="homeassistant2-new" +``` + +#### Server Cert + +``` +source /dev/stdin <<< "$(wget -qO- https://get.openziti.io/quick/ziti-cli-functions.sh)"; expressInstall +ziti pki create server --pki-root="${ZITI_PKI}" --ca-name "${ZITI_CA_NAME}" \ + --server-file "${FILE_NAME_ROOT}-server" \ + --dns "${DNS_ALLOW_LIST}" --ip "${IP_ALLOW_LIST}" \ + --server-name "${FILE_NAME_ROOT} server certificate" +``` + +#### Client Cert + +``` +ziti pki create client --pki-root="${ZITI_PKI}" --ca-name "${ZITI_CA_NAME}" \ + --client-file "${FILE_NAME_ROOT}-client" \ + --key-file "${FILE_NAME_ROOT}-server" \ + --client-name "${FILE_NAME_ROOT}" +``` + +### Edge / API Certs + +#### Update environment for edge certs +The following values are unique to edge/API plane certs as the control plane will have its own CA and certs. Update +these values as they pertain to your network PKI setup. + +``` +export ZITI_CA_NAME="150.136.141.199-intermediate" +export FILE_NAME_ROOT="150.136.141.199-new" +``` + +#### Server Cert + +``` +ziti pki create server --pki-root="${ZITI_PKI}" --ca-name "${ZITI_CA_NAME}" \ + --server-file "${FILE_NAME_ROOT}-server" \ + --dns "${DNS_ALLOW_LIST}" --ip "${IP_ALLOW_LIST}" \ + --server-name "${FILE_NAME_ROOT} server certificate" +``` + +#### Client Cert + +``` +ziti pki create client --pki-root="${ZITI_PKI}" --ca-name "${ZITI_CA_NAME}" \ + --client-file "${FILE_NAME_ROOT}-client" \ + --key-file "${FILE_NAME_ROOT}-server" \ + --client-name "${FILE_NAME_ROOT}" +``` + +## Update Controller Config + +If the original files were overwritten, no change is needed. However, if new files were generated alongside existing +files, the controller config `identity` and `web-->identity` sections need the `cert`, `server_cert`, and `key` fields +updated to point to the new client, server certs and key. Note that the key is auto-generated when creating the certs. + +Now, restart the controller and double check to ensure the certificates are now valid. \ No newline at end of file diff --git a/docusaurus/docs/guides/05-troubleshooting/pki-troubleshooting/troubleshoot-expired-certs.md b/docusaurus/docs/guides/05-troubleshooting/pki-troubleshooting/troubleshoot-expired-certs.md new file mode 100644 index 000000000..d1d479903 --- /dev/null +++ b/docusaurus/docs/guides/05-troubleshooting/pki-troubleshooting/troubleshoot-expired-certs.md @@ -0,0 +1,44 @@ +--- +title: Expired Certs +id: troubleshoot-expired-certs +--- + +The following are examples of how an invalid server cert will present itself on a running OpenZiti Network. There are certs +for the edge API and the control plane, so it's important to check for both as they will likely share the same expiry +date. + +## Behavior +An expired cert will manifest behavior in a few expected ways. +1. From an edge client or tunneler, an identity will show as `unavailable`. Reviewing the logs will generally indicate +some sort of certificate error. Something along the lines of `failed to verify certificate`, `certificate has expired`, +or something mentioning an error relating to `x509` +2. From a browser, if you attempt to access your controller, for example, by visiting +`https://controller-ip:1234/edge/client/v1/version` you may see an indication of a certificate error from the browser. +Ultimately, the controller is not accessible despite being online. +![img.png](invalid-certificate.png) +![img.png](browser-cert-viewer-invalid.png) +3. From the `ziti` CLI, if you attempt to log into the controller you will get an error to the effect of +`certificate is expired or invalid`, `unable to retrieve certificate authority`, or `bad certificate` +4. Viewing the controller logs, you may see errors similar to `TLS handshake error`, `bad certificate`, or +`certificate failure`. + +## Verification +The following commands are useful in viewing the status of the current certs for the control and edge/API planes. + +Substitute the address and port as necessary. +``` +# Example for edge/API plane +openssl s_client -connect 150.136.141.199:8441 | grep "NotBefore" + +# Example for control plane +openssl s_client -connect localhost:6262 | grep "NotBefore" +``` +### Example Output +As shown, the start and end dates for all certificates in the chain will be displayed. +``` +v:NotBefore: Sep 12 13:36:59 2023 GMT; NotAfter: Sep 11 13:37:55 2024 GMT +v:NotBefore: Aug 22 13:29:51 2022 GMT; NotAfter: Aug 19 13:30:41 2032 GMT +``` + +## Resolution +If you determine that your certs have expired. Check out the [Renew Client/Server Certs](renew-cert.md) guide. \ No newline at end of file diff --git a/docusaurus/docs/learn/core-concepts/pki.md b/docusaurus/docs/learn/core-concepts/pki.md index fab560aff..c5dca70a4 100644 --- a/docusaurus/docs/learn/core-concepts/pki.md +++ b/docusaurus/docs/learn/core-concepts/pki.md @@ -4,7 +4,7 @@ title: Public Key Infrastructure (PKI) sidebar_label: PKI --- -import PkiTroubleshootingMd from '../../guides/05-troubleshooting/pki-troubleshooting.md' +import PkiTroubleshootingMd from '../../guides/05-troubleshooting/pki-troubleshooting/pki-troubleshooting.md' All Ziti Networks leverage [Public Key Infrastructure (PKI)](https://en.wikipedia.org/wiki/Public_key_infrastructure) to provide secure network connections. This page is not intended to be a comprehensive guide. What it is, is a set of