From a65f84891c9bad2b401d1b03e5dac4b2ba2f4d2a Mon Sep 17 00:00:00 2001 From: Anthony Casagrande Date: Wed, 1 Nov 2023 10:33:52 -0700 Subject: [PATCH] feat: add docs on secure remote deployment of multiple device services Signed-off-by: Anthony Casagrande --- .../device/remote/Ch-RemoteSecure.md | 344 +++++++++++++++++- 1 file changed, 342 insertions(+), 2 deletions(-) diff --git a/docs_src/microservices/device/remote/Ch-RemoteSecure.md b/docs_src/microservices/device/remote/Ch-RemoteSecure.md index c9eb45381c..04ce9ca5bc 100644 --- a/docs_src/microservices/device/remote/Ch-RemoteSecure.md +++ b/docs_src/microservices/device/remote/Ch-RemoteSecure.md @@ -1,2 +1,342 @@ -# Remote deployment of device services in secure mode -Coming Soon. +# Remote deployment of multiple device service instances in secure mode + +In some use cases, devices are connected to nodes where core EdgeX services are not running. In these cases the appropriate device service +needs to run on the remote nodes where it can connect to the device(s) and communicate to the host node where the rest of the EdgeX services are running. + +This page provides an example of configuring and deploying multiple instances of [device-usb-camera](../services/device-usb-camera/General.md) +service using multiple nodes in EdgeX secure mode. + +## Example +This example uses 3 nodes for remote deployment. One of the nodes (host) is used to run all EdgeX core services in Docker and the other nodes (remote) +for running the device-usb-camera service either in Docker. This example can be further extended to run multiple instances of device-usb-camera +service in multiple nodes. + +## Pre-requisites +- 3 machines running Ubuntu 20.04 or newer OS, with docker installed. + - 1 "local" system for deploying all edgex core services in secure mode with docker + - 2 "remote" nodes for deploying USB service in secure mode with docker +- All dependencies such as docker, go, git, curl etc. installed on all 3 systems to run + edgex and usb services +- Local and remote nodes needs to be on the same network + + !!! note + This document explains without any VPN and Proxy configuration, if there is + any proxy and/or VPN then that needs to be configured appropriately for secure + communication between local and remote nodes + +This example assumes the following values; replace them as needed with your own values: + +- Local System IP: 192.168.4.103 +- Remote Node 1 IP: 192.168.4.104 +- Remote Node 2 IP: 192.168.4.105 +- Instance numbers for the services are 1 and 2: `device-usb-camera_2` and `device-usb-camera_2` + +## Local system setup (EdgeX Core) +This section will go over setting up the local node to run all the EdgeX Core services + + !!! note + Refer to sample [device virtual service deployment in secure mode](../../../security/Ch-RemoteDeviceServices.md) for general understanding + + +1. Set up the "local" node to be ready for deployment. Refer to [USB Service Setup](../services/device-usb-camera/walkthrough/setup.md) + for system requirements and dependencies such as Git, Docker, Docker Compose, etc. + +2. Next step is to install [EdgeX compose](https://github.com/edgexfoundry/edgex-compose) in the host node which will be used to run all EdgeX core services. So clone the `edgex-compose` + repository: + + ```bash + git clone https://github.com/edgexfoundry/edgex-compose.git + ``` + +3. Checkout the required version: + + ```bash + git checkout {{edgexversion}} + ``` + +### Generate SSH key pair for SSH tunnel +1. Navigate to `edgex-examples/security/remote_devices/spiffe_and_ssh` +2. Run `generate_keys.sh` script +3. Copy `remote` folder located at `edgex-examples/security/remote_devices/spiffe_and_ssh` to remote nodes + +### Edit docker-compose file +1. Navigate to `edgex-examples/security/remote_devices/spiffe_and_ssh/local` and edit + `docker-compose.yml` file + + !!! note + Below instructions explains to run 2 instances of usb service: instance 1 and instance 2) + +2. Change `host_ip` to ip address of local system for `consul`, `core-metadata`, `database` and `vault` + +```yaml +services: + consul: + ports: + - mode: ingress + host_ip: 192.168.4.103 # insert local machine's host ip here + target: 8500 + published: "8500" + protocol: tcp + + core-metadata: + ports: + - mode: ingress + host_ip: 192.168.4.103 # insert local machine's host ip here + target: 59881 + published: "59881" + protocol: tcp + database: + ports: + - mode: ingress + host_ip: 192.168.4.103 # insert local machine's host ip here + target: 6379 + published: "6379" + protocol: tcp + vault: + ports: + - mode: ingress + host_ip: 192.168.4.103 # insert local machine's host ip here + target: 8200 + published: "8200" + protocol: tcp +``` +1. Add/edit environment variables for `consul`, `security-proxy-setup` and `security-secretstore-setup` services + +```yaml + services: + consul: + environment: + EDGEX_ADD_REGISTRY_ACL_ROLES: "device-usb-camera_1,device-usb-camera_2" + + security-proxy-setup: + environment: + EDGEX_ADD_PROXY_ROUTE: "device-usb-camera_1.http://192.168.4.104:59983,device-usb-camera_2.http://192.168.4.105:59983" + ROUTES_DEVICE_USB_CAMERAL_HOST: "device-usb-camera_1,device-usb-camera_2" + + security-secretstore-setup: + environment: + EDGEX_ADD_KNOWN_SECRETS: "redisdb[app-rules-engine],redisdb[device-usb-camera_1],message-bus[device-usb-camera_1],redisdb[device-usb-camera_2],message-bus[device-usb-camera_2]" +``` + +1. Add multiple `device-ssh-proxy` service and configure with remote node ips + - copy `device-ssh-proxy` service and paste it again below itself. + - rename services to `device-ssh-proxy-1` and `device-ssh-proxy-2` respectively + - Modify the existing environment variables and other fields for `device-ssh-proxy-1` + + ```yaml + services: + device-ssh-proxy-1: + container_name: edgex-device-ssh-proxy-1 + environment: + SERVICE_HOST: edgex-device-usb-camera + SERVICE_PORT: 59983 + TUNNEL_HOST: 192.168.4.104 # use ip address of the first remote node! + hostname: edgex-device-ssh-proxy-1 + networks: + edgex-network: + aliases: + - edgex-device-usb-camera-1 + ports: + - 127.0.0.1:59983:59983/tcp + ``` + + - Edit environment variables and other fields for `device-ssh-proxy-2` + + ```yaml + services: + device-ssh-proxy-2: + container_name: edgex-device-ssh-proxy-2 + environment: + SERVICE_HOST: edgex-device-usb-camera + SERVICE_PORT: 59983 + TUNNEL_HOST: 192.168.4.105 # use ip address of the second remote node! + hostname: edgex-device-ssh-proxy-2 + networks: + edgex-network: + aliases: + - edgex-device-usb-camera-2 + ports: + - 127.0.0.1:59984:59983/tcp # notice the different port mapping to avoid conflicts + ``` + +2. Update to latest docker image for `security-spiffe-token-provider` + +```yaml + services: + security-spiffe-token-provider: + image: nexus3.edgexfoundry.org:10004/security-spiffe-token-provider:latest +``` + +### Build and run services +1. Navigate to edgex-examples/security/remote_devices/spiffe_and_ssh/lcoal +2. Build and start the services + ```bash + docker compose build + docker compose up -d + ``` +3. Run `docker ps -a` and ensure all containers started without crashing (Note: + `edgex-device-ssh-proxy-1` and `edgex-device-ssh-proxy-2` will keep restarting until services on remote node start.) + +### Update server entries +Edit and run `add-server-entry.sh` script for each usb service + +1. Navigate to `edgex-examples/security/remote_devices/spiffe_and_ssh/lcoal` + and edit `add-server-entry.sh` script like below. + + !!! example - Example for first device service + ```bash + docker exec -ti edgex-security-spire-config spire-server entry create \ + -socketPath /tmp/edgex/secrets/spiffe/private/api.sock \ + -parentID spiffe://edgexfoundry.org/spire/agent/x509pop/cn/ \ + remote-agent -dns "edgex-device-usb-camera" \ + -spiffeID spiffe://edgexfoundry.org/service/device-usb-camera_1 \ + -selector "docker:label:com.docker.compose.service:device-usb-camera_1" + ``` + +2. Save and then run `./add-server-entry.sh` + +3. Repeat previous steps for each additional device service + + !!! example - Example for second device service + ```bash + docker exec -ti edgex-security-spire-config spire-server entry create \ + -socketPath /tmp/edgex/secrets/spiffe/private/api.sock \ + -parentID spiffe://edgexfoundry.org/spire/agent/x509pop/cn/ \ + remote-agent -dns "edgex-device-usb-camera" \ + -spiffeID spiffe://edgexfoundry.org/service/device-usb-camera_2 \ + -selector "docker:label:com.docker.compose.service:device-usb-camera_2" + ``` +4. Save and then run `./add-server-entry.sh` again + +## Remote Node Setup + +1. Navigate to `edgex-examples/security/remote_devices/spiffe_and_ssh/remote` + and update `docker-compose.yml`. +2. Remove `device-virtual` service and add 2 `device-usb-service` like below, one for each instance. + +=== "First Remote Device Service" + + ```yaml + device-usb-camera_1: # device-usb-camera_ + # command: -cp -r -rsh ,,0.0.0.0 -i + command: -cp -r -rsh 192.168.4.104,192.168.4.103,0.0.0.0 -i 1 + container_name: edgex-device-usb-camera + device_cgroup_rules: + - c 81:* rw + depends_on: + - remote-spire-agent + environment: + EDGEX_SECURITY_SECRET_STORE: "true" + SECRETSTORE_HOST: 192.168.4.103 # + SECRETSTORE_PORT: "8200" + SECRETSTORE_RUNTIMETOKENPROVIDER_ENABLED: "true" + SECRETSTORE_RUNTIMETOKENPROVIDER_ENDPOINTSOCKET: /tmp/edgex/secrets/spiffe/public/api.sock + SECRETSTORE_RUNTIMETOKENPROVIDER_HOST: edgex-security-spiffe-token-provider + SECRETSTORE_RUNTIMETOKENPROVIDER_PORT: 59841 + SECRETSTORE_RUNTIMETOKENPROVIDER_PROTOCOL: https + SECRETSTORE_RUNTIMETOKENPROVIDER_REQUIREDSECRETS: redisdb + SECRETSTORE_RUNTIMETOKENPROVIDER_TRUSTDOMAIN: edgexfoundry.org + DEVICE_DISCOVERY_ENABLED: 'true' # enable or disable auto-discovery + DEVICE_DISCOVERY_INTERVAL: '5m' # configure the auto-discovery interval + hostname: edgex-device-usb-camera + image: nexus3.edgexfoundry.org:10004/device-usb-camera:latest + networks: + edgex-network: {} + ports: + - "192.168.4.104:59983:59983/tcp" # :59983:59983/tcp + - "8554:8554" + - "8000:8000" + read_only: true + restart: always + security_opt: + - no-new-privileges:true + user: 'root:root' + volumes: + # - /tmp/edgex/secrets/device-usb-camera_:/tmp/edgex/secrets/device-usb-camera_:ro,z + - /tmp/edgex/secrets/device-usb-camera_1:/tmp/edgex/secrets/device-usb-camera_1:ro,z + - /tmp/edgex/secrets/spiffe/public:/tmp/edgex/secrets/spiffe/public:ro,z + - /dev:/dev:ro + - /run/udev:/run/udev:ro + ``` + +=== "Second Remote Device Service" + + ```yaml + device-usb-camera_2: # device-usb-camera_ + # command: -cp -r -rsh ,,0.0.0.0 -i + command: -cp -r -rsh 192.168.4.105,192.168.4.103,0.0.0.0 -i 2 + container_name: edgex-device-usb-camera + device_cgroup_rules: + - c 81:* rw + depends_on: + - remote-spire-agent + environment: + EDGEX_SECURITY_SECRET_STORE: "true" + SECRETSTORE_HOST: 192.168.4.103 # + SECRETSTORE_PORT: "8200" + SECRETSTORE_RUNTIMETOKENPROVIDER_ENABLED: "true" + SECRETSTORE_RUNTIMETOKENPROVIDER_ENDPOINTSOCKET: /tmp/edgex/secrets/spiffe/public/api.sock + SECRETSTORE_RUNTIMETOKENPROVIDER_HOST: edgex-security-spiffe-token-provider + SECRETSTORE_RUNTIMETOKENPROVIDER_PORT: 59841 + SECRETSTORE_RUNTIMETOKENPROVIDER_PROTOCOL: https + SECRETSTORE_RUNTIMETOKENPROVIDER_REQUIREDSECRETS: redisdb + SECRETSTORE_RUNTIMETOKENPROVIDER_TRUSTDOMAIN: edgexfoundry.org + DEVICE_DISCOVERY_ENABLED: 'true' # enable or disable auto-discovery + DEVICE_DISCOVERY_INTERVAL: '5m' # configure the auto-discovery interval + hostname: edgex-device-usb-camera + image: nexus3.edgexfoundry.org:10004/device-usb-camera:latest + networks: + edgex-network: {} + ports: + - "192.168.4.105:59983:59983/tcp" # :59983:59983/tcp + - "8554:8554" + - "8000:8000" + read_only: true + restart: always + security_opt: + - no-new-privileges:true + user: 'root:root' + volumes: + # - /tmp/edgex/secrets/device-usb-camera_:/tmp/edgex/secrets/device-usb-camera_:ro,z + - /tmp/edgex/secrets/device-usb-camera_1:/tmp/edgex/secrets/device-usb-camera_1:ro,z + - /tmp/edgex/secrets/spiffe/public:/tmp/edgex/secrets/spiffe/public:ro,z + - /dev:/dev:ro + - /run/udev:/run/udev:ro + ``` + +3. Build and run service on remote node + - Navigate to `edgex-examples/security/remote_devices/spiffe_and_ssh/remote` + - `docker compose build` + - `docker compose up -d` + - Run `docker ps- a` to ensure all services started successfully +4) Wait for about 1 minute to establish ssh tunnel and to establish communication + between edgex server system and remote node +5. Run `docker logs edgex-device-usb-camera` to check usb service status. Ensure + usb service started successfully and detected and added connected usb cameras (if + any) to `edgex-core-metadata` + + +Note: If edgex services are restarted then remote node services must also be +restarted + +## Testing and execution of REST APIs +1. Download [get-api-gateway-token.sh](https://github.com/edgexfoundry/edgex-compose/blob/main/compose-builder/get-api-gateway-token.sh) script +2. Navigate to `edgex-examples/security/remote_devices/spiffe_and_ssh/lcoal` and + copy over the downloaded `get-api-gateway-token.sh` script +3. run `./get-api-gateway-token.sh` to retrieve secure token to run rest apis +4. Run curl command to get all connected usb cameras on remote node + ```bash + curl --location --request GET 'http://:59881/api/v3/device/service/name/device-usb-camera_' \ + --header 'Authorization: Bearer ' \ + --data-raw '' + ``` +5. Run curl command to get specific camera info + ```bash + curl --location --request GET 'http://:59882/api/v3/device/name//CameraInfo' \ + --header 'Authorization: Bearer ' \ + --data-raw '' + ``` + + !!! note + You can download the usb postman collection from [device-usb-service repo](https://github.com/edgexfoundry/device-usb-camera/tree/main/docs) and run + other APIs