Initial modifications to support Erlang applications

README.md

@@ -4,13 +4,14 @@
 
 ![Development](https://img.shields.io/badge/STATUS-Development_v0.3.0-blue) [![Build Status](https://github.com/thiagoesteves/deployex/workflows/Deployex%20CI/badge.svg)](https://github.com/thiagoesteves/deployex/actions/workflows/pr-ci.yml) 
 
-DeployEx is a lightweight tool designed for managing deployments in Elixir and Gleam applications without relying on additional deployment tools like Docker or Kubernetes. Its primary goal is to utilize the release package for executing full deployments or hot-upgrades, depending on the package's content, while leveraging OTP distribution for monitoring and data extraction.
+DeployEx is a lightweight tool designed for managing deployments in Beam applications (Elixir, Gleam and Erlang) without relying on additional deployment tools like Docker or Kubernetes. Its primary goal is to utilize the release package for executing full deployments or hot-upgrades, depending on the package's content, while leveraging OTP distribution for monitoring and data extraction.
 
 DeployEx acts as a central deployment runner, gathering crucial deployment data such as the current version and release package contents. The content of the release package enables it to run for a full deployment or a hot-upgrade. Meanwhile, on the development front, your CI/CD pipeline takes charge of crafting and updating packages for the target release. This integration ensures that DeployEx is always equipped with the latest packages, ready to facilitate deployments.
 
 DeployEx is currently used by:
  * [Calori Web Server](https://github.com/thiagoesteves/calori) for __Elixir__ applications and you can check it at [homepage](https://calori.com.br).
  * [Cochito Web Server](https://github.com/chouzar/cochito) for __Gleam__ applications and you can check it at [homepage](https://gleam.deployex.pro).
+ * [Erlsnake](https://github.com/chouzar/cochito) for __Erlang__ applications and you can check it at [homepage](https://gleam.deployex.pro).
 
 ![Deployment Architecture](docs/static/deployex.png)
 
@@ -25,6 +26,7 @@ Upon deployment, the following dashboard becomes available, offering access to l
  * Performs full deployments based solely on the release files generated by:
    - `mix release` for Elixir.
    - `gleam export` for Gleam.
+   - `rebar3 as prod tar` for Elixir.
  * Supports hot code reloading for Elixir applications using the [Jellyfish](https://github.com/thiagoesteves/jellyfish) library.
  * Supports the following cloud providers:
    - Amazon Web Services (AWS)
@@ -40,7 +42,7 @@ Upon deployment, the following dashboard becomes available, offering access to l
  * Allows access to current log files (stdout and stderr) for both monitored apps and DeployEx.
  * Provides access to the shell:
    - IEx shell for monitored Elixir apps and DeployEx.
-   - Erlang shell for monitored Gleam apps.
+   - Erlang shell for monitored Gleam/Erlang apps.
  * Provides installer script to be used with ubuntu hosts.
  * Provides status information per instance:
    - OTP connectivity
@@ -57,6 +59,7 @@ Upon deployment, the following dashboard becomes available, offering access to l
 ### Version 0.3.0
 
 - [X] 🚧 Add Gleam support.
+- [X] 🚧 Add Erlang support.
 
 ### Version 0.4.0
 
@@ -684,6 +687,193 @@ After making these changes, create and publish a new version `0.1.2` for `myglea
 
 ![mTLS Dashboard Gleam](docs/static/deployex_monitoring_app_gleam_tls.png)
 
+### Erlang
+
+For local testing, the root path used for distribution releases and versions is `/tmp/{monitored_app}`. Let's create the required release folders:
+```bash
+export monitored_app_name=myerlangapp
+mkdir -p /tmp/${monitored_app_name}/dist/${monitored_app_name}
+mkdir -p /tmp/${monitored_app_name}/versions/${monitored_app_name}/local/
+```
+
+Since Elixir is the default language for deployex, it will require set the respective values in the same terminal where deployex will run:
+```elixir
+export DEPLOYEX_MONITORED_APP_NAME=myerlangapp
+export DEPLOYEX_MONITORED_APP_LANG=erlang
+```
+
+It is important to note that for local deployments, DeployEx will use the path `/tmp/deployex` for local storage. This means you can delete the entire folder to reset any local version, history, or configurations.
+
+#### Creating an Erlang app (default name is `myerlangapp`)
+
+In this example, we create a brand new erlang app:
+
+```bash
+rebar3 new release myerlangapp
+cd myerlangapp
+```
+
+#### Add required release vars
+
+Open the args file `config/vm.args` and add the following lines, removing previous suggestions for sname and cookie:
+
+```bash
+-sname ${RELEASE_NODE}
+-setcookie ${RELEASE_COOKIE}
+${RELEASE_SSL_OPTIONS}
+
++K true
++A30
+```
+
+#### Generate a release
+Then you can compile and generate a release
+```bash
+rebar3 as prod tar
+```
+
+Pack the release and move it to the distributed folder and updated the version:
+```bash
+export app_name=myerlangapp
+cp _build/prod/rel/${app_name}/${app_name}-0.1.0.tar.gz /tmp/${app_name}/dist/${app_name}
+echo "{\"version\":\"0.1.0\",\"pre_commands\": [],\"hash\":\"local\"}" | jq > /tmp/${app_name}/versions/${app_name}/local/current.json
+```
+
+#### Running DeployEx and deploy the app
+
+Move back to the DeployEx project and run the command line with the required ENV vars. 
+
+*__NOTE:  All env vars that are available for DeployEx will also be available to the `monitored_app`__*
+```bash
+export DEPLOYEX_MONITORED_APP_NAME=myerlangapp
+export DEPLOYEX_MONITORED_APP_LANG=erlang
+export SECRET_KEY_BASE=e4CXwPpjrAJp9NbRobS8dXmOHfn0EBpFdhZlPmZo1y3N/BzW9Z/k7iP7FjMk+chi
+export PHX_SERVER=true
+iex --sname deployex --cookie cookie -S mix phx.server
+...
+
+[info] Update is needed at instance: 1 from: <no current set> to: 0.1.0
+[warning] HOT UPGRADE version NOT DETECTED, full deployment required, result: []
+[info] Full deploy instance: 1 deploy_ref: psukd1
+[info] Initialising monitor server for instance: 1
+[info] Ensure running requested for instance: 1 version: 0.1.0
+[info]  # Identified executable: /tmp/deployex/varlib/service/myerlangapp/1/current/bin/myerlangapp
+[info]  # Starting application
+[info]  # Running instance: 1, monitoring pid = #PID<0.790.0>, OS process = 22952 deploy_ref: psukd1
+[info]  # Application instance: 1 is running
+[info]  # Moving to the next instance: 2
+...
+iex(deployex@hostname)1>
+```
+
+You should then visit the application and check it is running [localhost:5001](http://localhost:5001/). Since you are not using mTLS, the dashboard should look like this:
+
+![No mTLS Dashboard Erlang](docs/static/deployex_monitoring_app_erlang_no_tls.png)
+
+Note that the __OTP-Nodes are connected__, but the __mTLS is not supported__. The __mTLS__ can be enabled and it will be covered ahead. Leave this terminal running and open a new one to compile and release the monitored app.
+
+#### Updating the application
+
+##### Full deployment
+
+In this scenario, the existing application will undergo termination, paving the way for the deployment of the new one. It's crucial to maintain the continuous operation of DeployEx throughout this process. Navigate to the `myerlangapp` project and increment the version in the `apps/myerlangapp/src/myerlangapp.app.src` and `rebar.config` files.
+
+1. Remove any previously generated files and generate a new release
+```bash
+rebar3 as prod tar
+```
+
+2. Now, *__keep DeployEx running in another terminal__* and copy the release file to the distribution folder and proceed to update the version accordingly:
+```bash
+export app_name=myerlangapp
+cp _build/prod/rel/${app_name}/${app_name}-0.1.1.tar.gz /tmp/${app_name}/dist/${app_name}
+echo "{\"version\":\"0.1.1\",\"pre_commands\": [],\"hash\":\"local\"}" | jq > /tmp/${app_name}/versions/${app_name}/local/current.json
+```
+
+3. You should then see the following messages in the DeployEx terminal while updating the app:
+```bash
+[info] Update is needed at instance: 1 from: 0.1.0 to: 0.1.1
+[warning] HOT UPGRADE version NOT DETECTED, full deployment required, result: []
+[info] Full deploy instance: 1 deploy_ref: oxnnwu
+[info] Requested instance: 1 to stop application pid: #PID<0.790.0>
+[info] Initialising monitor server for instance: 1
+[info] Ensure running requested for instance: 1 version: 0.1.1
+[info]  # Identified executable: /tmp/deployex/varlib/service/myerlangapp/1/current/bin/myerlangapp
+[info]  # Starting application
+[info]  # Running instance: 1, monitoring pid = #PID<0.852.0>, OS process = 23812 deploy_ref: oxnnwu
+[info]  # Application instance: 1 is running
+[info]  # Moving to the next instance: 2
+...
+```
+
+#### 🔑 Enhancing OTP Distribution Security with mTLS
+
+In order to improve security, mutual TLS (`mTLS` for short) can be employed to encrypt communication during OTP distribution. To implement this, follow these steps:
+
+1. Generate the necessary certificates, DeployEx has a good examples of how to create self-signed tls certificates:
+```bash
+cd deployex
+make tls-distribution-certs
+```
+
+2. Copy the generated certificates to the `/tmp` folder:
+```bash
+cp ca.crt /tmp
+cp deployex.crt /tmp
+cp deployex.key /tmp
+```
+
+3. Create the `inet_tls.conf` file with the appropriate paths, utilizing the command found in `rel/env.sh.eex` in deployex project:
+```bash
+export DEPLOYEX_OTP_TLS_CERT_PATH=/tmp
+
+test -f /tmp/inet_tls.conf || (umask 277
+ cd /tmp
+ cat >inet_tls.conf <<EOF
+[
+  {server, [
+    {certfile, "${DEPLOYEX_OTP_TLS_CERT_PATH}/deployex.crt"},
+    {keyfile, "${DEPLOYEX_OTP_TLS_CERT_PATH}/deployex.key"},
+    {cacertfile, "${DEPLOYEX_OTP_TLS_CERT_PATH}/ca.crt"},
+    {verify, verify_peer},
+    {secure_renegotiate, true}
+  ]},
+  {client, [
+    {certfile, "${DEPLOYEX_OTP_TLS_CERT_PATH}/deployex.crt"},
+    {keyfile, "${DEPLOYEX_OTP_TLS_CERT_PATH}/deployex.key"},
+    {cacertfile, "${DEPLOYEX_OTP_TLS_CERT_PATH}/ca.crt"},
+    {verify, verify_peer},
+    {secure_renegotiate, true},
+    {server_name_indication, disable}
+  ]}
+].
+EOF
+)
+```
+
+4. Since the default erlang application doesn't enable tls and ssh, you need to add these libraries, open the file `apps/myerlangapp/src/myerlangapp.app.src` and add inets and ssl:
+```bash
+    ...
+    {mod, {myerlangapp_app, []}},
+    {applications, [
+        kernel,
+        stdlib,
+        inets,
+        ssl
+    ]},
+    {env, []},
+    ...
+```bash
+
+5. To enable `mTLS` for DeployEx, set the appropriate Erlang options before running the application in the terminal:
+```bash
+ELIXIR_ERL_OPTIONS="-proto_dist inet_tls -ssl_dist_optfile /tmp/inet_tls.conf -setcookie cookie" iex --sname deployex -S mix phx.server
+```
+
+After making these changes, create and publish a new version `0.1.2` for `myerlangapp` and run the DeployEx with the command from item 5. After the deployment, you should see the following dashboard:
+
+![mTLS Dashboard Erlang](docs/static/deployex_monitoring_app_erlang_tls.png)
+
 ## 🔨 Throubleshooting
 
 ### Accessing DeployEx logs
@@ -720,7 +910,9 @@ tail -f /tmp/${monitored_app_name}/${monitored_app_name}/${monitored_app_name}-$
 tail -f /tmp/${monitored_app_name}/${monitored_app_name}/${monitored_app_name}-${instance}-stderr.log
 ```
 
-### Connecting to the monitored app IEX CLI (Elixir only)
+### Connecting to the monitored app manually
+
+#### Elixir
 
 ```bash
 export instance=1
@@ -733,6 +925,33 @@ export RELEASE_COOKIE=cookie
 /tmp/deployex/varlib/service/${monitored_app_name}/${instance}/current/bin/${monitored_app_name} remote
 ```
 
+#### Gleam
+
+```bash
+export instance=1
+export monitored_app_name=mygleamapp
+export hostname=??? # From the local machine
+# production
+erl -remsh ${monitored_app_name}-${instance}@${hostname} -setcookie cookie -proto_dist inet_tls -ssl_dist_optfile /tmp/inet_tls.conf
+# local test
+erl -remsh ${monitored_app_name}-${instance}@${hostname} -setcookie cookie -proto_dist inet_tls -ssl_dist_optfile /tmp/inet_tls.conf
+```
+
+#### Erlang
+
+```bash
+export instance=1
+export monitored_app_name=myerlangapp
+export RELX_REPLACE_OS_VARS=true
+export RELEASE_NODE=${monitored_app_name}-${instance}
+export RELEASE_COOKIE=cookie
+export RELEASE_SSL_OPTIONS="-proto_dist inet_tls -ssl_dist_optfile /tmp/inet_tls.conf" # If enabled
+# production
+/var/lib/deployex/service/${monitored_app_name}/${instance}/current/bin/${monitored_app_name} remote_console
+# local test
+/tmp/deployex/varlib/service/${monitored_app_name}/${instance}/current/bin/${monitored_app_name} remote_console
+```
+
 ## ❓How DeployEx handles services
 
 DeployEx operates by monitoring applications and versions using folders and files, treating the monitored app as a service:

lib/deployex/monitor/application.ex

@@ -94,8 +94,8 @@ defmodule Deployex.Monitor.Application do
 
   # This command is available during the hot upgrade. If it fails, the process will
   # restart and attempt a full deployment.
-  def handle_call({:run_pre_commands, pre_commands, app_bin_path}, _from, state) do
-    :ok = execute_pre_commands(state, pre_commands, app_bin_path)
+  def handle_call({:run_pre_commands, pre_commands, app_bin_state}, _from, state) do
+    :ok = execute_pre_commands(state, pre_commands, app_bin_state)
 
     {:reply, {:ok, pre_commands}, state}
   end
@@ -205,11 +205,11 @@ defmodule Deployex.Monitor.Application do
   end
 
   @impl true
-  def run_pre_commands(instance, pre_commands, app_bin_path) do
+  def run_pre_commands(instance, pre_commands, app_bin_state) do
     instance
     |> global_name()
     |> Enum.at(0)
-    |> Common.call_gen_server({:run_pre_commands, pre_commands, app_bin_path})
+    |> Common.call_gen_server({:run_pre_commands, pre_commands, app_bin_state})
   end
 
   @impl true
@@ -251,7 +251,7 @@ defmodule Deployex.Monitor.Application do
          version_map
        ) do
     monitore_app_lang = Storage.monitored_app_lang()
-    app_exec = executable_path(monitore_app_lang, instance, :current)
+    app_exec = Storage.bin_path(instance, monitore_app_lang, :current)
     version = version_map.version
 
     with true <- File.exists?(app_exec),
@@ -291,7 +291,7 @@ defmodule Deployex.Monitor.Application do
   # NOTE: Some commands need to run prior starting the application
   #       - Unset env vars from the deployex release to not mix with the monitored app release
   #       - Export suffix to add different snames to the apps
-  #       - Export phoenix listening port taht needs to be one per app
+  #       - Export listening port that needs to be one per app
   defp run_app_bin(language, instance, executable_path, command)
 
   defp run_app_bin("elixir", instance, executable_path, command) do
@@ -308,7 +308,33 @@ defmodule Deployex.Monitor.Application do
     """
   end
 
-  defp run_app_bin("gleam", instance, executable_path, _command) do
+  defp run_app_bin("erlang", instance, executable_path, "start") do
+    server_port = Storage.monitored_app_start_port() + (instance - 1)
+    path = Common.remove_deployex_from_path()
+    app_name = Storage.monitored_app_name()
+    cookie = Common.cookie()
+
+    ssl_options =
+      if Common.check_mtls() == :supported do
+        "-proto_dist inet_tls -ssl_dist_optfile /tmp/inet_tls.conf"
+      else
+        ""
+      end
+
+    """
+    unset $(env | grep '^RELEASE_' | awk -F'=' '{print $1}')
+    unset BINDIR ELIXIR_ERL_OPTIONS ROOTDIR
+    export PATH=#{path}
+    export RELX_REPLACE_OS_VARS=true
+    export RELEASE_NODE=#{app_name}-#{instance}
+    export RELEASE_COOKIE=#{cookie}
+    export RELEASE_SSL_OPTIONS=\"#{ssl_options}\"
+    export PORT=#{server_port}
+    #{executable_path} foreground
+    """
+  end
+
+  defp run_app_bin("gleam", instance, executable_path, "start") do
     server_port = Storage.monitored_app_start_port() + (instance - 1)
     app_name = Storage.monitored_app_name()
     path = Common.remove_deployex_from_path()
@@ -338,26 +364,24 @@ defmodule Deployex.Monitor.Application do
     """
   end
 
-  defp executable_path(language, instance, path)
-
-  defp executable_path("elixir", instance, :current) do
-    "#{Storage.current_path(instance)}/bin/#{Storage.monitored_app_name()}"
-  end
-
-  defp executable_path("elixir", instance, :new) do
-    "#{Storage.new_path(instance)}/bin/#{Storage.monitored_app_name()}"
-  end
+  defp run_app_bin(language, instance, _executable_path, command) do
+    msg =
+      "Running not supported for language: #{language}, instance: #{instance}, command: #{command}"
 
-  defp executable_path("gleam", instance, _path) do
-    "#{Storage.current_path(instance)}/erlang-shipment"
+    Logger.warning(msg)
+    "echo \"#{msg}\""
   end
 
   # credo:disable-for-lines:28
-  defp execute_pre_commands(_state, pre_commands, _bin_path) when pre_commands == [], do: :ok
+  defp execute_pre_commands(_state, pre_commands, _bin_state) when pre_commands == [], do: :ok
 
-  defp execute_pre_commands(%{instance: instance, status: status} = state, pre_commands, bin_path) do
+  defp execute_pre_commands(
+         %{instance: instance, status: status} = state,
+         pre_commands,
+         bin_state
+       ) do
     monitore_app_lang = Storage.monitored_app_lang()
-    migration_exec = executable_path(monitore_app_lang, instance, bin_path)
+    migration_exec = Storage.bin_path(instance, monitore_app_lang, bin_state)
 
     update_non_blocking_state(%{state | status: :pre_commands})