Adding Live Metrics feature with default retention period of 30 minutes

CHANGELOG.md

@@ -1,67 +1,18 @@
-# CHANGELOG (v0.3.0)
+# CHANGELOG (v0.4.0)
 
-## 0.3.0 🚀 (2025-01-06)
+## 0.4.0-rc-1 ()
 
-### Backwards incompatible changes for 0.2.0
- * Changed adapters/secrets to be configurable, new variables documentation added to the README.
- * Added new configuration for google credentials when using GCP, new variables documentation added to the README.
- * Changed storage to use term files instead jason format, updating to this version will require losing previous data.
+### Backwards incompatible changes for 0.3.0
+ * None
 
 ### Installer Actions
- * Update `deployex-config.json` to support new configuration
- * Update `deployex.sh` to be able to install using new configuration
+ * None
 
 ### Bug fixes
- * [[`PR-69`](https://github.com/thiagoesteves/deployex/pull/69)] Fixed bug that was rendering mode set when not required
- * [[`PR-62`](https://github.com/thiagoesteves/deployex/pull/62)] Modified application index to handle only its own node monitoring data
- * [[`PR-63`](https://github.com/thiagoesteves/deployex/pull/63)] Fixed blocking state for Monitor GenServer, when migrations were too long, the system couldn't fetch its state anymore. It is now under an ETS table and can be accessed without going to the Monitor GenServer.
- * [[`PR-60`](https://github.com/thiagoesteves/deployex/pull/60)] Fixed issues with auto-complete functionality in the IEx terminal and increased the log and terminal size.
- * [[`PR-59`](https://github.com/thiagoesteves/deployex/pull/59)] Fixing bug that was capturing letter v as Ctrl+v.
- * [[`PR-51`](https://github.com/thiagoesteves/deployex/pull/51)] Terminal copy/paste bug, terminal was pasting when copying code within it.
- * [[`Issue-47`](https://github.com/thiagoesteves/deployex/issues/47)] Application logs were not being appended
- * [[`PR-16`](https://github.com/thiagoesteves/deployex/pull/16)] Fixed an uptime bug that at deployex.
- * [[`c9bdc47`](https://github.com/thiagoesteves/deployex/commit/c9bdc47)] Fixed an uptime bug that incorrectly depended on previous version information.
- * Fixing exception when clicking in the app button
+ * None
 
 ### Enhancements
- * [[`PR-92`](https://github.com/thiagoesteves/deployex/pull/92)] Adding System Info bar to Applications and Live Tracing
- * [[`PR-83`](https://github.com/thiagoesteves/deployex/pull/83)] Adding Live logs option
- * [[`PR-84`](https://github.com/thiagoesteves/deployex/pull/84)] Refactoring Terminal Server
- * [[`PR-86`](https://github.com/thiagoesteves/deployex/pull/86)] Adding Live Observer option
- * [[`PR-88`](https://github.com/thiagoesteves/deployex/pull/88)] Adding Live Tracing option
- * [[`PR-91`](https://github.com/thiagoesteves/deployex/pull/91)] Updated liveview and OTP to 26.2.5.6
- * [[`PR-77`](https://github.com/thiagoesteves/deployex/pull/77)] Adding Erlang support
- * [[`PR-80`](https://github.com/thiagoesteves/deployex/pull/80)] Adding Erlang hot upgrade support
- * [[`PR-82`](https://github.com/thiagoesteves/deployex/pull/82)] Adding host Terminal (tmux) via Liveview
- * [[`PR-70`](https://github.com/thiagoesteves/deployex/pull/70)] Changed function listener to subscribe
- * [[`PR-71`](https://github.com/thiagoesteves/deployex/pull/71)] Added ex_docs and enhanced documentation
- * [[`PR-72`](https://github.com/thiagoesteves/deployex/pull/72)] Change deploy reference to string instead erlang reference
- * [[`PR-73`](https://github.com/thiagoesteves/deployex/pull/73)] Adding deploy_ref to the monitor global_name
- * [[`PR-75`](https://github.com/thiagoesteves/deployex/pull/75)] Adding Gleam support
- * [[`PR-58`](https://github.com/thiagoesteves/deployex/pull/58)] Updated terminal to allow more than one connection since authentication is required.
- * [[`PR-44`](https://github.com/thiagoesteves/deployex/pull/44)] New storage format (term), allowing a better map handling
- * [[`PR-49`](https://github.com/thiagoesteves/deployex/pull/49)] Adding authentication scheme
- * [[`PR-50`](https://github.com/thiagoesteves/deployex/pull/50)] Since authentication is required, there is noneed of typing the Erlang cookie
- * [[`PR-43`](https://github.com/thiagoesteves/deployex/pull/43)] New mode set functionality, the user can now set a specific version to be applied.
- * Multiple optimizations and improvements in organizations and context
- * Unit test added to achieve 100% of coverage
- * [[`00eb09f`](https://github.com/thiagoesteves/deployex/commit/00eb09f71e4ea25ef6a062edade9c95380fda74b)] Modify Monitor to use dynamic supervisors for start/stop instead of receiving direct commands via gen_server
- * [[`b91f640`](https://github.com/thiagoesteves/deployex/commit/b91f640a78a375ddfff310e1465ac962480dc7ee)] Implemented pre_commands functionality for running migrations
- * [[`PR-19`](https://github.com/thiagoesteves/deployex/pull/19)] Adding backoff delay pattern for retries and enhanced Monitor state handling
- * [[`32ac1b9`](https://github.com/thiagoesteves/deployex/commit/32ac1b9debdd7eff5f11aeb833b1616ae6d3f7e7)] Adding ability to copy/paste for the IEX terminal
- * [[`PR-21`](https://github.com/thiagoesteves/deployex/pull/21/files)] Modified aws secret manager name to deployex-${deployex_monitored_app_name}-${deployex_cloud_environment}-secrets
- * Modified ubuntu installer script to require a configuration json file instead of arguments
- * [[`PR-18`](https://github.com/thiagoesteves/deployex/pull/18/files)] Improvements for consistency
- * [[`769e69f`](https://github.com/thiagoesteves/deployex/commit/769e69f)] Created an installer script for ubuntu and added it to the release package
- * Modifying log view to keep the scroll position at the bottom
- * Adding stderr log file for deployex
- * Adding possibility to connect to the IEX terminal (including deployex)
- * Adding stderr and stdout logs for each app from liveview (including deployex)
- * Improved version badge and uptime status show
- * Fixed app card click
- * Adding try/catch for calling Monitor GenServer
- * Adding uptime feature for monitored apps.
- * Modified the application to be able to handle multiple instances for the monitored app
+ * [[`PR-93`](https://github.com/thiagoesteves/deployex/pull/93)] Adding Live Metrics feature with default retention period of 30 minutes
 
 # Host Binaries Available
 
@@ -73,5 +24,6 @@ This release includes binaries for the following Ubuntu versions:
  You can use these pre-built binaries, or you can build your own if preferred.
 
 # Previous Releases
+ * [0.3.0 🚀 (2025-01-06)](https://github.com/thiagoesteves/deployex/blob/0.3.0/CHANGELOG.md)
  * [0.2.0 🚀 (2024-05-23)](https://github.com/thiagoesteves/deployex/blob/0.2.0/CHANGELOG.md)
  * [0.1.0 🚀 (2024-05-06)](https://github.com/thiagoesteves/deployex/blob/0.1.0/changelog.md)

README.md

@@ -6,12 +6,12 @@
 
 DeployEx is a lightweight tool designed for managing deployments for Beam applications (Elixir, Gleam and Erlang) without relying on additional deployment tools like Docker or Kubernetes. Its primary goal is to utilize the release package for executing full deployments or hot-upgrades, depending on the package's content, while leveraging OTP distribution for monitoring and data extraction.
 
-DeployEx acts as a central deployment runner, gathering crucial deployment data such as the current version and release package contents. The content of the release package enables it to run for a full deployment or a hot-upgrade. Meanwhile, on the development front, your CI/CD pipeline takes charge of crafting and updating packages for the target release. This integration ensures that DeployEx is always equipped with the latest packages, ready to facilitate deployments.
+DeployEx serves as a centralized deployment runner, consolidating essential deployment information, including the current version and release package contents. Based on the package contents, it can execute either a full deployment or a hot-upgrade. On the development side, your CI/CD pipeline manages the creation and updating of packages for the target release.
 
 DeployEx is currently used by:
  * [Calori Web Server](https://github.com/thiagoesteves/calori) for __Elixir__ applications and you can check it at [homepage](https://calori.com.br).
- * [Cochito Web Server](https://github.com/chouzar/cochito) for __Gleam__ applications and you can check it at [homepage](https://gleam.deployex.pro).
- * [Snake Game with Cowboy](https://github.com/thiagoesteves/erlgame) for __Erlang__ applications and you can check it at [homepage](https://erlang.deployex.pro).
+ * [Cochito Web Server](https://github.com/chouzar/cochito) for __Gleam__ applications.
+ * [Snake Game with Cowboy](https://github.com/thiagoesteves/erlgame) for __Erlang__ applications.
 
 ![Deployment Architecture](docs/static/deployex.png)
 
@@ -33,6 +33,8 @@ Upon deployment, the following dashboard becomes available, providing easy acces
  * Supports the following cloud providers:
    - Amazon Web Services (AWS)
    - Google Cloud Provisioning (GCP)
+ * Supports live metrics with a retention time of 30 minutes, or until DeployEx restarts, as it relies on ETS tables for storage:
+   - Elixir applications using the [Telemetry Deployex](https://github.com/thiagoesteves/telemetry_deployex) library.
  * Provides rollback functionality if a monitored app version remains unstable for 10 minutes.
  * Rolled-back monitored app versions are ghosted, preventing their redeployment.
  * Ensures all instances remain connected to the OTP distribution, including DeployEx itself.
@@ -65,7 +67,7 @@ Upon deployment, the following dashboard becomes available, providing easy acces
 
 ### What is coming next
 
-- [ ] 🚧 Add telemetry support for DeployEx to capture metrics and telemetry via OTP distribution.
+- [X] 🚧 Add telemetry support for DeployEx to capture metrics and telemetry via OTP distribution.
 - [ ] 💤 Integrate CPU utilization monitoring from the OTP distribution.
 - [ ] 💤 Lazy deployments for Phoenix apps (Delay Endpoint start to allow fast switch for full deployments)
 - [ ] 💤 Continuous improvement in UI design.
@@ -81,6 +83,7 @@ Upon deployment, the following dashboard becomes available, providing easy acces
 
 | DeployEx version   |      Default major OTP version      | 
 |----------|-------------|
+| __0.4.0-rc1__ | __26.2.5.6__ |
 | __0.3.0__ | __26.2.5.6__ |
 
 ### Running the application

assets/js/app.js

@@ -118,6 +118,48 @@ hooks.ObserverEChart = {
   }
 }
 
+hooks.LiveMetricsEChart = {
+  mounted() {
+    selector = "#" + this.el.id
+
+    const dataConfig = JSON.parse(this.el.dataset.config)
+    const columns = JSON.parse(this.el.dataset.columns)
+
+    this.chart = echarts.init(this.el.querySelector(selector + "-chart"))
+    this.chart.setOption(dataConfig)
+    this.graph_cols = columns
+  },
+  updated() {
+    const dataConfig = JSON.parse(this.el.dataset.config)
+    const reset = JSON.parse(this.el.dataset.reset)
+    const columns = JSON.parse(this.el.dataset.columns)
+
+    if (reset) {
+      this.chart.setOption(dataConfig)
+
+    } else {
+      var option = this.chart.getOption();
+      var updatedXAxis = option.xAxis[0].data.concat(dataConfig.xAxis.data);
+      var updatedSeries = option.series.map((series, index) => {
+        // Concatenate the corresponding dataset to each series
+        return {
+          data: series.data.concat(dataConfig.series[index] ? dataConfig.series[index].data : [])
+        };
+      });
+
+      this.chart.setOption(
+        {
+          xAxis: { data: updatedXAxis },
+          series: updatedSeries
+        })
+    }
+    if (columns != this.columns) {
+      this.chart.resize()
+      this.columns = columns
+    }
+  }
+}
+
 let liveSocket = new LiveSocket("/live", Socket, {
   longPollFallbackMs: 2500,
   params: { _csrf_token: csrfToken },

config/config.exs

@@ -113,6 +113,10 @@ config :deployex, Deployex.Storage, adapter: Deployex.Storage.Local
 
 config :deployex, Deployex.Rpc, adapter: Deployex.Rpc.Local
 
+config :deployex, Deployex.Telemetry,
+  adapter: Deployex.Telemetry.Server,
+  data_retention_period: :timer.minutes(30)
+
 # Import environment specific config. This must remain at the bottom
 # of this file so it overrides the configuration defined above.
 import_config "#{config_env()}.exs"

config/test.exs

@@ -40,6 +40,11 @@ config :deployex, Deployex.OpSys, adapter: Deployex.OpSysMock
 # Config Mock for Rpc
 config :deployex, Deployex.Rpc, adapter: Deployex.RpcMock
 
+# Config Mock for Telemetry
+config :deployex, Deployex.Telemetry,
+  adapter: Deployex.TelemetryMock,
+  data_retention_period: :timer.minutes(1)
+
 config :deployex, Deployex.Deployment, delay_between_deploys_ms: 10
 
 # Disable swoosh api client as it is only required for production adapters.

coveralls.json

@@ -12,6 +12,7 @@
     "lib/deployex/storage/adapter.ex",
     "lib/deployex/rpc/adapter.ex",
     "lib/deployex/rpc/local.ex",
+    "lib/deployex/telemetry/adapter.ex",
     "lib/deployex/mailer.ex",
     "lib/deployex/macros.ex",
     "lib/deployex/observer/helper.ex",

docs/examples/local-elixir/README.md

@@ -39,8 +39,8 @@ export RELEASE_NODE=<%= @release.name %>${RELEASE_NODE_SUFFIX}
 # save the file :wq
 ```
 
-## The next steps are needed ONLY for Hot upgrades
-Add [Jellyfish](https://github.com/thiagoesteves/jellyfish) library __ONLY__ if the application will need hotupgrades
+## Steps for supporting Hot upgrades (Optional)
+Add [Jellyfish](https://github.com/thiagoesteves/jellyfish) library __ONLY__ for supporting hotupgrades:
 ```elixir
 def deps do
   [
@@ -49,7 +49,7 @@ def deps do
 end
 ```
 
-You also need to add the following lines in the mix project
+You also need to add the following lines in the `mix.exs` file:
 ```elixir
   def project do
     [
@@ -79,6 +79,33 @@ live_reload: [
 ]
 ```
 
+## Steps for allowing Live Metrics (Optional)
+
+Add [Telemetry Deployex](https://github.com/thiagoesteves/telemetry_deployex) library __ONLY__ for supporting live metrics:
+```elixir
+def deps do
+  [
+    {:telemetry_deployex, "~> 0.1.0-rc4"}
+  ]
+end
+```
+
+Open the telemetry file at `lib/myphoenixapp_web/telemetry.ex` and add the following line in the init function:
+
+```elixir
+@impl true
+def init(_arg) do
+  children = [
+    # Telemetry poller will execute the given period measurements
+    # every 10_000ms. Learn more here: https://hexdocs.pm/telemetry_metrics
+    {:telemetry_poller, measurements: periodic_measurements(), period: 10_000},
+    # Add reporters as children of your supervision tree.
+    {TelemetryDeployex, metrics: metrics()} <---------- Add here
+  ]
+  Supervisor.init(children, strategy: :one_for_one)
+end
+```
+
 ## Generate a release
 Then you can compile and generate a release
 ```bash

lib/deployex/application.ex

@@ -36,7 +36,7 @@ defmodule Deployex.Application do
     Supervisor.start_link(children, opts)
   end
 
-  # NOTE: Skip starting the development and status server when running tests.
+  # NOTE: Skip starting the these servers when running tests.
   if_not_test do
     alias Deployex.Deployment
 
@@ -49,7 +49,8 @@ defmodule Deployex.Application do
            timeout_rollback: Application.fetch_env!(:deployex, Deployment)[:timeout_rollback],
            schedule_interval: Application.fetch_env!(:deployex, Deployment)[:schedule_interval],
            name: Deployment
-         ]}
+         ]},
+        Deployex.Telemetry.Server
       ]
     end
   else

lib/deployex/telemetry.ex

@@ -0,0 +1,81 @@
+defmodule Deployex.Telemetry do
+  @moduledoc """
+  This module will provide telemetry abstraction
+  """
+
+  @behaviour Deployex.Telemetry.Adapter
+
+  defmodule Data do
+    @moduledoc """
+    Structure to handle the telemetry event
+    """
+    @type t :: %__MODULE__{
+            timestamp: non_neg_integer(),
+            value: integer() | float(),
+            unit: String.t(),
+            tags: map(),
+            measurements: map()
+          }
+
+    defstruct timestamp: nil,
+              value: "",
+              unit: "",
+              tags: %{},
+              measurements: %{}
+  end
+
+  ### ==========================================================================
+  ### Public functions
+  ### ==========================================================================
+
+  @doc """
+  This function pushes events to the Telemetry module, it is expected
+  to be called via RPC.
+  """
+  # coveralls-ignore-start
+  @spec push_data(any()) :: :ok
+  def push_data(event), do: default().push_data(event)
+  # coveralls-ignore-stop
+
+  @doc """
+  Subscribe for new keys notifications
+  """
+  @spec subscribe_for_new_keys() :: :ok | {:error, term}
+  def subscribe_for_new_keys, do: default().subscribe_for_new_keys()
+
+  @doc """
+  Subscribe for new data notifications for the respective node/key
+  """
+  @spec subscribe_for_new_data(String.t(), String.t()) :: :ok | {:error, term}
+  def subscribe_for_new_data(node, key), do: default().subscribe_for_new_data(node, key)
+
+  @doc """
+  Unsubscribe for new data notifications for the respective node/key
+  """
+  @spec unsubscribe_for_new_data(String.t(), String.t()) :: :ok
+  def unsubscribe_for_new_data(node, key), do: default().unsubscribe_for_new_data(node, key)
+
+  @doc """
+  Fetch data by node and key
+  """
+  @spec list_data_by_node_key(atom() | String.t(), String.t(), Keyword.t()) :: list()
+  def list_data_by_node_key(node, key, options),
+    do: default().list_data_by_node_key(node, key, options)
+
+  @doc """
+  List all keys registered for the respective instance
+  """
+  @spec get_keys_by_instance(integer()) :: list()
+  def get_keys_by_instance(instance), do: default().get_keys_by_instance(instance)
+
+  @doc """
+  Retrieve the repective noce registered for the passed instance
+  """
+  @spec node_by_instance(integer()) :: nil | atom()
+  def node_by_instance(instance), do: default().node_by_instance(instance)
+
+  ### ==========================================================================
+  ### Private functions
+  ### ==========================================================================
+  defp default, do: Application.fetch_env!(:deployex, __MODULE__)[:adapter]
+end

lib/deployex/telemetry/adapter.ex

@@ -0,0 +1,13 @@
+defmodule Deployex.Telemetry.Adapter do
+  @moduledoc """
+  Behaviour that defines the telemetry adapter callback
+  """
+
+  @callback push_data(any()) :: :ok
+  @callback subscribe_for_new_keys() :: :ok | {:error, term}
+  @callback subscribe_for_new_data(String.t(), String.t()) :: :ok | {:error, term}
+  @callback unsubscribe_for_new_data(String.t(), String.t()) :: :ok
+  @callback list_data_by_node_key(atom() | String.t(), String.t(), Keyword.t()) :: list()
+  @callback get_keys_by_instance(integer()) :: list()
+  @callback node_by_instance(integer()) :: nil | atom()
+end