Supporting deployments on zos4 nodes (#3497)

* Supporting deployments on zos4 nodes * Fixing tests, adding vmlight primitive and adding a match method * Adding networklight class in primitives * Updating networklight class in primitives * Adjusting FilterOptions to support features in automatic and manual node selection * Adding wiregaurd to filters * Adjusting getFeaturesFromFilters method and adding planetary, mycelium and wireguard to filters in all solutions * Updating networklight primitive class, create deployment and create methods to support zos4 deployments * Adjusting load, NodeExists, addNode and getFreeSubnet methods in network light primitive * Updating getFeaturesFromFilters method * Fixing reservation network error, updating k8s to support zos4 deployment, removing unnecessary comments and adding descriptive ones * Added a script to deploy on a zos4 node * Adding deployment flow doc * Resolving comments: removing unnecessary comments, adding wiregaurd in AddMachine filters and moving getFeatures method to utils * Updating network light primitive class * Revert "Resolving comments: removing unnecessary comments, adding wiregaurd in AddMachine filters and moving getFeatures method to utils" This reverts commit 17c3ec9. * Revert "Updating network light primitive class" This reverts commit 14ab9b9. * Added some changes in networklight primitive and added some more adjustments in _deleteMachineNetwork function * Fixing _deleteMachineNetwork function * Adjusting manual selection feature error, fixing the public_ip random error * Adjusting manual selection feature error, fixing the public_ip random error * Added the required features in all scripts' filters * Adjusting Supported interface in manage domains in case of zmachine-light * Adjust zos4 script and merging functionalites of delete zmachine and zmachine light in one condition to get rid of redundant code * Resolving comments: Addinf Features enum, using WorkloadTypes in ZmachineData, some updates in networklight primitive * Refactoring vm primitive file, adjusting features type in filtersOptions and fixing scripts * Fixing typo in deployment flow readme file * Fixing scripts unhandled promise error * Adding features to tests, corrects mispelled words and hiding wireguard in deployement data in case of zmachine light * Adding features to tests, corrects mispelled words and hiding wireguard in deployement data in case of zmachine light * Excluding node 259 in single_vm_zos4 script * adjusting features in zos4 script and updating deployment flow readme file * Checking against features rather than zos version while deploying, using workloadTypes enum and adding a semi-fix to loadgateway nodes * Fix: using json.stringify for filters so it doesn't enter an infinite loop when loading gateway nodes * Removing comments * Removing features from machinemodel
threefoldtech · Jan 13, 2025 · fd63f33 · fd63f33
1 parent 8dbc069
commit fd63f33
Show file tree

Hide file tree

Showing 86 changed files with 1,722 additions and 191 deletions.
diff --git a/packages/grid_client/docs/deployment_flow.md b/packages/grid_client/docs/deployment_flow.md
@@ -0,0 +1,117 @@
+# Deployment Flow
+
+This document outlines the process for determining whether to deploy on a zos3 or a zos4 node.
+
+## Machine Model Initialization
+
+The deployment process begins with initializing the machine model as follows:
+
+```ts
+const vms: MachinesModel = {
+  name: "newMY",
+  network: {
+    name: "hellotest",
+    ip_range: "10.249.0.0/16",
+    myceliumSeeds: [
+      {
+        nodeId: 168,
+        seed: "050d109829d8492d48bfb33b711056080571c69e46bfde6b4294c4c5bf468a76", //(HexSeed of length 32)
+      },
+    ],
+  },
+  machines: [
+    {
+      name: "testvmMY",
+      node_id: 168,
+      disks: [
+        {
+          name: "wedDisk",
+          size: 8,
+          mountpoint: "/testdisk",
+        },
+      ],
+      public_ip: false,
+      public_ip6: false,
+      planetary: true,
+      mycelium: true,
+      myceliumSeed: "1e1404279b3d", //(HexSeed of length 6)
+      cpu: 1,
+      memory: 1024 * 2,
+      rootfs_size: 0,
+      flist: "https://hub.grid.tf/tf-official-apps/base:latest.flist",
+      entrypoint: "/sbin/zinit init",
+      env: {
+        SSH_KEY: config.ssh_key,
+      },
+    },
+  ],
+  metadata: "",
+  description: "test deploying single VM with mycelium via ts grid3 client",
+};
+```
+
+## Deployment Execution
+
+- The next step is invoking the `deploy` function:
+  - Takes the `MachinesModel` object as a parameter
+  - Checks if a machine with the same name already exists and if so throws an error
+  - If not, then it calls the `_createDeployment` function along some othe functions
+  - Finally, the function returns the created contracts. If the deployment is on a zos3 node, then the wireguard configuration is returned as well.
+
+```ts
+await client.machines.deploy(vms);
+```
+
+- The `_createDeployment` function:
+  - Takes the `MachinesModel` object as a parameter
+  - Retrieves the features of the node using: `await this.rmb.request([nodeTwinId], "zos.system.node_features_get", "", 20, 3);`
+  - Examines the retrieved features to determine the network's primitive type (`Network` or `ZNetworkLight`) and initializes it accordingly.
+  - Sets the contractMetadata based on the network type.
+  - Invokes the `create` function
+
+```ts
+await this._createDeployment(options);
+```
+
+- The `create` function
+  - Validates or assigns IP addresses based on the network type.
+  - Determines network type (`network` or `network-light`) based on node features.
+  - Adds access points and updates network configurations as necessary.
+  - Initialize the VM primitive (`VMPrimitive` or `VMLightPrimitive`) based on the network type
+  - Configures the VM with networking, storage, and environment variables.
+  - Generates a Mycelium seed if not provided.
+  - Generate the deployments
+  - Finally, the function returns the created contracts. If the deployment is on a zos3 node, then the wireguard configuration is returned as well.
+
+```ts
+await this.vm.create(
+  machine.name,
+  machine.node_id,
+  machine.flist,
+  machine.cpu,
+  machine.memory,
+  machine.rootfs_size,
+  machine.disks!,
+  machine.public_ip,
+  machine.public_ip6!,
+  machine.planetary,
+  machine.mycelium,
+  machine.myceliumSeed!,
+  network,
+  options.network.myceliumSeeds!,
+  machine.entrypoint,
+  machine.env,
+  contractMetadata,
+  options.metadata,
+  options.description,
+  machine.qsfs_disks,
+  this.config.projectName,
+  options.network.addAccess,
+  options.network.accessNodeId,
+  machine.ip,
+  machine.corex,
+  machine.solutionProviderId!,
+  machine.zlogsOutput,
+  machine.gpus,
+);
+```
diff --git a/packages/grid_client/scripts/applications/casperlabs.ts b/packages/grid_client/scripts/applications/casperlabs.ts
@@ -1,4 +1,4 @@
-import { FilterOptions, GatewayNameModel, MachinesModel } from "../../src";
+import { Features, FilterOptions, GatewayNameModel, MachinesModel } from "../../src";
 import { config, getClient } from "../client_loader";
 import { log, pingNodes } from "../utils";
 
@@ -55,6 +55,7 @@ async function main() {
     sru: instanceCapacity.sru,
     availableFor: grid3.twinId,
     farmId: 1,
+    features: [Features.wireguard],
   };
   //GatewayNode Selection
   const gatewayQueryOptions: FilterOptions = {

diff --git a/packages/grid_client/scripts/applications/discourse.ts b/packages/grid_client/scripts/applications/discourse.ts
@@ -1,7 +1,7 @@
 import { Buffer } from "buffer";
 import TweetNACL from "tweetnacl";
 
-import { FilterOptions, GatewayNameModel, MachinesModel } from "../../src";
+import { Features, FilterOptions, GatewayNameModel, MachinesModel } from "../../src";
 import { config, getClient } from "../client_loader";
 import { log, pingNodes } from "../utils";
 
@@ -63,6 +63,7 @@ async function main() {
     sru: instanceCapacity.sru,
     availableFor: grid3.twinId,
     farmId: 1,
+    features: [Features.ip, Features.ipv4, Features.wireguard],
   };
   //GatewayNode Selection
   const gatewayQueryOptions: FilterOptions = {

diff --git a/packages/grid_client/scripts/applications/funkwhale.ts b/packages/grid_client/scripts/applications/funkwhale.ts
@@ -1,4 +1,4 @@
-import { FilterOptions, GatewayNameModel, MachinesModel } from "../../src";
+import { Features, FilterOptions, GatewayNameModel, MachinesModel } from "../../src";
 import { config, getClient } from "../client_loader";
 import { log, pingNodes } from "../utils";
 
@@ -55,6 +55,7 @@ async function main() {
     sru: instanceCapacity.sru,
     availableFor: grid3.twinId,
     farmId: 1,
+    features: [Features.wireguard],
   };
   //GatewayNode Selection
   const gatewayQueryOptions: FilterOptions = {

diff --git a/packages/grid_client/scripts/applications/mattermost.ts b/packages/grid_client/scripts/applications/mattermost.ts
@@ -1,4 +1,4 @@
-import { FilterOptions, GatewayNameModel, MachinesModel } from "../../src";
+import { Features, FilterOptions, GatewayNameModel, MachinesModel } from "../../src";
 import { config, getClient } from "../client_loader";
 import { log, pingNodes } from "../utils";
 
@@ -47,6 +47,8 @@ async function main() {
   const grid3 = await getClient(`mattermost/${name}`);
   const subdomain = "mm" + grid3.twinId + name;
   const instanceCapacity = { cru: 1, mru: 2, sru: 15 }; // Update the instance capacity values according to your requirements.
+  // Change to true if smtp is configured
+  const smtp = false;
 
   //VMNode Selection
   const vmQueryOptions: FilterOptions = {
@@ -55,6 +57,7 @@ async function main() {
     sru: instanceCapacity.sru,
     availableFor: grid3.twinId,
     farmId: 1,
+    features: smtp ? [Features.ip, Features.ipv4, Features.wireguard] : [],
   };
   //GatewayNode Selection
   const gatewayQueryOptions: FilterOptions = {

diff --git a/packages/grid_client/scripts/applications/nextcloud.ts b/packages/grid_client/scripts/applications/nextcloud.ts
@@ -1,4 +1,4 @@
-import { FilterOptions, GatewayNameModel, MachinesModel } from "../../src";
+import { Features, FilterOptions, GatewayNameModel, MachinesModel } from "../../src";
 import { config, getClient } from "../client_loader";
 import { log, pingNodes } from "../utils";
 
@@ -55,6 +55,7 @@ async function main() {
     sru: instanceCapacity.sru,
     availableFor: grid3.twinId,
     farmId: 1,
+    features: [Features.wireguard],
   };
   //GatewayNode Selection
   const gatewayQueryOptions: FilterOptions = {

diff --git a/packages/grid_client/scripts/applications/nodepilot.ts b/packages/grid_client/scripts/applications/nodepilot.ts
@@ -1,4 +1,4 @@
-import { FilterOptions, MachinesModel } from "../../src";
+import { Features, FilterOptions, MachinesModel } from "../../src";
 import { config, getClient } from "../client_loader";
 import { log, pingNodes } from "../utils";
 
@@ -35,6 +35,7 @@ async function main() {
     sru: instanceCapacity.sru,
     availableFor: grid3.twinId,
     farmId: 1,
+    features: [Features.ip, Features.ipv4, Features.wireguard],
   };
   const nodes = await grid3.capacity.filterNodes(vmQueryOptions);
   const vmNode = await pingNodes(grid3, nodes);

diff --git a/packages/grid_client/scripts/applications/peertube.ts b/packages/grid_client/scripts/applications/peertube.ts
@@ -1,4 +1,4 @@
-import { FilterOptions, GatewayNameModel, MachinesModel } from "../../src";
+import { Features, FilterOptions, GatewayNameModel, MachinesModel } from "../../src";
 import { config, getClient } from "../client_loader";
 import { log, pingNodes } from "../utils";
 
@@ -55,6 +55,7 @@ async function main() {
     sru: instanceCapacity.sru,
     availableFor: grid3.twinId,
     farmId: 1,
+    features: [Features.wireguard],
   };
   //GatewayNode Selection
   const gatewayQueryOptions: FilterOptions = {

diff --git a/packages/grid_client/scripts/applications/static_website.ts b/packages/grid_client/scripts/applications/static_website.ts
@@ -1,4 +1,4 @@
-import { FilterOptions, GatewayNameModel, GridClient, MachinesModel, NodeInfo } from "../../src";
+import { Features, FilterOptions, GatewayNameModel, GridClient, MachinesModel, NodeInfo } from "../../src";
 import { config, getClient } from "../client_loader";
 import { log, pingNodes } from "../utils";
 
@@ -55,6 +55,7 @@ async function main() {
     sru: instanceCapacity.sru,
     availableFor: grid3.twinId,
     farmId: 1,
+    features: [Features.wireguard],
   };
   //GatewayNode Selection
   const gatewayQueryOptions: FilterOptions = {

diff --git a/packages/grid_client/scripts/applications/subsquid.ts b/packages/grid_client/scripts/applications/subsquid.ts
@@ -1,4 +1,4 @@
-import { FilterOptions, GatewayNameModel, MachinesModel } from "../../src";
+import { Features, FilterOptions, GatewayNameModel, MachinesModel } from "../../src";
 import { config, getClient } from "../client_loader";
 import { log, pingNodes } from "../utils";
 
@@ -55,6 +55,7 @@ async function main() {
     sru: instanceCapacity.sru,
     availableFor: grid3.twinId,
     farmId: 1,
+    features: [Features.wireguard],
   };
   //GatewayNode Selection
   const gatewayQueryOptions: FilterOptions = {

diff --git a/packages/grid_client/scripts/applications/taiga.ts b/packages/grid_client/scripts/applications/taiga.ts
@@ -1,4 +1,4 @@
-import { FilterOptions, GatewayNameModel, MachinesModel } from "../../src";
+import { Features, FilterOptions, GatewayNameModel, MachinesModel } from "../../src";
 import { config, getClient } from "../client_loader";
 import { log, pingNodes } from "../utils";
 
@@ -55,6 +55,7 @@ async function main() {
     sru: instanceCapacity.sru,
     availableFor: grid3.twinId,
     farmId: 1,
+    features: [Features.wireguard],
   };
   //GatewayNode Selection
   const gatewayQueryOptions: FilterOptions = {

diff --git a/packages/grid_client/scripts/applications/wordpress.ts b/packages/grid_client/scripts/applications/wordpress.ts
@@ -1,4 +1,4 @@
-import { FilterOptions, GatewayNameModel, MachinesModel } from "../../src";
+import { Features, FilterOptions, GatewayNameModel, MachinesModel } from "../../src";
 import { config, getClient } from "../client_loader";
 import { log, pingNodes } from "../utils";
 
@@ -56,6 +56,7 @@ async function main() {
     sru: instanceCapacity.sru,
     availableFor: grid3.twinId,
     farmId: 1,
+    features: [Features.wireguard],
   };
   //GatewayNode Selection
   const gatewayQueryOptions: FilterOptions = {

diff --git a/packages/grid_client/scripts/orchestrators/caprover_leader.ts b/packages/grid_client/scripts/orchestrators/caprover_leader.ts
@@ -1,4 +1,4 @@
-import { FilterOptions, MachinesModel } from "../../src";
+import { Features, FilterOptions, MachinesModel } from "../../src";
 import { config, getClient } from "../client_loader";
 import { log } from "../utils";
 
@@ -34,6 +34,7 @@ async function main() {
     sru: 10,
     farmId: 1,
     availableFor: grid3.twinId,
+    features: [Features.wireguard, Features.ip, Features.ipv4],
   };
 
   const vms: MachinesModel = {

diff --git a/packages/grid_client/scripts/orchestrators/caprover_worker.ts b/packages/grid_client/scripts/orchestrators/caprover_worker.ts
@@ -1,4 +1,4 @@
-import { FilterOptions, MachinesModel } from "../../src";
+import { Features, FilterOptions, MachinesModel } from "../../src";
 import { config, getClient } from "../client_loader";
 import { log } from "../utils";
 
@@ -32,6 +32,7 @@ async function main() {
     mru: 4, // GB
     sru: 10,
     farmId: 1,
+    features: [Features.wireguard, Features.ip, Features.ipv4],
   };
 
   const vms: MachinesModel = {

diff --git a/packages/grid_client/scripts/orchestrators/kubernetes_leader.ts b/packages/grid_client/scripts/orchestrators/kubernetes_leader.ts
@@ -1,4 +1,4 @@
-import { FilterOptions, K8SModel } from "../../src";
+import { Features, FilterOptions, K8SModel } from "../../src";
 import { config, getClient } from "../client_loader";
 import { log } from "../utils";
 
@@ -33,6 +33,7 @@ async function main() {
     sru: 6,
     availableFor: grid3.twinId,
     farmId: 1,
+    features: [Features.wireguard],
   };
 
   const workerQueryOptions: FilterOptions = {

diff --git a/packages/grid_client/scripts/orchestrators/kubernetes_with_qsfs.ts b/packages/grid_client/scripts/orchestrators/kubernetes_with_qsfs.ts
@@ -1,4 +1,4 @@
-import { FilterOptions, GridClient, K8SModel, QSFSZDBSModel } from "../../src";
+import { Features, FilterOptions, GridClient, K8SModel, QSFSZDBSModel } from "../../src";
 import { config, getClient } from "../client_loader";
 import { log, pingNodes } from "../utils";
 
@@ -50,6 +50,7 @@ async function main() {
     sru: 6,
     availableFor: grid3.twinId,
     farmId: 1,
+    features: [Features.wireguard],
   };
 
   const qsfsQueryOptions: FilterOptions = {

diff --git a/packages/grid_client/scripts/orchestrators/kubernetes_worker.ts b/packages/grid_client/scripts/orchestrators/kubernetes_worker.ts
@@ -1,4 +1,4 @@
-import { AddWorkerModel, FilterOptions } from "../../src";
+import { AddWorkerModel, Features, FilterOptions } from "../../src";
 import { getClient } from "../client_loader";
 import { log } from "../utils";
 
@@ -34,6 +34,7 @@ async function main() {
     mru: 1, // GB
     sru: 10,
     farmId: 1,
+    features: [Features.wireguard],
   };
 
   const worker: AddWorkerModel = {