Merge pull request #15 from vectordotdev/silversupreme/multi-config-demo

Add a demo for multiple configs provided to a Vector instance.

.gitignore

@@ -1,2 +1,4 @@
 auth_token
-*.log
+*.log
+
+charts/

multi-config/Chart.lock

@@ -0,0 +1,6 @@
+dependencies:
+- name: vector
+  repository: https://helm.vector.dev
+  version: 0.2.2
+digest: sha256:ae9c41d335a82592b150e8511e6ae5dbe26fbb364399dad5ba764ca88dcdc824
+generated: "2021-12-23T18:59:04.637364547Z"

multi-config/Chart.yaml

@@ -0,0 +1,8 @@
+apiVersion: v2
+name: multi-config-vector
+version: 1.0.0
+description: A demonstration of multiple ConfigMaps being merged into a single Vector config.
+dependencies:
+  - name: vector
+    version: ~0.2
+    repository: https://helm.vector.dev

multi-config/Makefile

@@ -0,0 +1,15 @@
+start-local:
+	@echo "ℹ️  \033[36mRunning Vector locally to demonstrate how configs are loaded from a directory.\033[0m\n"
+	@vector -C ./vector
+
+start-k8s:
+	@echo "ℹ️  \033[36mRunning Vector in target k8s cluster to demonstrate merged configmaps.\033[0m"
+	@echo "⚠️  \033[33mNote that this will run in whatever your current kubectl target cluster is!\033[0m\n"
+	@helm install -n vector test .
+
+stop-k8s:
+	@echo "⚠️  \033[33mRemoving the test k8s Vector instance.\033[0m\n"
+	@helm uninstall -n vector test
+
+logs-k8s:
+	@kubectl logs -n vector test-vector-0

multi-config/README.md

@@ -0,0 +1,36 @@
+# Composing Vector Config Files
+This repo is a simple demonstration of providing Vector with multiple config files, which are merged into a ConfigMap 
+in the destination Kubernetes cluster. By leveraging this, users can break apart pieces of their Vector configs for
+management by multiple business units.
+
+## Constraints
+* **This solution works inside of a vanilla Kubernetes cluster.** This is intended as a simple solution that any
+  customer can easily hit the ground running with.
+* **Helm does not support templating in `values.yaml`.** [The method we use to provide extra config files to Vector does not support templated names](https://github.com/helm/helm/issues/2492),
+  which makes it difficult to make this a truly generic solution for your needs. In particular, this makes it
+  impossible to properly scope the ConfigMap names with `{{ .Release.Name }}` as you might wish, or to mount each config
+  file as a separate ConfigMap to avoid [file size limits](https://kubernetes.io/docs/concepts/configuration/configmap/#:~:text=A%20ConfigMap%20is%20not%20designed,separate%20database%20or%20file%20service.).
+
+## The Basics
+Multiple configs are provided to Vector by wrapping the official Helm chart with this one, which generates a ConfigMap
+from a directory and adds that to the supported `existingConfigMaps` for the Vector pod.
+
+To add configs to Vector in this setup, all you must do is add them to the `./vector` directory: both the locally-running
+Vector as well as the one installed by the Helm chart are configured to utilize all files there. Only `.yaml` files are
+supported for now.
+
+A simple `Makefile` is supplied which demonstrates the proper commands to test this setup both locally and through Helm.
+Teardown commands are also provided there for when testing is complete.
+
+The Helm chart was tested in `minikube v1.24.0 (76b94fb3c4e8ac5062daf70d60cf03ddcc0a741b)` running on Ubuntu 20.04.3 on a `c6g.2xlarge` EC2 instance.
+
+## Next Steps
+While useful, this setup does have limitations. In particular, the configs must be managed and provisioned inside of the
+same Kubernetes cluster that Vector runs in. Managing multiple clusters still imposes a non-trivial operational overhead.
+[Helm's template control structures](https://helm.sh/docs/chart_template_guide/control_structures/) could provide some
+ways of providing for multiple kinds of clusters/deployments from a single chart, although doing so is beyond the scope
+of this demo.
+
+Datadog's Observability Pipelines feature set (coming soon) can more easily provide config synchronization between a
+fleet of Vector instances, and allow remote management of Vector in an easy-to-use UI. If you're interested, please
+contact your TAM or Datadog Sales for more information on pricing and timelines.

multi-config/templates/configs.yaml

@@ -0,0 +1,9 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: configs
+data: 
+{{ range $path, $_ :=  .Files.Glob  "vector/**.yaml" }}
+  {{ $path | base }}: |
+{{ $.Files.Get $path | indent 4}}
+{{ end }}

multi-config/values.yaml

@@ -0,0 +1,3 @@
+vector:
+  existingConfigMaps:
+    - configs

multi-config/vector/global.yaml

@@ -0,0 +1,5 @@
+data_dir: /vector-data-dir
+api:
+  enabled: true
+  address: 127.0.0.1:8686
+  playground: false

multi-config/vector/one.yaml

@@ -0,0 +1,4 @@
+sources:
+  demo:
+    type: demo_logs
+    format: syslog

multi-config/vector/two.yaml

@@ -0,0 +1,7 @@
+sinks:
+  debug:
+    type: console
+    target: stderr
+    encoding: json
+    inputs:
+      - demo