Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add extension to kafka-cron project - port to minikube #206

Merged
merged 4 commits into from
May 20, 2024
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 30 additions & 0 deletions projects/kafka-cron/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -242,6 +242,36 @@ How many containers being down can our system tolerate?

What happens to the Kafka system logs and the metrics that our binaries export? Did our alerts fire? If not, consider how they could be improved - remember, the point of them is to tell us when something's wrong!

### Porting your system from docker-compose to minikube

In this project, and in several previous projects, you have used `docker-compose` to deploy our code, alongside dependencies (such as Kafka and
Zookeeper here). `docker-compose` is an extremely convenient tool for running a multi-part software stack locally (it also works well for running
integration tests in a Continuous Integration workflow as part of your development process). However, in most deployments, we want to be able
to run our code and its dependencies across more than one host, in order to scale horizontally and to be robust to single-node failures. For this,
`kubernetes` is a better tool. `Kubernetes`, like `docker-compose`, is a platform for running containerised applications, but where `docker-compose`
is focused on running a set of related containers on a single host, `kubernetes` is optimized for running services across many hosts.

Here are some introductions to Kubernetes:
* [What is Kubernetes](https://www.digitalocean.com/community/tutorials/an-introduction-to-kubernetes)
* [Kubernetes Basics](https://kubernetes.io/docs/tutorials/kubernetes-basics/) - explore the linked pages here

You may have already used `minikube` in one of the previous projects. `Minikube` is a local version of Kubernetes, which we can use to learn
(rather than needing to incur the expense of cloud services such as [EKS](https://aws.amazon.com/eks/) for learning on.)

Get your local `minikube setup` working by following [minikube start](https://minikube.sigs.k8s.io/docs/start/).
Follow the steps to install the program and run the `hello-minikube` application.

Once you have done this, you will need to convert your `docker-compose.yml` files to `kubernetes` deployment files.
There is a tool, [kompose](https://kubernetes.io/docs/tasks/configure-pod-container/translate-compose-kubernetes/) which may assist you.

Test that your system works as it did on `docker-compose`.

Learn your way around the `kubernetes` command-line tool, `kubectl` (see the [kubectl documentation](https://kubernetes.io/docs/reference/kubectl/)).
* How can you resize your service, i.e. change the number of running instances (pods)?
* How can you remove one instance of your service (a single pod)?
* How can you see the logs for your service?
* How can you see the log of Kubernetes' management operations?

### Dealing with long-running jobs and load (challenging)

What does our system do if someone submits a very long-running job? Try testing this with the `sleep` command.
Expand Down
Loading