Keda is a Kubernetes-based Event Driven Autoscaler which enables you to scale containers for processing events based on number of events.
Keda Connectors provide you readymade generic images to process events for standard tasks. For example if you are going to read messages from RabbitMQ and then call an HTTP endpoint then you can use RabbitMQ HTTP Connector. You just have to create a standard deployment manifest with all env variables needed for the pod.
These connectors are used in Fission project for integrating events to call functions via MQTrigger CRD.
| Connectot Name | Description |
|---|---|
| AWS SQS HTTP Connector | Reads message from AWS SQS and posts to a HTTP endpoint. |
| Kafka HTTP Connector | Consumes messages from Kafka topics and posts the message to an HTTP endpoint. |
| RabbitMQ HTTP Connector | Reads message from RabbitMQ and posts to a HTTP endpoint. Currently only the AMQP protocol is supported for consuming RabbitMQ messages. |
The job of the connector is to read messages from the topic, invoke a HTTP endpoint, and write response or error in the respective queues/topics.
-
Message Queue Trigger Spec fields are exposed as environment variables while creating deployment which will be utilized while creating consumer, producer, and during function invocation. Parse all required parameters required by the connector.
-
Create a Consumer for the queue.
-
Create a Producer for the queue.
-
Read messages from the queue using the consumer.
-
Create a POST request with the following headers using values specified during message queue trigger creation and headers from the consumed message.
{
"KEDA-Topic": Topic,
"KEDA-Response-Topic": ResponseTopic,
"KEDA-Error-Topic": ErrorTopic,
"Content-Type": ContentType
}
-
Invoke HTTP endpoint per consumed message in a retry loop using max retries parameter specified. You can reuse the already available method for calling HTTP endpoints: https://github.com/fission/keda-connectors/blob/master/common/util.go#L52
-
Write the response in response queue, if the reponse topic is specified.
-
If applicable then write error in error queue, if error topic is specified.
-
The final step would be to write a dockerfile to create docker image of the code.
Refer to Kafka HTTP Connector or RabbitMQ HTTP Connector for sample implementations.