Skip to content

Commit

Permalink
docs(orchestrator): improve README
Browse files Browse the repository at this point in the history
  • Loading branch information
batzionb committed Dec 29, 2024
1 parent 48407fc commit b8bdbe0
Show file tree
Hide file tree
Showing 5 changed files with 213 additions and 85 deletions.
122 changes: 71 additions & 51 deletions workspaces/orchestrator/README.md
Original file line number Diff line number Diff line change
@@ -1,57 +1,43 @@
# Orchestrator Plugin for Backstage

The Orchestrator for Backstage is a mechanism designed to facilitate the implementation and execution of developer self-service flows. It serves as a vital component that enhances and augments the existing scaffolder functionality of Backstage with a more flexible and powerful set of features including long-running and asynchronous flows.
The Orchestrator for Backstage is designed to enable self-service flows. It serves as a vital component that enhances and augments the existing scaffolder functionality of Backstage with a more flexible and powerful set of features including long-running and asynchronous flows.

The orchestrator works harmoniously with other Backstage components such as the Software Catalog, permissions, and plugins as well as others. By leveraging its capabilities, organizations can orchestrate and coordinate developer self-service flows effectively.

## Context
It utilizes [SonataFlow](https://sonataflow.org/), a powerful tool for building cloud-native workflow applications.

The Backstage Orchestrator plugin aims to provide a better option to Scaffolder, based on workflows to have a more flexible and powerful tool that addresses the need by streamlining and automating processes, allowing developers to focus more on coding and innovation.
The orchestrator works harmoniously with other Backstage components such as the Software Catalog, permissions, and plugins as well as others. By leveraging its capabilities, organizations can orchestrate and coordinate developer self-service flows effectively.

The orchestrator relies on [SonataFlow](https://sonataflow.org/), a powerful tool for building cloud-native workflow applications.
## Architecture Overview

The main idea is to keep the same user experience for users, leveraging the UI components, input forms, and flow that Scaffolder provides, this way it should be straightforward for users and transparent no matter whether using Templates or Workflows, both can live together being compatible with integration points.
The architecture adheres to standard Backstage plugin guidelines and requires the following plugins to be installed for proper functionality:

The orchestrator controls the flow orchestrating operations/tasks that may be executed in any external service including Scaffolder Actions, this way it is possible to leverage any existing Action hence Software Templates can be easily migrated to workflows opening the door to extend them to more complex use cases.
1. **`backstage-plugin-orchestrator`**

## Capabilities
- Provides the frontend interface.

**Advanced core capabilities**
2. **`backstage-plugin-orchestrator-backend`**
- Serves as a backend proxy between to SonataFlow.

- Stateful/long-lived
- Branching and parallelism
- Error management and compensation
- Event-driven supporting [CloudEvents](https://cloudevents.io)
- Audit logging
- Sub-flows
- Choreography
- Timer/timeout control
- Built-in powerful expression evaluation with JQ
- Low Code/No code
- Cloud-native architecture Kubernetes/OpenShift with Operator support
- OpenAPI / REST built-in integration etc.
In addition to the main plugins, the architecture includes several isolated plugins that are imported by the above plugins:

**Client-side tooling**
1. **`backstage-plugin-orchestrator-swf-editor-envelope`**

- Orchestration visualization / graphical editor
- Integration with service catalog/actions
- GitHub integration
- Form generation
- Runtime monitoring of instances
- Dashboards
- Potential custom integrations (user interaction, notifications, etc.)
- Hosts the workflow viewer.

2. **`backstage-plugin-orchestrator-common`**

## Devmode config
- Contains the backend OpenAPI specification along with autogenerated API documentation and client libraries.

3. **`backstage-plugin-orchestrator-form`**

## Install as a static plugin and run locally
- Provides the workflow execution form.

### Prerequisites
4. **`backstage-plugin-orchestrator-form-api`**
- Defines the API for extending the workflow execution form.
- For more details, see the [Extensible Form Documentation](./docs/extensibleForm.md).

- Docker up and running
## Install as a static plugin

Follows these instructions to install the orchestrator plugin on your own backstage environment. These instructions assume your code structure has the standard [backstage app structure](https://backstage.io/docs/getting-started/).
Follows these instructions to install the orchestrator plugin in a backstage environment. These instructions assume the code structure has the standard [backstage app structure](https://backstage.io/docs/getting-started/).

### Setting up the Orchestrator backend package

Expand Down Expand Up @@ -123,13 +109,21 @@ Follows these instructions to install the orchestrator plugin on your own backst
);
```


## Install as a dynamic plugin in Red Hat Developer Hub

Follow [these guidelines](https://github.com/rhdhorchestrator/orchestrator-helm-operator/blob/main/docs/release-1.3/existing-rhdh.md#install-the-orchestrator-operator) to install the orchestrator operator and configure the orchestrator as a dynamic plugin in Red Hat Developer Hub.

## Configuration

### Configuration for running locally
### Devmode local configuration

This configuration serves for running the application locally for development purposes.

#### Prerequisites

- Docker up and running

####

```yaml title="app-config.yaml"
backend:
Expand All @@ -151,33 +145,49 @@ orchestrator:
```
This configuration will trigger the following:
1. Cloning https://github.com/parodos-dev/backstage-orchestrator-workflows to /tmp/orchestrator/repository.
2. Running the sonataflow devmode container configured to load the workflows located in /tmp/orchestrator/repository.
2. Running the SonataFlow devmode container configured to load the workflows located in /tmp/orchestrator/repository.
> **Note:** /tmp/orchestrator needs to be available to docker.
> **Note:** /tmp/orchestrator needs to be accessible to docker.
The csp headers are required for the Workflow viewer to load.
### Production configuration
```yaml title="app-config.yaml"
backend:
csp:
script-src: ["'self'", "'unsafe-inline'", "'unsafe-eval'"]
script-src-elem: ["'self'", "'unsafe-inline'", "'unsafe-eval'"]
connect-src: ["'self'", 'http:', 'https:', 'data:']
orchestrator:
dataIndexService:
url: <url to SonataFlow data index>
```
The csp headers are required for the Workflow viewer to load.
## User interface
The user interface is accessible via the orchestrator button added in the Backstage sidebar. It provides a list of workflows and workflow runs, and an option to run the workflow and view the result.
The user interface is accessible via the orchestrator button added in the Backstage sidebar. It provides a list of workflows and the option to run the workflows and view the results.
![user interface](image.png)
![user interface](./docs/userInterface.png)
## Permissions
## Orchestrator API
The Orchestrator plugin protects its backend endpoints with the builtin permission mechanism and combines it with
the RBAC plugin. The result is control over what users can see or execute. Details available [here](./docs/Permissions.md).
The plugin provides OpenAPI `v2` endpoints definition to facilitate communication between the frontend and backend. This approach minimizes the data that needs to be sent to the frontend, provides flexibility and avoids dependencies on changes in the [CNCF serverless specification](https://github.com/serverlessworkflow/specification/blob/main/specification.md).
## Orchestrator API
The OpenAPI specification file is availabe [here](./plugins/orchestrator-common/src/openapi/openapi.yaml).
The backend plugin provides OpenAPI `v2` endpoints definition.

The OpenAPI specification documentation is available [here](./plugins/orchestrator-common/src/generated/docs/markdown/README.md)
The OpenAPI specification file is availabe in the [openapi spec file](./plugins/orchestrator-common/src/openapi/openapi.yaml).

The plugin provides an auto generated typescript client that can be used to call the API. To use it include the @red-hat-developer-hub/backstage-plugin-orchestrator-common plugin in your project. Refer to [OrchestratorClient.ts](./plugins/orchestrator/src/api/OrchestratorClient.ts#L59) as an example for how to use it.
Documentation is available in the autogenerated [documentation](./plugins/orchestrator-common/src/generated/docs/markdown/README.md)

The plugin provides an auto generated typescript client that can be used to call the API. To use it include the @red-hat-developer-hub/backstage-plugin-orchestrator-common plugin in your project. You can see in the [OrchestratorClient.ts](./plugins/orchestrator/src/api/OrchestratorClient.ts#L59) how the client can be used.

## audit log
## Audit log

The orchestrator backend has audit logs for all incoming requests.

Expand All @@ -186,8 +196,18 @@ For more information about audit logs in RHDH, please refer to [the official doc

## Extensible workflow execution form

The `orchestrator` plugin includes an extensible form for executing workflows. For detailed guidance see the [Extensible Workflow Execution Form Documentation]().
The `orchestrator` plugin includes an extensible form for executing workflows. Details are available in the [extensible form documentation](./docs/extensibleForm.md).

## Contributors

The orchestrator workspace is structured like a standard backstage application. To get it up and running locally run the following:

```
cd workspaces/orchestrator
yarn install
yarn dev
```

This will trigger a docker container run of devmode SonataFlow as described in [Devmode local configuration](#devmode-local-configuration).

## Maintainers
Follow these [instructions and guidelines](./Maintainers.md).
More development guidelines available in the [contributors documentation](./docs/Contributors.md).
Original file line number Diff line number Diff line change
@@ -1,8 +1,4 @@
# Initial setup



#### How to use the Git Hub identity provider
## How to use the Git Hub identity provider

The `workspaces/orchestrator/app-config.yaml` and the `workspaces/orchestrator/packages/app/src/App.tsx` files contain sufficient configuration to use the Git Hub identity provider.

Expand All @@ -17,41 +13,17 @@ export AUTH_GITHUB_CLIENT_SECRET=...fill
yarn dev
```


#### API Development instruction

Checkout the backstage-plugin
If you need to change the OpenAPI spec, edit the [openapi.yaml](../plugins/orchestrator-common/src/openapi/openapi.yaml) according to your needs.
After you update the spec, run:

`git clone [email protected]:red-hat-developer-hub/backstage-plugins.git`
`yarn --cwd plugins/orchestrator-common openapi:generate`

If you need to change the OpenAPI spec, edit the [openapi.yaml](https://github.com/redhat-developer/rhdh-plugins/blob/main/workspaces/orchestrator/plugins/orchestrator-common/src/openapi/openapi.yaml) according to your needs and then execute from the project root folder:

`yarn --cwd plugins/orchestrator-common openapi`

This command updates the [generated files](https://github.com/redhat-developer/rhdh-plugins/blob/main/workspaces/orchestrator/plugins/orchestrator-common/src/generated) including API, client and docs.
This command updates the [generated files](../plugins/orchestrator-common/src/generated/) including API, client and docs.

> NOTE: Do not manually edit auto-generated files
If you add a new component in the spec, then you need to export the generated typescript object [here](https://github.com/redhat-developer/rhdh-plugins/blob/main/workspaces/orchestrator/plugins/orchestrator-common/src/generated/client/api.ts). For example, if you define

```yaml
components:
schemas:
Person:
type: object
properties:
name:
type: string
surname:
type: string
```
then
```typescript
export type Person = components['schemas']['Person'];
```

When defining a new endpoint, you have to define the `operationId`.
That `id` is the one that you can use to implement the endpoint logic.

Expand Down
2 changes: 1 addition & 1 deletion workspaces/orchestrator/docs/Permissions.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ See https://casbin.org/docs/rbac for more information about casbin rules.

## Enable permissions

To enable permissions, you need to add the following in the [app-config file](../../../app-config.yaml):
To enable permissions, you need to add the following in the [app-config file](../app-config.yaml):

```
permission:
Expand Down
136 changes: 136 additions & 0 deletions workspaces/orchestrator/docs/extensibleForm.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,136 @@
# Extensible Workflow Execution Form

This capability enables developers to extend and customize the `react-jsonschema-form` workflow execution form component. It is designed to enable developers to implement a Backstage plugin that provides a custom decorator for the workflow execution form. This decorator supports overriding a selected set of [react-json-schema-form properties](https://rjsf-team.github.io/react-jsonschema-form/docs/api-reference/form-props) enabling the following features:

- **Custom Validations:** Extend default JSON schema validation with:
- **Synchronous Validation** via the `customValidate` property.
- **Asynchronous Validation** via the `getExtraErrors` property, for validations requiring backend calls.
- **Custom Components:** Replace default form components by overriding the `widgets` property.
- **Interdependent Field Values:** Manage complex inter-field dependencies using the `onChange` and `formData` properties.

The custom decorator is delivered via a factory method that leverages a [Backstage utility API](https://backstage.io/docs/api/utility-apis) provided by the orchestrator. To trigger the desired behavior, the workflow schema should include custom UI properties.

For reference, an example plugin can be found [here](https://github.com/parodos-dev/custom-form-example-plugin).

## API

To implement the API, include @red-hat-developer-hub/backstage-plugin-orchestrator-form-api package as a dependency.
This package provides the `FormExtensionsApi` interface and related types.

```typescript
export type FormDecoratorProps = Pick<
FormProps<JsonObject, JSONSchema7>,
'formData' | 'formContext' | 'widgets' | 'onChange' | 'customValidate'
> & {
getExtraErrors?: (
formData: JsonObject,
) => Promise<ErrorSchema<JsonObject>> | undefined;
};

export type FormDecorator = (
FormComponent: React.ComponentType<FormDecoratorProps>,
) => React.ComponentType;

export interface FormExtensionsApi {
getFormDecorator(schema: JSONSchema7): FormDecorator;
}
```

### Example API Implementation

```typescript
class CustomFormApi implements FormExtensionsApi {
getFormDecorator(schema: JSONSchema7) {
return (FormComponent: React.ComponentType<FormDecoratorProps>>) => {
const widgets = {CountryWidget}; // CountryWidget needs to be implemneted and imported
return () => <FormComponent widgets={widgets} />;
};
}
}
```

### Plugin Creation Example

```typescript
export const formApiFactory = createApiFactory({
api: orchestratorFormApiRef,
deps: {},
factory() {
return new CustomFormApi();
},
});

export const testFactoryPlugin = createPlugin({
id: 'custom-form-plugin',
apis: [formApiFactory],
});
```

### Schema example for above plugin

```typescript
{
"type": "object",
"properties": {
"personalDetails": {
"type": "object",
"title": "Personal Details",
"properties": {
"name": {
"type": "string",
"title": "Name"
},
"country": {
"type": "string",
"title": "Country",
"ui:widget": "CountryWidget"
}
}
},
"contactDetails": {
"type": "object",
"title": "Contact Details",
"properties": {
"email": {
"type": "string",
"title": "Email"
},
"phone": {
"type": "string",
"title": "Phone Number"
}
}
}
}
}
```

### Dynamic plugin configuration example

Add the following to backstage config to integrate the plugin:

```yaml
dynamicPlugins:
frontend:
custom-form-plugin:
apiFactories:
- importName: formApiFactory
```
### Referencing the custom behavior in the schema
The workflow execution schema adheres to the [json-schema](https://json-schema.org/) format, which allows for extending the schema with custom properties beyond the official specification. This flexibility enables the inclusion of additional [uiSchema](https://rjsf-team.github.io/react-jsonschema-form/docs/api-reference/uiSchema/) fields directly within the schema, as demonstrated in the example above.
### How It All Comes Together
The `orchestrator-form-react` plugin implements the form component for workflow execution. It integrates with the custom API provided by the developer's plugin to generate and customize the form. The `orchestrator` plugin then incorporates this form into the workflow execution page.

The `orchestrator-form-react` plugin handles the following key tasks:

- **Generating the UI Schema:** It extracts custom UI schema fields from the main schema, automatically generates the [uiSchema](https://rjsf-team.github.io/react-jsonschema-form/docs/api-reference/uiSchema/), and passes it to the `react-jsonschema-form` component, enabling advanced UI customizations.

- **Organizing Forms into Wizard-Style Steps:** If the schema is an object containing nested objects (i.e., the root is an object, and its properties are also objects), the plugin organizes the form into multiple steps. Each nested object becomes a separate step in a wizard-style interface. For example, the schema provided above results in two steps: _Personal Details_ and _Contact Details_.

The [`orchestrator-form-react`](https://github.com/janus-idp/backstage-plugins/tree/main/plugins/orchestrator-form-react) plugin is designed to operate independently of the main orchestrator plugin. This modularity allows developers to test and validate form behavior in a standalone Backstage development environment before integrating it with the full orchestrator setup.

To use this plugin, add the `@red-hat-developer-hub/backstage-plugin-orchestrator-form-react` package as a dependency in your project.
File renamed without changes

0 comments on commit b8bdbe0

Please sign in to comment.