diff --git a/.gitignore b/.gitignore index 485dee6..1076f2e 100644 --- a/.gitignore +++ b/.gitignore @@ -1 +1,3 @@ .idea + +.DS_Store diff --git a/rfc/new-artifacts-service/README.md b/rfc/new-artifacts-service/README.md new file mode 100644 index 0000000..2444138 --- /dev/null +++ b/rfc/new-artifacts-service/README.md @@ -0,0 +1,230 @@ +# Separate artifacts to new service + +| | | +|-|-| +| **Status** | **Proposed**, Accepted, Implemented, Obsolete | +| **RFC #** | _Update after PR has been made_ | +| **Author(s)** | [Nemesis Osorio](https://github.com/nemesisOsorio), [Sergio Quintero](https://github.com/sergio-quintero) | +| **SIG / WG** | _Applicable SIG(s) or Working Group(s)_ | + +## Overview + +Users with large context pipelines do not have a good experience with pipeline executions.
+Context and outputs in pipeline executions can contain large artifacts/objects such as k8s manifest (yaml/json files).
+Spinnaker passes these artifacts/objects in request and response bodies across some services.
+The idea is to create a new service to store these artifacts/objects +and only pass the `id`s of those artifacts/objects in the request/response bodies and context/outputs. + +Deck users do not click to see the produced artifact/object in every single pipeline execution. +With this approach, artifacts will be loaded on demand (lazy loading). + +### Goals and Non-Goals + +#### Goals + +* Improve Deck performance with large pipeline executions. +* Optimize message size over the wire. + +#### Non-Goals + +* Move [clouddriver-artifacts](https://github.com/spinnaker/clouddriver/blob/master/clouddriver-artifacts) +to the new service +* Refactor Angular code in Deck. +* Add new permissions to Fiat service. +* Avoid the duplication of artifacts when executing a pipeline multiple times without changes. + +## Motivation and Rationale + +* Deck is very slow and sometimes freezes when you are producing/consuming large artifacts/objects in pipeline executions. +* Smaller payloads are sent in request/response. +* Save serializing and deserializing computations. + +Early customers: +* Armory on behalf of its customers. + +## Timeline + +The proposed plan is to implement this by pipeline stages: + +* Milestone 1: (Targeting 1.32) Address stages with the most acute performance issues + * Introduce new Artifact service + * Bake (Manifest), both Helm and Kustomize providers + * Deploy (Manifest) +* Milestone 2: Analyze additional stages that exhibit similar performance patterns +* Milestone 3: (Targeting 1.33) Integrate additional stages with artifact service +* Milestone 4: Support to deploy artifacts service in Operator + +## Design + +Creation of a new microservice for artifacts/objects.
+We are proposing the following API: [Artifacts API](./artifacts-api.yml), you can render it using [Swagger Editor](https://editor-next.swagger.io/)
+Support of the following artifact types: +* base64 +* json (everything in the execution details has json format) + +```mermaid +--- +title: ER Diagram +--- +erDiagram + artifacts { + uuid id PK "artifact id" + varchar name "artifact name" + varchar type "base64, json" + json content "the large artifact" + timestamp created_at "creation date time" + } +``` + +Make changes in Deck components that are using the context/output to render artifacts/objects, +and only load those artifacts/objects if you click the button/link(lazy loading). + +Enable Fiat autoconfiguration and use application permissions for new artifacts endpoints. + +### Retention Policy + +Add a scheduler to automatically delete artifacts. The following options will be configurable: +* Retention time: How long the record will be kept in the database. +* Artifact cleanup job: Configurable cron job that deletes artifacts periodically in accordance to the retention time property. + +### Dependencies + +* A new database. + * Support for MySQL and PostgreSQL. + * Database changelogs managed by liquibase. +* Services involved + * Orca + * Clouddriver + * Rosco + * Deck + * Gate + +All changes will be implemented under a feature flag. + +```mermaid +--- +title: Artifacts Interactions +--- +flowchart TB + subgraph Artifacts + direction TB + Rest-Service --> DB + Scheduler --> DB + DB[(DB)] + end + Orca --> Artifacts + Clouddriver --> Artifacts + Rosco --> Artifacts + Gate --> Artifacts + Deck --> Gate + Artifacts --> Fiat +``` + +### Bake (Manifest) example + +![Bake Pipeline](./bake-pipeline.png "Bake Manifest Pipeline") + +```mermaid +sequenceDiagram +title Bake Manifest + participant Deck + participant Gate + participant Orca + participant Rosco + participant Artifacts + Deck->>+Gate: startPipelineExecution + Gate->>+Orca: startPipelineExecution + Orca-->>-Gate: #nbsp; + Gate-->>-Deck: #nbsp; + Orca->>+Rosco: bakeManifest + Rosco->>+Artifacts: createArtifact + Artifacts-->>-Rosco: artifactDetails + Rosco-->>-Orca: artifactDetails +``` + +```mermaid +sequenceDiagram +title Get Baked Manifest YAML + participant Deck + participant Gate + participant Artifacts + Deck->>+Gate: Baked Manifest YAML + Gate->>+Artifacts: getArtifactById + Artifacts-->>-Gate: artifact + Gate-->>-Deck: artifact +``` + +Diagrams: you can render them using [Mermaid Live Editor](https://mermaid.live/) +or the cli `mmdc -i README.md` + +## Drawbacks + +* This introduces a new service that adds complexity. +* Implementing it under a feature flag means we need to ensure it does not break existing users. + +## Alternatives + +An example of how it could be implemented in Orca without introducing a new service: + +1. Orca will request a bake operation to Rosco. + * Rosco performs a bake and returns the artifact in base64. + * Orca will receive the response and store the artifact. + * Orca will **only** add the artifactId to the pipeline context/outputs. +2. When deploying a Manifest + * Orca will fetch the artifact by id and add the artifact to the request. + * Clouddriver will continue to receive the manifest in the request body. + +Notes: +* Only Orca would change the behavior, other services remain the same. +* Add the new table to Orca's database schema. + +## Known Unknowns + +* Not all the stages/services that can produce/consume those artifacts/objects are known. +* How will artifacts be passed in automated triggers?
+ There seems to be no special treatment for trigger artifacts, this feature should be fine. +* How to migrate/support existing executions? + +## Security, Privacy, and Compliance + +* Enable Fiat autoconfiguration and use application permissions + +## Operations + +* The new service will require a new repository and integration to the Spinnaker CI/CD process. +* The database maintenance plan. + +## Risks + +A scenario when a user enables the feature flag to use the new artifacts service, +runs a pipeline with artifacts and later disables the feature, leaving orphaned artifacts. + +## Future Possibilities + +* Allow saving multiple artifacts at once +* Add other implementations for the persistence layer + * s3 +* Move [clouddriver-artifacts](https://github.com/spinnaker/clouddriver/blob/master/clouddriver-artifacts) +to the new artifacts service +* [Implement ACLs to artifacts](https://docs.spring.io/spring-security/reference/servlet/authorization/acls.html) + * Add permissions to artifact accounts in configuration, example: + + ```yaml + artifacts: + gitrepo: + enabled: true + accounts: + - name: git-repo-account + username: my-git-user + token: encrypted:k8s!n:my-secret!k:my-token + permissions: + READ: + - foo + - bar + WRITE: + - baz + ``` + + * Include the account in artifacts creation + * for "Embedded" artifacts(dynamically generated in Spinnaker, for example using SPeL) + we can use Application permissions diff --git a/rfc/new-artifacts-service/artifacts-api.yml b/rfc/new-artifacts-service/artifacts-api.yml new file mode 100644 index 0000000..c27abc9 --- /dev/null +++ b/rfc/new-artifacts-service/artifacts-api.yml @@ -0,0 +1,113 @@ +openapi: 3.0.3 +info: + title: Artifacts + description: |- + Artifacts or objects that could be large in pipeline executions + version: 0.0.0 +servers: + - url: https://items.spinnaker.io/api/v0 +tags: + - name: artifacts + description: Everything about your Artifacts +paths: + /artifacts: + post: + tags: + - artifacts + summary: Add a new artifact + description: Add a new artifact + operationId: addArtifact + requestBody: + description: Create a new Artifact + content: + application/json: + schema: + $ref: '#/components/schemas/Artifact' + required: true + responses: + '201': + description: Artifact created + content: + application/json: + schema: + $ref: '#/components/schemas/Artifact' + /artifacts/{artifactId}: + get: + tags: + - artifacts + summary: Find Artifact by id + description: Returns a single artifact + operationId: getArtifactById + parameters: + - name: artifactId + in: path + description: Id of artifact to return + required: true + schema: + type: integer + format: int64 + responses: + '200': + description: successful operation + content: + application/json: + schema: + $ref: '#/components/schemas/Artifact' + '404': + description: Artifact not found + delete: + tags: + - artifacts + summary: Deletes an Artifact + description: delete an artifact + operationId: deleteArtifact + parameters: + - name: artifactId + in: path + description: Artifact id to delete + required: true + schema: + type: integer + format: int64 + responses: + '204': + description: successful operation +components: + schemas: + Artifact: + required: + - name + - type + type: object + properties: + id: + type: string + format: uuid + name: + type: string + example: kustomize + content: + $ref: '#/components/schemas/Content' + type: + type: string + description: artifact type + enum: + - base64 + - json + createdAt: + type: string + format: date-time + Content: + description: content of the artifact + anyOf: + - type: string + - type: array + items: {} + - type: object + requestBodies: + Artifact: + description: Artifact + content: + application/json: + schema: + $ref: '#/components/schemas/Artifact' diff --git a/rfc/new-artifacts-service/bake-pipeline.png b/rfc/new-artifacts-service/bake-pipeline.png new file mode 100644 index 0000000..0a6b93d Binary files /dev/null and b/rfc/new-artifacts-service/bake-pipeline.png differ