diff --git a/.envrc b/.envrc index 636cfca..2e4fa14 100644 --- a/.envrc +++ b/.envrc @@ -1,4 +1,4 @@ -source ../.tmpenv # 2>/dev/null || true +source .tmpenv # 2>/dev/null || true ################################################################################## # bash bin export PATH=$PATH:.github/bin diff --git a/Pulumi.yaml b/Pulumi.yaml index 113e559..5e404a0 100644 --- a/Pulumi.yaml +++ b/Pulumi.yaml @@ -6,5 +6,5 @@ options: refresh: always runtime: name: python - options: - virtualenv: venv +# options: +# virtualenv: venv diff --git a/README.md b/README.md index 55fea9a..0d4d0e6 100644 --- a/README.md +++ b/README.md @@ -36,12 +36,12 @@ This repository is designed as a template, allowing you to quickly bootstrap new 1. **Clone the Repository to Your Local Machine:** - Once your new repository is created, clone it to your local machine using Git. +Once your new repository is created, clone it to your local machine using Git. - ```bash - git clone https://github.com/YourUsername/YourNewRepoName.git - cd YourNewRepoName - ``` +```bash +git clone https://github.com/YourUsername/YourNewRepoName.git +cd YourNewRepoName +``` 2. **Initialize the Development Environment:** @@ -66,12 +66,12 @@ This repository is designed as a template, allowing you to quickly bootstrap new 1. **Deploy the Infrastructure:** - Use the pre-configured tasks to deploy your infrastructure, as detailed in the Quickstart section. +Use the pre-configured tasks to deploy your infrastructure, as detailed in the Quickstart section. - ```bash - task kubernetes - task deploy - ``` +```bash +task kubernetes +task deploy +``` 2. **Build and Iterate:** @@ -81,13 +81,13 @@ This repository is designed as a template, allowing you to quickly bootstrap new 1. **Commit and Push:** - After making changes, commit them to your repository. +After making changes, commit them to your repository. - ```bash - git add . - git commit -m "Initial setup and configuration" - git push origin main - ``` +```bash +git add . +git commit -m "Initial setup and configuration" +git push origin main +``` 2. **Collaborate and Contribute:** @@ -130,49 +130,49 @@ Follow these steps to get your environment up and running: 3. **Open the Integrated Terminal:** - Use `` Ctrl + ` `` to open the VSCode integrated terminal. + Use `Ctrl + `` to open the VSCode integrated terminal. 4. **Authenticate Credentials:** - Login to Pulumi Cloud and other required services. +Login to Pulumi Cloud and other required services. - ```bash {"id":"01J5VC1KTJBR22WEDNSSGTNAX4","name":"login"} - task login - ``` +```bash {"name":"login"} +task login +``` 5. **Configure the Pulumi Stack:** - Set up Pulumi stack parameters. +Set up Pulumi stack parameters. - ```bash {"id":"01J5VC1KTJBR22WEDNSWYBKNQS","name":"configure"} - export ORGANIZATION="${GITHUB_USER:-${GITHUB_REPOSITORY_OWNER:-}}" - export DEPLOYMENT="${RepositoryName:-}" - task configure - ``` +```bash {"name":"configure"} +export ORGANIZATION="${GITHUB_USER:-${GITHUB_REPOSITORY_OWNER:-}}" +export DEPLOYMENT="${RepositoryName:-}" +task configure +``` 6. **Deploy Kubernetes:** - Deploy Kubernetes using Talos. +Deploy Kubernetes using Talos. - ```bash {"excludeFromRunAll":"true","id":"01J5VC1KTJBR22WEDNSX4RHEG2","name":"kubernetes"} - task kubernetes - ``` +```bash {"excludeFromRunAll":"true","name":"kubernetes"} +task kubernetes +``` 7. **Deploy the Platform:** - Deploy the KubeVirt PaaS infrastructure. +Deploy the KubeVirt PaaS infrastructure. - ```bash {"excludeFromRunAll":"true","id":"01J5VC1KTJBR22WEDNSZW7QADA","name":"deploy"} - task deploy - ``` +```bash {"excludeFromRunAll":"true","name":"deploy"} +task deploy +``` -10. **Cleanup:** +8. **Cleanup:** - Clean up all Kubernetes and Pulumi resources when you're done. +Clean up all Kubernetes and Pulumi resources when you're done. - ```bash {"excludeFromRunAll":"true","id":"01J5VC1KTJBR22WEDNT7BDRMAV","name":"clean"} - task clean-all - ``` +```bash {"excludeFromRunAll":"true","name":"clean"} +task clean-all +``` ## Contributing @@ -182,7 +182,7 @@ Contributions are welcome! This template is intended to evolve with the needs of Use the `act` tool to test GitHub Actions locally before pushing your changes. -```bash {"excludeFromRunAll":"true","id":"01J5VC1KTJBR22WEDNT92WYZEH"} +```bash {"excludeFromRunAll":"true"} task act ``` diff --git a/ROADMAP.md b/ROADMAP.md index 5d5583f..89cc8bf 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -13,6 +13,7 @@ - [Logging, Monitoring, and Alerting](#logging-monitoring-and-alerting) - [Networking](#networking) - [Cost Management](#cost-management) + 5. [Design Principles](#design-principles) 6. [Implementation Roadmap](#implementation-roadmap) - [Phase 1: Foundations](#phase-1-foundations) @@ -21,12 +22,13 @@ - [Phase 4: Application Onboarding](#phase-4-application-onboarding) - [Phase 5: Multi-Cloud Expansion](#phase-5-multi-cloud-expansion) - [Phase 6: Optimization and Scaling](#phase-6-optimization-and-scaling) + 7. [Roles and Responsibilities](#roles-and-responsibilities) 8. [Risks and Mitigation Strategies](#risks-and-mitigation-strategies) 9. [Conclusion](#conclusion) 10. [Appendices](#appendices) - - [A. Glossary](#a-glossary) - - [B. References](#b-references) + - [A. Glossary](#a-glossary) + - [B. References](#b-references) --- @@ -52,7 +54,6 @@ Centralized governance is achieved through policy propagation and centralized se - **Centralized Governance**: Maintain centralized policies, secrets, and configurations for consistent management across all environments. - **Scalability and Modularity**: Design for horizontal scalability and modularity to accommodate growth and technological changes. - ## Key Components ### Account Structure @@ -60,16 +61,18 @@ Centralized governance is achieved through policy propagation and centralized se #### Organizational Hierarchy - **Root Organization**: The top-level entity for each cloud provider. - - **Security OU**: - - **Log Archive Account**: Central repository for logs. - - **Security Tools Account**: Hosts security tools and services. - - **Infrastructure OU**: - - **Networking Account**: Manages shared networking resources. - - **Shared Services Account**: Houses services shared across the organization. - - **Applications OU**: - - **Development Accounts**: Environments for development teams. - - **Testing Accounts**: Isolated testing environments. - - **Production Accounts**: Live environments for production workloads. + - **Security OU**: + - **Log Archive Account**: Central repository for logs. + - **Security Tools Account**: Hosts security tools and services. + + - **Infrastructure OU**: + - **Networking Account**: Manages shared networking resources. + - **Shared Services Account**: Houses services shared across the organization. + + - **Applications OU**: + - **Development Accounts**: Environments for development teams. + - **Testing Accounts**: Isolated testing environments. + - **Production Accounts**: Live environments for production workloads. #### Account Provisioning @@ -80,63 +83,73 @@ Centralized governance is achieved through policy propagation and centralized se - **Centralized IAM**: Implement a unified IAM strategy across all cloud providers. - **Roles and Policies**: - - Define IAM roles with the principle of least privilege. - - Manage IAM policies and role assignments programmatically. + - Define IAM roles with the principle of least privilege. + - Manage IAM policies and role assignments programmatically. + - **User and Group Management**: - - Integrate with centralized identity providers (e.g., Azure AD, Okta). - - Group users by function and assign appropriate permissions. + - Integrate with centralized identity providers (e.g., Azure AD, Okta). + - Group users by function and assign appropriate permissions. ### Infrastructure as Code (IaC) - **Tooling**: Utilize a programming language (e.g., Python) with an IaC framework that supports multi-cloud provisioning. - **Repository Structure**: - - **Modular Design**: Create reusable modules for common infrastructure components. - - **Environment Separation**: Maintain separate configurations for development, testing, and production environments. + - **Modular Design**: Create reusable modules for common infrastructure components. + - **Environment Separation**: Maintain separate configurations for development, testing, and production environments. + - **CI/CD Integration**: - - Automate deployment pipelines with tools like Jenkins, GitHub Actions, or GitLab CI. - - Implement GitOps practices to ensure that Git is the single source of truth. + - Automate deployment pipelines with tools like Jenkins, GitHub Actions, or GitLab CI. + - Implement GitOps practices to ensure that Git is the single source of truth. ### Compliance and Governance - **Policy as Code**: - - Define compliance controls within the IaC configurations. - - Embed policies for standards like FISMA and NIST directly into code. + - Define compliance controls within the IaC configurations. + - Embed policies for standards like FISMA and NIST directly into code. + - **Automated Enforcement**: - - Use tagging and labeling to propagate compliance metadata to all resources. - - Implement automated checks during deployment to enforce compliance. + - Use tagging and labeling to propagate compliance metadata to all resources. + - Implement automated checks during deployment to enforce compliance. + - **Auditability**: - - Maintain detailed logs of infrastructure changes. - - Utilize version control history for audit trails. + - Maintain detailed logs of infrastructure changes. + - Utilize version control history for audit trails. ### Logging, Monitoring, and Alerting - **Centralized Logging**: - - Aggregate logs from all resources into centralized logging services. - - Ensure logs are stored securely and comply with data retention policies. + - Aggregate logs from all resources into centralized logging services. + - Ensure logs are stored securely and comply with data retention policies. + - **Monitoring Tools**: - - Deploy monitoring solutions (e.g., Prometheus, Grafana) to collect metrics. + - Deploy monitoring solutions (e.g., Prometheus, Grafana) to collect metrics. + - **Alerting Mechanisms**: - - Configure alerts for performance issues, security incidents, and compliance violations. - - Integrate with incident management systems for timely response. + - Configure alerts for performance issues, security incidents, and compliance violations. + - Integrate with incident management systems for timely response. ### Networking - **Standardized Network Topology**: - - Define network architectures using IaC for consistency. - - Include components like virtual networks, subnets, and routing configurations. + - Define network architectures using IaC for consistency. + - Include components like virtual networks, subnets, and routing configurations. + - **Security Controls**: - - Manage security groups, network access control lists (ACLs), and firewall rules programmatically. + - Manage security groups, network access control lists (ACLs), and firewall rules programmatically. + - **Cross-Cloud Connectivity**: - - Implement VPNs or cloud interconnects for secure communication between different cloud environments. + - Implement VPNs or cloud interconnects for secure communication between different cloud environments. ### Cost Management - **Cost Monitoring**: - - Implement tools to aggregate and analyze cost data across all cloud providers. + - Implement tools to aggregate and analyze cost data across all cloud providers. + - **Tagging for Cost Allocation**: - - Enforce tagging standards to facilitate cost tracking by project, environment, and department. + - Enforce tagging standards to facilitate cost tracking by project, environment, and department. + - **Budgeting and Alerts**: - - Set up cost thresholds and receive alerts to prevent budget overruns. + - Set up cost thresholds and receive alerts to prevent budget overruns. --- diff --git a/docs/README.md b/docs/README.md index 0182964..e44e9f2 100644 --- a/docs/README.md +++ b/docs/README.md @@ -26,7 +26,7 @@ Follow the steps below to set up your environment: Authenticate with your Pulumi account: -```bash {"id":"01J97M1349ZY70MQVHDE43DNY5","name":"login","tag":"setup"} +```bash {"name":"login","tag":"setup"} pulumi login && pulumi install ``` @@ -37,9 +37,9 @@ Use Pulumi to load environment variables, configuration files, and credentials: * NOTE: Replace ``, ``, and `` with your Pulumi organization, project name, and stack name. -```bash {"id":"01J97M1349ZY70MQVHDGAFVNEB","name":"load-environments-and-secrets","tag":"setup"} +```bash {"name":"load-environments-and-secrets","tag":"setup"} export ENVIRONMENT="containercraft/NavtecaAwsCredentialsConfigSmce/navteca-aws-credentials-config-smce" -eval $(pulumi env open --format=shell $ENVIRONMENT | tee ../.tmpenv; direnv allow) +eval $(pulumi env open --format=shell $ENVIRONMENT | tee .tmpenv; direnv allow) echo "Loaded environment $ENVIRONMENT" alias aws='aws --profile smdc-cba' @@ -50,7 +50,7 @@ alias aws='aws --profile smdc-cba' Get Caller Identity to verify your AWS identity: -```bash {"excludeFromRunAll":"true","id":"01J97M1349ZY70MQVHDFZQZQZT","name":"validate-aws-identity","tag":"validate-aws"} +```bash {"excludeFromRunAll":"true","name":"validate-aws-identity","tag":"validate-aws"} aws --profile smdc-cba sts get-caller-identity ``` @@ -59,7 +59,7 @@ aws --profile smdc-cba sts get-caller-identity Deploy the infrastructure as code (IaC) using Pulumi: -```bash {"id":"01J97M1349ZY70MQVHDFZQZQZT","name":"deploy-iac","tag":"setup"} +```bash {"name":"deploy-iac","tag":"setup"} git remote add origin https://github.com/containercraft/konductor || true git config remote.origin.url https://github.com/containercraft/konductor || true pulumi up --yes --stack containercraft/scip-ops-prod --skip-preview=true --refresh=true @@ -70,7 +70,7 @@ pulumi up --yes --stack containercraft/scip-ops-prod --skip-preview=true --refre Clone the SMCE CLI repository && Symlink `smce` cli: -```bash {"id":"01J97M1349ZY70MQVHDMSP1MHQ","name":"install-smce-cli","tag":"setup"} +```bash {"name":"install-smce-cli","tag":"setup"} cd ~ rm -rf ~/smce-cli ~/.local/bin/smce ln -sf $GIT_CONFIG ~/.gitconfig @@ -92,7 +92,7 @@ Set up AWS Multi-Factor Authentication (MFA): > TODO: enhance smce-cli to auto-export mfa env vars -```bash {"excludeFromRunAll":"true","id":"01J97M1349ZY70MQVHDT7NEEM1","name":"smce-aws-mfa","tag":"aws"} +```bash {"excludeFromRunAll":"true","name":"smce-aws-mfa","tag":"aws"} smce awsconfig mfa ``` @@ -101,7 +101,7 @@ smce awsconfig mfa List S3 buckets to confirm AWS access && Verify your AWS identity: -```bash {"excludeFromRunAll":"true","id":"01J97M1349ZY70MQVHE452WAP0","name":"validate-aws-s3-ls","tag":"validate-aws"} +```bash {"excludeFromRunAll":"true","name":"validate-aws-s3-ls","tag":"validate-aws"} aws s3 ls aws sts get-caller-identity @@ -111,21 +111,21 @@ aws sts get-caller-identity Update your kubeconfig file to interact with your EKS cluster: -```bash {"excludeFromRunAll":"true","id":"01J97M1349ZY70MQVHE78ZE70R","name":"aws-get-ops-kubeconfig","tag":"kubeconfig"} +```bash {"excludeFromRunAll":"true","name":"aws-get-ops-kubeconfig","tag":"kubeconfig"} aws eks update-kubeconfig --profile main --region us-east-1 --name smce-gitops ``` Generate a new Kubernetes configuration: -```bash {"excludeFromRunAll":"true","id":"01J97M1349ZY70MQVHDZVHZ3TQ","name":"generate-smce-kubeconfig","tag":"kubeconfig"} +```bash {"excludeFromRunAll":"true","name":"generate-smce-kubeconfig","tag":"kubeconfig"} smce kubeconfig generate ``` Generate an authentication token for the EKS cluster: -```bash {"excludeFromRunAll":"true","id":"01J97M1349ZY70MQVHE9A2GZGB","name":"generate-eks-auth-token","tag":"kubeconfig"} +```bash {"excludeFromRunAll":"true","name":"generate-eks-auth-token","tag":"kubeconfig"} aws eks get-token --region us-east-1 --cluster-name smce-gitops --output json ``` @@ -136,21 +136,21 @@ Replace `` with your MFA device's ARN and provide your MFA token List available Kubernetes contexts: -```bash {"excludeFromRunAll":"true","id":"01J97M1349ZY70MQVHEE3CJ1YZ","name":"validate-kubeconfig-context-list","tag":"kubeconfig"} +```bash {"excludeFromRunAll":"true","name":"validate-kubeconfig-context-list","tag":"kubeconfig"} kubectl --kubeconfig ~/.kube/smce config get-contexts ``` Retrieve the list of nodes in your Kubernetes cluster: -```bash {"excludeFromRunAll":"true","id":"01J97M1349ZY70MQVHEGY0QEQW","name":"validate-kube-get-nodes","tag":"kubeconfig"} +```bash {"excludeFromRunAll":"true","name":"validate-kube-get-nodes","tag":"kubeconfig"} kubectl --kubeconfig ~/.kube/smce get nodes ``` Check the Kubernetes client and server versions with verbose output: -```bash {"excludeFromRunAll":"true","id":"01J97M1349ZY70MQVHEHTZNV1Y","name":"validate-kube-get-version","tag":"kubeconfig"} +```bash {"excludeFromRunAll":"true","name":"validate-kube-get-version","tag":"kubeconfig"} kubectl version -v=8 ``` @@ -170,7 +170,7 @@ By following these steps, you've set up your environment to interact with AWS se **Note:** If you encounter authentication issues due to MFA requirements, test temporary session credentials using the following command: -```bash {"excludeFromRunAll":"true","id":"01J9CGCF9R0EWGHNN32BMZCGZY","name":"aws-sts-get-session-token","tag":"dbg"} +```bash {"excludeFromRunAll":"true","name":"aws-sts-get-session-token","tag":"dbg"} aws sts get-session-token \ --duration-seconds 129600 \ --profile default \ @@ -181,7 +181,7 @@ aws sts get-session-token \ ## Bonus: Launch Kubernetes in Docker -```bash {"excludeFromRunAll":"true","id":"01J9CGCF9R0EWGHNN32BMZCGZY","name":"task-run-kubernetes","tag":"tind"} +```bash {"excludeFromRunAll":"true","name":"task-run-kubernetes","tag":"tind"} cd .. task kubernetes diff --git a/pulumi/CALL_TO_ACTION.md b/pulumi/CALL_TO_ACTION.md index 89fa502..93dc579 100644 --- a/pulumi/CALL_TO_ACTION.md +++ b/pulumi/CALL_TO_ACTION.md @@ -1,50 +1,62 @@ ### Refactoring Enhancements to Consider **Modular Design**: - - Core functionalities are segregated into distinct files/modules, such as `config.py`, `deployment.py`, `resource_helpers.py`, etc. - - Each module follows a clear pattern with separate `types.py` and `deploy.py` files. + +- Core functionalities are segregated into distinct files/modules, such as `config.py`, `deployment.py`, `resource_helpers.py`, etc. +- Each module follows a clear pattern with separate `types.py` and `deploy.py` files. **Configuration Management**: - - Centralize configuration management using `config.py` to handle global settings. - - Use data classes for module configurations to ensure type safety and defaults. + +- Centralize configuration management using `config.py` to handle global settings. +- Use data classes for module configurations to ensure type safety and defaults. **Global Metadata Handling**: - - Implementation of a singleton pattern for managing global metadata (labels and annotations). - - Functions to generate and apply global metadata. + +- Implementation of a singleton pattern for managing global metadata (labels and annotations). +- Functions to generate and apply global metadata. **Consistency and Readability**: - - The existing TODO comments highlight areas needing reorganization and refactoring. - - Some modules including `kubevirt`, `cert_manager`, `hostpath_provisioner` and others deploy differently in terms of resource creation and dependency management, look for ways to improve consistency. + +- The existing TODO comments highlight areas needing reorganization and refactoring. +- Some modules including `kubevirt`, `cert_manager`, `hostpath_provisioner` and others deploy differently in terms of resource creation and dependency management, look for ways to improve consistency. **Centralized Configuration Loading**: - - Configuration loading and merging logic vary across modules. - - There is redundancy in fetching the latest versions for modules (e.g., `kubevirt`, `cert_manager`). Look for ways to reduce version fetching redundancy. + +- Configuration loading and merging logic vary across modules. +- There is redundancy in fetching the latest versions for modules (e.g., `kubevirt`, `cert_manager`). Look for ways to reduce version fetching redundancy. **Exception Handling**: - - Exception handling is partially implemented in some places, consistent and detailed error handling across all modules will improve reliability. + +- Exception handling is partially implemented in some places, consistent and detailed error handling across all modules will improve reliability. **Resource Helper Centralization**: - - Several helper functions like `create_namespace`, `create_custom_resource`, etc., provide common functionality but could be standardized further. - - Handling dependencies and resource transformations could be more DRY (Don't Repeat Yourself). + +- Several helper functions like `create_namespace`, `create_custom_resource`, etc., provide common functionality but could be standardized further. +- Handling dependencies and resource transformations could be more DRY (Don't Repeat Yourself). **Standardize Configuration Management**: - - Refactor configuration management to ensure consistency across all modules. - - Implement a common pattern for fetching the latest versions and configuration merging. -**Refactor `initialize_pulumi` Function**: - - Use data classes or named tuples instead of dictionaries for initialization components. - - Centralize and streamline initialization logic to reduce redundancy. +- Refactor configuration management to ensure consistency across all modules. +- Implement a common pattern for fetching the latest versions and configuration merging. + +__Refactor `initialize_pulumi` Function__: + +- Use data classes or named tuples instead of dictionaries for initialization components. +- Centralize and streamline initialization logic to reduce redundancy. **Enhance Error Handling and Logging**: - - Implement structured logging and consistent error handling across all the modules. - - Ensure all relevant operations are logged, and errors are informative. + +- Implement structured logging and consistent error handling across all the modules. +- Ensure all relevant operations are logged, and errors are informative. **Simplify Function Signatures and Improve Type Safety**: - - Refactor function signatures to use data classes and named tuples. This will improve readability and maintainability. + +- Refactor function signatures to use data classes and named tuples. This will improve readability and maintainability. **Centralize Shared Logic**: - - Standardize and centralize shared logic like version fetching, resource transformation, and compliance metadata generation. - - Use utility functions from `utils.py` and refactor repetitive logic across `deploy.py` files. + +- Standardize and centralize shared logic like version fetching, resource transformation, and compliance metadata generation. +- Use utility functions from `utils.py` and refactor repetitive logic across `deploy.py` files. ### Implementation Examples diff --git a/pulumi/COMPLIANCE.md b/pulumi/COMPLIANCE.md index cd354e7..f71dc63 100644 --- a/pulumi/COMPLIANCE.md +++ b/pulumi/COMPLIANCE.md @@ -9,6 +9,7 @@ 3. [Compliance Strategy](#compliance-strategy) - [Objectives](#objectives) - [Key Innovations](#key-innovations) + 4. [Implementation Details](#implementation-details) - [Compliance Metadata Management](#compliance-metadata-management) - [Automatic Propagation Mechanism](#automatic-propagation-mechanism) @@ -16,18 +17,22 @@ - [Configuration Schema](#configuration-schema) - [Version Management](#version-management) - [Stack Outputs and Reporting](#stack-outputs-and-reporting) + 5. [Developer Expectations and Best Practices](#developer-expectations-and-best-practices) - [Module Autonomy](#module-autonomy) - [Integration with Compliance Framework](#integration-with-compliance-framework) - [Coding Standards](#coding-standards) + 6. [User Experience (UX)](#user-experience-ux) - [Simplified Configuration](#simplified-configuration) - [Deployment Workflow](#deployment-workflow) - [Validation and Error Handling](#validation-and-error-handling) + 7. [Business Stakeholder Value](#business-stakeholder-value) - [Accelerated Time-to-Compliance](#accelerated-time-to-compliance) - [Auditability and Transparency](#auditability-and-transparency) - [Risk Reduction](#risk-reduction) + 8. [Conclusion](#conclusion) 9. [Appendix](#appendix) - [Glossary](#glossary) @@ -82,11 +87,11 @@ In the modern regulatory landscape, organizations face increasing pressure to co - **Configuration Dictionaries**: Compliance-related metadata is defined in a central configuration file (e.g., `Pulumi..yaml`) under a dedicated `compliance` section. - **Metadata Types**: - - **Regulatory Controls**: NIST, FISMA, ISO control identifiers. - - **Component Versions**: Versions of deployed components and dependencies. - - **Source Control Information**: Git repository URLs, commit hashes, branches. - - **Identity Information**: Cloud provider identities (e.g., AWS STS `GetCallerIdentity`, Kubernetes User and Service Account, etc. outputs). - - **Organizational Metadata**: Owner information, environment tags, project identifiers. + - **Regulatory Controls**: NIST, FISMA, ISO control identifiers. + - **Component Versions**: Versions of deployed components and dependencies. + - **Source Control Information**: Git repository URLs, commit hashes, branches. + - **Identity Information**: Cloud provider identities (e.g., AWS STS `GetCallerIdentity`, Kubernetes User and Service Account, etc. outputs). + - **Organizational Metadata**: Owner information, environment tags, project identifiers. **Example Configuration**: @@ -124,11 +129,12 @@ config: **Technical Implementation**: - **Core Compliance Module** (`core/compliance.py`): - - Provides functions to access compliance metadata. - - Supplies utility functions to format and apply metadata to resources. + - Provides functions to access compliance metadata. + - Supplies utility functions to format and apply metadata to resources. + - **Module Integration**: - - Modules import the core compliance utilities. - - Apply compliance metadata during resource instantiation. + - Modules import the core compliance utilities. + - Apply compliance metadata during resource instantiation. **Example in Module Deployment**: @@ -169,13 +175,14 @@ def deploy_aws_module( **Example Tags and Labels**: - **Tags**: - - `compliance:nist-controls=AC-2,IA-5` - - `compliance:owner=compliance-team@example.com` - - `compliance:project-id=proj-12345` + - `compliance:nist-controls=AC-2,IA-5` + - `compliance:owner=compliance-team@example.com` + - `compliance:project-id=proj-12345` + - **Labels**: - - `compliance/nist-controls: "AC-2,IA-5"` - - `compliance/owner: "compliance-team@example.com"` - - `compliance/project-id: "proj-12345"` + - `compliance/nist-controls: "AC-2,IA-5"` + - `compliance/owner: "compliance-team@example.com"` + - `compliance/project-id: "proj-12345"` ### Configuration Schema @@ -378,7 +385,7 @@ The Konductor IaC codebase embodies a strategic approach to compliance managemen - **ISO/IEC 27001 Information Security Management**: [https://www.iso.org/isoiec-27001-information-security.html](https://www.iso.org/isoiec-27001-information-security.html) - **Pulumi Documentation**: [https://www.pulumi.com/docs/](https://www.pulumi.com/docs/) - **Pydantic Documentation**: [https://pydantic-docs.helpmanual.io/](https://pydantic-docs.helpmanual.io/) -- **AWS STS GetCallerIdentity**: [https://docs.aws.amazon.com/STS/latest/APIReference/API_GetCallerIdentity.html](https://docs.aws.amazon.com/STS/latest/APIReference/API_GetCallerIdentity.html) +- __AWS STS GetCallerIdentity__: [https://docs.aws.amazon.com/STS/latest/APIReference/API_GetCallerIdentity.html](https://docs.aws.amazon.com/STS/latest/APIReference/API_GetCallerIdentity.html) --- diff --git a/pulumi/README.md b/pulumi/README.md index 92998ef..4fbef03 100644 --- a/pulumi/README.md +++ b/pulumi/README.md @@ -10,16 +10,20 @@ Welcome to the **Konductor DevOps Template Infrastructure as Code (IaC) new proj - [Introduction](#introduction) - [Developer & Architecture Ethos](#developer--architecture-ethos) - - [Prime Directive](#prime-directive) - - [Developer Directives](#developer-directives) + - [Prime Directive](#prime-directive) + - [Developer Directives](#developer-directives) + - [Getting Started](#getting-started) - - [Prerequisites](#prerequisites) - - [Setting Up Your Environment](#setting-up-your-environment) + - [Prerequisites](#prerequisites) + - [Setting Up Your Environment](#setting-up-your-environment) + - [Developer Imperatives](#developer-imperatives) - - [Detailed Breakdown](#detailed-breakdown) + - [Detailed Breakdown](#detailed-breakdown) + - [Developing New Modules](#developing-new-modules) - - [Directory Structure](#directory-structure) - - [Creating a New Module](#creating-a-new-module) + - [Directory Structure](#directory-structure) + - [Creating a New Module](#creating-a-new-module) + - [Common Utilities](#common-utilities) - [Version Control](#version-control) - [Contributing to the Project](#contributing-to-the-project) @@ -47,9 +51,7 @@ Quality is not just about the product or code; it's about creating an enjoyable ### Developer Directives 1. **Improve Code Maintainability**: Write code that is structured, organized, and easy to understand. Prioritize readability, reusability, and extensibility. - 2. **Optimize Performance**: Ensure that the code performs efficiently and respects configurations. Avoid executing inactive or unnecessary code. - 3. **Establish Standard Practices**: Develop consistent approaches to configuration handling, module deployment, and code organization to guide future development. --- @@ -72,41 +74,41 @@ Follow these steps to set up your environment: 1. **Clone the Repository** - ```bash - git clone https://github.com/ContainerCraft/Kargo.git - cd Kargo/pulumi - ``` +```bash +git clone https://github.com/ContainerCraft/Kargo.git +cd Kargo/pulumi +``` 2. **Install Python Dependencies** - ```bash - pip install -r requirements.txt - ``` +```bash +pip install -r requirements.txt +``` 3. **Initialize Pulumi Stack** - ```bash - pulumi stack init dev - ``` +```bash +pulumi stack init dev +``` 4. **Configure Pulumi** - Set your Kubernetes context and any necessary configuration options. +Set your Kubernetes context and any necessary configuration options. - ```bash - pulumi config set --path kubernetes.kubeconfig - # Set other configuration options as needed - ``` +```bash +pulumi config set --path kubernetes.kubeconfig +# Set other configuration options as needed +``` 5. **Deploy the Stack** - Preview and deploy your changes. +Preview and deploy your changes. - ```bash - pulumi up - ``` +```bash +pulumi up +``` - Follow the prompts to confirm the deployment. +Follow the prompts to confirm the deployment. --- @@ -116,151 +118,152 @@ Follow these steps to set up your environment: 1. **User Experience (UX)** - - **Clear Error Messages**: Provide meaningful error messages to help users resolve issues. - - **Uniform Logging**: Use consistent logging practices to make debugging easier. +- **Clear Error Messages**: Provide meaningful error messages to help users resolve issues. - ```python - pulumi.log.info(f"Deploying module: {module_name}") - ``` +- **Uniform Logging**: Use consistent logging practices to make debugging easier. + +```python +pulumi.log.info(f"Deploying module: {module_name}") +``` 2. **Developer Experience (DX)** - - **Documentation**: Include comprehensive docstrings and comments in your code. +- **Documentation**: Include comprehensive docstrings and comments in your code. - ```python - def deploy_module(...): - """ - Deploys a module based on configuration. +```python +def deploy_module(...): + """ + Deploys a module based on configuration. - Args: - module_name (str): Name of the module. - config (pulumi.Config): Pulumi configuration object. - ... + Args: + module_name (str): Name of the module. + config (pulumi.Config): Pulumi configuration object. + ... - Returns: - None - """ - ``` + Returns: + None + """ +``` - - **Examples**: Provide example configurations and usage in the documentation to help others understand how to use your code. +- **Examples**: Provide example configurations and usage in the documentation to help others understand how to use your code. 3. **Configurable Modules** - - **Pulumi Stack Configuration**: Use the Pulumi config object to allow users to customize module configurations. +- **Pulumi Stack Configuration**: Use the Pulumi config object to allow users to customize module configurations. - ```python - module_config = config.get_object("module_name") or {} - ``` +```python +module_config = config.get_object("module_name") or {} +``` 4. **Module Data Classes** - - **Typed Data Classes**: Use `dataclass` to encapsulate configurations clearly. +- **Typed Data Classes**: Use `dataclass` to encapsulate configurations clearly. - ```python - from dataclasses import dataclass +```python +from dataclasses import dataclass - @dataclass - class KubeVirtConfig: - namespace: str = "default" - ``` +@dataclass +class KubeVirtConfig: + namespace: str = "default" +``` 5. **Sane Defaults in Data Classes** - - **Sensible Defaults**: Set reasonable default values to minimize the need for user configuration. +- **Sensible Defaults**: Set reasonable default values to minimize the need for user configuration. - ```python - @dataclass - class CertManagerConfig: - namespace: str = "cert-manager" - install_crds: bool = True - ``` +```python +@dataclass +class CertManagerConfig: + namespace: str = "cert-manager" + install_crds: bool = True +``` 6. **User Configuration Handling** - - **Merge Configurations**: Combine user-provided configurations with defaults to ensure all necessary parameters are set. +- **Merge Configurations**: Combine user-provided configurations with defaults to ensure all necessary parameters are set. - ```python - @staticmethod - def merge(user_config: Dict[str, Any]) -> 'CertManagerConfig': - default_config = CertManagerConfig() - for key, value in user_config.items(): - if hasattr(default_config, key): - setattr(default_config, key, value) - else: - pulumi.log.warn(f"Unknown configuration key '{key}' in cert_manager config.") - return default_config - ``` +```python +@staticmethod +def merge(user_config: Dict[str, Any]) -> 'CertManagerConfig': + default_config = CertManagerConfig() + for key, value in user_config.items(): + if hasattr(default_config, key): + setattr(default_config, key, value) + else: + pulumi.log.warn(f"Unknown configuration key '{key}' in cert_manager config.") + return default_config +``` 7. **Simple Function Signatures** - - **Reduce Parameters**: Keep function signatures minimal by encapsulating configurations within data classes. +- **Reduce Parameters**: Keep function signatures minimal by encapsulating configurations within data classes. - ```python - def deploy_module(config_module: ModuleConfig, ...) - ``` +```python +def deploy_module(config_module: ModuleConfig, ...) +``` 8. **Type Annotations** - - **Enhance Readability**: Use type annotations to clarify expected parameter types and return values. +- **Enhance Readability**: Use type annotations to clarify expected parameter types and return values. - ```python - def deploy_module(module_name: str, config: pulumi.Config) -> None: - ``` +```python +def deploy_module(module_name: str, config: pulumi.Config) -> None: +``` 9. **Safe Function Signatures** - - **Type Safety**: Use consistent type checks and raise meaningful errors when types don't match expectations. +- **Type Safety**: Use consistent type checks and raise meaningful errors when types don't match expectations. - ```python - if not isinstance(module_name, str): - raise TypeError("module_name must be a string") - ``` +```python +if not isinstance(module_name, str): + raise TypeError("module_name must be a string") +``` 10. **Streamlined Entrypoint** - - **Encapsulate Logic**: Keep the top-level code minimal and encapsulate logic within functions. +- **Encapsulate Logic**: Keep the top-level code minimal and encapsulate logic within functions. - ```python - if __name__ == "__main__": - main() - ``` +```python +if __name__ == "__main__": + main() +``` 11. **Reuse and Deduplicate Code** - - **Central Utilities**: Place reusable code in the `core` module to maintain consistency and reduce duplication. +- **Central Utilities**: Place reusable code in the `core` module to maintain consistency and reduce duplication. - ```python - from core.utils import sanitize_label_value, extract_repo_name - ``` +```python +from core.utils import sanitize_label_value, extract_repo_name +``` 12. **Version Control Dependencies** - - **Manage Versions**: Control component versions within configuration files to maintain consistency across deployments. +- **Manage Versions**: Control component versions within configuration files to maintain consistency across deployments. - ```python - default_versions = load_default_versions(config) - ``` +```python +default_versions = load_default_versions(config) +``` 13. **Transparency** - - **Informative Outputs**: Export configuration and version information for visibility and auditing. +- **Informative Outputs**: Export configuration and version information for visibility and auditing. - ```python - pulumi.export("versions", versions) - ``` +```python +pulumi.export("versions", versions) +``` 14. **Conditional Execution** - - **Avoid Unnecessary Execution**: Only load and execute modules that are enabled in the configuration. +- **Avoid Unnecessary Execution**: Only load and execute modules that are enabled in the configuration. - ```python - if module_enabled: - deploy_func(...) - ``` +```python +if module_enabled: + deploy_func(...) +``` 15. **Remove Deprecated Code** - - **Maintain a Clean Codebase**: Remove obsolete features and update code to align with current best practices. + - **Maintain a Clean Codebase**: Remove obsolete features and update code to align with current best practices. --- @@ -292,133 +295,133 @@ kargo/ 1. **Define Configuration** - Create a `types.py` file in your module directory to define the configuration data class: - - ```python - from dataclasses import dataclass, field - from typing import Optional, Dict, Any - - @dataclass - class NewModuleConfig: - version: Optional[str] = None - namespace: str = "default" - labels: Dict[str, str] = field(default_factory=dict) - annotations: Dict[str, Any] = field(default_factory=dict) - - @staticmethod - def merge(user_config: Dict[str, Any]) -> 'NewModuleConfig': - default_config = NewModuleConfig() - for key, value in user_config.items(): - if hasattr(default_config, key): - setattr(default_config, key, value) - else: - pulumi.log.warn(f"Unknown configuration key '{key}' in new_module config.") - return default_config - ``` +Create a `types.py` file in your module directory to define the configuration data class: + +```python +from dataclasses import dataclass, field +from typing import Optional, Dict, Any + +@dataclass +class NewModuleConfig: + version: Optional[str] = None + namespace: str = "default" + labels: Dict[str, str] = field(default_factory=dict) + annotations: Dict[str, Any] = field(default_factory=dict) + + @staticmethod + def merge(user_config: Dict[str, Any]) -> 'NewModuleConfig': + default_config = NewModuleConfig() + for key, value in user_config.items(): + if hasattr(default_config, key): + setattr(default_config, key, value) + else: + pulumi.log.warn(f"Unknown configuration key '{key}' in new_module config.") + return default_config +``` 2. **Implement Deployment Logic** - Define the deployment logic in `deploy.py`: - - ```python - import pulumi - import pulumi_kubernetes as k8s - from typing import List, Dict, Any, Tuple, Optional - - from core.metadata import get_global_labels, get_global_annotations - from core.resource_helpers import create_namespace - from .types import NewModuleConfig - - def deploy_new_module( - config_new_module: NewModuleConfig, - global_depends_on: List[pulumi.Resource], - k8s_provider: k8s.Provider, - ) -> Tuple[Optional[str], Optional[pulumi.Resource]]: - # Create Namespace - namespace_resource = create_namespace( - name=config_new_module.namespace, - labels=config_new_module.labels, - annotations=config_new_module.annotations, - k8s_provider=k8s_provider, - depends_on=global_depends_on, - ) - - # Implement specific resource creation logic - # ... - - return config_new_module.version, namespace_resource - ``` - -3. **Update `__main__.py`** - - Include your module in the main deployment script: - - ```python - from typing import List, Dict, Any - import pulumi - from pulumi_kubernetes import Provider - - from core.deployment import initialize_pulumi, deploy_module - from core.config import export_results - - def main(): - try: - init = initialize_pulumi() - - config = init["config"] - k8s_provider = init["k8s_provider"] - versions = init["versions"] - configurations = init["configurations"] - default_versions = init["default_versions"] - global_depends_on = init["global_depends_on"] - - modules_to_deploy = ["cert_manager", "kubevirt", "new_module"] # Add your module here - - deploy_modules( - modules=modules_to_deploy, - config=config, - default_versions=default_versions, - global_depends_on=global_depends_on, - k8s_provider=k8s_provider, - versions=versions, - configurations=configurations, - ) - - compliance_config = init.get("compliance_config", {}) - export_results(versions, configurations, compliance_config) - - except Exception as e: - pulumi.log.error(f"Deployment failed: {str(e)}") - raise - - if __name__ == "__main__": - main() - ``` +Define the deployment logic in `deploy.py`: + +```python +import pulumi +import pulumi_kubernetes as k8s +from typing import List, Dict, Any, Tuple, Optional + +from core.metadata import get_global_labels, get_global_annotations +from core.resource_helpers import create_namespace +from .types import NewModuleConfig + +def deploy_new_module( + config_new_module: NewModuleConfig, + global_depends_on: List[pulumi.Resource], + k8s_provider: k8s.Provider, + ) -> Tuple[Optional[str], Optional[pulumi.Resource]]: + # Create Namespace + namespace_resource = create_namespace( + name=config_new_module.namespace, + labels=config_new_module.labels, + annotations=config_new_module.annotations, + k8s_provider=k8s_provider, + depends_on=global_depends_on, + ) + + # Implement specific resource creation logic + # ... + + return config_new_module.version, namespace_resource +``` + +3. __Update `__main__.py`__ + +Include your module in the main deployment script: + +```python +from typing import List, Dict, Any +import pulumi +from pulumi_kubernetes import Provider + +from core.deployment import initialize_pulumi, deploy_module +from core.config import export_results + +def main(): + try: + init = initialize_pulumi() + + config = init["config"] + k8s_provider = init["k8s_provider"] + versions = init["versions"] + configurations = init["configurations"] + default_versions = init["default_versions"] + global_depends_on = init["global_depends_on"] + + modules_to_deploy = ["cert_manager", "kubevirt", "new_module"] # Add your module here + + deploy_modules( + modules=modules_to_deploy, + config=config, + default_versions=default_versions, + global_depends_on=global_depends_on, + k8s_provider=k8s_provider, + versions=versions, + configurations=configurations, + ) + + compliance_config = init.get("compliance_config", {}) + export_results(versions, configurations, compliance_config) + + except Exception as e: + pulumi.log.error(f"Deployment failed: {str(e)}") + raise + +if __name__ == "__main__": + main() +``` 4. **Document Your Module** - Create a `README.md` file in your module directory to document its purpose, configuration options, and usage instructions. +Create a `README.md` file in your module directory to document its purpose, configuration options, and usage instructions. - ```markdown - # New Module +```markdown +# New Module - Description of your module. +Description of your module. - ## Configuration +## Configuration - - **version** *(string)*: The version of the module to deploy. - - **namespace** *(string)*: The Kubernetes namespace where the module will be deployed. - - **labels** *(dict)*: Custom labels to apply to resources. - - **annotations** *(dict)*: Custom annotations to apply to resources. +- **version** *(string)*: The version of the module to deploy. +- **namespace** *(string)*: The Kubernetes namespace where the module will be deployed. +- **labels** *(dict)*: Custom labels to apply to resources. +- **annotations** *(dict)*: Custom annotations to apply to resources. - ## Usage +## Usage - Example of how to configure and deploy the module. +Example of how to configure and deploy the module. - ## Additional Information +## Additional Information - Any additional details or resources. - ``` +Any additional details or resources. +``` --- @@ -469,9 +472,7 @@ Manage module versions and dependencies within configuration files, such as `def We welcome contributions from the community! Here's how you can help: - **Report Issues**: If you encounter any bugs or have feature requests, please open an issue on GitHub. - - **Submit Pull Requests**: If you'd like to contribute code, fork the repository and submit a pull request. - - **Improve Documentation**: Help us enhance this guide and other documentation to make it more accessible. --- diff --git a/pulumi/ROADMAP.md b/pulumi/ROADMAP.md index 9bc5bf0..8365582 100644 --- a/pulumi/ROADMAP.md +++ b/pulumi/ROADMAP.md @@ -9,10 +9,12 @@ - [Part 1: Refactoring AWS Module to Align with Kubernetes Modules](#part-1-refactoring-aws-module-to-align-with-kubernetes-modules) - [Part 2: Adjusting Version Handling to Be Exclusive to Kubernetes Modules](#part-2-adjusting-version-handling-to-be-exclusive-to-kubernetes-modules) - [Part 3: Improving Configuration Schema and Type System Using Pydantic](#part-3-improving-configuration-schema-and-type-system-using-pydantic) + 5. [Detailed Implementation Steps](#detailed-implementation-steps) - [Part 1 Steps](#part-1-steps) - [Part 2 Steps](#part-2-steps) - [Part 3 Steps](#part-3-steps) + 6. [Additional Considerations](#additional-considerations) 7. [Conclusion](#conclusion) 8. [Appendix: Understanding Pydantic](#appendix-understanding-pydantic) @@ -40,16 +42,18 @@ This document serves as a comprehensive roadmap and educational resource for ref ### Current Situation - **Kubernetes Modules**: - - Each module resides under `pulumi/modules//`. - - Contains `types.py` for configuration, `deploy.py` for deployment logic, and `README.md` for documentation. - - Deployment functions have a consistent signature and return values. - - Dynamic module discovery and deployment are handled via `core/deployment.py` and `__main__.py`. + + - Each module resides under `pulumi/modules//`. + - Contains `types.py` for configuration, `deploy.py` for deployment logic, and `README.md` for documentation. + - Deployment functions have a consistent signature and return values. + - Dynamic module discovery and deployment are handled via `core/deployment.py` and `__main__.py`. - **AWS Module**: - - Does not conform to the structure of Kubernetes modules. - - Has a different code organization and deployment function signature. - - Integration with the core module and dynamic discovery is inconsistent. - - Version handling is not aligned with the strategy used for Kubernetes modules. + + - Does not conform to the structure of Kubernetes modules. + - Has a different code organization and deployment function signature. + - Integration with the core module and dynamic discovery is inconsistent. + - Version handling is not aligned with the strategy used for Kubernetes modules. ### Issues Identified @@ -65,29 +69,29 @@ This document serves as a comprehensive roadmap and educational resource for ref - **Goal**: Restructure the AWS module to match the directory and code structure of Kubernetes modules, ensuring consistency across the codebase. - **Actions**: - - Move AWS module files under `pulumi/modules/aws/`. - - Define configuration data classes in `types.py` without unnecessary version attributes. - - Update the deployment logic in `deploy.py` to match the standard function signature. - - Modify `__main__.py` and core modules to include and deploy the AWS module consistently. + - Move AWS module files under `pulumi/modules/aws/`. + - Define configuration data classes in `types.py` without unnecessary version attributes. + - Update the deployment logic in `deploy.py` to match the standard function signature. + - Modify `__main__.py` and core modules to include and deploy the AWS module consistently. ### Part 2: Adjusting Version Handling to Be Exclusive to Kubernetes Modules - **Goal**: Ensure that version locking mechanisms are exclusive to Kubernetes modules and remove unnecessary version interfaces from cloud provider modules like AWS. - **Actions**: - - Update `core/config.py` and `core/deployment.py` to handle versioning exclusively for Kubernetes modules. - - Adjust module discovery functions to accommodate modules with and without versioning. - - Remove version handling code from cloud provider modules. - - Optionally, implement a utility to extract versions from `requirements.txt` for logging purposes. + - Update `core/config.py` and `core/deployment.py` to handle versioning exclusively for Kubernetes modules. + - Adjust module discovery functions to accommodate modules with and without versioning. + - Remove version handling code from cloud provider modules. + - Optionally, implement a utility to extract versions from `requirements.txt` for logging purposes. ### Part 3: Improving Configuration Schema and Type System Using Pydantic - **Goal**: Enhance the configuration schema by using Pydantic models for type safety, validation, and flexibility, allowing module maintainers to define complex configurations independently. - **Actions**: - - Integrate Pydantic into the project. - - Each module defines its own Pydantic configuration model in `types.py`. - - Centralize configuration loading and validation in `core/config.py`. - - Update deployment functions to use validated configuration objects. - - Provide clear error reporting and documentation. + - Integrate Pydantic into the project. + - Each module defines its own Pydantic configuration model in `types.py`. + - Centralize configuration loading and validation in `core/config.py`. + - Update deployment functions to use validated configuration objects. + - Provide clear error reporting and documentation. --- @@ -105,9 +109,10 @@ This document serves as a comprehensive roadmap and educational resource for ref - Create the directory `pulumi/modules/aws/` if it doesn't exist. - Move existing AWS module files into this directory: - - `types.py`: Defines configuration data classes. - - `deploy.py`: Contains deployment logic. - - `README.md`: Provides module documentation. + - `types.py`: Defines configuration data classes. + - `deploy.py`: Contains deployment logic. + - `README.md`: Provides module documentation. + - Ensure the directory contains an `__init__.py` file to make it a Python package. #### Step 2: Define Configuration Data Classes in `types.py` @@ -149,7 +154,7 @@ class AWSConfig(BaseModel): #### Step 3: Update Deployment Logic in `deploy.py` -**Action**: Update the AWS module's `deploy.py` to contain a deployment function `deploy_aws_module` with a consistent signature and remove any version handling. +__Action__: Update the AWS module's `deploy.py` to contain a deployment function `deploy_aws_module` with a consistent signature and remove any version handling. **Reasoning**: Aligning the function signature with other modules ensures consistent deployment patterns and simplifies the core deployment logic. @@ -198,7 +203,7 @@ def deploy_aws_module( #### Step 4: Modify `__main__.py` to Include AWS Module -**Action**: Update `pulumi/__main__.py` to include and deploy the AWS module consistently with other modules. +__Action__: Update `pulumi/__main__.py` to include and deploy the AWS module consistently with other modules. **Reasoning**: This ensures that the AWS module is integrated into the deployment process in the same way as Kubernetes modules. @@ -278,7 +283,7 @@ if __name__ == "__main__": #### Step 5: Adjust `core/deployment.py` to Handle Providers and Versioning -**Action**: Modify `deploy_module` in `pulumi/core/deployment.py` to handle modules with and without versioning. +__Action__: Modify `deploy_module` in `pulumi/core/deployment.py` to handle modules with and without versioning. **Reasoning**: The core deployment function needs to accommodate both Kubernetes modules (which use versioning) and cloud provider modules (which do not). @@ -336,7 +341,7 @@ def deploy_module( #### Step 6: Update `core/config.py` to Handle Versioning Exclusively for Kubernetes Modules -**Action**: Adjust `get_module_config` to include version information only for Kubernetes modules. +__Action__: Adjust `get_module_config` to include version information only for Kubernetes modules. **Reasoning**: Cloud provider modules do not need version information in their configuration. @@ -390,7 +395,7 @@ def get_module_config( #### Step 7: Adjust Module Discovery Functions -**Action**: Ensure `discover_config_class` and `discover_deploy_function` work correctly for all modules. +__Action__: Ensure `discover_config_class` and `discover_deploy_function` work correctly for all modules. **Reasoning**: These functions need to dynamically import the appropriate classes and functions for each module, regardless of whether they handle versioning. @@ -429,10 +434,11 @@ def discover_deploy_function(module_name: str) -> Callable: **Implementation**: - **In `types.py`**: - - Ensure no `version` field is present in the configuration models. + - Ensure no `version` field is present in the configuration models. + - **In `deploy.py`**: - - Ensure deployment functions do not return version information. - - Remove any logic that deals with versioning. + - Ensure deployment functions do not return version information. + - Remove any logic that deals with versioning. **Explanation**: @@ -478,15 +484,15 @@ def get_module_version_from_requirements(module_name: str) -> Optional[str]: - Install Pydantic: - ```bash - pip install pydantic - ``` +```bash +pip install pydantic +``` - Add to `requirements.txt`: - ``` - pydantic>=1.8.2 - ``` +``` +pydantic>=1.8.2 +``` #### Step 11: Define Base Configuration Classes @@ -515,40 +521,40 @@ class BaseConfig(BaseModel): - **For AWS Module**: - ```python - # pulumi/modules/aws/types.py +```python +# pulumi/modules/aws/types.py - from pydantic import BaseModel, root_validator - from typing import Optional, List, Dict, Any +from pydantic import BaseModel, root_validator +from typing import Optional, List, Dict, Any - class AWSConfig(BaseModel): - enabled: bool = False - profile: Optional[str] = None - region: str - account_id: Optional[str] = None - landingzones: List[Dict[str, Any]] = [] +class AWSConfig(BaseModel): + enabled: bool = False + profile: Optional[str] = None + region: str + account_id: Optional[str] = None + landingzones: List[Dict[str, Any]] = [] - @root_validator - def check_region(cls, values): - region = values.get('region') - if not region: - raise ValueError('region must be specified for AWS module') - return values - ``` + @root_validator + def check_region(cls, values): + region = values.get('region') + if not region: + raise ValueError('region must be specified for AWS module') + return values +``` - **For Kubernetes Module**: - ```python - # pulumi/modules/cert_manager/types.py +```python +# pulumi/modules/cert_manager/types.py - from pydantic import BaseModel +from pydantic import BaseModel - class CertManagerConfig(BaseModel): - enabled: bool = False - version: str = "latest" - namespace: str = "cert-manager" - install_crds: bool = True - ``` +class CertManagerConfig(BaseModel): + enabled: bool = False + version: str = "latest" + namespace: str = "cert-manager" + install_crds: bool = True +``` **Explanation**: @@ -641,14 +647,15 @@ def deploy_aws_module( **Implementation**: - Errors are caught in `get_module_config` and logged with detailed information. + - Example error message: - ``` - Configuration error in module 'aws': - 1 validation error for AWSConfig - region - field required (type=value_error.missing) - ``` +``` +Configuration error in module 'aws': +1 validation error for AWSConfig +region + field required (type=value_error.missing) +``` #### Step 16: Document Configuration Schemas @@ -660,34 +667,37 @@ def deploy_aws_module( - **Example for AWS Module**: - ```markdown - # AWS Module Configuration - - ## Configuration Schema - - ```yaml - aws: - enabled: true - profile: "default" - region: "us-west-2" - account_id: "123456789012" - landingzones: - - name: "tenant1" - email: "tenant1@example.com" - # Add other fields as needed - ``` - - ## Configuration Fields - - - **enabled** *(bool)*: Enable or disable the AWS module. - - **profile** *(string)*: AWS CLI profile to use. - - **region** *(string, required)*: AWS region. - - **account_id** *(string)*: AWS account ID. - - **landingzones** *(list)*: List of landing zone configurations. - - **name** *(string)*: Name of the landing zone. - - **email** *(string)*: Email associated with the landing zone. - - *...* - ``` +```markdown +# AWS Module Configuration + +## Configuration Schema + +```yaml +aws: + enabled: true + profile: "default" + region: "us-west-2" + account_id: "123456789012" + landingzones: + - name: "tenant1" + email: "tenant1@example.com" + # Add other fields as needed +``` + +## Configuration Fields + +- **enabled** *(bool)*: Enable or disable the AWS module. +- **profile** *(string)*: AWS CLI profile to use. +- **region** *(string, required)*: AWS region. +- __account_id__ _(string)_: AWS account ID. +- **landingzones** *(list)*: List of landing zone configurations. + - **name** *(string)*: Name of the landing zone. + - **email** *(string)*: Email associated with the landing zone. + - *...* + +``` + +``` **Explanation**: @@ -711,8 +721,8 @@ def deploy__module( # Deployment logic ``` -- **Kubernetes Modules**: Return a tuple `(version, primary_resource)`. -- **Cloud Provider Modules**: Return the `primary_resource`. +- __Kubernetes Modules__: Return a tuple `(version, primary_resource)`. +- __Cloud Provider Modules__: Return the `primary_resource`. ### Code Comments and Docstrings @@ -765,29 +775,29 @@ To ensure that all developers are comfortable with Pydantic and its usage in the 1. **Installation**: - ```bash - pip install pydantic - ``` +```bash +pip install pydantic +``` 2. **Defining a Model**: - ```python - from pydantic import BaseModel +```python +from pydantic import BaseModel - class ModuleConfig(BaseModel): - enabled: bool = False - name: str - version: str = "latest" - ``` +class ModuleConfig(BaseModel): + enabled: bool = False + name: str + version: str = "latest" +``` 3. **Validation and Usage**: - ```python - try: - config = ModuleConfig(**user_input) - except ValidationError as e: - print(e) - ``` +```python +try: + config = ModuleConfig(**user_input) +except ValidationError as e: + print(e) +``` #### Advanced Features diff --git a/pulumi/ROADMAP_Addendum.md b/pulumi/ROADMAP_Addendum.md index ff9e5fa..f7b89a6 100644 --- a/pulumi/ROADMAP_Addendum.md +++ b/pulumi/ROADMAP_Addendum.md @@ -1,4 +1,5 @@ # Konductor IaC Template Repository Refactor and Enhancement Roadmap + > **Technical Blueprint Addendum** --- @@ -13,23 +14,26 @@ - [Part 1: Aligning AWS Module with Kubernetes Modules](#part-1-aligning-aws-module-with-kubernetes-modules) - [Part 2: Modular Version Handling](#part-2-modular-version-handling) - [Part 3: Enhancing Configuration Management with Pydantic](#part-3-enhancing-configuration-management-with-pydantic) + 6. [Detailed Implementation Plan](#detailed-implementation-plan) - [Part 1 Implementation Steps](#part-1-implementation-steps) - [Part 2 Implementation Steps](#part-2-implementation-steps) - [Part 3 Implementation Steps](#part-3-implementation-steps) + 7. [Technical Considerations](#technical-considerations) - [Dependency Management](#dependency-management) - [Error Handling and Logging](#error-handling-and-logging) - [Testing Strategy](#testing-strategy) - [Documentation Standards](#documentation-standards) - [Security Implications](#security-implications) + 8. [Risks and Mitigations](#risks-and-mitigations) 9. [Timeline and Milestones](#timeline-and-milestones) 10. [Conclusion](#conclusion) 11. [Appendices](#appendices) - - [Appendix A: Pydantic Overview](#appendix-a-pydantic-overview) - - [Appendix B: Code Samples](#appendix-b-code-samples) - - [Appendix C: Glossary](#appendix-c-glossary) + - [Appendix A: Pydantic Overview](#appendix-a-pydantic-overview) + - [Appendix B: Code Samples](#appendix-b-code-samples) + - [Appendix C: Glossary](#appendix-c-glossary) --- @@ -61,23 +65,25 @@ Konductor is an IaC platform built using Pulumi and Python, designed to streamli ### Kubernetes Modules - **Structure**: - - Reside under `pulumi/modules//`. - - Include `types.py`, `deploy.py`, and `README.md`. - - Use `dataclasses` for configuration models. - - Deployment functions have consistent signatures and return types. + - Reside under `pulumi/modules//`. + - Include `types.py`, `deploy.py`, and `README.md`. + - Use `dataclasses` for configuration models. + - Deployment functions have consistent signatures and return types. + - **Version Handling**: - - Version locking mechanisms are in place. - - Versions are managed via configuration. + - Version locking mechanisms are in place. + - Versions are managed via configuration. ### AWS Module - **Structure**: - - Does not conform to the standard module structure. - - Lacks separation of concerns (configuration vs. deployment logic). - - Deployment function signatures differ from Kubernetes modules. + - Does not conform to the standard module structure. + - Lacks separation of concerns (configuration vs. deployment logic). + - Deployment function signatures differ from Kubernetes modules. + - **Version Handling**: - - Unnecessary version interfaces are present. - - Versions are managed via `requirements.txt`, making internal version handling redundant. + - Unnecessary version interfaces are present. + - Versions are managed via `requirements.txt`, making internal version handling redundant. ### Issues Identified @@ -122,12 +128,14 @@ Konductor is an IaC platform built using Pulumi and Python, designed to streamli **Technical Details**: - **Create Directory**: - - Ensure `pulumi/modules/aws/` exists. + - Ensure `pulumi/modules/aws/` exists. + - **Organize Files**: - - Move existing AWS code files (`aws_deploy.py`, `aws_types.py`, etc.) into `pulumi/modules/aws/`. - - Rename files to `deploy.py` and `types.py` to match the convention. + - Move existing AWS code files (`aws_deploy.py`, `aws_types.py`, etc.) into `pulumi/modules/aws/`. + - Rename files to `deploy.py` and `types.py` to match the convention. + - **Initialize Module**: - - Add `__init__.py` to `pulumi/modules/aws/`. + - Add `__init__.py` to `pulumi/modules/aws/`. **Expected Outcome**: @@ -141,34 +149,37 @@ Konductor is an IaC platform built using Pulumi and Python, designed to streamli **Technical Details**: - **Use Pydantic**: - - Import `BaseModel` from `pydantic`. + + - Import `BaseModel` from `pydantic`. + - **Define Configuration Class**: - ```python - from pydantic import BaseModel, Field, root_validator - from typing import Optional, List, Dict, Any +```python +from pydantic import BaseModel, Field, root_validator +from typing import Optional, List, Dict, Any - class AWSConfig(BaseModel): - enabled: bool = False - profile: Optional[str] = None - region: str - account_id: Optional[str] = None - landingzones: List[Dict[str, Any]] = Field(default_factory=list) +class AWSConfig(BaseModel): + enabled: bool = False + profile: Optional[str] = None + region: str + account_id: Optional[str] = None + landingzones: List[Dict[str, Any]] = Field(default_factory=list) - @root_validator - def validate_region(cls, values): - if not values.get('region'): - raise ValueError('The "region" field is required for AWSConfig.') - return values - ``` + @root_validator + def validate_region(cls, values): + if not values.get('region'): + raise ValueError('The "region" field is required for AWSConfig.') + return values +``` - **Explanation**: - - `enabled`: Determines if the module is active. - - `profile`: AWS CLI profile. - - `region`: Required AWS region. - - `account_id`: Optional AWS account ID. - - `landingzones`: List of landing zone configurations. - - `validate_region`: Ensures `region` is provided. + + - `enabled`: Determines if the module is active. + - `profile`: AWS CLI profile. + - `region`: Required AWS region. + - `account_id`: Optional AWS account ID. + - `landingzones`: List of landing zone configurations. + - `validate_region`: Ensures `region` is provided. **Expected Outcome**: @@ -176,45 +187,46 @@ Konductor is an IaC platform built using Pulumi and Python, designed to streamli #### Step 3: Update Deployment Logic in `deploy.py` -**Action**: Create `deploy_aws_module` function in `pulumi/modules/aws/deploy.py` with a consistent signature. +__Action__: Create `deploy_aws_module` function in `pulumi/modules/aws/deploy.py` with a consistent signature. **Technical Details**: - **Define Deployment Function**: - ```python - from typing import List, Dict, Any - import pulumi - import pulumi_aws as aws - from .types import AWSConfig - - def deploy_aws_module( - config: AWSConfig, - global_depends_on: List[pulumi.Resource], - providers: Dict[str, Any], - ) -> pulumi.Resource: - aws_provider = providers.get('aws') - if not aws_provider: - raise ValueError("AWS provider not found.") - - # Example AWS resource creation - s3_bucket = aws.s3.Bucket( - resource_name='my_bucket', - bucket='my-unique-bucket-name', - opts=pulumi.ResourceOptions( - provider=aws_provider, - depends_on=global_depends_on, - ) - ) - - return s3_bucket - ``` +```python +from typing import List, Dict, Any +import pulumi +import pulumi_aws as aws +from .types import AWSConfig + +def deploy_aws_module( + config: AWSConfig, + global_depends_on: List[pulumi.Resource], + providers: Dict[str, Any], + ) -> pulumi.Resource: + aws_provider = providers.get('aws') + if not aws_provider: + raise ValueError("AWS provider not found.") + + # Example AWS resource creation + s3_bucket = aws.s3.Bucket( + resource_name='my_bucket', + bucket='my-unique-bucket-name', + opts=pulumi.ResourceOptions( + provider=aws_provider, + depends_on=global_depends_on, + ) + ) + + return s3_bucket +``` - **Explanation**: - - The function signature aligns with other modules. - - It accepts the validated `config` object. - - Uses the `aws_provider` from the `providers` dictionary. - - Returns the primary resource without version information. + + - The function signature aligns with other modules. + - It accepts the validated `config` object. + - Uses the `aws_provider` from the `providers` dictionary. + - Returns the primary resource without version information. **Expected Outcome**: @@ -223,43 +235,43 @@ Konductor is an IaC platform built using Pulumi and Python, designed to streamli #### Step 4: Modify `__main__.py` to Include AWS Module -**Action**: Update `pulumi/__main__.py` to deploy the AWS module consistently. +__Action__: Update `pulumi/__main__.py` to deploy the AWS module consistently. **Technical Details**: - **Initialize AWS Provider**: - ```python - # Inside main function - aws_config, aws_enabled = get_module_config('aws', config) - aws_provider = None - if aws_enabled: - from pulumi_aws import Provider as AWSProvider - aws_provider = AWSProvider( - 'aws_provider', - profile=aws_config.profile, - region=aws_config.region, - ) - ``` +```python +# Inside main function +aws_config, aws_enabled = get_module_config('aws', config) +aws_provider = None +if aws_enabled: + from pulumi_aws import Provider as AWSProvider + aws_provider = AWSProvider( + 'aws_provider', + profile=aws_config.profile, + region=aws_config.region, + ) +``` - **Update Providers Dictionary**: - ```python - providers = { - 'k8s': k8s_provider, - 'aws': aws_provider, - # Add other providers as needed - } - ``` +```python +providers = { + 'k8s': k8s_provider, + 'aws': aws_provider, + # Add other providers as needed +} +``` - **Include AWS Module in Deployment**: - ```python - modules_to_deploy = [ - 'aws', - # Other modules... - ] - ``` +```python +modules_to_deploy = [ + 'aws', + # Other modules... +] +``` **Expected Outcome**: @@ -268,48 +280,49 @@ Konductor is an IaC platform built using Pulumi and Python, designed to streamli #### Step 5: Adjust `core/deployment.py` to Handle Providers and Versioning -**Action**: Modify `deploy_module` to handle modules with and without versioning. +__Action__: Modify `deploy_module` to handle modules with and without versioning. **Technical Details**: -- **Update `deploy_module` Function**: - - ```python - def deploy_module( - module_name: str, - config: pulumi.Config, - global_depends_on: List[pulumi.Resource], - providers: Dict[str, Any], - versions: Dict[str, str], - configurations: Dict[str, Dict[str, Any]] - ) -> None: - config_obj, module_enabled = get_module_config(module_name, config) - - if module_enabled: - deploy_func = discover_deploy_function(module_name) - result = deploy_func( - config=config_obj, - global_depends_on=global_depends_on, - providers=providers, - ) - - if module_name in KUBERNETES_MODULES: - version, primary_resource = result - versions[module_name] = version - else: - primary_resource = result - - configurations[module_name] = {"enabled": module_enabled} - global_depends_on.append(primary_resource) - else: - log.info(f"Module {module_name} is not enabled.") - ``` +- __Update `deploy_module` Function__: + +```python +def deploy_module( + module_name: str, + config: pulumi.Config, + global_depends_on: List[pulumi.Resource], + providers: Dict[str, Any], + versions: Dict[str, str], + configurations: Dict[str, Dict[str, Any]] +) -> None: + config_obj, module_enabled = get_module_config(module_name, config) + + if module_enabled: + deploy_func = discover_deploy_function(module_name) + result = deploy_func( + config=config_obj, + global_depends_on=global_depends_on, + providers=providers, + ) + + if module_name in KUBERNETES_MODULES: + version, primary_resource = result + versions[module_name] = version + else: + primary_resource = result + + configurations[module_name] = {"enabled": module_enabled} + global_depends_on.append(primary_resource) + else: + log.info(f"Module {module_name} is not enabled.") +``` - **Explanation**: - - Checks if the module is enabled. - - For Kubernetes modules, expects a `(version, resource)` tuple. - - For other modules, expects a single resource. - - Updates `versions` only for Kubernetes modules. + + - Checks if the module is enabled. + - For Kubernetes modules, expects a `(version, resource)` tuple. + - For other modules, expects a single resource. + - Updates `versions` only for Kubernetes modules. **Expected Outcome**: @@ -319,43 +332,44 @@ Konductor is an IaC platform built using Pulumi and Python, designed to streamli #### Step 6: Update `core/config.py` to Handle Versioning Exclusively for Kubernetes Modules -**Action**: Adjust `get_module_config` to inject version information only for Kubernetes modules. +__Action__: Adjust `get_module_config` to inject version information only for Kubernetes modules. **Technical Details**: - **Define Kubernetes Modules List**: - ```python - KUBERNETES_MODULES = [ - 'cert_manager', - 'kubevirt', - 'multus', - # Other Kubernetes modules... - ] - ``` - -- **Update `get_module_config` Function**: - - ```python - def get_module_config( - module_name: str, - config: pulumi.Config, - ) -> Tuple[Any, bool]: - module_config_dict = config.get_object(module_name) or {} - module_enabled = module_config_dict.get('enabled', False) - - # Inject version for Kubernetes modules - if module_name in KUBERNETES_MODULES: - default_versions = load_default_versions() - module_config_dict['version'] = module_config_dict.get('version', default_versions.get(module_name)) - - # Import and instantiate the module's configuration class - # ... (same as before) - ``` +```python +KUBERNETES_MODULES = [ + 'cert_manager', + 'kubevirt', + 'multus', + # Other Kubernetes modules... +] +``` + +- __Update `get_module_config` Function__: + +```python +def get_module_config( + module_name: str, + config: pulumi.Config, + ) -> Tuple[Any, bool]: + module_config_dict = config.get_object(module_name) or {} + module_enabled = module_config_dict.get('enabled', False) + + # Inject version for Kubernetes modules + if module_name in KUBERNETES_MODULES: + default_versions = load_default_versions() + module_config_dict['version'] = module_config_dict.get('version', default_versions.get(module_name)) + + # Import and instantiate the module's configuration class + # ... (same as before) +``` - **Explanation**: - - Only Kubernetes modules receive a `version` in their configuration. - - Non-Kubernetes modules are unaffected. + + - Only Kubernetes modules receive a `version` in their configuration. + - Non-Kubernetes modules are unaffected. **Expected Outcome**: @@ -363,32 +377,32 @@ Konductor is an IaC platform built using Pulumi and Python, designed to streamli #### Step 7: Adjust Module Discovery Functions -**Action**: Ensure `discover_config_class` and `discover_deploy_function` work for all modules. +__Action__: Ensure `discover_config_class` and `discover_deploy_function` work for all modules. **Technical Details**: -- **Update `discover_config_class`**: - - ```python - def discover_config_class(module_name: str) -> Type[BaseModel]: - types_module = importlib.import_module(f"modules.{module_name}.types") - for name, obj in inspect.getmembers(types_module): - if inspect.isclass(obj) and issubclass(obj, BaseModel): - return obj - raise ValueError(f"No Pydantic BaseModel found in modules.{module_name}.types") - ``` - -- **Update `discover_deploy_function`**: - - ```python - def discover_deploy_function(module_name: str) -> Callable: - deploy_module = importlib.import_module(f"modules.{module_name}.deploy") - function_name = f"deploy_{module_name}_module" - deploy_function = getattr(deploy_module, function_name, None) - if not deploy_function: - raise ValueError(f"No deploy function named '{function_name}' found in modules.{module_name}.deploy") - return deploy_function - ``` +- __Update `discover_config_class`__: + +```python +def discover_config_class(module_name: str) -> Type[BaseModel]: + types_module = importlib.import_module(f"modules.{module_name}.types") + for name, obj in inspect.getmembers(types_module): + if inspect.isclass(obj) and issubclass(obj, BaseModel): + return obj + raise ValueError(f"No Pydantic BaseModel found in modules.{module_name}.types") +``` + +- __Update `discover_deploy_function`__: + +```python +def discover_deploy_function(module_name: str) -> Callable: + deploy_module = importlib.import_module(f"modules.{module_name}.deploy") + function_name = f"deploy_{module_name}_module" + deploy_function = getattr(deploy_module, function_name, None) + if not deploy_function: + raise ValueError(f"No deploy function named '{function_name}' found in modules.{module_name}.deploy") + return deploy_function +``` **Expected Outcome**: @@ -401,10 +415,11 @@ Konductor is an IaC platform built using Pulumi and Python, designed to streamli **Technical Details**: - **In `types.py`**: - - Ensure no `version` field is present. + - Ensure no `version` field is present. + - **In `deploy.py`**: - - Ensure deployment functions do not return version information. - - Remove any logic related to version checking or handling. + - Ensure deployment functions do not return version information. + - Remove any logic related to version checking or handling. **Expected Outcome**: @@ -418,22 +433,23 @@ Konductor is an IaC platform built using Pulumi and Python, designed to streamli - **Utility Function**: - ```python - def get_module_version_from_requirements(module_name: str) -> Optional[str]: - try: - with open('requirements.txt', 'r') as f: - for line in f: - if module_name in line: - parts = line.strip().split('==') - if len(parts) == 2: - return parts[1] - except Exception as e: - pulumi.log.warn(f"Error reading requirements.txt: {e}") - return None - ``` +```python +def get_module_version_from_requirements(module_name: str) -> Optional[str]: + try: + with open('requirements.txt', 'r') as f: + for line in f: + if module_name in line: + parts = line.strip().split('==') + if len(parts) == 2: + return parts[1] + except Exception as e: + pulumi.log.warn(f"Error reading requirements.txt: {e}") + return None +``` - **Usage**: - - For logging or documentation, not for configuration. + + - For logging or documentation, not for configuration. **Expected Outcome**: @@ -449,15 +465,15 @@ Konductor is an IaC platform built using Pulumi and Python, designed to streamli - **Install Pydantic**: - ```bash - pip install pydantic - ``` +```bash +pip install pydantic +``` - **Update `requirements.txt`**: - ``` - pydantic>=1.8.2 - ``` +``` +pydantic>=1.8.2 +``` **Expected Outcome**: @@ -465,22 +481,23 @@ Konductor is an IaC platform built using Pulumi and Python, designed to streamli #### Step 11: Define Base Configuration Classes -**Action**: Create `BaseConfig` in `pulumi/core/base_config.py` for common fields. +__Action__: Create `BaseConfig` in `pulumi/core/base_config.py` for common fields. **Technical Details**: - **Define `BaseConfig`**: - ```python - from pydantic import BaseModel +```python +from pydantic import BaseModel - class BaseConfig(BaseModel): - enabled: bool = False - ``` +class BaseConfig(BaseModel): + enabled: bool = False +``` - **Explanation**: - - Provides a common `enabled` field. - - Modules can inherit from `BaseConfig`. + + - Provides a common `enabled` field. + - Modules can inherit from `BaseConfig`. **Expected Outcome**: @@ -494,31 +511,31 @@ Konductor is an IaC platform built using Pulumi and Python, designed to streamli - **For AWS Module**: - ```python - from pydantic import BaseModel, Field, root_validator - from typing import Optional, List, Dict, Any +```python +from pydantic import BaseModel, Field, root_validator +from typing import Optional, List, Dict, Any - class AWSConfig(BaseConfig): - profile: Optional[str] = None - region: str - account_id: Optional[str] = None - landingzones: List[Dict[str, Any]] = Field(default_factory=list) +class AWSConfig(BaseConfig): + profile: Optional[str] = None + region: str + account_id: Optional[str] = None + landingzones: List[Dict[str, Any]] = Field(default_factory=list) - @root_validator - def validate_region(cls, values): - if not values.get('region'): - raise ValueError('The "region" field is required for AWSConfig.') - return values - ``` + @root_validator + def validate_region(cls, values): + if not values.get('region'): + raise ValueError('The "region" field is required for AWSConfig.') + return values +``` - **For Kubernetes Module**: - ```python - class CertManagerConfig(BaseConfig): - version: str = "latest" - namespace: str = "cert-manager" - install_crds: bool = True - ``` +```python +class CertManagerConfig(BaseConfig): + version: str = "latest" + namespace: str = "cert-manager" + install_crds: bool = True +``` **Expected Outcome**: @@ -526,35 +543,35 @@ Konductor is an IaC platform built using Pulumi and Python, designed to streamli #### Step 13: Centralize Configuration Loading and Validation -**Action**: Update `get_module_config` in `core/config.py` to use Pydantic models. +__Action__: Update `get_module_config` in `core/config.py` to use Pydantic models. **Technical Details**: - **Update Function**: - ```python - def get_module_config( - module_name: str, - config: pulumi.Config, - ) -> Tuple[Any, bool]: - module_config_dict = config.get_object(module_name) or {} - module_enabled = module_config_dict.get('enabled', False) +```python +def get_module_config( + module_name: str, + config: pulumi.Config, + ) -> Tuple[Any, bool]: + module_config_dict = config.get_object(module_name) or {} + module_enabled = module_config_dict.get('enabled', False) - if module_name in KUBERNETES_MODULES: - default_versions = load_default_versions() - module_config_dict['version'] = module_config_dict.get('version', default_versions.get(module_name)) + if module_name in KUBERNETES_MODULES: + default_versions = load_default_versions() + module_config_dict['version'] = module_config_dict.get('version', default_versions.get(module_name)) - types_module = importlib.import_module(f"modules.{module_name}.types") - ModuleConfigClass = getattr(types_module, f"{module_name.capitalize()}Config") + types_module = importlib.import_module(f"modules.{module_name}.types") + ModuleConfigClass = getattr(types_module, f"{module_name.capitalize()}Config") - try: - config_obj = ModuleConfigClass(**module_config_dict) - except ValidationError as e: - pulumi.log.error(f"Configuration error in module '{module_name}':\n{e}") - raise + try: + config_obj = ModuleConfigClass(**module_config_dict) + except ValidationError as e: + pulumi.log.error(f"Configuration error in module '{module_name}':\n{e}") + raise - return config_obj, module_enabled - ``` + return config_obj, module_enabled +``` **Expected Outcome**: @@ -568,23 +585,23 @@ Konductor is an IaC platform built using Pulumi and Python, designed to streamli - **Example for AWS Module**: - ```python - def deploy_aws_module( - config: AWSConfig, - global_depends_on: List[pulumi.Resource], - providers: Dict[str, Any], - ) -> pulumi.Resource: - aws_provider = providers.get('aws') - if not aws_provider: - raise ValueError("AWS provider not found.") +```python +def deploy_aws_module( + config: AWSConfig, + global_depends_on: List[pulumi.Resource], + providers: Dict[str, Any], + ) -> pulumi.Resource: + aws_provider = providers.get('aws') + if not aws_provider: + raise ValueError("AWS provider not found.") - # Use configuration values directly - # Example: region = config.region - # ... + # Use configuration values directly + # Example: region = config.region + # ... - # Deployment logic - # ... - ``` + # Deployment logic + # ... +``` **Expected Outcome**: @@ -596,22 +613,22 @@ Konductor is an IaC platform built using Pulumi and Python, designed to streamli **Technical Details**: -- **In `get_module_config`**: +- __In `get_module_config`__: - ```python - except ValidationError as e: - pulumi.log.error(f"Configuration error in module '{module_name}':\n{e}") - raise - ``` +```python +except ValidationError as e: + pulumi.log.error(f"Configuration error in module '{module_name}':\n{e}") + raise +``` - **User Feedback**: - ``` - Configuration error in module 'aws': - 1 validation error for AWSConfig - region - The "region" field is required for AWSConfig. (type=value_error) - ``` +``` +Configuration error in module 'aws': +1 validation error for AWSConfig +region + The "region" field is required for AWSConfig. (type=value_error) +``` **Expected Outcome**: @@ -625,25 +642,28 @@ Konductor is an IaC platform built using Pulumi and Python, designed to streamli - **Include Configuration Examples**: - ```markdown - ## Configuration Schema +```markdown +## Configuration Schema + +```yaml +aws: + enabled: true + profile: "default" + region: "us-west-2" + # Other fields... +``` + +``` - ```yaml - aws: - enabled: true - profile: "default" - region: "us-west-2" - # Other fields... - ``` - ``` +``` - **Explain Each Field**: - - **enabled**: Enables or disables the module. - - **profile**: AWS CLI profile name. - - **region**: AWS region (required). - - **account_id**: AWS account ID. - - **landingzones**: List of landing zone configurations. + - **enabled**: Enables or disables the module. + - **profile**: AWS CLI profile name. + - **region**: AWS region (required). + - __account_id__: AWS account ID. + - **landingzones**: List of landing zone configurations. **Expected Outcome**: @@ -685,11 +705,13 @@ Konductor is an IaC platform built using Pulumi and Python, designed to streamli ## Risks and Mitigations - **Risk**: Breaking Changes During Refactoring - - **Mitigation**: Implement changes incrementally and test thoroughly. + - **Mitigation**: Implement changes incrementally and test thoroughly. + - **Risk**: Incompatibility with Existing Configurations - - **Mitigation**: Provide migration guides and support legacy configurations temporarily. + - **Mitigation**: Provide migration guides and support legacy configurations temporarily. + - **Risk**: Learning Curve for Pydantic - - **Mitigation**: Provide training and resources for the development team. + - **Mitigation**: Provide training and resources for the development team. --- @@ -698,15 +720,19 @@ Konductor is an IaC platform built using Pulumi and Python, designed to streamli 1. **Week 1**: - Integrate Pydantic into the project. - Update `core/config.py` and base classes. + 2. **Week 2**: - Refactor AWS module to align with Kubernetes modules. - Remove version handling from cloud provider modules. + 3. **Week 3**: - Update Kubernetes modules to use Pydantic. - Adjust core deployment functions. + 4. **Week 4**: - Comprehensive testing and bug fixing. - Update documentation and provide training. + 5. **Week 5**: - Final review and deployment to production. - Post-deployment monitoring and support. diff --git a/pulumi/core/README.md b/pulumi/core/README.md index 2af0724..9d4cbeb 100644 --- a/pulumi/core/README.md +++ b/pulumi/core/README.md @@ -9,15 +9,17 @@ Welcome to the **Core Module** of the Kargo KubeVirt Kubernetes PaaS project! Th - [Introduction](#introduction) - [Getting Started](#getting-started) - [Core Module Overview](#core-module-overview) - - [Module Structure](#module-structure) - - [Key Components](#key-components) + - [Module Structure](#module-structure) + - [Key Components](#key-components) + - [Detailed Explanation of Core Files](#detailed-explanation-of-core-files) - - [config.py](#configpy) - - [deployment.py](#deploymentpy) - - [metadata.py](#metadatapy) - - [resource_helpers.py](#resource_helperspy) - - [types.py](#typespy) - - [utils.py](#utilspy) + - [config.py](#configpy) + - [deployment.py](#deploymentpy) + - [metadata.py](#metadatapy) + - [resource_helpers.py](#resource_helperspy) + - [types.py](#typespy) + - [utils.py](#utilspy) + - [Best Practices](#best-practices) - [Troubleshooting and FAQs](#troubleshooting-and-faqs) - [Contributing to the Core Module](#contributing-to-the-core-module) @@ -36,25 +38,32 @@ The Core Module is the heart of the Kargo KubeVirt Kubernetes PaaS project. It p If you're new to Kargo or DevOps, start here! - **Prerequisites**: - - Basic understanding of Python and Kubernetes. - - [Pulumi CLI](https://www.pulumi.com/docs/get-started/) installed. - - Access to a Kubernetes cluster (minikube, kind, or cloud-based). + + - Basic understanding of Python and Kubernetes. + - [Pulumi CLI](https://www.pulumi.com/docs/get-started/) installed. + - Access to a Kubernetes cluster (minikube, kind, or cloud-based). - **Setup Steps**: - 1. **Clone the Repository**: - ```bash - git clone https://github.com/ContainerCraft/Kargo.git - cd Kargo/pulumi - ``` - 2. **Install Dependencies**: - ```bash - pip install -r requirements.txt - ``` - 3. **Configure Pulumi**: - ```bash - pulumi login - pulumi stack init dev - ``` + +1. **Clone the Repository**: + +```bash +git clone https://github.com/ContainerCraft/Kargo.git +cd Kargo/pulumi +``` + +2. **Install Dependencies**: + +```bash +pip install -r requirements.txt +``` + +3. **Configure Pulumi**: + +```bash +pulumi login +pulumi stack init dev +``` --- @@ -138,10 +147,10 @@ deploy_module('kubevirt', init['config'], ...) - **Singleton Pattern**: Ensures a single source of truth for metadata. - **Metadata Functions**: - - `set_global_labels(labels)` - - `set_global_annotations(annotations)` - - `get_global_labels()` - - `get_global_annotations()` + - `set_global_labels(labels)` + - `set_global_annotations(annotations)` + - `get_global_labels()` + - `get_global_annotations()` **Usage Example**: diff --git a/pulumi/modules/README.md b/pulumi/modules/README.md index 93ccb67..77be9d2 100644 --- a/pulumi/modules/README.md +++ b/pulumi/modules/README.md @@ -10,19 +10,22 @@ Welcome to the **Konductor IaC Platform Engineering User Guide**. This document - [Design Principles](#design-principles) - [Code Structure](#code-structure) - [Configuration Management with Pydantic](#configuration-management-with-pydantic) - - [Why Pydantic?](#why-pydantic) - - [Integration Strategy](#integration-strategy) + - [Why Pydantic?](#why-pydantic) + - [Integration Strategy](#integration-strategy) + - [Module Development Guide](#module-development-guide) - - [1. Module Configuration](#1-module-configuration) - - [2. Defining Configuration Models](#2-defining-configuration-models) - - [3. Module Deployment Logic](#3-module-deployment-logic) - - [4. Updating `__main__.py`](#4-updating-__main__py) - - [5. Best Practices](#5-best-practices) + - [1. Module Configuration](#1-module-configuration) + - [2. Defining Configuration Models](#2-defining-configuration-models) + - [3. Module Deployment Logic](#3-module-deployment-logic) + - [4. Updating `__main__.py`](#4-updating-__main__py) + - [5. Best Practices](#5-best-practices) + - [Example Module: Cert Manager](#example-module-cert-manager) - - [Configuration Schema](#configuration-schema) - - [Configuration Model](#configuration-model) - - [Deployment Logic](#deployment-logic) - - [Integration in `__main__.py`](#integration-in-__main__py) + - [Configuration Schema](#configuration-schema) + - [Configuration Model](#configuration-model) + - [Deployment Logic](#deployment-logic) + - [Integration in `__main__.py`](#integration-in-__main__py) + - [Conclusion](#conclusion) --- @@ -47,13 +50,13 @@ Konductor is a Pulumi-based Infrastructure as Code (IaC) platform designed to st ## Code Structure -- **`__main__.py`**: The entry point of the Pulumi program. Handles global configurations, provider setup, and module deployments. +- __`__main__.py`__: The entry point of the Pulumi program. Handles global configurations, provider setup, and module deployments. - **`core/`**: Contains shared utilities and libraries, such as configuration management (`config.py`), deployment orchestration (`deployment.py`), and metadata handling (`metadata.py`). -- **`modules//`**: Each module resides in its own directory under `modules/`, containing its specific configuration models (`types.py`) and deployment logic (`deploy.py`). -- **`modules//types.py`**: Defines Pydantic models for module configurations with default values and validation logic. -- **`modules//deploy.py`**: Contains the module-specific deployment logic, taking in the validated configuration and returning relevant outputs. -- **`modules//*.py`**: Contains additional module-specific scripts or utilities, if needed. -- **`modules//README.md`**: Module-specific documentation with configuration options, features, and usage instructions. +- __`modules//`__: Each module resides in its own directory under `modules/`, containing its specific configuration models (`types.py`) and deployment logic (`deploy.py`). +- __`modules//types.py`__: Defines Pydantic models for module configurations with default values and validation logic. +- __`modules//deploy.py`__: Contains the module-specific deployment logic, taking in the validated configuration and returning relevant outputs. +- __`modules//*.py`__: Contains additional module-specific scripts or utilities, if needed. +- __`modules//README.md`__: Module-specific documentation with configuration options, features, and usage instructions. - **`requirements.txt`**: Lists the dependencies for the project, including Pydantic and cloud provider SDKs. --- @@ -80,7 +83,7 @@ Konductor is a Pulumi-based Infrastructure as Code (IaC) platform designed to st ### 1. Module Configuration - **Purpose**: Retrieve and validate the module's configuration using Pydantic models. -- **Implementation**: Use the `get_module_config` function in `core/config.py`. +- __Implementation__: Use the `get_module_config` function in `core/config.py`. ```python # core/config.py diff --git a/pulumi/modules/aws/DEVELOPER.md b/pulumi/modules/aws/DEVELOPER.md index c551e64..72aa63b 100644 --- a/pulumi/modules/aws/DEVELOPER.md +++ b/pulumi/modules/aws/DEVELOPER.md @@ -22,17 +22,19 @@ This document is intended for developers who want to contribute to the Konductor The project is organized into modular components to ensure scalability and maintainability. Below is an overview of the key components: -- **`__main__.py`**: The entry point of the Pulumi program. +- __`__main__.py`__: The entry point of the Pulumi program. - **`core/`**: Contains shared utilities and libraries. - - **`config.py`**: Handles configuration loading and validation using Pydantic. - - **`deployment.py`**: Manages deployment orchestration and module integration. - - **`metadata.py`**: Manages global metadata, labels, and annotations. - - **`utils.py`**: Provides generic utility functions. + - **`config.py`**: Handles configuration loading and validation using Pydantic. + - **`deployment.py`**: Manages deployment orchestration and module integration. + - **`metadata.py`**: Manages global metadata, labels, and annotations. + - **`utils.py`**: Provides generic utility functions. + - **`modules/`**: Contains individual modules, each in its own directory. - - **`modules//types.py`**: Defines Pydantic models for module configurations. - - **`modules//deploy.py`**: Contains module-specific deployment logic. - - **`modules//*.py`**: Contains other module-specific functions and components. - - **`modules//README.md`**: Module-specific documentation. + - __`modules//types.py`__: Defines Pydantic models for module configurations. + - __`modules//deploy.py`__: Contains module-specific deployment logic. + - __`modules//*.py`__: Contains other module-specific functions and components. + - __`modules//README.md`__: Module-specific documentation. + - **`requirements.txt`**: Lists the dependencies for the project. ### Example Directory Structure @@ -78,10 +80,10 @@ pulumi/ ### Naming Conventions -- **Files and Directories**: Use `snake_case` for file and directory names. +- __Files and Directories__: Use `snake_case` for file and directory names. - **Classes**: Use `PascalCase` for class names. -- **Variables and Functions**: Use `snake_case` for variable and function names. -- **Constants**: Use `UPPER_SNAKE_CASE` for constants. +- __Variables and Functions__: Use `snake_case` for variable and function names. +- __Constants__: Use `UPPER_SNAKE_CASE` for constants. ### Configuration Management @@ -92,15 +94,18 @@ pulumi/ ### Module Integration - **Deployment Functions**: Follow the standard function signature for deployment functions. - ```python - def deploy_( - config: ModuleNameConfig, - global_depends_on: List[pulumi.Resource], - providers: Dict[str, Any], - ) -> pulumi.Resource: - # Deployment logic - ``` -- **Configuration Loading**: Use the `get_module_config` function to load and validate configurations. + +```python +def deploy_( + config: ModuleNameConfig, + global_depends_on: List[pulumi.Resource], + providers: Dict[str, Any], +) -> pulumi.Resource: + # Deployment logic +``` + +- __Configuration Loading__: Use the `get_module_config` function to load and validate configurations. + - **Providers**: Access required providers from the `providers` dictionary passed to deployment functions. --- @@ -110,34 +115,45 @@ pulumi/ ### Fork and Clone the Repository 1. **Fork the repository** on GitHub. + 2. **Clone the forked repository** to your local machine. - ```sh - git clone https://github.com/your-username/konductor.git - cd konductor/pulumi - ``` + +```sh +git clone https://github.com/your-username/konductor.git +cd konductor/pulumi +``` ### Set Up Development Environment 1. **Create a virtual environment**: - ```sh - python -m venv venv - source venv/bin/activate - ``` + +```sh +python -m venv venv +source venv/bin/activate +``` + 2. **Install dependencies**: - ```sh - pip install -r requirements.txt - ``` + +```sh +pip install -r requirements.txt +``` + 3. **Configure Pulumi**: - ```sh - pulumi login - pulumi stack init dev - ``` + +```sh +pulumi login +pulumi stack init dev +``` + 4. **Set Up Cloud Provider Credentials**: - - For AWS: - ```sh - aws configure - ``` - - For other providers, follow their respective setup instructions. + +- For AWS: + +```sh +aws configure +``` + +- For other providers, follow their respective setup instructions. ### Create a Feature Branch @@ -155,14 +171,17 @@ git checkout -b feature/new-module ### Commit and Push 1. **Commit your changes** with clear and concise messages. - ```sh - git add . - git commit -m "Add new module: description" - ``` + +```sh +git add . +git commit -m "Add new module: description" +``` + 2. **Push to your fork**: - ```sh - git push origin feature/new-module - ``` + +```sh +git push origin feature/new-module +``` ### Create a Pull Request @@ -177,32 +196,38 @@ git checkout -b feature/new-module ### Example: Adding a New Module 1. **Create Module Directory**: - ```sh - mkdir modules/new_module - touch modules/new_module/__init__.py - ``` + +```sh +mkdir modules/new_module +touch modules/new_module/__init__.py +``` + 2. **Define Configuration Model**: - ```python - # modules/new_module/types.py - from pydantic import BaseModel +```python +# modules/new_module/types.py + +from pydantic import BaseModel + +class NewModuleConfig(BaseModel): + enabled: bool = False + # ... other configuration fields ... +``` - class NewModuleConfig(BaseModel): - enabled: bool = False - # ... other configuration fields ... - ``` 3. **Implement Deployment Logic**: - ```python - # modules/new_module/deploy.py - - def deploy_new_module( - config: NewModuleConfig, - global_depends_on: List[pulumi.Resource], - providers: Dict[str, Any], - ) -> pulumi.Resource: - # Deployment logic - ``` -4. **Update `__main__.py`**: + +```python +# modules/new_module/deploy.py + +def deploy_new_module( + config: NewModuleConfig, + global_depends_on: List[pulumi.Resource], + providers: Dict[str, Any], +) -> pulumi.Resource: + # Deployment logic +``` + +4. __Update `__main__.py`__: - Add the module to `modules_to_deploy`. - Ensure the module is correctly loaded and deployed. diff --git a/pulumi/modules/aws/ROADMAP.md b/pulumi/modules/aws/ROADMAP.md index 65f26df..9a77b14 100644 --- a/pulumi/modules/aws/ROADMAP.md +++ b/pulumi/modules/aws/ROADMAP.md @@ -1,9 +1,11 @@ # Project Roadmap: Comprehensive AWS Organization and IAM Management with Pulumi ## Introduction + This project aims to develop a comprehensive AWS Organization and IAM Management program using Pulumi and Python. It will focus on demonstrating high-quality AWS infrastructure automation setups, including AWS Organizations, Control Tower, IAM policies, roles, users, groups, and associated permissions. The primary audience for this project is senior platform engineering teams who require real-world, scalable, and modular infrastructure automation solutions. ## Goals and Objectives + - Demonstrate the setup of an AWS Organization with nested Organizational Units (OUs). - Implement AWS Control Tower for multi-account governance. - Configure and manage IAM roles, policies, users, and groups. @@ -12,20 +14,25 @@ This project aims to develop a comprehensive AWS Organization and IAM Management ## Plan of Action ### 1. Code Structure and Modules + 1. **Core Module**: Handles configuration management, deployment orchestration, metadata management, and utility functions. 2. **Infrastructure Module**: Manages AWS-specific resources, including AWS Organization, Control Tower, IAM roles, policies, users, groups, and their permissions. 3. **Utils Module**: Provides helper functions for common tasks and resource transformations. ### 2. Defining Configuration Structures + 1. **Configuration File (Pulumi.aws.yaml)**: + - Centralize all configurations, including AWS Organization details, Control Tower settings, IAM configurations, and global tags. - Use nested structures to define tenant accounts, workloads, and associated tags. 2. **Type-Safe Data Classes**: + - Define data classes for AWS configurations using Python's `dataclasses` module. - Ensure type safety and clarity by using well-defined data structures. ### 3. AWS Organization Setup + 1. **Creating AWS Organizations**: - Define a function to create AWS Organizations with all features enabled. @@ -64,10 +71,12 @@ def create_organizational_units(organization: aws.organizations.Organization, co ``` ### 4. AWS Control Tower Setup + 1. **Enabling AWS Control Tower**: - Placeholder function to enable AWS Control Tower (hypothetical until AWS provides full programmatic support). ### 5. IAM Management + 1. **Creating IAM Roles and Policies**: - Define functions to create IAM roles for Control Tower and other specific needs. - Attach necessary policies to these roles. @@ -139,6 +148,7 @@ def create_iam_resources(tenant_provider: aws.Provider, tenant_account: aws.orga ``` ### 6. Deploying Workloads in Tenant Accounts + 1. **Deploying Various Workloads**: - Define functions for deploying EKS clusters, RDS instances, Lambda functions, etc. - Utilize tenant-specific AWS Providers for these deployments. @@ -177,6 +187,7 @@ def deploy_lambda_function(tenant_provider: aws.Provider, tenant_account: aws.or ``` ### 7. Secrets Management + 1. **Creating Secrets in AWS Secrets Manager**: - Define a function to create secrets in AWS Secrets Manager for tenant accounts. @@ -206,6 +217,7 @@ def create_secrets(tenant_provider: aws.Provider, tenant_account: aws.organizati ``` ### 8. Main Execution and Resource Management + 1. **Main Execution Function**: - Define the main function to call all other functions in sequence, ensuring appropriate dependencies and order of execution. @@ -253,18 +265,25 @@ if __name__ == "__main__": ``` ### 9. Documentation and Best Practices + 1. **Comprehensive Docstrings and Comments**: + - Ensure all functions and classes are well-documented with detailed docstrings and inline comments. 2. **Developer Guide**: + - Create a detailed developer guide to explain the project structure, codebase, and contribution guidelines. ### 10. Testing and Validation + 1. **Unit Testing**: + - Write unit tests for critical functions to ensure functionality and prevent regressions. 2. **Integration Testing**: + - Deploy the infrastructure in a test environment and validate its correctness and robustness. 3. **Error Handling**: + - Implement robust error handling and logging to aid in troubleshooting and debugging. diff --git a/pulumi/modules/cert_manager/README.md b/pulumi/modules/cert_manager/README.md index e9c090e..24bb224 100644 --- a/pulumi/modules/cert_manager/README.md +++ b/pulumi/modules/cert_manager/README.md @@ -1,4 +1,5 @@ # Cert Manager Module Guide + # TODO: Convert from Kargo to generalized Konductor Framework Template repo docs content Welcome to the **Cert Manager Module** for the Konductor IaC Framework! This guide is tailored for both newcomers to DevOps and experienced developers, providing a comprehensive overview of how to deploy and configure the Cert Manager module within the Kargo platform. @@ -12,14 +13,17 @@ Welcome to the **Cert Manager Module** for the Konductor IaC Framework! This gui - [Getting Started](#getting-started) - [Enabling the Module](#enabling-the-module) - [Configuration Options](#configuration-options) - - [Default Settings](#default-settings) - - [Customizing Your Deployment](#customizing-your-deployment) + - [Default Settings](#default-settings) + - [Customizing Your Deployment](#customizing-your-deployment) + - [Module Components Explained](#module-components-explained) - - [Namespace Creation](#namespace-creation) - - [Helm Chart Deployment](#helm-chart-deployment) - - [Self-Signed Cluster Issuer Setup](#self-signed-cluster-issuer-setup) + - [Namespace Creation](#namespace-creation) + - [Helm Chart Deployment](#helm-chart-deployment) + - [Self-Signed Cluster Issuer Setup](#self-signed-cluster-issuer-setup) + - [Using the Module](#using-the-module) - - [Example Usage](#example-usage) + - [Example Usage](#example-usage) + - [Troubleshooting and FAQs](#troubleshooting-and-faqs) - [Additional Resources](#additional-resources) - [Conclusion](#conclusion) @@ -52,17 +56,22 @@ The Cert Manager module automates the management of SSL/TLS certificates in your ### Setup Steps 1. **Navigate to the Kargo Pulumi Directory**: - ```bash - cd Kargo/pulumi - ``` + +```bash +cd Kargo/pulumi +``` + 2. **Install Dependencies**: - ```bash - pip install -r requirements.txt - ``` + +```bash +pip install -r requirements.txt +``` + 3. **Initialize Pulumi Stack**: - ```bash - pulumi stack init dev - ``` + +```bash +pulumi stack init dev +``` --- @@ -95,7 +104,7 @@ pulumi config set --path cert_manager.enabled true The module is designed to work out-of-the-box with default settings: - **Namespace**: `cert-manager` -- **Version**: Defined in `default_versions.json` +- __Version__: Defined in `default_versions.json` - **Cluster Issuer Name**: `cluster-selfsigned-issuer` - **Install CRDs**: `true` @@ -108,8 +117,8 @@ You can tailor the module to fit your specific needs by customizing its configur - **enabled** *(bool)*: Enable or disable the module. - **namespace** *(string)*: Kubernetes namespace for cert-manager. - **version** *(string)*: Helm chart version to deploy. Use `'latest'` to fetch the most recent stable version. -- **cluster_issuer** *(string)*: Name of the ClusterIssuer resource. -- **install_crds** *(bool)*: Whether to install Custom Resource Definitions. +- __cluster_issuer__ _(string)_: Name of the ClusterIssuer resource. +- __install_crds__ _(bool)_: Whether to install Custom Resource Definitions. #### Example Custom Configuration diff --git a/pulumi/modules/kubevirt/README.md b/pulumi/modules/kubevirt/README.md index 2869321..12cf168 100644 --- a/pulumi/modules/kubevirt/README.md +++ b/pulumi/modules/kubevirt/README.md @@ -11,14 +11,17 @@ Welcome to the **KubeVirt Module** for the Kargo KubeVirt Kubernetes PaaS! This - [Getting Started](#getting-started) - [Enabling the Module](#enabling-the-module) - [Configuration Options](#configuration-options) - - [Default Settings](#default-settings) - - [Customizing Your Deployment](#customizing-your-deployment) + - [Default Settings](#default-settings) + - [Customizing Your Deployment](#customizing-your-deployment) + - [Module Components Explained](#module-components-explained) - - [Namespace Creation](#namespace-creation) - - [Operator Deployment](#operator-deployment) - - [Custom Resource Configuration](#custom-resource-configuration) + - [Namespace Creation](#namespace-creation) + - [Operator Deployment](#operator-deployment) + - [Custom Resource Configuration](#custom-resource-configuration) + - [Using the Module](#using-the-module) - - [Example Usage](#example-usage) + - [Example Usage](#example-usage) + - [Troubleshooting and FAQs](#troubleshooting-and-faqs) - [Additional Resources](#additional-resources) - [Conclusion](#conclusion) @@ -51,17 +54,22 @@ The KubeVirt module enables you to run virtual machines (VMs) within your Kubern ### Setup Steps 1. **Navigate to the Kargo Pulumi Directory**: - ```bash - cd Kargo/pulumi - ``` + +```bash +cd Kargo/pulumi +``` + 2. **Install Dependencies**: - ```bash - pip install -r requirements.txt - ``` + +```bash +pip install -r requirements.txt +``` + 3. **Initialize Pulumi Stack**: - ```bash - pulumi stack init dev - ``` + +```bash +pulumi stack init dev +``` --- @@ -92,7 +100,7 @@ pulumi config set --path kubevirt.enabled true ### Default Settings - **Namespace**: `kubevirt` -- **Version**: Defined in `default_versions.json` +- __Version__: Defined in `default_versions.json` - **Use Emulation**: `false` (suitable for bare-metal environments) ### Customizing Your Deployment @@ -102,7 +110,7 @@ pulumi config set --path kubevirt.enabled true - **enabled** *(bool)*: Enable or disable the module. - **namespace** *(string)*: Kubernetes namespace for KubeVirt. - **version** *(string)*: Specific version to deploy. Use `'latest'` for the most recent stable version. -- **use_emulation** *(bool)*: Enable if running in a nested virtualization environment. +- __use_emulation__ _(bool)_: Enable if running in a nested virtualization environment. - **labels** *(dict)*: Custom labels for resources. - **annotations** *(dict)*: Custom annotations for resources. @@ -144,7 +152,7 @@ Deploys the KubeVirt operator. Defines the KubeVirt CustomResource to configure KubeVirt settings. -- **Emulation Mode**: Controlled by `use_emulation`. +- __Emulation Mode__: Controlled by `use_emulation`. - **Feature Gates**: Enables additional features like `HostDevices` and `ExpandDisks`. - **SMBIOS Configuration**: Sets metadata for virtual machines. @@ -166,7 +174,7 @@ pulumi up **Q1: Virtual machines are not starting.** -- **A**: Ensure that your nodes support virtualization. If running in a VM without the `/dev/kvm` device, set `use_emulation` to `true`. +- __A__: Ensure that your nodes support virtualization. If running in a VM without the `/dev/kvm` device, set `use_emulation` to `true`. **Q2: Deployment fails with version errors.**