diff --git a/cloud_computing/cloud_computing_intro/index.md b/cloud_computing/cloud_computing_intro/index.md new file mode 100644 index 00000000..d0ae1989 --- /dev/null +++ b/cloud_computing/cloud_computing_intro/index.md @@ -0,0 +1,76 @@ +--- +name: Introduction to Cloud Computing +id: intro_to_cloud_computing +dependsOn: [ +] +files: [ +] +summary: | + This course covers the basics of cloud computing, including key concepts, service models, and major providers. + It also highlights the primary benefits of cloud computing, while addressing common challenges faced in its adoption. +--- + + +## Cloud Computing: A High-Level Overview + +Cloud computing refers to the delivery of computing services —including servers, storage, databases, networking, software, analytics, and intelligence— over the Internet ("the cloud") to offer faster innovation, flexible resources, and economies of scale. It enables organizations to access resources without the need to own or manage physical infrastructure, allowing them to innovate rapidly, adapt to changing workloads, and grow efficiently while minimizing costs. By leveraging the cloud, organizations can focus more on delivering value and less on managing IT infrastructure, enhancing their ability to compete in an increasingly digital world. + +## Key Features of Cloud Computing + +Cloud computing offers a range of key features that set it apart from traditional IT infrastructure. These features enhance flexibility, ease of access, and operational efficiency, making cloud solutions well-suited for businesses with diverse needs and varying scales. + +1. **On-Demand Self-Service:** Users can easily provision resources such as virtual servers or storage without requiring human intervention from the provider. +2. **Access Anytime, Anywhere:** Cloud services are accessible over the Internet from anywhere using standard tools like web browsers and APIs. +3. **Shared Resources:** Cloud providers pool resources to serve multiple users, dynamically allocating computing power and storage as needed. +4. **Scalability:** Resources can be scaled up or down automatically based on demand, providing agility for fluctuating workloads. +5. **Pay-as-You-Go:** Users only pay for the resources they consume, making cloud infrastructure cost efficient and avoiding the expense of over provisioning. + + +## Cloud Service Models + +Cloud computing is delivered through three primary models, each offering different levels of control and responsibility: + +1. **Software as a Service (SaaS):** SaaS provides fully managed software applications over the Internet, eliminating the need for installation or maintenance. Examples include Gmail and Microsoft Office 365. + +2. **Platform as a Service (PaaS):** PaaS offers a platform for developers to build, deploy, and manage applications without managing the underlying infrastructure. Examples include AWS Elastic Beanstalk, Microsoft Azure App Service, and Google App Engine. + +3. **Infrastructure as a Service (IaaS):** IaaS Provides virtualized computing resources like virtual machines and storage, giving users full control over operating systems and applications. Examples include AWS EC2, Microsoft Azure Virtual Machines, and Google Compute Engine. + + +## Leading Cloud Providers + +Several cloud providers offer a wide range of services to meet the diverse needs of organizations: + +1. **Amazon Web Services (AWS):** The largest and most comprehensive cloud provider, offering a wide range of services, including computing, storage, databases, and machine learning. AWS provides global infrastructure with high availability and performance. + +2. **Microsoft Azure:** A cloud platform integrated with Microsoft’s ecosystem of products, Azure provides services for building, managing, and scaling applications, as well as seamless integration with Windows, Office 365, and Azure Active Directory. + +3. **Google Cloud Platform (GCP):** Known for its strengths in data analytics, AI, and machine learning, GCP offers reliable cloud services through Google's global network. GCP is ideal for organizations looking to leverage cutting-edge technologies. + +## Benefits of Cloud Computing + +Cloud computing provides many significant advantages that allow organizations to stay agile, reduce costs, and scale their operations efficiently. These benefits make cloud technology a compelling choice for businesses of all sizes. + +1. **Cost Savings:** Organizations no longer need to invest in and maintain physical infrastructure, reducing both capital expenditure (CapEx) and operational expenditure (OpEx), while benefiting from a flexible pay-as-you-go model. +2. **Scalability and Flexibility:** Cloud services can scale seamlessly to accommodate growing workloads, providing businesses with the agility to meet changing demands. +3. **Global Accessibility:** With cloud services available from anywhere, businesses can access resources across geographic locations. +4. **High Reliability:** Cloud providers use redundant infrastructure with built-in fault tolerance that automatically takes over in case of failure, ensuring smooth operation and minimizing downtime. This guarantees reliable performance and continuous availability of services. +5. **Faster Innovation:** The cloud empowers businesses to test new ideas and roll out services more quickly by providing immediate access to resources. + + +## Challenges of Cloud Computing + +While cloud computing offers numerous benefits, there are also challenges that organizations must consider. These challenges can impact decision-making and require careful planning to mitigate risks. + +1. **Vendor Lock-In:** Once integrated with a cloud provider’s services, it can be challenging to switch providers due to proprietary technologies and costs. +2. **Security and Privacy Concerns:** Storing data off-premises introduces security risks, and businesses must ensure that robust data protection policies are in place. +3. **Downtime and Outages:** Despite high availability, cloud services can still face outages or disruptions, which could affect business operations. +4. **Limited Control:** While cloud services offer flexibility, users have less control over the infrastructure, and customization options may be limited. + +## Conclusion and Next Steps + +Cloud computing is transforming industries by providing scalable, cost-effective, and globally accessible resources. While the cloud offers significant benefits, such as reduced costs and faster innovation, businesses must also be mindful of challenges like vendor lock-in and security concerns. By understanding both the advantages and potential drawbacks, organizations can make informed decisions and fully harness the power of cloud technology. + +Next, we will dive into hands-on exercises where you'll apply these concepts and gain practical experience with real-world cloud solutions. + + diff --git a/cloud_computing/index.md b/cloud_computing/index.md new file mode 100644 index 00000000..7e79b084 --- /dev/null +++ b/cloud_computing/index.md @@ -0,0 +1,18 @@ +--- +id: cloud_computing +name: Cloud Computing +dependsOn: [ + introductory_courses.python +] +courses: [ + cloud_computing_intro, + migrating_app_to_cloud +] +summary: | + + An introduction to cloud computing, exploring foundational concepts and the offerings of major providers, with a focus on practical applications for compute, storage, and databases. Additionally, a hands-on course focused on migrating an on-premises application to the cloud, outlining the essential steps and considerations. + +--- + + An introduction to cloud computing, exploring foundational concepts and the offerings of major providers, with a focus on practical applications for compute, storage, and databases. Additionally, a hands-on course focused on migrating an on-premises application to the cloud, outlining the essential steps and considerations. + diff --git a/cloud_computing/migrating_app_to_cloud/app_local_storage.md b/cloud_computing/migrating_app_to_cloud/app_local_storage.md new file mode 100644 index 00000000..c516a4ac --- /dev/null +++ b/cloud_computing/migrating_app_to_cloud/app_local_storage.md @@ -0,0 +1,139 @@ +--- +name: Running the Image Processing Application Locally +dependsOn: [] +tags: [] +learningOutcomes: +--- + +This image processing application is designed to store and process images using local storage, with a SQLite database specifically for metadata management. It provides an excellent introduction to handling file uploads, processing images, and organizing metadata effectively. + +### Prerequisites + +Before getting started with the image processing application, ensure that the following software is installed on your system: + +1. **Git:** Required to clone the repository via command line from GitHub (optional if downloading the repository from the GitHub web interface as a ZIP file). [Download Git](https://git-scm.com/downloads) +2. **Python:** The application is built and tested with Python 3.11; therefore, it is recommended to use this version. However, it should also be compatible with the latest versions of Python. [Download Python](https://www.python.org/downloads/) +3. **Pip:** Required to install the Python dependencies. Pip typically comes bundled with Python, but make sure it's installed and up to date (`pip --version`). [Pip Installation Guide](https://pip.pypa.io/en/stable/installation/) +4. **Docker:** If you prefer to run the application inside a Docker container. [Download Docker](https://www.docker.com/products/docker-desktop/) +5. **IDE:** You can use any IDE to edit and navigate the code, but **Visual Studio Code** is recommended for its flexibility and extensive range of helpful extensions (e.g., Docker) that enhance development and troubleshooting. [Download Visual Studio Code](https://code.visualstudio.com/) + + +### Cloning the Application from GitHub + +To get started with the image processing application, follow these steps to clone the repository and set it up locally: + +1. Clone the repository from GitHub: +```bash +git clone https://github.com/Oxford-Research-Cloud-Competency-Centre/image-processing-app.git +``` + +2. Navigate to the project directory: +```bash +cd image-processing-app +``` + +After cloning the repository, open the project in your preferred IDE. Take some time to explore the project structure and get familiar with each component. The next section will provide an overview of the application, and afterward, we'll dive into the details of running it, covering each part of the setup and execution process. + + +### Application Overview + +The application consists of three main components: + +1. **`app.py`** - Manages the Flask web interface, providing endpoints for file uploads and image downloads, while handling image processing and metadata storage. +2. **`image_processor.py`** - Processes images by converting them to grayscale and resizing them. +3. **`database.py`** - Manages the SQLite database, storing metadata for original and processed images. + +These components work together to create a cohesive image processing system. + + +### Core Components of the Image Processing Application + +We will now discuss these components in detail, outlining their roles, and how they work together to form the complete image processing application. + +#### 1. **`app.py`: Flask Application** + +The central component, `app.py`, handles the core functionality of the web interface, routing, and interactions with both the image processing module and the database. It manages the following: + +- **File Uploads**: Users upload images through the web interface, and these images are saved in the `images/uploads` directory. The images are stored directly on the system running the application, ensuring that the raw files are accessible for processing. +- **Image Processing**: The component calls the `process_image()` function from `image_processor.py` to process images. +- **Directory Management**: The component ensures the `images/uploads` and `images/processed` directories exist, creating them if necessary. +- **Routing**: Manages routes for displaying metadata, handling uploads, and downloading processed images: + - `/`: Displays a dashboard with image metadata. + - `/upload`: Handles the image upload and processing. + - `/processed/`: Enables downloading of processed images. + +#### 2. **`image_processor.py`: Image Processing** + +This module handles the image transformation, performing two operations: + +- **Grayscale Conversion**: Converts the uploaded image to grayscale using [OpenCV](https://opencv.org/). +- **Resizing**: Resizes the grayscale image to 300x300 pixels. + +The processed images are stored in the `processed` directory. + +#### 3. `database.py`: SQLite Database Management + +The `database.py` module handles interactions with the SQLite database, which stores essential metadata for each uploaded and processed image. Key metadata fields include original and processed filenames, image sizes, upload dates, and optional descriptions, providing a comprehensive record of each image. + +**Main functions in `database.py`:** + +- **`init_db()`**: Initializes the database by creating the `ImageMetadata` table if it doesn't already exist. The `metadata.db` file is automatically generated in the project directory upon initialization. +- **`insert_metadata()`**: Adds metadata for both original and processed images, supporting detailed tracking. +- **`get_metadata()`**: Retrieves stored metadata for display on the application's dashboard. + +Once an image is processed, `app.py` saves it in the `images/processed` directory on the system. Both `uploads` and `processed` directories are automatically created when the application runs, if they don't already exist. + +By managing and retrieving image metadata efficiently, `database.py` enables the application to track details like: + +- Filenames for original and processed images. +- Image sizes. +- Upload date and time. +- Optional descriptions. + +This setup ensures efficient metadata storage and retrieval, contributing to the application's smooth functionality and user experience. + + +### Running the Application + +The application can be run in two different ways, depending on your setup: + +#### 1. **Running Locally with Python:** + +To run the application locally on your system: + +1. Make sure the repository is cloned and you are inside the home directory of the application. +2. Install the necessary Python dependencies: + ```bash + pip install -r requirements.txt + ``` +3. Run the Flask application: + ```bash + python app.py + ``` + +The application will be accessible at http://localhost:5000 + +#### 2. **Running with Docker:** + +Alternatively, you can run the application inside a Docker container: + +1. Build the Docker image: + ```bash + docker build -t image-processing-app . + ``` +2. Run the Docker container: + ```bash + docker run -p 5000:5000 image-processing-app + ``` +This will make the application available at http://localhost:5000, without the need to install Python or any other dependencies directly on your machine. + + +### Application Workflow + +1. **Image Upload:** Users upload images via a form on the main page. These images are saved in the `uploads` directory. +2. **Image Processing:** After an image is uploaded, the `image_processor.py` module converts it to grayscale and resizes it. The processed image is stored in the `processed` directory. +3. **Metadata Management:** Metadata for both the original and processed images—such as filename, size, and upload date—is stored in the SQLite database. This information is viewable on the application's dashboard. +4. **Download Processed Images:** Users can download the processed images directly from the dashboard. + + +This locally hosted image processing application provides an introduction to file uploads, image processing, and database management. It integrates **Flask** for web routing and user interaction, **OpenCV** for image processing, and **SQLite** for managing image metadata. In later sections of this course, we will expand the application by transitioning from local storage and databases to AWS services such as **S3** and **DynamoDB**, creating a scalable, cloud-native solution. diff --git a/cloud_computing/migrating_app_to_cloud/app_migrate_to_aws.md b/cloud_computing/migrating_app_to_cloud/app_migrate_to_aws.md new file mode 100644 index 00000000..952f5dbb --- /dev/null +++ b/cloud_computing/migrating_app_to_cloud/app_migrate_to_aws.md @@ -0,0 +1,437 @@ +--- +name: Application Migration to AWS with Elastic Beanstalk +dependsOn: [] +tags: [] +learningOutcomes: +--- + +To overcome the limitations of running a local application with cloud services, we will migrate the entire image processing application to AWS, where it will be deployed and managed using Elastic Beanstalk. Migrating the application to AWS allows you to fully leverage cloud scalability, performance, and automation. Elastic Beanstalk takes care of provisioning the required infrastructure and scaling your application as needed. + +### Prerequisites + +In addition to the prerequisites covered in **Integrating AWS Services with the Local Application** (e.g., Git, Python, Pip, Docker, AWS CLI), you will also need to install and configure the **AWS Elastic Beanstalk CLI** (`awsebcli`). + +> **Note:** Before setting up Elastic Beanstalk, make sure that the **AWS CLI** is installed and configured with valid credentials. This includes your **AWS Access Key ID**, **AWS Secret Access Key**, and **Default region name**. These credentials are essential for interacting with AWS services, including Elastic Beanstalk. + + +#### Install the AWS Elastic Beanstalk CLI + +Once the AWS CLI is configured, you can proceed to install the AWS Elastic Beanstalk (EB) Command Line Interface (CLI), which allows you to deploy and manage applications in Elastic Beanstalk using terminal commands. + +Follow the instructions in the [official AWS Elastic Beanstalk CLI installation guide](https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/eb-cli3-install.html) to install the EB CLI for your specific operating system. + +Once the AWS EB CLI is installed, you can confirm the installation by running the following command in the terminal: + +```bash +eb --version +``` + +This command should display the version of the EB CLI, indicating that the installation was successful. + + +### Setting Up AWS Resources + +Before cloning and running the application, you need to create an S3 bucket and a DynamoDB table on AWS, as the application will use these resources for storage and metadata management. + +#### 1. **Creating an S3 Bucket:** + +The S3 bucket will be used to store the images uploaded by the users and processed by the application. + +1. Open the AWS Management Console and navigate to the [S3 service](https://s3.console.aws.amazon.com/s3/home). Please make sure you are in the correct AWS region. You can select the region by clicking on the region dropdown in the top right corner of the console. Refer to the image below, where the region selection is highlighted with a red box for guidance on changing the region + + ![AWS Region Selection](fig/S3_Region.jpg) + +2. Click on **Create bucket**. Refer to the image below if you need guidance, where the "Create bucket" option is highlighted with a red box: + + ![Where is S3 Create Button](fig/Where_S3_Create_Button.jpg) + +3. Provide a unique bucket name (e.g., `ar-image-processing-bucket`). It's recommended to include your initials or another unique identifier in the bucket name to avoid naming conflicts. Refer to the image below, where the field for entering the bucket name is highlighted with a red box: + + ![S3 Bucket Name Input](fig/Create_S3_Bucket.jpg) + +4. Leave the other settings as default unless you have specific requirements. However, it's important to understand and consider the following options: + +#### Object Ownership: + +- **Object Ownership** defines the access control behavior for objects uploaded to the bucket. By default, Amazon disables Access Control Lists (ACLs), which is the recommended approach for most use cases as it simplifies permissions management. Disabling ACLs ensures that the bucket owner automatically owns all objects, even if they are uploaded by different AWS accounts. + +#### Block Public Access: + +- **Block Public Access** settings are critical for securing your bucket. By default, *Block all public access* is selected, which is highly recommended unless you need to expose objects publicly. This setting prevents any accidental exposure of sensitive data stored in the bucket. + +#### Bucket Versioning: + +- **Bucket Versioning** is disabled by default. Enabling versioning allows you to retain multiple versions of an object, which is useful for backup and recovery scenarios. If versioning is important to your use case, you can enable it after creating the bucket. + +#### Default Encryption: + +- **Server-Side Encryption (SSE)** is automatically applied to new objects stored in the bucket. By default, **SSE-S3** (Server-Side Encryption with Amazon S3 managed keys) is selected. This ensures that your data is encrypted at rest without requiring you to manage encryption keys. + + You also have the option to choose: + + - **SSE-KMS**: Server-side encryption with AWS Key Management Service (KMS) keys, which allows you to have more control over encryption keys. + - **DSSE-KMS**: Dual-layer encryption using AWS KMS keys for even greater security. This option applies two separate layers of encryption to your data, although it might incur additional costs. + +#### Bucket Key: + +- If you select **SSE-KMS**, you will also have the option to enable the **S3 Bucket Key**, which reduces the cost of encryption by minimizing calls to AWS KMS. Enabling S3 Bucket Keys can optimize costs when encrypting large volumes of data. This option is automatically enabled by default but can be disabled if necessary. Note that it is not supported for DSSE-KMS. + +5. Click **Create bucket** once you're comfortable with these settings. Refer to the image below for guidance, where the "Create bucket" button is highlighted with a red box: + + ![AWS Create Bucket Confirmation](fig/S3_Create_Button_Confirm.jpg) + +#### 2. **Creating a DynamoDB Table:** + +The DynamoDB table will be used to store metadata related to the images. + +1. Open the AWS Management Console and navigate to the [DynamoDB service](https://console.aws.amazon.com/dynamodb/home). Please make sure you are in the correct AWS region. You can select the region by clicking on the region dropdown in the top right corner of the console. Refer to the image below, where the region selection is highlighted with a red box for guidance on changing the region. + + ![DynamoDB Region Selection](fig/DynamoDB_Region.jpg) + +2. In the left panel, click on **Tables**. This will display the tables panel on the right-hand side, where you'll see a **Create table** button. Click on **Create table** to begin setting up your new table. Refer to the images below for guidance, with each relevant section highlighted in a red box: + + - ![DynamoDB Tables Panel](fig/DynamoDB_Where_Tables.jpg) + - ![DynamoDB Create Table Button](fig/DynamoDB_Create_Table_Button.jpg) + +3. Provide a table name (e.g., `ar-ImageMetadata`) and set the **Partition key** to `id` (String). It's recommended to include your initials or another unique identifier in the table name to avoid naming conflicts. Refer to the image below for guidance, where the table name and partition key fields are highlighted with a red box: + + ![DynamoDB Table Name and Partition Key Setup](fig/DynamDB_Table_Details.jpg) + +4. You can leave the other settings as default unless you need advanced configuration. +5. Click **Create table** to finalize the setup. Refer to the image below for guidance, where the "Create table" button is highlighted with a red box: + + ![DynamoDB Create Table Confirmation](fig/DynamoDB_Create_Table_Confirm.jpg) + + +Once these resources are set up, you can proceed to clone the application and configure it to use the S3 bucket and DynamoDB table. + + +### Cloning the Application from GitHub + +To get started with the image processing application, follow these steps to clone the repository and set it up locally: + +1. Clone the repository from GitHub: +```bash +git clone https://github.com/Oxford-Research-Cloud-Competency-Centre/image-processing-cloud-app +``` + +2. Navigate to the project directory: +```bash +cd image-processing-cloud-app +``` + + +Once cloned, and before running the application, you need to create a `.env` file in the root directory of the project. This file will store environment variables such as the AWS region, S3 bucket name, and DynamoDB table name. These variables will be used by the application to connect to the appropriate AWS resources and manage image storage and metadata effectively. Below are the required entries for the `.env` file: + + +```plaintext +REGION_NAME = 'YOUR_AWS_REGION' +S3_BUCKET = 'YOUR_AWS_S3_BUCKET_NAME' +IMAGE_METADATA_TABLE = 'YOUR_AWS_DYNAMODB_TABLE_NAME' +``` + +> **Note:** Please replace the values above with those specific to your AWS resources. Ensure you use the correct region, S3 bucket name, and DynamoDB table name that you created. + + +### Deploying the Application to AWS Elastic Beanstalk + +After cloning the repository and navigating to the project directory, follow these steps to deploy the application using Elastic Beanstalk, leveraging the existing configuration files. + +### Step 1: Verify Configuration Files + +When deploying a Flask application to AWS Elastic Beanstalk, you need certain configuration files to set up the environment correctly and to define how AWS should handle the application and platform settings. Two essential files are **`flask.conf`** in `.ebextensions/` and **`config.yml`** in `.elasticbeanstalk/`. + +1. **`flask.conf` in `.ebextensions/`** + +The `.ebextensions` folder contains configuration files that AWS Elastic Beanstalk reads during deployment to customize the environment. The `flask.conf` file specifically configures the environment for running a Flask application on Elastic Beanstalk. + +**Contents of `flask.conf`:** +```yaml +option_settings: + aws:elasticbeanstalk:application:environment: + PYTHONPATH: "/var/app/current:$PYTHONPATH" + aws:elasticbeanstalk:container:python: + WSGIPath: "app.app:app" +``` + +- **`option_settings`**: Specifies configuration options for Elastic Beanstalk. + +- **`aws:elasticbeanstalk:application:environment:`**: Sets environment variables for the application. Here, the `PYTHONPATH` is set to include `/var/app/current`, ensuring that the Python environment includes the application's directory, which is essential for importing modules correctly. + +- **`aws:elasticbeanstalk:container:python:`**: Configures the Python container. The `WSGIPath` option defines the entry point for the Flask application. In this case, `"app.app:app"` points to a Python file and a Flask application instance named `app` inside the `app.py` file in the `app` directory. This setting tells the web server where to find the application's entry point, allowing it to serve HTTP requests to the Flask app. + +2. `config.yml` in `.elasticbeanstalk/` + +The `config.yml` file within `.elasticbeanstalk/` contains settings for Elastic Beanstalk's CLI, including the application name, region, default platform, and EC2 settings. This file is essential for redeployments, ensuring consistent environment and platform settings. + +**Contents of `config.yml`:** +```yaml +branch-defaults: + default: + environment: null + group_suffix: null +environment-defaults: + Image-processing-cloud-app-env: + branch: null + repository: null +global: + application_name: image-processing-cloud-app + default_ec2_keyname: ar-fsl-keypair + default_platform: Python 3.11 running on 64bit Amazon Linux 2023 + default_region: eu-west-2 + include_git_submodules: true + instance_profile: null + platform_name: null + platform_version: null + profile: null + sc: null + workspace_type: Application +``` + +- **`branch-defaults`**: Defines default settings for any branches in version control systems (e.g., Git). Setting these to `null` here means there are no specific branch configurations. + +- **`environment-defaults`**: Holds configurations for specific environment names. In this example, `Image-processing-cloud-app-env` is set, which is the name of the environment Elastic Beanstalk will deploy. + +- **`global`**: Contains general settings for the application: + - **`application_name`**: Specifies the name of the Elastic Beanstalk application (e.g., `image-processing-cloud-app`). This helps identify the app in the AWS Console. + - **`default_ec2_keyname`**: The name of the EC2 key pair used for SSH access to application instances, here set to `ar-fsl-keypair`. + - **`default_platform`**: Defines the platform and runtime environment (Python 3.11 on Amazon Linux 2023), ensuring compatibility with the Flask app. + - **`default_region`**: Sets the AWS region (e.g., `eu-west-2` for London), determining the location of resources. + - **`include_git_submodules`**: When set to `true`, Elastic Beanstalk includes any Git submodules, which is helpful if the application has dependencies managed as submodules. + - **`workspace_type`**: Indicates the type of workspace, in this case, `Application`, meaning it's an Elastic Beanstalk application workspace. + +Each of these configurations ensures the application is deployed and managed predictably on Elastic Beanstalk, allowing for easier troubleshooting and a standardized setup. + +### Step 1.5: Create and use your own EC2 key + +Create an EC2 key and use it to replace **ar-fsl-keypair** in **config.yml** + +```yaml +global: + application_name: image-processing-cloud-app + default_ec2_keyname: ar-fsl-keypair +``` + +![Creating an EC2 key](fig/Create_EC2_Key.png) + +Choose key name and settings + +![Creating an EC2 key](fig/Create_EC2_Key_Part2.png) + +The key file will be downloaded when you press "Create key pair" + +### Step 2: Initialize the Elastic Beanstalk Application + +With your `.elasticbeanstalk/config.yml` file already configured, initializing the application is straightforward. Follow these steps: + +1. Open your terminal and navigate to your project directory. +2. Run the following command to initialize the Elastic Beanstalk application: + + ```bash + eb init + ``` + +During this step, Elastic Beanstalk reads the configuration settings from `config.yml` and sets up the necessary application properties. You may be prompted to confirm some details or make minor adjustments, but no additional setup is needed. + +To verify the application creation, go to the [AWS Elastic Beanstalk Console](https://console.aws.amazon.com/elasticbeanstalk). Click on **Applications** in the left panel to see the list of applications. This will display the application you have just initialized. Refer to the image below, where **Applications** is highlighted in red for guidance: + +![Elastic Beanstalk Applications](fig/EB_Applications.jpg) + +> **Note:** This step only initializes the application. It does not yet create the environment or deploy the application. + +### Step 3: Create the Environment and Deploy the Application + +Now, you will create the environment and perform the initial deployment of your application: + +Run the following command to create a new environment and deploy your application: + +```bash +eb create image-processing-cloud-app-env +``` + +This command will: + +- Set up the necessary AWS resources based on your configuration, including: + - **S3 bucket** for storing application versions. + - **Security groups** for managing access to resources. + - **CloudWatch logs** for monitoring and logging application activity. + - **EC2 instance** to host the application. + - **Load balancer** to distribute incoming traffic across multiple instances (if applicable). + +> ❗ **Error:** If your account cannot use Launch Configurations, an error message may appear at this stage. +> You can modify the command like this to force usage of Launch Templates instead: +> +> ```bash +> eb create image-processing-cloud-app-env --enable-spot +> ``` +> The AWS Spot Instances feature allows users to take advantage of unused EC2 capacity at significantly reduced costs compared to On-Demand prices. However, Spot Instances can be interrupted by AWS with little notice when the capacity is reclaimed. +> +> Using AWS Spot Instances with Launch Templates solves the issue faced by new AWS accounts, which are restricted from using the older Launch Configurations. AWS has deprecated Launch Configurations in favor of Launch Templates to enable more advanced features and flexibility. New accounts are often provisioned under policies that enforce modern practices, requiring Launch Templates for any EC2 instance configuration. + +This process can take a few minutes, during which Elastic Beanstalk will provision these resources and configure the environment. Once the environment is fully set up and the application is deployed, you will see a confirmation message on the console similar to: + +`2024-11-05 10:19:21 INFO Successfully launched environment: YOUR_ENVIRONMENT_NAME` + +This message indicates that the environment and application are now live, and your setup is complete. This process is the initial deployment, where Elastic Beanstalk sets up and deploys your application code for the first time. + +To verify the environment creation, go to the [AWS Elastic Beanstalk Console](https://console.aws.amazon.com/elasticbeanstalk). Click on **Environments** in the left panel to see the list of environments, as highlighted in red in the image below. + +![Elastic Beanstalk Environments](fig/EB_Environments.jpg) + +This view shows details such as: +- **Environment name**: The name of your environment. +- **Health status**: Displayed in green with **"OK,"** indicating that the environment is running smoothly. +- **Application name**: Associated with the environment. +- **Platform**: The platform on which the application runs. +- **Platform state**: Displays as **"Supported"** in green, meaning the platform version is actively maintained and will continue to receive updates and security patches from AWS. +- **Domain**: The unique domain endpoint for accessing the environment. +- **Running version**: The current deployed version of your application. +- **Tier name**: Typically set as "Web Server" for web applications. + +![Elastic Beanstalk Environment Details](fig/EB_Environment_Details.jpg) + +These details help you monitor and confirm that your environment is correctly configured and live are are shown in the image below for your reference. + +For any future updates to your application code, use the following command: + +```bash +eb deploy +``` + +The `eb deploy` command pushes changes to your existing environment without recreating resources, allowing you to update your application efficiently. + + +### Step 4: Monitor and Manage the Application + +Elastic Beanstalk provides several commands to help you monitor and manage your application after deployment. Use the following commands to keep track of its status, troubleshoot issues, and access it directly. + +- **To check the application's status:** + + ```bash + eb status + ``` + +This command provides real-time information on the health and status of your Elastic Beanstalk environment, including details about the environment state, health, and any recent events. Use it regularly to ensure your application is running smoothly. If you prefer using the AWS console, you can also check the environment status in the [AWS Elastic Beanstalk Console](https://console.aws.amazon.com/elasticbeanstalk). Go to **Environments**, select your environment, and view the status and health details displayed on the overview page. + +- **To view logs and troubleshoot:** + + ```bash + eb logs + ``` + + This command retrieves log files from the EC2 instances in your environment. Logs can help you troubleshoot errors, review access patterns, and monitor performance metrics. For more detailed information, use `eb logs --all` to pull logs from all instances or `eb logs --zip` to download them in a compressed format for offline analysis. + + +- **To open the application in a browser:** + +There are two ways to access your application: + +1. **Directly from the Elastic Beanstalk Console**: Go to your application in the [AWS Elastic Beanstalk Console](https://console.aws.amazon.com/elasticbeanstalk) and click on the **Domain** link, highlighted in red in the image below. This link takes you directly to the URL of your environment. + + ![Elastic Beanstalk Domain Link](fig/Eb_Application_Domain.jpg) + +2. **Using the EB CLI**: Run the following command in your console: + + ```bash + eb open + ``` + +This command opens your application in the default web browser, taking you directly to the URL of your environment. It's a quick way to verify that the application is accessible and running as expected. + +After accessing your application, you can start using it. The application allows you to upload images, which are stored in the S3 bucket you created for this setup. Additionally, metadata associated with each image (e.g., filename, size, upload date) is saved in your DynamoDB table. This setup ensures that both the images and their metadata are stored securely and efficiently, enabling you to manage and retrieve image data through your application. + + +### Step 5: Terminate the Environment and Application + +Once you have finished managing and monitoring your application, you must terminate the environment and remove the application completely. Follow these steps: + +- **To terminate the environment:** + +This will remove the running environment, including EC2 instances and other associated resources: + +```bash +eb terminate +``` + +When you run this command, Elastic Beanstalk will prompt you to confirm the termination by typing the environment name. This step ensures the termination is intentional, as it will delete all associated resources for that environment. This process takes a few minutes to complete. + +> **Note:** Terminating the environment does not delete the application itself. To fully remove the application, follow these steps in the AWS Console: + +1. Go to the [AWS Elastic Beanstalk Console](https://console.aws.amazon.com/elasticbeanstalk). +2. In the left navigation panel, click on **Applications** to see a list of applications. +3. Select the application you wish to delete. +4. Click on **Actions** and choose **Delete Application**. +5. Confirm the deletion by typing the application name when prompted. + +This process will permanently remove the application and all associated data in Elastic Beanstalk. + +- **To terminate the environment and delete the application:** + +This command will terminate all environments and delete the application along with associated versions and configurations: + +```bash +eb terminate --all +``` + +The `--all` flag ensures that all related resources, including environments, saved configurations, and the application itself, are fully removed. + +> **Note:** The `--all` flag also removes the `.elasticbeanstalk` directory, which contains your `config.yml` file. Be cautious when using this option, as any local configuration settings in `.elasticbeanstalk/config.yml` will be deleted, requiring reinitialization if you decide to recreate the application later. + +### Step 6: Resource Cleanup + +Once you have terminated your Elastic Beanstalk environment, it's important to clean up any resources you manually created, such as the S3 bucket and DynamoDB table, as well as resources Elastic Beanstalk created to store application versions. This helps you avoid unnecessary costs. Follow the steps below to safely delete these resources. + +1. **Delete the S3 Bucket (Application Storage)**: + - Go to the [S3 console](https://s3.console.aws.amazon.com/s3/home) and locate the bucket you created specifically for storing images. + - Select the bucket, then click on **Empty** to delete all objects within it. Refer to the image below, where the **Empty** button is highlighted in red for guidance: + + ![S3 Empty Bucket](fig/S3_Empty.jpg) + + - After clicking **Empty**, a new page will open. To confirm, type "permanently delete" in the text input field, then click on **Empty** again to finalize. Refer to the image below for guidance, where the **confirmation field** and **Empty** button are highlighted in red: + + ![S3 Confirm Empty](fig/S3_Confirm_Empty.jpg) + + - After emptying the bucket, return to the bucket's main page and click on **Delete** to remove the bucket itself. Refer to the image below, where the **Delete** button is highlighted in red: + + ![S3 Delete Bucket](fig/S3_Delete.jpg) + + - A new window will open. To confirm the bucket deletion, enter the bucket name in the text input field, then click on **Delete Bucket**. Refer to the image below, where the **confirmation field** and **Delete Bucket** button are highlighted in red: + + ![S3 Confirm Delete Bucket](fig/S3_Confirm_Delete.jpg) + + - **Please Note:** It may take a few moments for AWS to fully delete the bucket. Wait for a confirmation message indicating that the bucket has been deleted to ensure the resource is fully removed. + +2. **Delete the Elastic Beanstalk S3 Bucket (Application Versions)**: + Elastic Beanstalk creates an additional S3 bucket (with a name starting with `elasticbeanstalk--`) to store application versions, logs, and other deployment artifacts. The `eb terminate` command does not delete this bucket automatically. + + > **Why is this bucket not deleted automatically?** + > Elastic Beanstalk keeps this bucket as a safeguard against accidental data loss. This bucket stores valuable information, such as application versions and logs, which can be essential for tasks like troubleshooting, re-deployment, or audit purposes. By retaining this bucket, AWS allows users to access and manage these artifacts independently, deciding if and when to delete them based on their needs. + + To delete this bucket manually: + + 1. Open the [Amazon S3 console](https://s3.console.aws.amazon.com/s3/home). + 2. Locate the bucket that begins with `elasticbeanstalk--`. + 3. Follow the same steps as above to **Empty** and **Delete** the bucket, ensuring all deployment versions and logs are permanently removed. + + This step completes the cleanup of all Elastic Beanstalk-related resources. + +3. **Delete the DynamoDB Table**: + - Navigate to the [DynamoDB console](https://console.aws.amazon.com/dynamodb/home). + - In the **Tables** section, locate the table you created (e.g., `ar-ImageMetadata`). + - Select the table, then click on **Delete table** to remove it. Refer to the image below, where the **Delete table** button is highlighted in red: + + ![DynamoDB Delete Table](fig/DynamoDB_Delete.jpg) + + - After clicking **Delete table**, a new window will open asking for confirmation. Type **confirm** in the text input field, then click on the **Delete** button to finalize the deletion. Refer to the image below, where the **confirmation field** and **Delete** button are highlighted in red: + + ![DynamoDB Confirm Delete Table](fig/DynamoDB_Confirm_Delete.jpg) + + - **Please Note:** It may take a few moments for AWS to fully delete the table. Wait until you see a confirmation message indicating that the table has been deleted to ensure it is completely removed. + +> **Important:** Be sure to back up any necessary data from these resources before deleting them, as this action is irreversible. + + + + + + diff --git a/cloud_computing/migrating_app_to_cloud/app_onprem_limitation.md b/cloud_computing/migrating_app_to_cloud/app_onprem_limitation.md new file mode 100644 index 00000000..285987b5 --- /dev/null +++ b/cloud_computing/migrating_app_to_cloud/app_onprem_limitation.md @@ -0,0 +1,46 @@ +--- +name: Drawbacks of Running a Local Application with Remote AWS Services +dependsOn: [] +tags: [] +learningOutcomes: +--- + +While integrating AWS services like S3 and DynamoDB with a locally running application provides access to the power of the cloud, there are several significant drawbacks that can impact performance, reliability, and scalability: + +#### 1. **Increased Latency** + +When running the application locally, every request to cloud-based services like S3 (for storage) or DynamoDB (for databases) must travel over the Internet. This introduces **network latency**, which can significantly slow down response times, especially if the cloud resources are hosted in regions far from the local machine. As the application grows in complexity and the number of interactions with AWS services increases, this latency can become more noticeable and degrade user experience. + +#### 2. **Dependence on Internet Connectivity** + +The local application depends entirely on a **stable and fast Internet connection** to communicate with AWS services. Any disruption in connectivity, slow network speeds, or outages can cause the application to fail to access essential resources like files in S3 or data in DynamoDB. This dependency can lead to downtime or inconsistent behavior, which is particularly problematic for production environments or critical applications where reliability is key. + +#### 3. **Limited Scalability** + +While AWS services like S3 and DynamoDB are highly scalable by design, the **local machine** running the application is not. As your application grows, whether in the number of users or the volume of data processed, the local environment may struggle to keep up. You are constrained by the processing power, memory, and storage limitations of your local hardware. This lack of scalability can result in performance bottlenecks and reduce the overall efficiency of your application. + +#### 4. **Management Overhead** + +Running an application locally means that you are responsible for maintaining the runtime environment, dependencies, and configurations. Keeping your local machine up-to-date with the necessary libraries and security patches adds to the **operational burden**, especially as the project evolves or becomes more complex. + +#### 5. **Lack of Automatic Failover and High Availability** + +When running an application locally, you miss out on the **high availability** and **automatic failover** features offered by cloud platforms. If your local machine crashes or encounters hardware issues, the application will be completely unavailable until the problem is resolved. In contrast, AWS provides built-in redundancy and failover mechanisms to ensure continuous availability of services. + +### Motivation for Migrating to AWS Elastic Beanstalk + +To overcome the challenges of running applications locally, AWS offers **Elastic Beanstalk**, a fully managed service that simplifies the process of deploying, managing, and scaling web applications and services. Elastic Beanstalk handles the infrastructure provisioning, load balancing, auto-scaling, and monitoring, allowing developers to focus on writing code without worrying about the underlying infrastructure. + +Elastic Beanstalk offers several key benefits: + +- **Reduced Latency**: By running the application in the same cloud environment as your AWS services (such as S3 and DynamoDB), the latency is greatly reduced since communication happens within the AWS network, resulting in faster performance. + +- **Improved Reliability**: Elastic Beanstalk automatically provisions the necessary resources in a cloud environment, removing the dependency on local hardware and internet connectivity, and ensuring the application is always accessible. + +- **Effortless Scalability**: Elastic Beanstalk automatically handles scaling your application based on demand. Whether you need more processing power or memory, it adjusts the infrastructure to meet your needs, without requiring manual intervention. + +- **Simplified Management**: With Elastic Beanstalk, AWS manages much of the underlying infrastructure, including patching and updates, so you can focus on developing your application without worrying about server maintenance. + +- **Built-in High Availability**: Elastic Beanstalk offers automatic failover and high availability, which ensures that your application remains operational even in the event of infrastructure failures. + +In summary, while running an application locally can work for development or small-scale use cases, the limitations in performance, scalability, availability, and reliability become apparent as the application grows. Migrating to AWS Elastic Beanstalk allows you to eliminate these limitations and fully leverage the power of AWS to deliver a high-performing, scalable, and resilient application. diff --git a/cloud_computing/migrating_app_to_cloud/app_remote_storage.md b/cloud_computing/migrating_app_to_cloud/app_remote_storage.md new file mode 100644 index 00000000..91eee5ee --- /dev/null +++ b/cloud_computing/migrating_app_to_cloud/app_remote_storage.md @@ -0,0 +1,248 @@ +--- +name: Integrating AWS Services with the Local Application +dependsOn: [] +tags: [] +learningOutcomes: +--- + +### Image Processing Application Using S3 and DynamoDB + +This is a modified version of the image processing application. Instead of relying on local storage and databases, this version utilizes AWS services, specifically **S3** for image storage and **DynamoDB** for metadata management. + +### Prerequisites + +In addition to the prerequisites (e.g., Git, Python, Pip, Docker, etc.) covered in **Running the Image Processing Application Locally**, we also need to install and configure the **AWS Command Line Interface (CLI)** to enable interaction with AWS services. + +#### Step 1: Install the AWS CLI + +The AWS Command Line Interface (CLI) allows you to interact with AWS services from your local machine using commands in the terminal. Depending on your operating system (OS), follow the instructions in this [AWS CLI installation guide](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) to install the AWS CLI. Once the AWS CLI is installed, you can confirm the installation by running the following command in the terminal: + +```bash +aws --version +``` + +#### Step 2: Configure the AWS CLI + +After installation, you need to configure the AWS CLI with your credentials so that it can interact with your AWS account. To accomplish that, you need to confure the following steps: + +1. Open a terminal or Command Prompt and run the following command: + +```bash +aws configure +``` + +2. You will be prompted to enter the following information: + +- **AWS Access Key ID:** This is a key that allows you to access AWS services. You can obtain it by logging into the AWS Management Console, navigating to Identity and Access Management (IAM), and creating a new user with programmatic access. Once created, the Access Key ID will be provided to you. + +- **AWS Secret Access Key:** This is the secret part of the Access Key pair. It's provided along with the Access Key ID when you create the user with programmatic access. Ensure you store this key securely, as you won't be able to view it again. + +- **Default region name:** This is the AWS region where you want your AWS services to operate. Examples include: + + - `eu-west-1` (Ireland) + - `eu-west-2` (London) + - `eu-west-3` (Paris) + +- **Default output format:** Choose how you want the CLI to display the results. Options include json, text, and table. + + Example input: + + ```bash + AWS Access Key ID [None]: AAKIEXAMPLE + AWS Secret Access Key [None]: wJalFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY + Default region name [None]: eu-west-2 + Default output format [None]: json + ``` + +#### Step 3: Test the Configuration + +To verify that the AWS CLI is correctly configured, try listing your S3 buckets: + +```bash +aws s3 ls +``` + +If the configuration is successful, you will see a list of your S3 buckets. If there are any issues, you will likely see an error message indicating the problem. + + +### Setting Up AWS Resources + +Before cloning and running the application, you need to create an S3 bucket and a DynamoDB table on AWS, as the application will use these resources for storage and metadata management. + +#### 1. **Creating an S3 Bucket:** + +The S3 bucket will be used to store the images uploaded by the users and processed by the application. + +1. Open the AWS Management Console and navigate to the [S3 service](https://s3.console.aws.amazon.com/s3/home). Please make sure you are in the correct AWS region. You can select the region by clicking on the region dropdown in the top right corner of the console. Refer to the image below, where the region selection is highlighted with a red box for guidance on changing the region + + ![AWS Region Selection](S3_Region.jpg) + +2. Click on **Create bucket**. Refer to the image below if you need guidance, where the "Create bucket" option is highlighted with a red box: + + ![Where is S3 Create Button](Where_S3_Create_Button.jpg) + +3. Provide a unique bucket name (e.g., `ar-image-processing-bucket`). It's recommended to include your initials or another unique identifier in the bucket name to avoid naming conflicts. Refer to the image below, where the field for entering the bucket name is highlighted with a red box: + + ![S3 Bucket Name Input](Create_S3_Bucket.jpg) + +4. Leave the other settings as default unless you have specific requirements. However, it's important to understand and consider the following options: + +#### Object Ownership: + +- **Object Ownership** defines the access control behavior for objects uploaded to the bucket. By default, Amazon disables Access Control Lists (ACLs), which is the recommended approach for most use cases as it simplifies permissions management. Disabling ACLs ensures that the bucket owner automatically owns all objects, even if they are uploaded by different AWS accounts. + +#### Block Public Access: + +- **Block Public Access** settings are critical for securing your bucket. By default, *Block all public access* is selected, which is highly recommended unless you need to expose objects publicly. This setting prevents any accidental exposure of sensitive data stored in the bucket. + +#### Bucket Versioning: + +- **Bucket Versioning** is disabled by default. Enabling versioning allows you to retain multiple versions of an object, which is useful for backup and recovery scenarios. If versioning is important to your use case, you can enable it after creating the bucket. + +#### Default Encryption: + +- **Server-Side Encryption (SSE)** is automatically applied to new objects stored in the bucket. By default, **SSE-S3** (Server-Side Encryption with Amazon S3 managed keys) is selected. This ensures that your data is encrypted at rest without requiring you to manage encryption keys. + + You also have the option to choose: + + - **SSE-KMS**: Server-side encryption with AWS Key Management Service (KMS) keys, which allows you to have more control over encryption keys. + - **DSSE-KMS**: Dual-layer encryption using AWS KMS keys for even greater security. This option applies two separate layers of encryption to your data, although it might incur additional costs. + +#### Bucket Key: + +- If you select **SSE-KMS**, you will also have the option to enable the **S3 Bucket Key**, which reduces the cost of encryption by minimizing calls to AWS KMS. Enabling S3 Bucket Keys can optimize costs when encrypting large volumes of data. This option is automatically enabled by default but can be disabled if necessary. Note that it is not supported for DSSE-KMS. + +5. Click **Create bucket** once you are comfortable with these settings. Refer to the image below for guidance, where the "Create bucket" button is highlighted with a red box: + + ![AWS Create Bucket Confirmation](S3_Create_Button_Confirm.jpg) + + +#### 2. **Creating a DynamoDB Table:** + +The DynamoDB table will be used to store metadata related to the images. + +1. Open the AWS Management Console and navigate to the [DynamoDB service](https://console.aws.amazon.com/dynamodb/home). Please make sure you are in the correct AWS region. You can select the region by clicking on the region dropdown in the top right corner of the console. Refer to the image below, where the region selection is highlighted with a red box for guidance on changing the region. + + ![DynamoDB Region Selection](DynamoDB_Region.jpg) + +2. In the left panel, click on **Tables**. This will display the tables panel on the right-hand side, where you will see a **Create table** button. Click on **Create table** to begin setting up your new table. Refer to the images below for guidance, with each relevant section highlighted in a red box: + + - ![DynamoDB Tables Panel](DynamoDB_Where_Tables.jpg) + - ![DynamoDB Create Table Button](DynamoDB_Create_Table_Button.jpg) + +3. Provide a table name (e.g., `ar-ImageMetadata`) and set the **Partition key** to `id` (String). It's recommended to include your initials or another unique identifier in the table name to avoid naming conflicts. Refer to the image below for guidance, where the table name and partition key fields are highlighted with a red box: + + ![DynamoDB Table Name and Partition Key Setup](DynamDB_Table_Details.jpg) + +4. You can leave the other settings as default unless you need advanced configuration. +5. Click **Create table** to finalize the setup. Refer to the image below for guidance, where the "Create table" button is highlighted with a red box: + + ![DynamoDB Create Table Confirmation](DynamoDB_Create_Table_Confirm.jpg) + + +Once these resources are set up, you can proceed to clone the application and configure it to use the S3 bucket and DynamoDB table. + + +### Cloning the Application from GitHub + +To get started with the image processing application, follow these steps to clone the repository and set it up locally: + +1. Clone the repository from GitHub: +```bash +git clone https://github.com/Oxford-Research-Cloud-Competency-Centre/image-processing-hybrid-app +``` + +2. Navigate to the project directory: +```bash +cd image-processing-hybrid-app +``` + +Once cloned, and before running the application, you need to create a `.env` file in the root directory of the project. This file will store environment variables such as the AWS region, S3 bucket name, and DynamoDB table name. These variables will be used by the application to connect to the appropriate AWS resources and manage image storage and metadata effectively. Below are the required entries for the `.env` file: + + +```plaintext +REGION_NAME = 'YOUR_AWS_REGION' +S3_BUCKET = 'YOUR_AWS_S3_BUCKET_NAME' +IMAGE_METADATA_TABLE = 'YOUR_AWS_DYNAMODB_TABLE_NAME' +``` + +> **Note:** Please replace the values above with those specific to your AWS resources. Ensure you use the correct region, S3 bucket name, and DynamoDB table name that you created. + + +### Running the Application + +The application can be run in two different ways, depending on your setup: + +#### 1. **Running Locally with Python:** + +To run the application locally on your system: + +1. Make sure the repository is cloned and you are inside the home directory of the application. +2. Install the necessary Python dependencies: + ```bash + pip install -r requirements.txt + ``` +3. Run the Flask application: + ```bash + python -m app.app + ``` + +The application will be accessible at http://localhost:5000 + +#### 2. **Running with Docker:** + +Alternatively, you can run the application inside a Docker container: + +1. Build the Docker image: + ```bash + docker build -t image-processing-hybrid-app . + ``` +2. Run the Docker container: + ```bash + docker run -p 5000:5000 -e AWS_ACCESS_KEY_ID=your_access_key -e AWS_SECRET_ACCESS_KEY=your_secret_key -e AWS_DEFAULT_REGION=your_aws_region image_processing_cloud_app_local + ``` +You can retrieve your AWS credentials (e.g., AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_DEFAULT_REGION) by checking your ~/.aws/credentials file. Your application inside the container can now access these credentials from the environment variables, and AWS SDKs will automatically pick them up. + +This will make the application available at http://localhost:5000, without the need to install Python or any other dependencies directly on your machine. + +### Resource Cleanup + +Once you've set up and used the S3 and DynamoDB resources for this application, it's important to clean up these resources to avoid unnecessary costs. Follow the steps below to delete the resources. + +1. **Delete S3 Bucket**: + - Go to the [S3 console](https://s3.console.aws.amazon.com/s3/home) and locate the bucket you created for this application. + - Select the bucket, then click on **Empty** to delete all objects within the bucket. Refer to the image below, where the **Empty** button is highlighted in red for guidance: + + ![S3 Empty Bucket](S3_Empty.jpg) + + - After clicking **Empty**, a new page will open. To confirm deletion, type "permanently delete" in the text input field, then click on **Empty** again to finalize. Refer to the image below for guidance, where the **confirmation field** and **Empty button** are highlighted in red: + + ![S3 Confirm Empty](S3_Confirm_Empty.jpg) + + - After emptying the bucket, return to the bucket's main page and click on **Delete** to remove the bucket itself. Refer to the image below, where the **Delete** button is highlighted in red: + + ![S3 Delete Bucket](S3_Delete.jpg) + + - A new window will open. To confirm the bucket deletion, enter the bucket name in the text input field, then click on **Delete Bucket**. Refer to the image below, where the **confirmation field** and **Delete Bucket** button are highlighted in red: + + ![S3 Confirm Delete Bucket](S3_Confirm_Delete.jpg) + + - **Please Note:** It may take a few moments for AWS to fully delete the bucket. Wait until you see a confirmation message indicating that the bucket has been deleted to ensure the resource is fully removed. + +2. **Delete DynamoDB Table**: + - Navigate to the [DynamoDB console](https://console.aws.amazon.com/dynamodb/home). + - In the **Tables** section, locate the table you created (e.g., `ar-ImageMetadata`). + - Select the table, then click on **Delete table** to remove it. Refer to the image below, where the **Delete table** button is highlighted in red: + + ![DynamoDB Delete Table](DynamoDB_Delete.jpg) + + - After clicking **Delete table**, a new window will open asking for confirmation. Type **confirm** in the text input field, then click on the **Delete** button to finalize the deletion. Refer to the image below, where the **confirmation field** and **Delete** button are highlighted in red: + + ![DynamoDB Confirm Delete Table](DynamoDB_Confirm_Delete.jpg) + + - **Please Note:** It may take a few moments for AWS to fully delete the table. Wait until you see a confirmation message indicating that the table has been deleted to ensure the resource is completely removed. + + +> **Note:** Ensure you have backed up any necessary data from these resources before deleting them, as this action is irreversible. + + diff --git a/cloud_computing/migrating_app_to_cloud/app_usecase.md b/cloud_computing/migrating_app_to_cloud/app_usecase.md new file mode 100644 index 00000000..094a56bd --- /dev/null +++ b/cloud_computing/migrating_app_to_cloud/app_usecase.md @@ -0,0 +1,27 @@ +--- +name: Application Use Case Overview +dependsOn: [] +tags: [] +learningOutcomes: +--- + +For this course, we will use a running application as a practical example to demonstrate key concepts. The **Image Processing Application** will serve as our use case, showcasing how to manage file uploads, image processing, and metadata storage. While image processing applications can often be complex, this is a streamlined, lightweight version specifically designed and developed to demonstrate essential functionality without adding the complexity of a full-scale image processing system. + + +## Image Processing Application + +The running application (i.e., the image processing application) is built using [Flask](https://flask.palletsprojects.com/en/stable/), a lightweight web framework that simplifies the development of web applications by providing essential tools for routing, request handling, and templating. The application allows users to upload images, after which the image processing component kicks in to perform necessary transformations on the images. Both the original and processed versions are then stored. Additionally, the application saves metadata—such as filenames, sizes, and upload dates—in a [SQLite](https://www.sqlite.org/) database, showcasing file handling and database interaction. + + +### Application Features and Workflow + +We will walk through the main features and the workflow of the application, from image upload to metadata storage and display. + +1. **Upload and Processing**: Users upload images through the web interface. Uploaded images are saved in the `uploads` directory, and metadata is stored in the SQLite database. +2. **Image Processing**: Images are converted to grayscale and resized to 300x300 pixels. The processed images are saved in the `processed` directory. +3. **Metadata Storage**: Details of both the original and processed images are stored in the SQLite database. +4. **Display and Download**: A dashboard displays all image metadata, and users can download processed images from the web interface. + + + + diff --git a/cloud_computing/migrating_app_to_cloud/aws_services.md b/cloud_computing/migrating_app_to_cloud/aws_services.md new file mode 100644 index 00000000..0aac3a38 --- /dev/null +++ b/cloud_computing/migrating_app_to_cloud/aws_services.md @@ -0,0 +1,50 @@ +--- +name: Introduction to Key AWS Services +dependsOn: [] +tags: [] +learningOutcomes: + - Identify the core AWS services and their purposes. + - Explain the key features and benefits of AWS services. + - Understand how these services enable scalable applications. +--- + +This section covers the fundamentals of AWS services like EC2, S3, DynamoDB, and Elastic Beanstalk. These services are the foundation of many modern cloud applications, allowing organizations to build scalable, resilient, and high-performance applications. + +## AWS Core Services + +In this section, we will dive into the core AWS services that form the building blocks of cloud infrastructure. Each service offers unique capabilities, and together, they provide a comprehensive platform for developing scalable and reliable applications in the cloud. + +1. **Amazon Elastic Compute Cloud (EC2):** + Amazon EC2 provides scalable virtual servers (known as instances) in the cloud. These + instances allow you to run applications in a secure and resizable computing environment, similar to a physical server but without the overhead of managing hardware. EC2 offers the flexibility to scale up or down based on demand, making it ideal for a wide range of workloads. + + **Key Features:** + - **Instance Types:** EC2 provides various instance types optimized for different use cases, from general-purpose to compute, memory, and GPU-optimized instances. + - **Auto Scaling:** Automatically adjusts the number of instances based on your application's needs, ensuring optimal performance while controlling costs. + - **Security:** EC2 instances can be secured using security groups and network access control lists (ACLs), controlling inbound and outbound traffic. + +2. **Amazon Simple Storage Service (S3):** + Amazon S3 is a highly scalable object storage service, designed to store data from anywhere on the web. It is ideal for storing large files like media, backups, and application data. Amazon S3 is known for its 11 nines of durability, meaning for every ten million objects stored, you can expect to incur an average loss of a single object once every 10,000 years. This ensures that your data is highly secure and redundant across multiple locations. + + **Key Features:** + - **Data Durability and Availability:** Data is automatically replicated across multiple locations, ensuring high availability. + - **Versioning and Lifecycle Policies:** S3 supports version control and policies to manage storage costs by transitioning old or infrequently used data to cheaper storage classes. + - **Security:** Supports encryption and access control through policies, ensuring that your data is secure. + +3. **Amazon DynamoDB:** + Amazon DynamoDB is a fully managed NoSQL database service designed for high-performance, low-latency applications. It is ideal for storing and retrieving large volumes of structured data quickly. + + **Key Features:** + - **Performance:** DynamoDB provides fast and predictable performance, making it suitable for high-traffic web and mobile applications. + - **Seamless Scaling:** DynamoDB automatically scales up and down to handle changes in traffic without manual intervention. + - **High Availability:** DynamoDB's replication across multiple AWS regions ensures your data is highly available and durable. + +4. **AWS Elastic Beanstalk:** + AWS Elastic Beanstalk simplifies the deployment and management of applications in the AWS cloud. It automatically handles the infrastructure, including scaling, load balancing, and monitoring, so you can focus on writing code without worrying about servers or other underlying infrastructure. + + **Key Features:** + - **Automatic Provisioning:** Elastic Beanstalk automatically sets up and configures the required resources (e.g., EC2 instances, load balancers) to run your application. + - **Scaling:** Automatically adjusts the number of resources allocated to your application to meet traffic demands. + - **Monitoring:** Elastic Beanstalk integrates with AWS CloudWatch to provide real-time monitoring and performance metrics for your application. + +By understanding these key services, you will be ready to work with real-world applications that utilize the cloud's full potential. diff --git a/cloud_computing/migrating_app_to_cloud/conclusion.md b/cloud_computing/migrating_app_to_cloud/conclusion.md new file mode 100644 index 00000000..3dc139d7 --- /dev/null +++ b/cloud_computing/migrating_app_to_cloud/conclusion.md @@ -0,0 +1,18 @@ +--- +name: Conclusion +dependsOn: [] +tags: [] +learningOutcomes: +--- + +The goal of this course was to familiarize you with cloud computing, particularly with using AWS services to enhance an application’s scalability and performance. We took a hands-on approach, beginning with a **local setup** and gradually transitioning to a full cloud deployment. + +The course was structured into three main parts: + +1. **Local Application Setup**: We started with an **Application Use Case Overview** of a local image processing application built with **Flask** to handle the web interface and **SQLite** to store metadata locally. This setup introduced you to fundamental processes like file handling, image processing, and metadata management in a Python-based application. + +2. **Integrating AWS Services with a Local Application**: In this part, we kept the application running locally but integrated it with **AWS Services** for remote data storage and management. We introduced **S3** for scalable image storage and **DynamoDB** for managing metadata in the cloud, allowing the local application to leverage these cloud resources. This setup demonstrated how AWS services could enhance durability and accessibility even for applications not fully deployed on the cloud. We also explored the **Drawbacks of Running a Local Application with Remote AWS Services**, noting that full migration to AWS could optimize performance by reducing latency and streamlining management. + +3. **Migrating the Application to AWS with Elastic Beanstalk**: In the final part, we migrated the application fully to AWS using **Elastic Beanstalk**, creating a cloud-native, fully managed environment. This migration shifted responsibilities for scaling, resource allocation, and management to Elastic Beanstalk, providing a production-ready deployment with minimal overhead and demonstrating the advantages of running the application directly alongside its data in the cloud. + +Through this course, you’ve gained practical skills and theoretical knowledge in building, integrating, and deploying applications on AWS. These foundational skills provide a strong base for further exploration in developing scalable, high-performing applications on the cloud. diff --git a/cloud_computing/migrating_app_to_cloud/fig/Create_EC2_Key.png b/cloud_computing/migrating_app_to_cloud/fig/Create_EC2_Key.png new file mode 100644 index 00000000..9ad3a2df Binary files /dev/null and b/cloud_computing/migrating_app_to_cloud/fig/Create_EC2_Key.png differ diff --git a/cloud_computing/migrating_app_to_cloud/fig/Create_EC2_Key_Part2.png b/cloud_computing/migrating_app_to_cloud/fig/Create_EC2_Key_Part2.png new file mode 100644 index 00000000..56059745 Binary files /dev/null and b/cloud_computing/migrating_app_to_cloud/fig/Create_EC2_Key_Part2.png differ diff --git a/cloud_computing/migrating_app_to_cloud/fig/Create_S3_Bucket.jpg b/cloud_computing/migrating_app_to_cloud/fig/Create_S3_Bucket.jpg new file mode 100644 index 00000000..f07be51c Binary files /dev/null and b/cloud_computing/migrating_app_to_cloud/fig/Create_S3_Bucket.jpg differ diff --git a/cloud_computing/migrating_app_to_cloud/fig/DynamDB_Table_Details.jpg b/cloud_computing/migrating_app_to_cloud/fig/DynamDB_Table_Details.jpg new file mode 100644 index 00000000..6d4d693c Binary files /dev/null and b/cloud_computing/migrating_app_to_cloud/fig/DynamDB_Table_Details.jpg differ diff --git a/cloud_computing/migrating_app_to_cloud/fig/DynamoDB_Confirm_Delete.jpg b/cloud_computing/migrating_app_to_cloud/fig/DynamoDB_Confirm_Delete.jpg new file mode 100644 index 00000000..a805858e Binary files /dev/null and b/cloud_computing/migrating_app_to_cloud/fig/DynamoDB_Confirm_Delete.jpg differ diff --git a/cloud_computing/migrating_app_to_cloud/fig/DynamoDB_Create_Table_Button.jpg b/cloud_computing/migrating_app_to_cloud/fig/DynamoDB_Create_Table_Button.jpg new file mode 100644 index 00000000..53374360 Binary files /dev/null and b/cloud_computing/migrating_app_to_cloud/fig/DynamoDB_Create_Table_Button.jpg differ diff --git a/cloud_computing/migrating_app_to_cloud/fig/DynamoDB_Create_Table_Confirm.jpg b/cloud_computing/migrating_app_to_cloud/fig/DynamoDB_Create_Table_Confirm.jpg new file mode 100644 index 00000000..088ac012 Binary files /dev/null and b/cloud_computing/migrating_app_to_cloud/fig/DynamoDB_Create_Table_Confirm.jpg differ diff --git a/cloud_computing/migrating_app_to_cloud/fig/DynamoDB_Delete.jpg b/cloud_computing/migrating_app_to_cloud/fig/DynamoDB_Delete.jpg new file mode 100644 index 00000000..b1fe585c Binary files /dev/null and b/cloud_computing/migrating_app_to_cloud/fig/DynamoDB_Delete.jpg differ diff --git a/cloud_computing/migrating_app_to_cloud/fig/DynamoDB_Where_Tables.jpg b/cloud_computing/migrating_app_to_cloud/fig/DynamoDB_Where_Tables.jpg new file mode 100644 index 00000000..d9955515 Binary files /dev/null and b/cloud_computing/migrating_app_to_cloud/fig/DynamoDB_Where_Tables.jpg differ diff --git a/cloud_computing/migrating_app_to_cloud/fig/DynamoDB_region.jpg b/cloud_computing/migrating_app_to_cloud/fig/DynamoDB_region.jpg new file mode 100644 index 00000000..4edab588 Binary files /dev/null and b/cloud_computing/migrating_app_to_cloud/fig/DynamoDB_region.jpg differ diff --git a/cloud_computing/migrating_app_to_cloud/fig/EB_Application_Domain.jpg b/cloud_computing/migrating_app_to_cloud/fig/EB_Application_Domain.jpg new file mode 100644 index 00000000..d8fb0ce8 Binary files /dev/null and b/cloud_computing/migrating_app_to_cloud/fig/EB_Application_Domain.jpg differ diff --git a/cloud_computing/migrating_app_to_cloud/fig/EB_Applications.jpg b/cloud_computing/migrating_app_to_cloud/fig/EB_Applications.jpg new file mode 100644 index 00000000..17de0f08 Binary files /dev/null and b/cloud_computing/migrating_app_to_cloud/fig/EB_Applications.jpg differ diff --git a/cloud_computing/migrating_app_to_cloud/fig/EB_Environment_Details.jpg b/cloud_computing/migrating_app_to_cloud/fig/EB_Environment_Details.jpg new file mode 100644 index 00000000..85f384e4 Binary files /dev/null and b/cloud_computing/migrating_app_to_cloud/fig/EB_Environment_Details.jpg differ diff --git a/cloud_computing/migrating_app_to_cloud/fig/EB_Environments.jpg b/cloud_computing/migrating_app_to_cloud/fig/EB_Environments.jpg new file mode 100644 index 00000000..98215b3e Binary files /dev/null and b/cloud_computing/migrating_app_to_cloud/fig/EB_Environments.jpg differ diff --git a/cloud_computing/migrating_app_to_cloud/fig/S3_Confirm_Delete.jpg b/cloud_computing/migrating_app_to_cloud/fig/S3_Confirm_Delete.jpg new file mode 100644 index 00000000..574b92e7 Binary files /dev/null and b/cloud_computing/migrating_app_to_cloud/fig/S3_Confirm_Delete.jpg differ diff --git a/cloud_computing/migrating_app_to_cloud/fig/S3_Confirm_Empty.jpg b/cloud_computing/migrating_app_to_cloud/fig/S3_Confirm_Empty.jpg new file mode 100644 index 00000000..1ddfaa32 Binary files /dev/null and b/cloud_computing/migrating_app_to_cloud/fig/S3_Confirm_Empty.jpg differ diff --git a/cloud_computing/migrating_app_to_cloud/fig/S3_Create_Button_Confirm.jpg b/cloud_computing/migrating_app_to_cloud/fig/S3_Create_Button_Confirm.jpg new file mode 100644 index 00000000..4637e647 Binary files /dev/null and b/cloud_computing/migrating_app_to_cloud/fig/S3_Create_Button_Confirm.jpg differ diff --git a/cloud_computing/migrating_app_to_cloud/fig/S3_Delete.jpg b/cloud_computing/migrating_app_to_cloud/fig/S3_Delete.jpg new file mode 100644 index 00000000..35172490 Binary files /dev/null and b/cloud_computing/migrating_app_to_cloud/fig/S3_Delete.jpg differ diff --git a/cloud_computing/migrating_app_to_cloud/fig/S3_Empty.jpg b/cloud_computing/migrating_app_to_cloud/fig/S3_Empty.jpg new file mode 100644 index 00000000..9c9b63b0 Binary files /dev/null and b/cloud_computing/migrating_app_to_cloud/fig/S3_Empty.jpg differ diff --git a/cloud_computing/migrating_app_to_cloud/fig/S3_Region.jpg b/cloud_computing/migrating_app_to_cloud/fig/S3_Region.jpg new file mode 100644 index 00000000..e3c6a9b8 Binary files /dev/null and b/cloud_computing/migrating_app_to_cloud/fig/S3_Region.jpg differ diff --git a/cloud_computing/migrating_app_to_cloud/fig/Where_S3_Create_Button.jpg b/cloud_computing/migrating_app_to_cloud/fig/Where_S3_Create_Button.jpg new file mode 100644 index 00000000..099be1de Binary files /dev/null and b/cloud_computing/migrating_app_to_cloud/fig/Where_S3_Create_Button.jpg differ diff --git a/cloud_computing/migrating_app_to_cloud/index.md b/cloud_computing/migrating_app_to_cloud/index.md new file mode 100644 index 00000000..84cd6a5b --- /dev/null +++ b/cloud_computing/migrating_app_to_cloud/index.md @@ -0,0 +1,19 @@ +--- +name: Migrating Application from an On-Premise Environment to AWS +id: migrating_app_to_cloud +dependsOn: [ +] +files: [ + app_usecase.md, + app_local_storage.md, + aws_services.md, + app_remote_storage.md, + app_onprem_limitation.md, + app_migrate_to_aws.md, + conclusion.md +] +summary: | + This course covers the basics of AWS with a focus on services like EC2, S3, DynamoDB, and Elastic Beanstalk. These services are utilized to build a scalable, resilient, and high-performance cloud application. +--- + + diff --git a/index.md b/index.md index 7c6d3757..28637dd2 100644 --- a/index.md +++ b/index.md @@ -9,5 +9,6 @@ themes: [ introductory_courses, scientific_computing, libraries, + cloud_computing, ] ---