Skip to content
This repository has been archived by the owner on Jul 8, 2024. It is now read-only.

Latest commit

 

History

History
executable file
·
66 lines (43 loc) · 4.17 KB

architecture.md

File metadata and controls

executable file
·
66 lines (43 loc) · 4.17 KB

StarThinker Architecture

Overview

At the core of StarThinker are python functions with a JSON defined parameter set. The JSON allows various interfaces such as UI and command line. The python functions in their simplicity allow for deployment on many back ends. The architecture packaged in this project is fully extensible and customizable.

This is a typical pattern deployed for large scale job processing. It is designed to get any team started using both open source and Google Cloud technologies. StarThinker can be leveraged as a whole or in pieces, the central contract is the JSON/Python script task combination.

Application

Architecture View Architecture Larger / View Application Slides

Components

UI ( Enterprise Deployment )

Light weight open source Django Framework using open source Materialize CSS for the interface. Manages recipes and authentication using a browser interface. Ideal for deploying in a multiuser enterprise programmatic media teams.

  1. User Account - Google oAuth allows recipes to execute with user's permissions.
  2. Project Service - Google Cloud Project Service Credentials allow recipes to access cloud resources.
  3. Recipes - Assemble scripts into scheduled recipes without coding.
  4. Tasks - Each recipe is a collection of tasks, parameters are parsed and stored in a database by the UI.
  5. Database - Databse used to store user created recipes for execution by job worker.

Job Worker ( Enterprise Deployment )

Scalable back end for pulling and executing jobs. Multiple machines can be deployed with several workers each. Collision detection is built in, each worker will reserve tasks. If a worker stops working, other workers will begin pulling dropped tasks. The deploy script offers 4 scale options, that can be extended to any size budget permitting. The worker has full test coverage for safe maintenance.

  1. Database - Database polled by worker requesting recipe jobs.
  2. Job - A task returned to the worker containing all information to run it.
  3. Worker - The code deployable on a virtual machine at scale, to execute each task.

Command Line ( Optional Developer Support Interface )

Command line for creating, testing, and executing scripts.

  1. Script Developer - The author of every script needs only to provide a JSON script file and python function that is tested. This simplifies and speeds up development by seperating it from the UI and workers.
  2. JSON Parameter - Definition of parameters for python function, understood by all components of StarThinker.
  3. Python Function - Any python function for any purpose.

Airflow Cloud Composer ( Optional Alternate Deployment )

Connector for deploying recipes to Apache Airflow.

  1. Dag Recipes - List of all StarThinker recipes as Airflow DAGs.
  2. Dag Generator - Utility for generating Dags from recipes.
  3. Dag Factory - Python function doing the actual translation of JSON into a DAG.
  4. Dag - A wrapper for a python function in Airflow.
  5. Cloud Composer - Google Cloud instance of Apache Airflow.
  6. Airflow Framework - Project documentation and tutorials.

© 2019 Google LLC - Apache License, Version 2.0