v0.1.1 Release (#168)

* Condense slack messages into one * Updating Data Monitors to be a dropdown * Update documentation images * Fix execution timestamp update error * Fix incorrect zscore calculations * Decrease dockerfile size * Add GA tag to our documentation * Zscores bug fix * Add sampling for data points instead of backfill on timestamp * Add freshness metric check for timestamp columns * update readme copy and images * Show healthiness status on a metric * Fix metric id equality join * Add algolia search * Update postgresql class * Add to user setup process * Block usage on preferences set * Update docs colors * Add issues page and model * Add webhook integration support * add sqlalchemy-bigquery dependency * add bigquery logo & update image folder structure * add support for bigquery datasource in UI * add bigquery driver * update logo link & fix class overrides * fix timestamp cast for bq freshness monitor * add BigQuery to docs * fix redshift freshness monitor syntax * Fix missing monitors * remove timestamp filter on frontend * fix metric type detection for bigquery * fix bq config * update contributing docs * Revert "add BigQuery to docs" This reverts commit 9b426a5. * Add bigquery Co-authored-by: Kevin Unkrich <[email protected]>
monosidev · Apr 13, 2022 · 6c60387 · 6c60387
1 parent ea63cae
commit 6c60387
Show file tree

Hide file tree

Showing 71 changed files with 1,565 additions and 291 deletions.
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -23,15 +23,7 @@ Bug reports help make monosi better for everyone. To create a bug report, [raise
 
 ## Development Environment Setup
 
-To get the project setup locally and be ready for development:
-
-1. Fork the repository
-2. Clone the repository to your development machine & navigate into it
-3. Start a python virtual environment `python3 -m virtualenv .venv && source .venv/bin/activate`
-4. Run `make init`
-5. Install the current dev version by running `python3 -m setup.py install`
-
-You are now running the development version of monosi. If you have any questions, reach out on Slack
+Please follow the [instructions outlined in the Monosi docs](https://docs.monosi.dev/docs/contributing/local-development) to get set up with a local instance of the project for development.
 
 ## First-time contributors
 
@@ -51,6 +43,6 @@ If you want to speak to us before doing lots of work, just ping us in [Slack](ht
 
 ## Questions
 
-Come say hi to us on [Slack](https://monosi.dev/slack)! :wave:
+Come say hi to us on [Slack](https://monosi.dev/slack)! 👋
 
 Or if you prefer, email us at [[email protected]](mailto:[email protected]) — we're always happy to chat!
diff --git a/README.md b/README.md
@@ -20,9 +20,16 @@
   <a href="https://monosi.dev/slack">Join the Data Reliability Engineering Community</a>
 </p>
 
-Monosi is a stand-alone & extensible platform for data observability and monitoring. It ensures data quality by automatically monitoring your data stack for anomalies, alerting you of issues, and analyzing the root cause.
 
-This project is an OSS alternative to proprietary data quality and observability systems. Get started monitoring your data in [less than 10 minutes](https://docs.monosi.dev/docs/user-guide/getting-started)
+Monosi offers data quality monitoring as a service to root cause data issues with end-to-end observability.
+
+🏆 Ensure data quality
+
+🚨 Monitor your data & alert on anomalies
+
+🎛 Analyze root cause of data issues
+
+This project is an OSS alternative to proprietary data quality and observability systems. Get started monitoring your data in [less than 10 minutes](https://docs.monosi.dev/docs/user-guide/getting-started).
 
 
 ## Installation
@@ -53,10 +60,9 @@ Start the UI through Docker and quickly connect your data sources and alert inte
 
 <img alt="web interface" src="/documentation/static/img/example.gif"/>
 
-
 Get alerts in slack when Monosi detects anomalies in defined monitors.
 
-![monosi slack alerts](https://www.monosi.dev/images/Monosi_Slack_Alert.svg)
+<img alt="monosi slack alert" src="/documentation/static/img/slack_alert.svg">
 
 ### Own your stack
 Avoid integration pitfalls with fragmented, legacy tools by using open source & prevent vendor lock-in by owning your (meta)data. 

diff --git a/deployment/docker/Dockerfile.api b/deployment/docker/Dockerfile.api
@@ -4,6 +4,7 @@ WORKDIR /app
 COPY ./requirements.txt ./
 COPY ./requirements.api.txt ./
 COPY ./src .
+RUN rm -rf ./src/ui
 RUN pip install -r ./requirements.txt
 RUN pip install -r ./requirements.api.txt
 

diff --git a/documentation/docs/contributing/local-development.md b/documentation/docs/contributing/local-development.md
@@ -4,5 +4,39 @@ title: Local Development Setup
 sidebar_label: Local Development
 ---
 
-Learn how to build Monosi from source using the following YouTube guide:
-https://www.youtube.com/watch?v=OsVKKo71Z70
+The directions below outline the process for setting up Monosi for local development. To get started:
+
+1. Create a fork of the Monosi repository to your personal GitHub account
+2. Clone the fork URL that you have created to your local machine (`git clone <fork-url>`)
+
+Monosi can be installed either through docker or through a local dependency setup.
+## Running with Docker
+
+1. Ensure that you have installed docker and it is running on your machine
+2. In a terminal, navigate to the cloned repository
+3. Run `make compose-build` from the base of the directory to build a local instance of Monosi
+4. Run `make compose-run` from the base of the directory to run the local build
+5. Navigate to `http://localhost:3000` and you will see the Monosi UI
+
+For any changes you make to the code locally, you can test them by re-running steps 3 & 4.
+## Running without Docker
+
+1. Ensure that you have Python3, Node, and Yarn on your machine
+2. In a terminal, navigate to the cloned repository
+
+For the server: 
+1. Create a Python virtualenv by running `virtualenv .venv`
+
+2. Activte the virtualenv by running `source .venv/bin/activate`
+3. Install the Monosi dependencies by running `python3 setup.py install`
+4. Navigate to the server directory `cd src/server`
+5. Run `flask run`
+6. You should see the server startup and become accessible at `http://localhost:5000`
+
+For the client:
+1. Navigate to the ui directory `cd src/ui`
+2. Run `yarn && yarn start`
+3. The React application should start on `localhost:3000`
+
+
+If there's any problems with the setup, please send us a message in the [Slack](https://monosi.dev/slack) or over [email](mailto:[email protected]). 
diff --git a/documentation/docs/contributing/overview.md b/documentation/docs/contributing/overview.md
@@ -3,5 +3,51 @@ id: contributing-overview
 title: Contributing Overview
 sidebar_label: Overview
 ---
+# Contributing to monosi
 
-We would love to have you contribute to Monosi. You can start by joining our [Slack community](https://monosi.dev/slack) to learn more about how to get involved.
+Thank you for considering contributing to monosi! There are many types of contributions you can make, including bug reports and fixes, writing tutorials, and larger feature requests or changes.
+
+Before you contribute, make sure to read these guidelines thoroughly, so that you can get your pull request reviewed and finalized as quickly as possible. 
+
+- [Reporting Issues](#reporting-issues)
+- [Feature requests](#feature-requests)
+- [Development Environment Setup](#development-environment-setup)
+- [First-time contributors](#first-time-contributors)
+- [Creating a PR](#creating-a-pr)
+- [Questions](#questions)
+
+> **Note:** we're a young project and our contribution process is actively evolving. If you'd like to suggest or discuss changes to this document or the process in general, we're very open to suggestions and would appreciate if you reach out on [Slack](https://monosi.dev/slack) or at [[email protected]](mailto:[email protected])! To suggest additions to this document, feel free to raise an issue or make a PR with the changes.
+
+## Reporting Issues
+
+Bug reports help make monosi better for everyone. To create a bug report, [raise an issue](https://github.com/monosidev/monosi/issues/new?assignees=&labels=bug&template=bug_report.md&title=) using the "Bug Report" template. This template will provide you with a structure so we can best recreate the issue and promptly respond. Please search within our issues before raising a new one to make sure you're not raising a duplicate.
+
+## Feature requests
+
+[Raise an issue](https://github.com/monosidev/monosi/issues/new?assignees=&labels=enhancement&template=feature_request.md&title=) using the "Feature Request" template and make sure it's tagged as an `enhancement`. We love every idea. Please give us as much context on the "why" as possible.
+
+## Development Environment Setup
+
+Please follow the instructions outlined in the [local development](https://docs.monosi.dev/docs/contributing/local-development) guide to get set up with a local instance of the project for development.
+
+## First-time contributors
+
+> **Note:** if you're a first-time contributor, we recommend that you [follow this tutorial](http://makeapullrequest.com/) to learn how to start contributing. 
+
+- [Good first issues](https://github.com/monosidev/monosi/labels/good%20first%20issue)
+
+If you'd like to be assigned an issue, feel free to reach out on [Slack](https://monosi.dev/slack) or over email, or you can simply comment on an issue you'd like to work on. 
+
+## Creating a PR
+
+Create a new pull request from your personal fork of the project if you want to make an update. We would prefer that you tag an existing issue or open a new issue describing the update first if possible!
+
+For now, please request Kevin (@unkrich) or Ivan (@iporollo) to review your PR.
+
+If you want to speak to us before doing lots of work, just ping us in [Slack](https://monosi.dev/slack) or email [[email protected]](mailto:[email protected])
+
+## Questions
+
+Come say hi to us on [Slack](https://monosi.dev/slack)! 👋
+
+Or if you prefer, email us at [[email protected]](mailto:[email protected]) — we're always happy to chat!
diff --git a/documentation/docs/integrations/bigquery.md b/documentation/docs/integrations/bigquery.md
@@ -4,4 +4,17 @@ title: BigQuery Integration
 sidebar_label: BigQuery
 ---
 
-More information on how to integrate with BigQuery is coming soon. If you’re interested in this option, please follow along in the [GitHub Issue](https://github.com/monosidev/monosi/issues/33).
+Monosi supports BigQuery data warehouse connections. Currently, only authentication through a `service_account.json` file is supported. If you do not have a service account, please follow the instructions to create one [here](https://cloud.google.com/docs/authentication/getting-started).
+
+If you require other forms of authentication, please [open an issue](https://github.com/monosidev/monosi/issues/new?assignees=&labels=&template=feature_request.md).
+
+<img src="/img/integrations/bigquery_connection.png" alt="BigQuery Integration" />
+
+## Configuration Details
+
+| Key             | Description                                              | Required |
+| --------------- | -------------------------------------------------------- | -------- |
+| project         | The name of the BigQuery project you want to connect     | Yes      |
+| dataset         | The name of the dataset you want to connect              | Yes      |
+| service_account | The `service_account.json` file associated with your IAM | Yes      |
+
diff --git a/documentation/docusaurus.config.js b/documentation/docusaurus.config.js
@@ -108,12 +108,12 @@ module.exports = {
       `,
     },
     gtag: {
-      trackingID: "",
+      trackingID: "G-M6MW4MGZS8",
     },
     algolia: {
-      apiKey: "api", // TODO: INSERT UPDATE
+      apiKey: "53bfbe8a5227e168e89437292698846f",
+      appId: "GNDLCIWQCJ",
       indexName: "monosi",
-      // contextualSearch: true, // Optional, If you different version of docs etc (v1 and v2) doesn't display dup results
     },
   },
   presets: [

diff --git a/documentation/sidebars.js b/documentation/sidebars.js
@@ -19,7 +19,7 @@ module.exports = {
         {
           type: "category",
           label: "Data Monitors",
-          collapsible: false,
+          collapsible: true,
           collapsed: false,
           items: [
             "user-guide/table-health",
@@ -47,8 +47,9 @@ module.exports = {
           collapsed: true,
           items: [
             "integrations/snowflake",
-            "integrations/postgresql",
             "integrations/redshift",
+            "integrations/postgresql",
+            "integrations/bigquery",
           ]
         },
         {

diff --git a/documentation/src/components/Integrations.js b/documentation/src/components/Integrations.js
@@ -49,7 +49,7 @@ export const Integrations = () => {
           to={useBaseUrl("/docs/integrations/webhooks")}
           className=" scale flex flex-col items-center justify-center space-y-3 rounded-lg bg-[color:var(--ifm-card-background-color)] p-6 text-center shadow-lg"
         >
-          <img className="h-16 w-16" src="https://cdn.worldvectorlogo.com/logos/webhooks.svg" alt="Webhook logo" />
+          <img className="h-16 w-16" src="/img/webhooks.svg" alt="Webhook logo" />
           <p className="text-lg">Webhooks</p>
         </Link>
         <Link
@@ -63,28 +63,28 @@ export const Integrations = () => {
           className=" scale flex flex-col items-center justify-center space-y-3 rounded-lg bg-[color:var(--ifm-card-background-color)] p-6 text-center shadow-lg"
           disabled
           >
-          <img className="h-16 w-16" src="https://cdn.worldvectorlogo.com/logos/google-bigquery-logo-1.svg" alt="BigQuery logo" />
+          <img className="h-16 w-16" src="/img/bigquery.svg" alt="BigQuery logo" />
           <p className="text-lg">Google BigQuery</p>
         </Link>
         <Link
           disabled
           className=" scale flex flex-col items-center justify-center space-y-3 rounded-lg bg-[color:var(--ifm-card-background-color)] p-6 text-center shadow-lg"
         >
-          <img className="h-16 w-16" src="https://asset.brandfetch.io/idofJOT4bu/id3jbLcFnO.png" alt="dbt Logo" />
+          <img className="h-16 w-16" src="/img/dbt.png" alt="dbt Logo" />
           <p className="text-lg">dbt</p>
         </Link>
         <Link
           disabled
           className=" scale flex flex-col items-center justify-center space-y-3 rounded-lg bg-[color:var(--ifm-card-background-color)] p-6 text-center shadow-lg"
         >
-          <img className="h-16" src="https://seeklogo.com/images/G/google-looker-logo-B27BD25E4E-seeklogo.com.png" alt="Looker logo" />
+          <img className="h-16" src="/img/looker.png" alt="Looker logo" />
           <p className="text-lg">Looker</p>
         </Link>
         <Link
           disabled
           className=" scale flex flex-col items-center justify-center space-y-3 rounded-lg bg-[color:var(--ifm-card-background-color)] p-6 text-center shadow-lg"
         >
-          <img className="h-16 w-16" src="https://iconape.com/wp-content/files/cw/345108/svg/345108.svg" alt="Mode logo" />
+          <img className="h-16 w-16" src="/img/mode.svg" alt="Mode logo" />
           <p className="text-lg">Mode</p>
         </Link>
         <Link
@@ -98,35 +98,35 @@ export const Integrations = () => {
           disabled
           className=" scale flex flex-col items-center justify-center space-y-3 rounded-lg bg-[color:var(--ifm-card-background-color)] p-6 text-center shadow-lg"
         >
-          <img className="h-16" src="https://financeandbusiness.ucdavis.edu/sites/g/files/dgvnsk4871/files/styles/sf_landscape_16x9/public/images/article/tableau_logo.png?h=c673cd1c&itok=5J3wvhE8" alt="Tableau logo" />
+          <img className="h-16" src="/img/tableau.png" alt="Tableau logo" />
           <p className="text-lg">Tableau</p>
         </Link>
         <Link
           disabled
           className=" scale flex flex-col items-center justify-center space-y-3 rounded-lg bg-[color:var(--ifm-card-background-color)] p-6 text-center shadow-lg"
         >
-          <img className="h-16 w-16" src="https://download.logo.wine/logo/MySQL/MySQL-Logo.wine.png" alt="MySQL logo" />
+          <img className="h-16 w-16" src="/img/mysql.png" alt="MySQL logo" />
           <p className="text-lg">MySQL</p>
         </Link>
         <Link
           disabled
           className=" scale flex flex-col items-center justify-center space-y-3 rounded-lg bg-[color:var(--ifm-card-background-color)] p-6 text-center shadow-lg"
         >
-          <img className="h-16 w-16" src="https://avatars.githubusercontent.com/u/33643075?s=280&v=4" alt="Airflow logo" />
+          <img className="h-16 w-16" src="/img/airflow.png" alt="Airflow logo" />
           <p className="text-lg">Airflow</p>
         </Link>
         <Link
           disabled
           className=" scale flex flex-col items-center justify-center space-y-3 rounded-lg bg-[color:var(--ifm-card-background-color)] p-6 text-center shadow-lg"
         >
-          <img className="h-16 w-16" src="https://symbols.getvecta.com/stencil_91/7_pagerduty-icon.32302d0739.png" alt="PagerDuty logo" />
+          <img className="h-16 w-16" src="/img/pagerduty.png" alt="PagerDuty logo" />
           <p className="text-lg">PagerDuty</p>
         </Link>
         <Link
           disabled
           className=" scale flex flex-col items-center justify-center space-y-3 rounded-lg bg-[color:var(--ifm-card-background-color)] p-6 text-center shadow-lg"
         >
-          <img className="h-16 w-16" src="https://cdn.iconscout.com/icon/free/png-256/power-bi-3244521-2701891.png" alt="Power BI logo" />
+          <img className="h-16 w-16" src="/img/powerbi.png" alt="Power BI logo" />
           <p className="text-lg">Power BI</p>
         </Link>
       </div>

diff --git a/documentation/src/components/Intro.js b/documentation/src/components/Intro.js
@@ -8,7 +8,7 @@ function SDKs() {
       <div className="flex items-center space-x-4">
         <svg
           width="13"
-          className="mb-4 h-10 w-10 rounded-lg bg-[color:var(--ifm-color)] p-2 text-[color:var(--ifm-background-color)]"
+          className="mb-4 h-10 w-10 rounded-lg bg-[color:var(--ifm-color-highlight)] p-2 text-[color:var(--ifm-background-color)]"
           height="14"
           viewBox="0 0 13 14"
           fill="none"
@@ -50,14 +50,14 @@ function SDKs() {
         <li className="">
           <Link
             className="flex items-center space-x-3 hover:underline"
-            to={useBaseUrl("/docs/integrations/postgresql")}
+            to={useBaseUrl("/docs/integrations/bigquery")}
           >
             <img
               className="h-8 w-8 transition hover:scale-110"
-              src="/img/postgresql.svg"
-              alt="PostgreSQL logo"
+              src="/img/bigquery.svg"
+              alt="BigQuery logo"
             />
-            <p className="font-semibold">PostgreSQL</p>
+            <p className="font-semibold">BigQuery</p>
           </Link>
         </li>
 
@@ -93,7 +93,7 @@ function Server() {
     <div className="flex flex-col rounded-lg bg-[color:var(--ifm-card-background-color)] p-5 shadow">
       <div className="flex items-center space-x-4">
         <svg
-          className="mb-4 h-10 w-10 rounded-lg bg-[color:var(--ifm-color)] p-2 text-[color:var(--ifm-background-color)]"
+          className="mb-4 h-10 w-10 rounded-lg bg-[color:var(--ifm-color-highlight)] p-2 text-[color:var(--ifm-background-color)]"
           width="16"
           height="13"
           viewBox="0 0 16 13"
@@ -172,7 +172,7 @@ function Tools() {
     <div className="flex flex-col rounded-lg bg-[color:var(--ifm-card-background-color)] p-5 shadow">
       <div className="flex items-center space-x-4">
         <svg
-          className="mb-4 h-10 w-10 rounded-lg bg-[color:var(--ifm-color)] p-2 text-[color:var(--ifm-background-color)]"
+          className="mb-4 h-10 w-10 rounded-lg bg-[color:var(--ifm-color-highlight)] p-2 text-[color:var(--ifm-background-color)]"
           viewBox="0 0 15 15"
           fill="currentColor"
           xmlns="http://www.w3.org/2000/svg"

diff --git a/documentation/src/components/shared/Button.js b/documentation/src/components/shared/Button.js
@@ -5,7 +5,7 @@ export const Button = ({children, type, name, className}) => {
     <button
       type={type}
       name={name}
-      className={`flex items-center justify-center rounded-md border border-transparent bg-[color:var(--ifm-color)] px-5 py-3 text-base font-medium  text-[color:var(--ifm-background-color)]
+      className={`flex items-center justify-center rounded-md border border-transparent bg-[color:var(--ifm-color-highlight)] px-5 py-3 text-base font-medium  text-[color:var(--ifm-background-color)]
       hover:opacity-90 focus:outline-none focus:ring-2 focus:ring-[color:var(--ifm-color)] focus:ring-offset-2 ${className}`}
     >
       {children}

diff --git a/documentation/src/css/custom.css b/documentation/src/css/custom.css
@@ -32,6 +32,7 @@ html[data-theme="dark"] {
   --ifm-blockquote-color: #eeeeee;
   /* --ifm-code-background: var(--ifm-color-primary-dark); */
   --ifm-color: #ffffff;
+  --ifm-color-highlight: #ffffff;
   --ifm-background-color: #141414;
   --ifm-footer-background-color: #000;
   --ifm-card-background-color: #1a1a1a;
@@ -49,7 +50,8 @@ html[data-theme="light"] {
   /* --ifm-code-background: var(--ifm-color-primary-lightest); */
   --ifm-color-input-background: #ffffff;
   --ifm-color: #270d5b;
-  --ifm-background-color: #f9f7f1;
+  --ifm-color-highlight: #7344d2;
+  --ifm-background-color: #f8f9fa;
   --ifm-footer-background-color: #ffffff;
   --ifm-card-background-color: #ffffff;
   --ifm-badge-background-color: rgba(239, 246, 255);

diff --git a/documentation/static/img/airflow.png b/documentation/static/img/airflow.png