Docs (#451)

* initializing docs

* updating documentation

* satelliteImages docs, remove worldview folder, remove tileserver

* Adding more docs

* Fix typos/misc whitespace issues

* Fix other typos

* Update docs/DevDocs/Architecture.md

Co-authored-by: Mike VanDenburgh <[email protected]>

* Update docs/DevDocs/Architecture.md

Co-authored-by: Mike VanDenburgh <[email protected]>

* Update docs/DevDocs/GettingStarted.md

Co-authored-by: Mike VanDenburgh <[email protected]>

* Update docs/DevDocs/GettingStarted.md

Co-authored-by: Mike VanDenburgh <[email protected]>

* Update docs/DevDocs/GettingStarted.md

Co-authored-by: Mike VanDenburgh <[email protected]>

* Update docs/DevDocs/SatelliteImages.md

Co-authored-by: Mike VanDenburgh <[email protected]>

* Update docs/UserGuide/LeftSidePanel.md

Co-authored-by: Mike VanDenburgh <[email protected]>

* Update docs/UserGuide/UsingRDWatch.md

Co-authored-by: Mike VanDenburgh <[email protected]>

* Update docs/DevDocs/SatelliteImages.md

Co-authored-by: Mike VanDenburgh <[email protected]>

* Update docs/DevDocs/SatelliteImages.md

Co-authored-by: Mike VanDenburgh <[email protected]>

* Update .github/workflows/cd.yaml

Co-authored-by: Mike VanDenburgh <[email protected]>

* Build docs in CI (#458)

This will cause CI to fail if the docs build fails.

---------

Co-authored-by: Mike VanDenburgh <[email protected]>
Co-authored-by: Mike VanDenburgh <[email protected]>

.github/workflows/cd.yaml

@@ -53,3 +53,20 @@ jobs:
           platforms: linux/amd64
           push: true
           tags: ${{ steps.image-tags.outputs.result }}
+  docs:
+    name: Deploy docs
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v2
+      with:
+        # "ref" specifies the branch to check out.
+        # "github.event.release.target_commitish" is a global variable and specifies the branch the release targeted
+        ref: ${{ github.event.release.target_commitish }}
+
+    # Deploy docs
+    - name: Deploy docs
+      uses: mhausenblas/[email protected]
+      env:
+        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        CONFIG_FILE: mkdocs.yml
+        EXTRA_PACKAGES: build-base

.github/workflows/ci.yaml

@@ -70,7 +70,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        tox-env: [test, check-migrations]
+        tox-env: [test, check-migrations, builddocs]
     services:
       postgres:
         image: postgis/postgis:latest

README.md

@@ -30,9 +30,9 @@ RDWATCH (**R**esonantGeo**D**ata **WATCH**) is a modern web application that off
 
 ## Documentation
 
-* [Development](https://github.com/ResonantGeoData/RD-WATCH/blob/main/DEVELOPMENT.md)
+* [Getting Started](https://ResonantGeoData.github.io/RD-WATCH/DevDocs/GettingStarted/)
 
-* [Data Ingestion](https://github.com/ResonantGeoData/RD-WATCH/blob/main/INGESTION.md)
+* [Data Ingestion](https://ResonantGeoData.github.io/RD-WATCH/DevDocs/IngestingData/)
 
 
 ## Related Work

docker-compose.override.yaml

@@ -1,5 +1,6 @@
 services:
   django:
+    platform: linux/amd64
     build:
       dockerfile: Dockerfile
       target: dev
@@ -33,6 +34,7 @@ services:
       - minio
 
   celery_worker:
+    platform: linux/amd64
     build:
       dockerfile: Dockerfile
       target: dev
@@ -114,23 +116,5 @@ services:
         # Run the npm development script
         npm run dev
       "
-  vectorserver:
-    image: node:20
-    container_name: rd-watch-vectorserver
-    working_dir: /usr/tileserver
-    environment:
-        RDWATCH_POSTGRESQL_URI: "${RDWATCH_POSTGRESQL_URI-postgresql://rdwatch:secretkey@postgresql:5432/rdwatch}"
-        PORT: "8001"
-        RDWATCH_REDIS_URI: "${RDWATHC_REDIS_URI-redis://redis:6379}"
-        RDWATCH_POSTGRESQL_SCORING_URI: "${RDWATCH_POSTGRESQL_SCORING_URI-postgresql://scoring:secretkey@scoredb:5432/scoring}"
-        RDWATCH_SUPPRESS_VECTOR_TILE_LOGGING: "true" # prevent console logging of information for dev server
-    volumes:
-      - ./tileserver:/usr/tileserver
-    ports:
-      - "8001:8001"
-      # required for the git display of the commit hash
-    command: >
-      sh -c "npm install && npm run dev"
-
 volumes:
   celery-SAM-model:

docs/DevDocs/Architecture.md

@@ -0,0 +1,58 @@
+# Architecture
+
+## Overview
+
+The basic RDWATCH systems is based on utilizing docker containers to create a full contained application for local development and deployment.
+
+- Django is used as the back-end ORM with Postgres/PostGIS as a database.  There are multiple Django apps.  One for base RDWATCH and one for the Scoring data.
+- Celery is used to execute and manage long running tasks.
+- MinIO/S3 is used as object storage for cropped/chipped satellite images.
+- Vue based front-end utilizing Vuetify for components and Maplibre for vector tile rendering
+
+## Django Overview
+
+### Core
+
+This is the base application for RDWATCH.  All endpoints that don't contain `api/v1/scoring/*` will utilize the base Core Application.
+This is also the mode location for all satellite image storage and downloading progress.
+
+### Scoring
+
+When the Environment Variable `RDWATCH_POSTGRESQL_SCORING_URI` is set the system expects to be connected to a second Database.  This database is based of the T&E metrics scoring database.  The RDWATCH team doesn't have direct control over the schema of this database so any data that is modified or stored should be placed in the RDWATCH database (This includes data like Stored Satellite Images or downloading progress).
+
+For reading from the database Django Models are created but the `managed` property in the `Meta` class is set to False.  This allows for reading from the database using similar django querysets and orm logic while not having django actively manage the system.  This means that when the DB changes the models need to be updated.
+
+The standard `rdwatch/core/views` endpoints for visualizing model-runs, sites, siteobservations are all mirrored in the Scoring application so they can access information directly from the scoring database.
+
+### Tasks
+
+Each app **core** and **scoring** has it's own tasks used in celery.  These are tasks that will download GeoJSON for an entire model run as well as downloading satellite images.  All information regarding satellite images are stored in the core rdwatch database because this project doesn't have access to modify the **scoring** database.
+
+## Stack Links
+
+### Django
+
+A single Django application (`rdwatch`) for the backend. Source code is in the ["rdatch"](https://github.com/ResonantGeoData/RD-WATCH/tree/main/rdwatch) folder.
+
+- [Django](https://docs.djangoproject.com/) with [GeoDjango](https://docs.djangoproject.com/en/5.0/ref/contrib/gis/)
+- [Django Ninja](https://django-ninja.dev/)
+- [Poetry](https://python-poetry.org/docs/) for dependency management
+
+### Vue
+
+The Vue-based SPA frontend. Source code is in the ["vue"](https://github.com/ResonantGeoData/RD-WATCH/tree/phase-ii/vue) folder.
+
+- [Vue 3](https://vuejs.org/guide/introduction.html)
+- [Vuetify](https://vuetifyjs.com/en/)
+- [MapLibre GL JS](https://maplibre.org/maplibre-gl-js-docs/api/)
+- [npm](https://docs.npmjs.com/) for dependency management
+
+### Services
+
+Services the application requires.
+
+- [NGINX Unit](https://unit.nginx.org/): serves both the Django application backend and the bundled static assets from the Vue frontend
+- [PostgreSQL](https://www.postgresql.org/docs/) and [PostGIS](http://www.postgis.net/documentation/): the RDWATCH database
+- [Redis](https://redis.io/docs/): caching and a job queue for celery
+- [MinIO/S3](https://min.io/): storage for satellite images for faster browsing
+- [Celery](https://docs.celeryq.dev/en/stable/getting-started/introduction.html): long running tasks for image chipping, downloading and compressing geoJSONs.

docs/DevDocs/GettingStarted.md

@@ -0,0 +1,154 @@
+# Getting Started
+
+This document gives an overview of the code contained in this monorepo and the recommended development setup.
+
+## Develop with Docker (recommended)
+This is the simplest configuration for developers to start with.
+
+1. Make a copy of `template.env` and call it `.env`.
+2. Set the environment variables in `.env`.
+3. Run `docker compose up` to start the Django development server and Celery worker, plus all backing services
+   like PostGIS, Redis, RabbitMQ, etc.
+4. Run `docker compose run --rm django poetry run django-admin migrate` to apply database migrations.
+5. Run `docker compose run --rm django poetry run django-admin loaddata lookups` to initialize your database with required data.
+7. Optionally, create an account for the Django admin (http://localhost:8000/admin) by running `docker compose run --rm django poetry --directory django run django-admin createsuperuser`
+8. If running the docker compose by default a client development server should be started at http://localhost:8080/
+9. On first login you will be redirected to the `Adminstrator` page.  This is for logging in.  Afterwards you should be able to redirect back to either http://localhost:8080/ or http://localhost:3000/.  The deployed version will automatically redirect.
+9. If doing local Client Development, start the client development server:
+   ```sh
+   cd vue
+   npm install
+   npm run dev
+   ```
+   The server will be started at http://localhost:3000 as to not conflict with the docker compose development service
+   When finished, use `Ctrl+C`
+
+## Develop Natively
+This configuration still uses Docker to run attached services in the background,
+but allows developers to run Python code on their native system.
+
+### Initial Setup
+1. Make a copy of `template.env` and call it `.env`.
+2. Set the environment variables in `.env`.
+3. Run `docker compose -f ./docker-compose.yaml up -d`
+4. Install Python 3.11
+5. Install
+   [`psycopg2` build prerequisites](https://www.psycopg.org/docs/install.html#build-prerequisites)
+6. Install Poetry
+7. Run `poetry --directory django install`
+8. Run the following command to configure your environment: `source ./dev/export-env.sh dev/.env.docker-compose-native ./dev/export-env.sh .env`
+10. Optionally, create an account for the Django admin (http://localhost:8000/admin) by running `poetry --directory django run django-admin createsuperuser`
+
+### Run Application
+1. Ensure `docker compose -f ./docker-compose.yaml up -d` is still active
+2. Run:
+   1. `source ./dev/export-env.sh dev/.env.docker-compose-native`
+   2. `source ./dev/export-env.sh .env`
+   3. `poetry run --directory django django/src/manage.py migrate`
+   4. `poetry run --directory django django/src/manage.py loaddata lookups`
+   5. `poetry run --directory django django/src/manage.py runserver`
+3. Run in a separate terminal:
+   1. `source ./dev/export-env.sh`
+   2. `poetry run --directory django celery --app rdwatch.celery worker --loglevel INFO --without-heartbeat`
+4. Run in another separate terminal:
+   1. `source ./dev/export-env.sh`
+   2. `poetry run --directory django celery --app rdwatch.celery beat --loglevel INFO`
+5. When finished, run `docker compose stop`
+6. To destroy the stack and start fresh, run `docker compose down`
+   1. Note: this command does not destroy docker volumes, such as those associated with the postgresql and minio services. To destroy those as well, run `docker compose down -v`.
+
+### A note on database migrations
+Note that database migrations are _not_ run automatically. Anytime a new migration is introduced, you must run the following command to apply it:
+
+`poetry --directory django run django-admin migrate`
+
+## Type support for ".vue" imports in VS Code
+
+Enable ["takeover mode"](https://github.com/johnsoncodehk/volar/discussions/471) for Volar.
+
+1. Disable built-in TypeScript extension:
+   1. Open the Command Palette (<kbd>⌘</kbd> <kbd>⇧</kbd> <kbd>P</kbd> or <kbd>Ctrl</kbd> <kbd>Shift</kbd> <kbd>P</kbd>) and run `>Extensions: Show Built-in Extensions` command
+   2. Find "TypeScript and JavaScript Language Features", right click and select "Disable (Workspace)"
+2. Reload VS Code
+
+## Stack
+
+The key software used to build the application.
+
+### Django
+
+A single Django application (`rdwatch`) for the backend. Source code is in the ["django"](https://github.com/ResonantGeoData/RD-WATCH/tree/phase-ii/django) folder.
+
+- [Django 4](https://docs.djangoproject.com/en/4.1/contents/) with [GeoDjango](https://docs.djangoproject.com/en/4.0/ref/contrib/gis/)
+- [Django Ninja](https://django-ninja.dev/)
+- [Poetry](https://python-poetry.org/docs/) for dependency management
+
+### Vue
+
+The Vue-based SPA frontend. Source code is in the ["vue"](https://github.com/ResonantGeoData/RD-WATCH/tree/phase-ii/vue) folder.
+
+- [Vue 3](https://vuejs.org/guide/introduction.html)
+- [Vuetify](https://vuetifyjs.com/en/)
+- [MapLibre GL JS](https://maplibre.org/maplibre-gl-js-docs/api/)
+- [npm](https://docs.npmjs.com/) for dependency management
+
+### Services
+
+Services the application requires.
+
+- [NGINX Unit](https://unit.nginx.org/): serves both the backend and the bundled static assets ([Dockerfile](https://github.com/ResonantGeoData/RD-WATCH/blob/phase-ii/Dockerfile))
+- [PostgreSQL 14](https://www.postgresql.org/docs/14/index.html) and [PostGIS 3.2](http://www.postgis.net/documentation/): data warehouse ([Dockerfile](https://github.com/ResonantGeoData/RD-WATCH/blob/phase-ii/docker/services/postgresql/Dockerfile))
+- [Redis 7](https://redis.io/docs/): caching (and maybe in the future as a job queue) ([Dockerfile](https://github.com/ResonantGeoData/RD-WATCH/blob/phase-ii/docker/services/redis/Dockerfile))
+
+### Ingesting Data
+
+#### Loading Ground Truth Data
+
+Within the ./scripts directory is a python script named `loadGroundTruth.py`.  This file can be used in conjunction with the ground truth annotaitons located in the annotation Repo:
+[Annotation Repo](https://smartgitlab.com/TE/annotations)
+Running a command like `python loadGroundTruth.py ~/AnnotationRepoLocation --skip_region` will load all of the annotations for the ground truth while skipping the regions.
+
+
+#### Loading Single Model Runs
+Within the ./scripts directory is a python script named `loadModelRuns.py`.  This can be used to load a folder filled with geojson data into the system by using a command like:
+
+```
+python loadModelRuns.py 'KR_0001' "./site_models/KR_R001_*.geojson" --title Test_Eval_12 --performer_shortcode 'KIT' --eval_num 12 --eval_run_num 0
+```
+Within this python file at the top is the rgd_endpoint variable which needs to be set to the server URL and port for where RGD is hosted.  By default this assumes running locally with `http://localhost:8000`
+Be sure that the system is up and running before running the commands.
+The above command will load the data in the site_models/KR_R001 files and give it the title 'Test_Eval_12'.  The eval_num and eval_run_num aren't required unless the scoring database is going to be connected to the system.  Within the script there is
+
+### Scoring
+
+The [Metrics and Test Framework](https://smartgitlab.com/TE/metrics-and-test-framework#creating-a-metrics-database) can be used in addition with RGD to display scores from results.
+In development mode a scoring Database is automatically initialized at URI: `postgresql+psycopg2://scoring:secretkey@localhost:5433/scoring`
+
+To score data:
+- Clone the [Metrics and Test Framework](https://smartgitlab.com/TE/metrics-and-test-framework) repo.
+- In the Metrics and Test Framework repo:
+  - Copy the `alembic_example.ini` to `alembic.ini` and set the `sqlalchemy.url = postgresql+psycopg2://scoring:secretkey@localhost:5433/scoring`
+  - Run `pip install -e .` to install the metrics-and-test-framework package
+  - Run `alembic upgrade head` to initialize the scoring database schema
+  - Execute the scoring code from inside the metrics and test framework:
+```
+  python -m iarpa_smart_metrics.run_evaluation \
+               --roi KR_R001 \
+               --gt_dir ../annotations/site_models/ \
+               --rm_dir ../KR_R001/region_models/ \
+               --sm_dir ../KR_R001/site_models/ \
+               --output_dir ../KR_R001/output \
+               --eval_num 12 \
+               --eval_run_num 0 \
+               --performer kit \
+               --no-viz \
+               --no-viz-detection-table \
+               --no-viz-comparison-table \
+               --no-viz-associate-metrics \
+               --no-viz-activity-metrics \
+               --sequestered_id KR_R001 \
+               --db_conn_str postgresql+psycopg2://scoring:secretkey@localhost:5433/scoring
+```
+- the rm_dir and sm_dir shgould be your test annotaitons.
+- gt annotations can be retrieved from the [Annotation Repo](https://smartgitlab.com/TE/annotations)
+- be sure to set the val_num and eval_run_num and remember them when ingesting data into RGD.  The region, eval_num, eval_run_num and performer are used to connect data loaded in RGD to the scoring data.