Release/1.0.0 (#365)

* chore: update portal resources to 1Gi mem
* Feature/224 (#231)
* #224 feat: Add about dialog
* #224 chore: Add about images
* #224 fix: Update about text
* #224 fix: removed typo '\'
Co-authored-by: Zoran Sinnema <[email protected]>
* #233 feat: Add coming soon messages (#235)
* #233 feat: Add coming soon messages
* #233 feat: disabled the comming soon items
* chore: pin sqlalchemy version <2, v2 has issues with pgpool2
* Feature/244 247 (#251)
* #244 feat: WIP add single file upload flow
* #244 feat: WIP - Update UI
* #244 feat: Update api
* #244 feat: Update UI error feedback
* #244 fix: Fix generate_population_cells command
* #244 fix: Fix initialize_single_file_upload command
* #244 fix: Fix get cells filepath from single file
* #247 chore: Update persistence files location (population and experiment)
* #247 feat: Add download experiment cells backend
* #247 fix: Fix experiment cells download
* #247 feat: Make zip file dynamic, depending on active populations
* #247 feat: Add frontend UI button to download active populations data
* #244 chore: Update cordmap
* #247 chore: Update api
* chore: Remove unsuccessful config fix
* chore: Fixate pandas version
* #244 feat: Refactor out common ground on population upload

* Feature/252 (#253)

* #252 feat: Add population registration strategy pattern

* #252 chore: Rename strategy classes

* fix: issue with migration order, first we need to add the is_fiducial field since it's needed during the image generation

* #254 chore: Upgrade requirements to meet ch 2.1 django version

* #254 chore: Reset migrations to fix build issues

* Feature/254 (#259)

* #254 chore: Upgrade requirements to meet ch 2.1 django version

* #254 chore: Reset migrations to fix build issues

* #255 feat: Allow adding populations to existent experiment (#263)

* chore: Upgrade itk version (#267)

* Feature/fix build (#268)

* chore: Upgrade itk version

* chore: Disable pip version check for cordmap install

* chore: Add step to install problematic package standalone

* Update README.md

* Update README.md use Python 3.9

* Update README.md

* Update README.md added deployment section

* 277 Empty Tags are also displayed in My Experiments (#299)

* #277 added check for tags

* #277 fixed misalignment

* chore: Update requirements (#300)

* chore: Remove geppetto-meta from codefresh-dev pipeline (#301)

* #257 experiment title added (#298)

* #257 experiment title added

* #257 changed the way of rendering fetched experiment

* #257 deleted unnecessary code

* Docs update with generate canal, grid, lamina, annotation images script (#302)

* #SALK-update-docs generate salk cord images

* docs update image generate script

* Update README position

* #257 fix: Update shared header to only fetch for experiment when needed

* #257 fix: Rollback conditional rendering of the experiment (#303)

* 284 Add hierarchy to neuronal populations --tbe-- (#308)

* #284 changes made

* #284 population entry ui changed

* #284 code cleaning

* #284 small changes

* #284 changes made

* #develop changed some styling

* Revert "#develop changed some styling"

This reverts commit a85b1f8.

* #256 rename frontend without permission check

* Feature/258: Delete Experiment  (#307)

* #258 adding delete experiment card

* #258 feat: integrate delete API and refresh experiments

* #258 rename modal

* #258 title and text changed for modal

* #258 cleanup

* #258 BE check and FE response error handling

* #258 cleanup

* Feature/243 (#275)

* #255 feat: Allow adding populations to existent experiment

* #261 feat: Add instant feedback on population updates

* #246 feat: WIP - Add residential populations

* #246 fix: Fix residential populations race condition

* #246 fix: Fix UI misalignments on resident populations

* #246 fix: Fix color picker not opening

* #246 fix: Fix residential populations race condition

* #243 feat: WIP - Update heatmap to contour plot

* #246 feat: Update sidebar design

* #246 feat: Update sidebar width

* #246 feat: Add complete data for residential populations; Update migration

* #246 chore: Remove unnecessary print

* #243 feat: WIP - Update contour creator

* #242 feat: WIP - Add atlas upscale service

* #243 feat: WIP - Update probability map calculations

* #242 fix: Fix interpolate_atlas_section

* #242 chore: Rename annotation to grey_and_white_matter

* #242 fix: Fix canal image offset

* #242 feat: Upscale grid dimensions

* #242 fix: Fix gray and white matter image generation

* #242 fix: Update hidden canvas dimensions

* #242 fix: Update white and grey matter image generation

* #242 fix: Fix centroids plot points position

* #242 fix: Update centroids plot points size

* #243 feat: Add contour plot image creator

* #243 feat: Add overlay to contours

* #243 feat: Rename probability map overlay to contour plot

* #243 chore: Improve resources management for plotting

* #243 chore: Rename density map type

* #243 chore: Add management command to update static images of all populations

* chore: Update geppetto-meta

* chore: Update codefresh-dev

* chore: Update codefresh-dev

* chore: Add missing icon

* chore: Update local setup script

* #243 feat: Make contour plot creator generate 2 images

* #243 feat: Add grid to contour plot UI control

* #242 feat: Update 3D meshes

* #243 fix: Update contour plot creator

* #243 feat: Update residential populations

* #243 fix: Fix typo in neuronal location switch

* #243 feat: Update contour levels

* #243 feat: Consider intensity on drawColoredImage

* #242 chore: Add 3D meshes

* #chore: Remove deprecated annotation changes

* #274 fix: Update download populations to work with residential populations

* #243 feat: Update static images creators

* #243 chore: Update residential populations data

* #243 fix: Update drawColoredImageCallback to allow for skipping intensity changes

* #243 chore: Update residential populations

* #243 fix: Fix background image generation

* #284 changes made

* #257 feat: Update cordmap

* #284 population entry ui changed

* #284 code cleaning

* #284 small changes

* #284 changes made

* #284 Integrating the populations with hierarchy accordion

* fix: Fix indentation typo

* #285 chore: Fix itk

* #284 cleanup and split function into two

* #284 disable subpopulation color change

* #285 fix: Fix indentation typo

* feat: Add residential populations back

* #243 chore: WIP - Merge with develop

* #284 feat: Connect show all

* #284 feat: Connect parent switch

* #284 feat: Connect color picker

* #284 feat: Connect download populations

* #284 fix: Fix population grouping

* #284 chore: Fix linting

* #243-b ui changes made

* 290 Add z-axis position in the 2D viewer (#311)

* #develop changed some styling

* Revert "#develop changed some styling"

This reverts commit a85b1f8.

* #290 added indicator cord image and label

---------

Co-authored-by: afonso pinto <[email protected]>

* Feature/276 (#315)

* #276 chore: Add mesh splitting to build pipeline

* #276 chore: Update residential populations migration

* #276 chore: Undo local dev change

* fix: Import real model instead of historic version

---------

Co-authored-by: Aiga115 <[email protected]>
Co-authored-by: D. Gopal Krishna <[email protected]>
Co-authored-by: Aigul Zhyldyzbekova <[email protected]>

* chore: Update cordmap to d0ef8af (#316)

* 273 Editing an experiment tag leads to blank page (#317)

* #273_c edit functionality added

* #273_c changes made

* fix: Fix delete tag

* chore: Fix linting errors

---------

Co-authored-by: afonso pinto <[email protected]>

* #256 - TBE push rename population

* Feature/291-fix - Change the dot size of populations - NEW (after changes) (#320)

* #255 feat: Allow adding populations to existent experiment

* #261 feat: Add instant feedback on population updates

* #246 feat: WIP - Add residential populations

* #246 fix: Fix residential populations race condition

* #246 fix: Fix UI misalignments on resident populations

* #246 fix: Fix color picker not opening

* #246 fix: Fix residential populations race condition

* #243 feat: WIP - Update heatmap to contour plot

* #246 feat: Update sidebar design

* #246 feat: Update sidebar width

* #246 feat: Add complete data for residential populations; Update migration

* #246 chore: Remove unnecessary print

* #243 feat: WIP - Update contour creator

* #242 feat: WIP - Add atlas upscale service

* #243 feat: WIP - Update probability map calculations

* #242 fix: Fix interpolate_atlas_section

* #242 chore: Rename annotation to grey_and_white_matter

* #242 fix: Fix canal image offset

* #242 feat: Upscale grid dimensions

* #242 fix: Fix gray and white matter image generation

* #242 fix: Update hidden canvas dimensions

* #242 fix: Update white and grey matter image generation

* #242 fix: Fix centroids plot points position

* #242 fix: Update centroids plot points size

* #243 feat: Add contour plot image creator

* #243 feat: Add overlay to contours

* #243 feat: Rename probability map overlay to contour plot

* #243 chore: Improve resources management for plotting

* #243 chore: Rename density map type

* #243 chore: Add management command to update static images of all populations

* chore: Update geppetto-meta

* chore: Update codefresh-dev

* chore: Update codefresh-dev

* chore: Add missing icon

* chore: Update local setup script

* #243 feat: Make contour plot creator generate 2 images

* #243 feat: Add grid to contour plot UI control

* #242 feat: Update 3D meshes

* #243 fix: Update contour plot creator

* #243 feat: Update residential populations

* #243 fix: Fix typo in neuronal location switch

* #243 feat: Update contour levels

* #243 feat: Consider intensity on drawColoredImage

* #242 chore: Add 3D meshes

* #chore: Remove deprecated annotation changes

* #274 fix: Update download populations to work with residential populations

* #243 feat: Update static images creators

* #243 chore: Update residential populations data

* #243 fix: Update drawColoredImageCallback to allow for skipping intensity changes

* #243 chore: Update residential populations

* #243 fix: Fix background image generation

* #284 changes made

* #257 feat: Update cordmap

* #284 population entry ui changed

* #284 code cleaning

* #284 small changes

* #284 changes made

* #284 Integrating the populations with hierarchy accordion

* fix: Fix indentation typo

* #285 chore: Fix itk

* #284 cleanup and split function into two

* #284 disable subpopulation color change

* #285 fix: Fix indentation typo

* feat: Add residential populations back

* #243 chore: WIP - Merge with develop

* #284 feat: Connect show all

* #284 feat: Connect parent switch

* #284 feat: Connect color picker

* #284 feat: Connect download populations

* #284 fix: Fix population grouping

* #284 chore: Fix linting

* #243-b ui changes made

* #291 dot size modal

* 290 Add z-axis position in the 2D viewer (#311)

* #develop changed some styling

* Revert "#develop changed some styling"

This reverts commit a85b1f8.

* #290 added indicator cord image and label

---------

Co-authored-by: afonso pinto <[email protected]>

* #291 feat: dot size adjustment for each population and all at once.

* #291 feat: improving UI

* #291 threeDViewer without active refresh

* #291 correct import from PIL instead of tkinter

* #291-fix: UI fixed

* #291-fix: Adding fixes and threeDViewer data flow

* #291 active refresh to threeDViewer

active-refresh - on change of the value in the dotsize dialog, it instantly reflects in the threeDViewer

* #291-fix cleanup

* #291 remove cached changes from 243

git rm --cached -r --ignore-unmatch applications/portal/frontend/src/assets/atlas/

* #291-fix type change cleanup

---------

Co-authored-by: afonso <[email protected]>
Co-authored-by: Aiga115 <[email protected]>
Co-authored-by: Aigul Zhyldyzbekova <[email protected]>

* #256 - rename population and subpopulation APIs

* #256 - rename API body logic

* #256 change openapi spec

* #256 openapi spec change

* #256 add parentName to subpopulations

* #256 logic integration

* #256 cleanup

* #288 temporary changes made

* #288 changes made

* #288 style changes

* #288 style changes made

* Feature/285 experiment registration (#324)

* #255 feat: Allow adding populations to existent experiment

* #261 feat: Add instant feedback on population updates

* #246 feat: WIP - Add residential populations

* #246 fix: Fix residential populations race condition

* #246 fix: Fix UI misalignments on resident populations

* #246 fix: Fix color picker not opening

* #246 fix: Fix residential populations race condition

* #243 feat: WIP - Update heatmap to contour plot

* #246 feat: Update sidebar design

* #246 feat: Update sidebar width

* #246 feat: Add complete data for residential populations; Update migration

* #246 chore: Remove unnecessary print

* #243 feat: WIP - Update contour creator

* #242 feat: WIP - Add atlas upscale service

* #243 feat: WIP - Update probability map calculations

* #242 fix: Fix interpolate_atlas_section

* #242 chore: Rename annotation to grey_and_white_matter

* #242 fix: Fix canal image offset

* #242 feat: Upscale grid dimensions

* #242 fix: Fix gray and white matter image generation

* #242 fix: Update hidden canvas dimensions

* #242 fix: Update white and grey matter image generation

* #242 fix: Fix centroids plot points position

* #242 fix: Update centroids plot points size

* #243 feat: Add contour plot image creator

* #243 feat: Add overlay to contours

* #243 feat: Rename probability map overlay to contour plot

* #243 chore: Improve resources management for plotting

* #243 chore: Rename density map type

* #243 chore: Add management command to update static images of all populations

* chore: Update geppetto-meta

* chore: Update codefresh-dev

* chore: Update codefresh-dev

* chore: Add missing icon

* chore: Update local setup script

* #243 feat: Make contour plot creator generate 2 images

* #243 feat: Add grid to contour plot UI control

* #242 feat: Update 3D meshes

* #243 fix: Update contour plot creator

* #243 feat: Update residential populations

* #243 fix: Fix typo in neuronal location switch

* #243 feat: Update contour levels

* #243 feat: Consider intensity on drawColoredImage

* #242 chore: Add 3D meshes

* #chore: Remove deprecated annotation changes

* #274 fix: Update download populations to work with residential populations

* #243 feat: Update static images creators

* #243 chore: Update residential populations data

* #243 fix: Update drawColoredImageCallback to allow for skipping intensity changes

* #243 chore: Update residential populations

* #243 fix: Fix background image generation

* #284 changes made

* #257 feat: Update cordmap

* #284 population entry ui changed

* #284 code cleaning

* #284 small changes

* #284 changes made

* #284 Integrating the populations with hierarchy accordion

* fix: Fix indentation typo

* #285 chore: Fix itk

* #284 cleanup and split function into two

* #284 disable subpopulation color change

* #285 fix: Fix indentation typo

* feat: Add residential populations back

* #243 chore: WIP - Merge with develop

* #284 feat: Connect show all

* #284 feat: Connect parent switch

* #284 feat: Connect color picker

* #284 feat: Connect download populations

* #284 fix: Fix population grouping

* #284 chore: Fix linting

* 290 Add z-axis position in the 2D viewer (#311)

* #develop changed some styling

* Revert "#develop changed some styling"

This reverts commit a85b1f8.

* #290 added indicator cord image and label

---------

Co-authored-by: afonso pinto <[email protected]>

* #276 chore: Add mesh splitting to build pipeline

* #276 chore: Update residential populations migration

* #276 chore: Undo local dev change

* fix: Import real model instead of historic version

* 285 feat: WIP - Change registration workflow

* #285 feat: Fix typos on new registration flow

* fix: Hotfix cordmap

* chore: Remove debugging logs

* chore: Refactor code according to PR feedback

---------

Co-authored-by: Aiga115 <[email protected]>
Co-authored-by: D. Gopal Krishna <[email protected]>
Co-authored-by: Aigul Zhyldyzbekova <[email protected]>

* fix: Correct parameters order (#325)

* #288 style changes

* #288 pagination style change

* #288 changes made

* #288 style changes made

* #288 changes made

* Feature/294 (#326)

* #294 fix: Update experiment update conditions

* fix: Fix registration of fiducial experiments

* feat: Move populations to error state if cordmap doesn't produce cell files

* fix: Rollback custom cordmap changes

* #294 fix: Improve update experiment logic

* #288 details viewer changes made

* #288 models and migrations

* #288 utils and requirements

* #288 api definition

* Feature/286 (#330)

* #294 fix: Update experiment update conditions

* fix: Fix registration of fiducial experiments

* feat: Move populations to error state if cordmap doesn't produce cell files

* fix: Rollback custom cordmap changes

* #294 fix: Improve update experiment logic

* #286 feat: Change csv headers at download time

* fix: Update population cells file location

* #328 nested elements ui changes

* #288 resolving comments

* #288 pdf ui changes

* #256 - Check multiple @

* #288 changes to pdf ui styling

* #256 New changes - untested

* #288 changes made

* #288 changes made

* #288 changing naming of variables

* #288 category functionality changed

* #288 removing files changed

* #288 icons small changes

* #256 rename cleanning

* #256 update the openapi spec

removing the subpopulation concept from BE

* #256 FE integrate the API for rename 

Using the same API. Only risk could be - when any of the API fail and that could create another population.

* #256 cleanup

* #288 rename variables

* #288 BE changes

* #288 BE cleanup

* #288 dockerfile and openapi change

* #288 pdf-scripts and openapi spec

* #288 passing mesh info change in 3dviewer

* #288 pdf viewer

* #288 BE updates for permissions

* #288 cleanup

* #288 migrations updated and merged

* #288 cleanup

* #288 feat: WIP - Update PDF upload and render logic

* #288 feat: Update PDF upload and api

* #288 pagination styling full view

* #288 fix: Remove renderTextLayer

* #335 Fix active population download text

* #342 replace the tags API

* #343 implement copy link + disable the unimplemented feature - share experiments

* #342 the search feature

* #342 filtering and cleanup

* #256 naming correction and tuple

* #256 snackbar for error in rename

* #256 Do not allow to rename when not finished

* #338 changing permissions for pdf upload

* #256 simplifying is_user_owner

* #341 changes made

* #256 chore: Fix linter

* chore: Add workaround to see DetailsViewer

* #341 feat: Add hasEditPermission to DetailsViewer

* #341 chnages made

* #341 small change

* #341 delete and pagination handled

* #341 title change made

* #288 fix: Fix typo in double press check

* #288 fix: Fix incorrect grouping

* #288 fix: Fix population patch

* #341 fix: Update prop name

* #350 fix: Wait for all children color change before updating

* fix: Fix snackbar typo

* #256 fix: Update population name only if request is successful

* #342 fix tags option

* #342 filter dropdown fix styling

* chore: Fix linting problem

* #341 changes made

* #352: comment the non-implemented sections from My Profile dialog

Instead of disabling , commenting it out for later implementation.

* chore: Remove placeholder user image

* #337 changed dialog component to popover

* #341 search issue solved

* #356 fix: Fix tagsOptions incorrect mapping

* #340 disabled prop for show all switch is added

* chore: Fix linting and add safe check on user

* #258 refactor: Define has_object_destroy_permission instead of custom rest endpoint logic

* fix: Update experiment permissions

* fix: Update pdf permissions

* fix: Update pdf permissions

* #340 changes made

* chore: updated README

* Update cordmap to latest sow4 (#366)

* chore: Update latest cordmap (e9e12e6)

* chore: Add cordmap column swap back

---------

Co-authored-by: Zoran Sinnema <[email protected]>
Co-authored-by: Afonso Pinto <[email protected]>
Co-authored-by: Aigul Zhyldyzbekova <[email protected]>
Co-authored-by: D. Gopal Krishna <[email protected]>
Co-authored-by: Aiga115 <[email protected]>
Co-authored-by: D. Gopal Krishna <[email protected]>

.gitignore

@@ -11,3 +11,7 @@ skaffold.yaml
 applications/workspaces/backend/static
 .overrides
 **/*.sqlite3
+
+applications/portal/frontend/src/assets/atlas/salk_cord_10um/*
+applications/portal/backend/persistent/*
+applications/portal/backend/data/*

README.md

@@ -7,23 +7,51 @@ The SALK Mouse Cord Atlas
 ### Prerequisites
 
 The SALK deployment is built on top of [CloudHarness](https://github.com/MetaCell/cloud-harness).
-The deployment process is based on Python 3.7+ scripts. It is recommended to setup a virtual 
-environment first.
+CloudHarness is a base infrastructure and common utilities for microservice applications deployed on Kubernetes.
+
+CloudHarness and SALK use Python 3.9, it is recommended to setup a virtual environment first.
 
 With conda: 
 ```bash
-conda create --name salk python=3.7
+conda create --name salk python=3.9
 conda activate salk
 ```
 
 To install CloudHarness:
 
 ```
-git clone --branch release/1.0.0 https://github.com/MetaCell/cloud-harness.git
+git clone --branch release/2.1.0 https://github.com/MetaCell/cloud-harness.git
 cd cloud-harness
 pip install -r requirements.txt
 ```
 
+
+
+## Deploy to a K8s cluster
+
+Make sure that you are using the correct K8s context (check `kubectl config get-contexts`)
+Or set your context using `kubectl config set-context`
+
+Requirements:
+* helm installed (https://helm.sh/docs/intro/install/)
+* kubectl installed (https://kubernetes.io/docs/tasks/tools/)
+* skaffold installed (https://skaffold.dev/docs/install/)
+
+
+```bash
+conda activate salk
+
+# prepare the Helm chart
+harness-deployment cloud-harness . -t <docker tag> -d <the.domain.com> -r <docker registry> -rs <docker registry secret> -n <namespace> -e prod -i portal
+
+# build the images
+skaffold build
+
+# install the application
+helm upgrade salk ./deployment/helm --install --reset-values --version 0.0.1 --namespace <namespace> --values ./deployment/helm/values.yaml --timeout 600s
+```
+
+
 ## Development setup
 
 Minikube is recommended to setup locally. The procedure is different depending on where Minikube is installed.
@@ -33,7 +61,7 @@ Run the `setup.sh` script to setup your minikube cluster.
 The script will also build all Docker images and installs the deployment using `skaffold`
 
 Requirements:
-* minikube installed
-* kubectl installed
-* skaffold installed
-* helm installed
+* helm installed (https://helm.sh/docs/intro/install/)
+* kubectl installed (https://kubernetes.io/docs/tasks/tools/)
+* minikube installed (https://minikube.sigs.k8s.io/docs/start/)
+* skaffold installed (https://skaffold.dev/docs/install/)

applications/portal/.gitignore

@@ -2,4 +2,5 @@ public
 dist
 node_modules
 .vscode
-
+backend/static/*
+frontend/src/assets/atlas/*

applications/portal/Dockerfile

@@ -18,6 +18,7 @@ RUN python3 -m pip install -r requirements.txt --no-cache-dir --upgrade
 
 COPY backend/requirements.txt backend/setup.py .
 RUN python3 -m pip install -e .
+RUN apt-get update && apt-get install poppler-utils -y
 
 COPY backend ${APP_DIR}
 RUN python3 manage.py collectstatic --noinput
@@ -28,16 +29,18 @@ RUN mkdir -p ${APP_DIR}/static/www
 RUN python3 manage.py generate_canal_images salk_cord_10um
 RUN python3 manage.py generate_grid_images salk_cord_10um
 RUN python3 manage.py generate_lamina_images salk_cord_10um
-RUN python3 manage.py generate_annotation_images salk_cord_10um
+RUN python3 manage.py generate_grey_and_white_matter_images salk_cord_10um
+RUN python3 manage.py split_meshes salk_cord_10um
 
 # compile frontend
-FROM node:16-alpine as frontend
+FROM node:18-alpine as frontend
 
 ENV BUILDDIR=/builddir
 ENV APP_DIR=/usr/src/app
 
 COPY --from=backend ${APP_DIR}/persistent/ ${BUILDDIR}/src/assets/atlas/salk_cord_10um/
-RUN rm -rf ${BUILDDIR}/src/assets/atlas/salk_cord_10um/*.sqlite3
+RUN rm -rf ${BUILDDIR}/src/assets/atlas/salk_cord_10um/populations/*
+RUN rm -rf ${BUILDDIR}/src/assets/atlas/salk_cord_10um/*.json && rm -rf ${BUILDDIR}/src/assets/atlas/salk_cord_10um/*.sqlite3
 COPY frontend/package.json ${BUILDDIR}/package.json
 COPY frontend/yarn.lock ${BUILDDIR}/yarn.lock
 WORKDIR ${BUILDDIR}

applications/portal/README.md

@@ -35,6 +35,15 @@ sudo ln -s "$NVM_DIR/versions/node/$(nvm version)/bin/npm" "/usr/local/bin/npm"
 The local webpack dev server uses [keycloak_dev.json](src/assets/keycloak_dev.json) for connecting to the keycloak accounts system.
 Please check this file and change the domain according to your setup.
 
+
+### Generate SALK Cord images
+
+To generate SALK Cord images, once all the requirements for backend is installed, activate the virtual environment. And run the following command to generate the Salk Cord images .
+```
+./generate_images.sh
+```
+
+
 ### Self Signed Certificates
 
 When running on a local minikube please make sure you import the generated cacert certificates. The certificate file is most likely to be found here ./deployment/helm/resources/certs/cacert.crt