Ease of Install and Upgrade for Sunbird Ed

[beepdot]

Dear All,

We have come up with a plan to simplify and ease the installation and upgrade process for Sunbird Ed. I would request each of the BB owners to have a look at this confluence page.

We have listed few points on how to achieve this and have planned it in two phases. Please comment back in this discussion thread if you have any queries or suggestions.

We will also add a detailed execution plan, technical design document and a roadmap for each of these items in a couple of days.

@Krishnaj20 Can you please tag the required audience in this discussion?

@SanthoshVasabhaktula @vrayulu @gandham-santhosh

[Krishnaj20]

@rhwarrier @reshmi-nair @pallakartheekreddy @vinukumar-vs @parthlawate @coolbung @surendrasinghs @luckynvdu @anandvarada @vijiurs @aishwaryashikshalokam @Prateek-slokam @aks30 @varsha-sl @MeghnaSL -
Each BB also needs to do some work for ED to meet the goals outlined in the document. Please go thru and add any questions/feedback you may have.

[gandham-santhosh]

Please find the RFC link for Ed one click installation https://project-sunbird.atlassian.net/browse/ED-881 . As Ed has dependency on other BB's we will first start working on one click installers for Lern, InQuiry and CoKreat and then create installer for Ed leveraging the dependent BB installer. Obsrv BB already have one click installer cc: @anandp504 and Knwlg BB team will be working on one click installer cc: @pallakartheekreddy
We have created stories for Lern one click installers and same will be done for the other BB's.

We will also setup a meeting soon to walk through the approach and get feedback from the community. Please feel free to add any questions/feedback

[Krishnaj20]

@gandham-santhosh - thanks. It will be good to know the timelines your team is targeting to complete this initiative.

[gandham-santhosh]

Hi Team, we had a review meeting yesterday about the same. Please find the recording of the session
https://us06web.zoom.us/rec/share/XWp9nkNfjgb_feqlBop-SX7xyEhRfj3L_RAEs-6EMOmPAfji5P__mFzFp67RIHH7.o35zFjNvWvQIFoSX
Passcode: v%QUXre7

Summary:

• Objective of One click installer is to make adopter experience Sunbird quickly
• Focus is on completing the Ed one click installer by 31st March
• Follow Helm umbrella and sub chart approach
• Publish container images and Helm charts
• Try to deploy all the applications including databases to the k8s cluster
• We will have documentation to setup scale and manage dev/qa/prod like environments​
• Automation of cloud infra provisioning and automated publishing of artifacts (container images, Helm charts) will be taken up in the next phase

Please feel free to comment here if you have any questions and feedback

[rhwarrier]

@gandham-santhosh will this subsume one click installers for other building blocks as well, or will that be taken up as a different exercise?
cc: @reshmi-nair

[gandham-santhosh]

Hi Team, I am sorry to inform you all that there is been a delay in completing one click installation of Ed due to non availability of team members as planned and other interrupts. I would like to call the status as RED.

We are currently assessing the situation and working on a revised timeline. I will provide you with an updated timeline as soon as possible. Thank you for your understanding

[Krishnaj20]

@gandham-santhosh - it will be helpful if you can share a summary of what activities are complete and what are pending.

[gandham-santhosh]

Completed items:

• Helm charts for cert-registry, cert, certificateapi, certificatesign, discussionsmw, groups, learner, lms, nodebb, notification, registry, opa-policies, user-access-key, admin-utils, Lern-flink jobs, Cassandra, redis, Neo4j, Postgres, Elasticsearch, kafka, keycloak

In-progress items:

• Initialising the databases and non-containerised services with schema/initial data
  Cassandra, NEo4j, Elasticsearch, Postgres, Kafka and Keycloak
• Installer bash script for Lern BB

TODO items:

• Write helm charts for InQuiry services (Question set, Question set publisher)
• Write helm charts for Ed microservices (Portal, Kong, Nginx, ML-services)
• Create Ed bash installer
• Create bash installer for Observ to leverage the Helm charts from Observ one click installer and integrate with Ed installer
• Publish final docker images, helm artifacts for each BB
• Integrating with Knowlg one click installer
• Integrating with functional configuration postman collection
• Testing and verify the Ed installer as a whole

[gandham-santhosh]

Hi Team,
Please find the below deliverables done so far

Microservices	Path to Helm chart/Script
cert-registry	https://github.com/Sunbird-Lern/certificate-registry/tree/helmchart/helm
cert	https://github.com/Sunbird-Lern/cert-service/tree/helmchart/helm
certificateapi	https://github.com/Sunbird-Lern/certificate-registry/tree/helmchart/helm
certificatesign	https://github.com/Sunbird-Lern/certificate-registry/tree/helmchart/helm
discussionsmw	https://github.com/Sunbird-Lern/discussions-middleware/tree/helmchart/helm
groups	https://github.com/Sunbird-Lern/groups-service/tree/helm-chart/helm
learner	https://github.com/Sunbird-Lern/sunbird-lms-service/tree/helmchart/helm
lms	https://github.com/Sunbird-Lern/sunbird-course-service/tree/helmchart/helm
nodebb	https://github.com/Sunbird-Lern/sunbird-nodebb/tree/helmchart/helm
notification	https://github.com/Sunbird-Lern/sunbird-notification-service/tree/helmchart/helm
registry	https://github.com/Sunbird-Lern/certificate-registry/tree/helmchart/helm/registry
admin-utils	https://github.com/Sunbird-Lern/sunbird-apimanager-util/tree/helmchart/helm
Lern Flink jobs	https://github.com/Sunbird-Lern/data-pipeline/tree/helmchart/helm/lern-flink-jobs
Lern Installer	https://github.com/project-sunbird/sunbird-devops/blob/oneclickinstaller/oneclickinstaller
Cassandra and data initialisation	https://github.com/Sunbird-Ed/sunbird-helmchart-common/tree/main/cassandra
Redis	https://github.com/Sunbird-Ed/sunbird-helmchart-common/tree/main/redis
Postgres and data initilisation	https://github.com/Sunbird-Ed/sunbird-helmchart-common/tree/main/postgresql
Elasticsearch	https://github.com/Sunbird-Ed/sunbird-helmchart-common/tree/main/elasticsearch
Kafka setup and initialisation	https://github.com/Sunbird-Ed/sunbird-helmchart-common/tree/main/kafka
keycloak	https://github.com/Sunbird-Ed/sunbird-helmchart-common/tree/main/keycloak

In-progress items:

• Write helm charts for InQuiry services (Question set, Question set publisher)
• Create bash installer for Observ to leverage the Helm charts from Observ one click installer and integrate with Ed installer
• Write helm charts for Ed microservices (Portal, Kong, Nginx, ML-services)
• Publish final docker images, helm artifacts for each BB
• Creating postman collection for functional configurations

TODO Items:

• Create Ed bash installer
• Integrating with Knowlg one click installer
• Integrating with functional configuration postman collection
• Testing and verify the Ed installer as a whole

We are targeting to complete this activity by end of April

[gandham-santhosh]

Apologies. Missed updating the status of last week deliverable. Here are the deliverables as of 6th April 2023:

Microservices	Path to Helm chart/Script
Observ flink jobs	https://github.com/santhosh-tg/obsrv-automation/tree/helmchart/flink
Druid operator	https://github.com/santhosh-tg/obsrv-automation/tree/helmchart/druid-operator
Druid setup	https://github.com/santhosh-tg/obsrv-automation/tree/helmchart/druid-cluster
InQuiry assessment	https://github.com/saiakhil46/inquiry-api-service/tree/helmchart/helm/assessment
InQuiry flink jobs	https://github.com/saiakhil46/data-pipeline-1/tree/helmchart/helm/inquiry-flink-jobs
Neo4J	https://github.com/saiakhil46/sunbird-helmchart-common/tree/main/neo4j
Lern global vars	https://github.com/project-sunbird/sunbird-devops/blob/oneclickinstaller/oneclickinstaller/global-values.yaml
Knowlg Helm charts	https://github.com/aimansharief/knowledge-platform/tree/knowlg-oneclick/knowlg-automation/helm_charts

In-progress items:
Create bash installer for Observ to leverage the Helm charts from Observ one click installer and integrate with Ed installer
Write helm charts for Ed microservices (Portal, Kong, Nginx, ML-services)
Publish final docker images, helm artifacts for each BB
Creating postman collection for functional configurations

TODO Items:
Create Ed bash installer
Integrating with Knowlg one click installer
Integrating with functional configuration postman collection
Testing and verify the Ed installer as a whole

[gandham-santhosh]

We are currently working on

• Integrating functional configurations (Content creation workflow) as part of easy installer
• Benchmarking the top used consumption API's
  We will share an update by end of this week regarding these activities

@manjirisulekavadi @vinukumar-vs @Krishnaj20 @chitranshu-keshav @santhosh-tg @tsprasath

[gandham-santhosh]

Hi Team,
We completed integrating with the Postman collection of content creation functional configurations last Friday and integration to the courses is in progress. @vinukumar-vs and team are working on postman collection for certificates