Code

CNPG DBConfig#2

Code

Open

yuyiguo wants to merge 125 commits intoyuyiguo:DB-update-2024-05-22yuyiguo/rubin-rucio-deploy:DB-update-2024-05-22from slaclab:mainslaclab/rubin-rucio-deploy:mainCopy head branch name to clipboard

+1,864-1,886

Commits

Commits on May 13, 2024

• Add hermes section to server to prevent ActiveMQ message queuing in messages table

  Thysk

  committed
  2f22061

  Copy full SHA for 2f22061

• Add hermes section with used services_list into server values

  Thysk

  committed
  44df417

  Copy full SHA for 44df417

• Add opensearch values and make files to be incorperated into Rubin Rucio build

  Thysk

  committed
  7205ee6

  Copy full SHA for 7205ee6

Commits on May 20, 2024

• Add opensearch to kustomization and makefiles

  Thysk

  committed
  ebc0a7f

  Copy full SHA for ebc0a7f

• add opensearch version to makefile

  Thysk

  committed
  703e04a

  Copy full SHA for 703e04a

• add opensearch repo to helm command

  Thysk

  committed
  36ee360

  Copy full SHA for 36ee360

• uncomment ns.yaml in kustomization

  Thysk

  committed
  9a5f770

  Copy full SHA for 9a5f770

Commits on May 22, 2024

• Fix opensearch directory depth, and repo typo in top level prod kustomize

  Thysk

  committed
  5f67052

  Copy full SHA for 5f67052

• typo and added opensearch to apply

  Thysk

  committed
  527d054

  Copy full SHA for 527d054

• Merge branch 'slaclab:main' into add-opensearch

  Thysk

  authored
  5fa43ce

  Copy full SHA for 5fa43ce

• Merge branch 'slaclab:main' into main

  Thysk

  authored
  aed8d81

  Copy full SHA for aed8d81

• Merge branch 'slaclab:main' into Prevent-activeMQ-messages

  Thysk

  authored
  18c3d8e

  Copy full SHA for 18c3d8e

• remove emptyDir from podSecurityPolicy.volumes

  Thysk

  committed
  70258b5

  Copy full SHA for 70258b5

• remove secret that was unused in my deployment

  Thysk

  committed
  c6a4414

  Copy full SHA for c6a4414

Commits on May 23, 2024

• reduce VmMaxMap count to match cluster

  Thysk

  committed
  2be5da3

  Copy full SHA for 2be5da3

• updated to use opensearch version 2.14

  Thysk

  committed
  6a2cdfb

  Copy full SHA for 6a2cdfb

• Reenable HermesK and Kafka support. Needs both to work.

  bjwhite-fnal

  committed
  c7a6644

  Copy full SHA for c7a6644

• Merge pull request #27 from bjwhite-fnal/main

  Show description for c5ecf18
  bjwhite-fnal

  authored
  c5ecf18

  Copy full SHA for c5ecf18

• Make patchfile naming consistent because I keep grepping for the wrong thing.

  bjwhite-fnal

  committed
  80c2f73

  Copy full SHA for 80c2f73

• Merge pull request #28 from bjwhite-fnal/main

  Show description for b369804
  bjwhite-fnal

  authored
  b369804

  Copy full SHA for b369804

Commits on May 24, 2024

• Merge pull request #24 from Thysk/Prevent-activeMQ-messages

  Show description for a215d20
  bjwhite-fnal

  authored
  a215d20

  Copy full SHA for a215d20

Commits on May 29, 2024

• enable sysctlInit to set sysctl settings using init container, and reset MaxMapCount to default

  Thysk

  committed
  cd458fe

  Copy full SHA for cd458fe

• add sysctlInit and resetMaxMapCount to default

  Thysk

  committed
  bce1d8e

  Copy full SHA for bce1d8e

• Merge branch 'main' of https://github.com/Thysk/rubin-rucio-deploy into main

  Thysk

  committed
  aa4418e

  Copy full SHA for aa4418e

• change elastic_endpoint to opensearch cluster, add use_ssl to false for messaging-hermes

  Thysk

  committed
  3e9b17a

  Copy full SHA for 3e9b17a

• change the RUCIO_CFG_HERMES_ELASTIC_USERNAME to match the opensearch username

  Thysk

  committed
  dd5084c

  Copy full SHA for dd5084c

Commits on May 30, 2024

• make change to server elastic_endpoint as well

  Thysk

  committed
  709104f

  Copy full SHA for 709104f

• Testing most recent version of HermesK

  bjwhite-fnal

  committed
  85be30a

  Copy full SHA for 85be30a

• Add env var to turn on HermesK.

  bjwhite-fnal

  committed
  3122984

  Copy full SHA for 3122984

Commits on Jun 5, 2024

• Merge pull request #25 from Thysk/add-opensearch

  Show description for 983b8f9
  bjwhite-fnal

  authored
  983b8f9

  Copy full SHA for 983b8f9

Commits on Jun 6, 2024

• Add hermes section to server to prevent ActiveMQ message queuing in messages table

  Thysk

  committed
  50372e1

  Copy full SHA for 50372e1

• Merge branch 'main' of https://github.com/Thysk/rubin-rucio-deploy into main

  Thysk

  committed
  8e66c9d

  Copy full SHA for 8e66c9d

• Merge branch 'slaclab:main' into main

  Thysk

  authored
  564b231

  Copy full SHA for 564b231

• Increase persistant volume request from 30 to 300GB as requested by Wei

  Thysk

  committed
  2369f30

  Copy full SHA for 2369f30

• Merge pull request #33 from yuyiguo/DB-update-2024-05-22

  Show description for 1f9f0e8
  bjwhite-fnal

  authored
  1f9f0e8

  Copy full SHA for 1f9f0e8

• Merge pull request #30 from Thysk/main

  Show description for e6e9cee
  bjwhite-fnal

  authored
  e6e9cee

  Copy full SHA for e6e9cee

• Merge pull request #29 from Thysk/Hermes-output-from-ElasticSearch-to-OpenSearch

  Show description for a84a554
  bjwhite-fnal

  authored
  a84a554

  Copy full SHA for a84a554

Commits on Jun 7, 2024

• some input format changed and some commented out for testing.

  yuyiguo

  committed
  43f6191

  Copy full SHA for 43f6191

Commits on Jun 12, 2024

• Give the hermesk daemon the proxy it was looking for.

  bjwhite-fnal

  committed
  6c3454a

  Copy full SHA for 6c3454a

• Use the bootstrap ip.

  bjwhite-fnal

  committed
  185b626

  Copy full SHA for 185b626

Commits on Jun 17, 2024

• Syntax fix.

  bjwhite-fnal

  committed
  5128a1e

  Copy full SHA for 5128a1e

• Mount ca-cert volume to Hermes.

  bjwhite-fnal

  committed
  4036817

  Copy full SHA for 4036817

• Disable HermesK for production.

  bjwhite-fnal

  committed
  90f2c39

  Copy full SHA for 90f2c39

• Enable ca cert mount for Automatix in production.

  bjwhite-fnal

  committed
  eae897e

  Copy full SHA for eae897e

Commits on Jun 20, 2024

• add SLAC_BUTLER_DISK & SLAC_BUTLER_DEV_DISK to hermes hermes topic list

  s-sajid-ali

  committed
  808d162

  Copy full SHA for 808d162

Commits on Jun 21, 2024

• update postgreSQL DB to 14.12-5 from 14.4-7

  yuyiguo

  committed
  a5da5d0

  Copy full SHA for a5da5d0

Commits on Jun 24, 2024

• Merge pull request #35 from yuyiguo/2024-06-18-dev-DBConfg

  Show description for 26c6fe9
  bjwhite-fnal

  authored
  26c6fe9

  Copy full SHA for 26c6fe9

• Merge pull request #34 from s-sajid-ali/add_rses_to_hermes

  Show description for d372b01
  bjwhite-fnal

  authored
  d372b01

  Copy full SHA for d372b01

Commits on Jul 24, 2024

• Updates to HermesK and Kafka stuff.

  bjwhite-fnal

  committed
  dba46e4

  Copy full SHA for dba46e4

• Update server to send Kafka messages.

  bjwhite-fnal

  committed
  cba95db

  Copy full SHA for cba95db

• Reconciliate with upstream.

  bjwhite-fnal

  committed
  b1da0f0

  Copy full SHA for b1da0f0

• Merge pull request #37 from bjwhite-fnal/main

  Show description for 442e9cd
  bjwhite-fnal

  authored
  442e9cd

  Copy full SHA for 442e9cd

Commits on Jul 29, 2024

• updated/added FTS groupBulk to 50 (the default rucio bulk is 100).

  yuyiguo

  committed
  c1ff810

  Copy full SHA for c1ff810

Commits on Jul 30, 2024

• Add HermesK configuration to production. Ready for production workflow testing.

  bjwhite-fnal

  committed
  88e9391

  Copy full SHA for 88e9391

• Merge pull request #39 from bjwhite-fnal/main

  Show description for eb8e0ee
  bjwhite-fnal

  authored
  eb8e0ee

  Copy full SHA for eb8e0ee

• Merge pull request #38 from yuyiguo/update-groupBulk-20240729

  Show description for b4fc25a
  bjwhite-fnal

  authored
  b4fc25a

  Copy full SHA for b4fc25a

• Remove Open/Elasticsearch. Opensearch is going to its own vCluster in the future.

  bjwhite-fnal

  committed
  350363f

  Copy full SHA for 350363f

• Merge remote-tracking branch 'upstream/main'

  bjwhite-fnal

  committed
  69955c4

  Copy full SHA for 69955c4

• Merge pull request #40 from bjwhite-fnal/main

  Show description for d569b49
  bjwhite-fnal

  authored
  d569b49

  Copy full SHA for d569b49

Commits on Aug 6, 2024

• Remove open/elasticsearch stuff from top level Makefile.

  bjwhite-fnal

  committed
  52a0860

  Copy full SHA for 52a0860

• Add SLAC_BUTLER_DISK for production.

  bjwhite-fnal

  committed
  c1f6b6f

  Copy full SHA for c1f6b6f

• Merge pull request #41 from bjwhite-fnal/main

  Show description for 3a63163
  bjwhite-fnal

  authored
  3a63163

  Copy full SHA for 3a63163

Commits on Sep 17, 2024

• Update HermesK topic_list to values specified in DM-46355

  bjwhite-fnal

  committed
  8474406

  Copy full SHA for 8474406

• Merge remote-tracking branch 'upstream/main'

  bjwhite-fnal

  committed
  b1b291a

  Copy full SHA for b1b291a

• Merge pull request #42 from bjwhite-fnal/main

  Show description for d1cdb26
  bjwhite-fnal

  authored
  d1cdb26

  Copy full SHA for d1cdb26

Commits on Sep 18, 2024

• Re enable elasitic service and change endpoint to new OS and indexes for PROD and DEV

  

  thysk

  committed
  54e9389

  Copy full SHA for 54e9389

• correct secrets for hermes

  

  thysk

  committed
  28f2cdb

  Copy full SHA for 28f2cdb

• Update deployed Kafka topics to match HermesK config in DM-46355.

  bjwhite-fnal

  committed
  fe7d735

  Copy full SHA for fe7d735

• Merge pull request #44 from bjwhite-fnal/main

  Show description for 1de0a29
  bjwhite-fnal

  authored
  1de0a29

  Copy full SHA for 1de0a29

Commits on Sep 25, 2024

• Merge pull request #43 from Thysk/Enable-OS-and-point-to-index

  Show description for 091df3a
  bjwhite-fnal

  authored
  091df3a

  Copy full SHA for 091df3a

Commits on Sep 30, 2024

• add logstash for dev and prod environments

  

  thysk

  committed
  bd7f464

  Copy full SHA for bd7f464

Commits on Oct 2, 2024

• Merge pull request #45 from Thysk/add-logstash-for-monitoring

  Show description for 04afee5
  bjwhite-fnal

  authored
  04afee5

  Copy full SHA for 04afee5

Commits on Oct 7, 2024

• Upgrade dev deployment to 35.4.0.

  bjwhite-fnal

  committed
  c74f052

  Copy full SHA for c74f052

• Upgrade prod to 35.4.0.

  bjwhite-fnal

  committed
  4601c37

  Copy full SHA for 4601c37

• Merge pull request #46 from bjwhite-fnal/main

  Show description for cbf62dc
  bjwhite-fnal

  authored
  cbf62dc

  Copy full SHA for cbf62dc

Commits on Oct 8, 2024

• Increase Hermes chunk size.

  bjwhite-fnal

  committed
  caf32d0

  Copy full SHA for caf32d0

• Merge pull request #47 from bjwhite-fnal/main

  Show description for 9437515
  bjwhite-fnal

  authored
  9437515

  Copy full SHA for 9437515

• Update Hermes bulk further.

  bjwhite-fnal

  committed
  90e340e

  Copy full SHA for 90e340e

• Merge pull request #48 from bjwhite-fnal/main

  Show description for 5eb51f5
  bjwhite-fnal

  authored
  5eb51f5

  Copy full SHA for 5eb51f5

Commits on Oct 9, 2024

• Adjust kafka Topic config. Increase PVCs to 5G for Kafka.

  bjwhite-fnal

  committed
  e599693

  Copy full SHA for e599693

• Merge pull request #49 from bjwhite-fnal/main

  Show description for 0028120
  bjwhite-fnal

  authored
  0028120

  Copy full SHA for 0028120

Commits on Oct 14, 2024

• Fix typo in automatix config.

  bjwhite-fnal

  committed
  32c30fb

  Copy full SHA for 32c30fb

• Merge pull request #50 from bjwhite-fnal/main

  Show description for dc5c894
  bjwhite-fnal

  authored
  dc5c894

  Copy full SHA for dc5c894

Commits on Oct 23, 2024

• Move prod logstash directory to match dev.

  bjwhite-fnal

  committed
  70bb175

  Copy full SHA for 70bb175

• Remove unused Elasticsearch deployment. Exists in its own vCluster now.

  bjwhite-fnal

  committed
  f5e6e2f

  Copy full SHA for f5e6e2f

• Merge pull request #51 from bjwhite-fnal/main

  Show description for 25f10bc
  bjwhite-fnal

  authored
  25f10bc

  Copy full SHA for 25f10bc

Commits on Nov 13, 2024

• connect dve db to pg16 DB and disabled pg14.

  yuyiguo

  committed
  07e3daf

  Copy full SHA for 07e3daf

Commits on Nov 14, 2024

• Merge pull request #53 from yuyiguo/pg16dev

  Show description for 58ba251
  bjwhite-fnal

  authored
  58ba251

  Copy full SHA for 58ba251

• Remove cruft.

  bjwhite-fnal

  committed
  27eb3be

  Copy full SHA for 27eb3be

• Remove Elastic stuff from dev makefile.

  bjwhite-fnal

  committed
  3a6eb59

  Copy full SHA for 3a6eb59

• Remove elastic stuff from dev.

  bjwhite-fnal

  committed
  5a0c831

  Copy full SHA for 5a0c831

• Sync up.

  bjwhite-fnal

  committed
  b367723

  Copy full SHA for b367723

• Merge pull request #54 from bjwhite-fnal/main

  Show description for e98b4d0
  bjwhite-fnal

  authored
  e98b4d0

  Copy full SHA for e98b4d0

Commits on Nov 15, 2024

• Fix postgres/kustomization.yaml namespace.

  bjwhite-fnal

  committed
  49212a2

  Copy full SHA for 49212a2

• Merge pull request #55 from bjwhite-fnal/main

  Show description for 59cc3b8
  bjwhite-fnal

  authored
  59cc3b8

  Copy full SHA for 59cc3b8

• Update DB cluster/pooler to have 16 in name, so we don't start a second cluster.

  bjwhite-fnal

  committed
  082e149

  Copy full SHA for 082e149

• Merge pull request #56 from bjwhite-fnal/main

  Show description for bbb7b81
  bjwhite-fnal

  authored
  bbb7b81

  Copy full SHA for bbb7b81

• Fix the CNPG DB container image.

  bjwhite-fnal

  committed
  ec61036

  Copy full SHA for ec61036

• Merge pull request #57 from bjwhite-fnal/main

  Show description for 7a3a496
  bjwhite-fnal

  authored
  7a3a496

  Copy full SHA for 7a3a496

Commits on Nov 18, 2024

• Increase production Postgres replicas to 3.

  bjwhite-fnal

  committed
  cfeed63

  Copy full SHA for cfeed63

Commits on Nov 19, 2024

• Upgrade dev to 35.6.0

  bjwhite-fnal

  committed
  f1e3011

  Copy full SHA for f1e3011

• Upgrade prod to 35.6.0

  bjwhite-fnal

  committed
  45a865a

  Copy full SHA for 45a865a

Commits on Nov 20, 2024

• Update prod Kafka bootstrap URL for HermesK.

  bjwhite-fnal

  committed
  49a571c

  Copy full SHA for 49a571c

• Increase KafkaTopic retention period to 6 hours.

  bjwhite-fnal

  committed
  5181a0e

  Copy full SHA for 5181a0e

Commits on Nov 21, 2024

• Update HermesK to 35.6.0.1

  bjwhite-fnal

  committed
  915a1dd

  Copy full SHA for 915a1dd

• Pin production Kafka bootstrap IP.

  bjwhite-fnal

  committed
  00ba427

  Copy full SHA for 00ba427

Commits on Dec 3, 2024

• We will only running automatix against the *_DATA_DISK in all the four facilities.

  

  yuyiguo

  authored and

  bjwhite-fnal

  committed
  9c79142

  Copy full SHA for 9c79142

• Increase Postgres memory for production.

  bjwhite-fnal

  committed
  e82d221

  Copy full SHA for e82d221

Commits on Dec 9, 2024

• Upgrade helm charts for dev server/daemons.

  bjwhite-fnal

  committed
  34a3bb6

  Copy full SHA for 34a3bb6

• Upgrade helm charts for prod server/daemons.

  bjwhite-fnal

  committed
  a7f7b86

  Copy full SHA for a7f7b86

Commits on Dec 16, 2024

• Turn on WebUI for dev.

  bjwhite-fnal

  committed
  068bc3d

  Copy full SHA for 068bc3d

Commits on Dec 17, 2024

• Update for use of usdf-fts3-dev.

  bjwhite-fnal

  committed
  ca94768

  Copy full SHA for ca94768

Commits on Dec 30, 2024

• Switch DEV to using fetch-crl-cron for initContainer CA certificate injection.

  bjwhite-fnal

  committed
  3ae5c96

  Copy full SHA for 3ae5c96

• Switch PROD to using fetch-crl-cron for initContainer CA certificate injection.

  bjwhite-fnal

  committed
  659b175

  Copy full SHA for 659b175

Commits on Jan 2, 2025

• Change dev server to use SSLCACertificatePath instead of SSLCACertificateFile.

  bjwhite-fnal

  committed
  460900d

  Copy full SHA for 460900d

Commits on Jan 3, 2025

• Turn back on gridcert patch for some containers.

  bjwhite-fnal

  committed
  66e9e7c

  Copy full SHA for 66e9e7c

• Change prod server to use SSLCACertificatePath instead of SSLCACertificateFile.

  bjwhite-fnal

  committed
  b1603d9

  Copy full SHA for b1603d9

Commits on Jan 6, 2025

• Switch dev grid certificates to PVC, and update with CronJob.

  bjwhite-fnal

  committed
  19d0c8a

  Copy full SHA for 19d0c8a

• Switch prod grid certificates to PVC, and update with CronJob.

  bjwhite-fnal

  committed
  9200532

  Copy full SHA for 9200532

Commits on Jan 13, 2025

• Set sleepTime for conveyor-poller/finisher.

  bjwhite-fnal

  committed
  799d812

  Copy full SHA for 799d812

• Dev: Disable useDeprecatedImplicitSecrets.

  bjwhite-fnal

  committed
  24dcbcc

  Copy full SHA for 24dcbcc

Commits on Jan 14, 2025

• Fix hermes constant mount for reaper.

  bjwhite-fnal

  committed
  2aee70a

  Copy full SHA for 2aee70a

• Prod: Dev: Disable useDeprecatedImplicitSecrets. Enable webui.

  bjwhite-fnal

  committed
  ae6ab6c

  Copy full SHA for ae6ab6c

Commits on Jan 17, 2025

• Dev WebUI CORS fix.

  bjwhite-fnal

  committed
  299a9c9

  Copy full SHA for 299a9c9

Commits on Feb 28, 2025

• Update CNPH to 1.25, migrate DB to Postgres 16.6 and fixed helm generation with explicit namespace.

  

  yuyiguo

  authored and

  bjwhite-fnal

  committed
  e4d6414

  Copy full SHA for e4d6414