Code

Initial commit for RAG in py-ml#427

Code

Open

hmumtazz wants to merge 42 commits intoopensearch-project:mainopensearch-project/opensearch-py-ml:mainfrom hmumtazz:rag_pipelinehmumtazz/opensearch-py-ml:rag_pipelineCopy head branch name to clipboard

+8,182-5

Commits

Commits on Nov 21, 2024

• Initial commit for RAG pipeline scripts

  Show description for dbbd4f5
  hmumtazz

  committed
  dbbd4f5

  Copy full SHA for dbbd4f5

• Added Licence Header and fixed .gitingore file

  Show description for dc94feb
  hmumtazz

  committed
  dc94feb

  Copy full SHA for dc94feb

• Added comments to understand code

  Show description for cba567c
  hmumtazz

  committed
  cba567c

  Copy full SHA for cba567c

• Remove sensitive config file

  Show description for 3481c3e
  hmumtazz

  committed
  3481c3e

  Copy full SHA for 3481c3e

• Simplify the selection process for the ef_construction parameter by offering a suggested default value with the flexibility for users to enter a custom value if needed.

  Show description for e6eee30
  hmumtazz

  committed
  e6eee30

  Copy full SHA for e6eee30

• Allows Customer to register model via CLI, fixed embedding generation, addressed comments, fixed upload csv method

  Show description for 8b7aa0e
  hmumtazz

  committed
  8b7aa0e

  Copy full SHA for 8b7aa0e

• Sign-off on all previous work

  Show description for 2635751
  hmumtazz

  committed
  2635751

  Copy full SHA for 2635751

Commits on Nov 22, 2024

• Enhance RAG pipeline functionality and user experience

  Show description for 88cf68b
  hmumtazz

  committed
  88cf68b

  Copy full SHA for 88cf68b

Commits on Nov 25, 2024

• Created Ingest Pipline for chunking.- Merge custom setup.py with the pre-existing one, allowing users to run a single setup process

  Show description for ad1ea97
  hmumtazz

  committed
  ad1ea97

  Copy full SHA for ad1ea97

Commits on Nov 29, 2024

• Removed hard coded LLM model, allowed for Opensoure integration, users can now specify their model payload through JSON object

  Show description for da0ce7c
  hmumtazz

  committed
  da0ce7c

  Copy full SHA for da0ce7c

• Remove requirements.txt and setup.py from Git tracking

  Show description for fc2ace8
  hmumtazz

  committed
  fc2ace8

  Copy full SHA for fc2ace8

• Update setup.py file

  Show description for fe83b25
  hmumtazz

  committed
  fe83b25

  Copy full SHA for fe83b25

• Remove .gitignore file from rag pipeline directory

  Show description for bea75e0
  hmumtazz

  committed
  bea75e0

  Copy full SHA for bea75e0

• Organized Model registration into classes

  Show description for f5b74ed
  hmumtazz

  committed
  f5b74ed

  Copy full SHA for f5b74ed

• Remove base_model.py from rag pipeline

  Show description for da834d2
  hmumtazz

  committed
  da834d2

  Copy full SHA for da834d2

• Licence header

  Show description for 9c45e43
  hmumtazz

  committed
  9c45e43

  Copy full SHA for 9c45e43

Commits on Dec 2, 2024

• Updated user setup, Added Semantic search for Managed service using ML Models, Added Neural search for non managed service., Added UT, Update to Changelog to reflect recent changes and to past test…

  Show description for a61a840
  hmumtazz

  committed
  a61a840

  Copy full SHA for a61a840

• fixed failing test in AIConnector tests

  Show description for 8db5e9f
  hmumtazz

  committed
  8db5e9f

  Copy full SHA for 8db5e9f

Commits on Dec 3, 2024

• Update test_SecretsHelper

  Show description for aa42922
  hmumtazz

  authored
  aa42922

  Copy full SHA for aa42922

• fixed license header test_SageMakerModel.py

  Show description for 5984c19
  hmumtazz

  authored
  5984c19

  Copy full SHA for 5984c19

• fixed license header rag.py

  Show description for 4503a55
  hmumtazz

  authored
  4503a55

  Copy full SHA for 4503a55

Commits on Dec 5, 2024

• Added seperate class for embedding generation, removed nominee text parametrs, added more functionality to user setup with knn index, fixed chunking proccessor to actually chunk rather, more intuti…

  Show description for b1b98ab
  hmumtazz

  committed
  b1b98ab

  Copy full SHA for b1b98ab

• Correctly leveraged existing methods in AI connector class, without having method wrappers around them

  Show description for d7e4713
  hmumtazz

  committed
  d7e4713

  Copy full SHA for d7e4713

• Removed duplicate method, and deleted unused method

  Show description for c02fe1b
  hmumtazz

  committed
  c02fe1b

  Copy full SHA for c02fe1b

• Fixed chunking pipeline, query was not generating due to mismatchd vector fields, like name etc, updated knn index details and chunking details

  Show description for d877bca
  hmumtazz

  committed
  d877bca

  Copy full SHA for d877bca

• Remove config.ini from repository and add to .gitignore

  Show description for f0194ec
  hmumtazz

  committed
  f0194ec

  Copy full SHA for f0194ec

• Update rag_setup.py to remove neural search line and serverless

  Show description for d354b25
  hmumtazz

  authored
  d354b25

  Copy full SHA for d354b25

• Updated query.py to reflect search changes

  Show description for 1ca4221
  hmumtazz

  committed
  1ca4221

  Copy full SHA for 1ca4221

• Missing innit file

  Show description for 8b97791
  hmumtazz

  committed
  8b97791

  Copy full SHA for 8b97791

Commits on Dec 6, 2024

• missing init file

  Show description for fe6f8c5
  hmumtazz

  committed
  fe6f8c5

  Copy full SHA for fe6f8c5

• Fixed domain ARN/domain name error, changed directory, for IAM, SecretsHelper

  Show description for 34f7fd4
  hmumtazz

  committed
  34f7fd4

  Copy full SHA for 34f7fd4

• Updated AIConnectorHelper to use existing methods, like get task, and creating a connector

  Show description for 39a1883
  hmumtazz

  committed
  39a1883

  Copy full SHA for 39a1883

• Updated License Headers

  Show description for 2dfa609
  hmumtazz

  committed
  2dfa609

  Copy full SHA for 2dfa609

Commits on Dec 7, 2024

• Fixed UT, Ran Lint nox, fixed code and unused enviroment variable issues.

  Show description for 2c40ce0
  hmumtazz

  committed
  2c40ce0

  Copy full SHA for 2c40ce0

Commits on Dec 11, 2024

• Deleted serverless.py from rag functionality as we are not using it

  Show description for 6a846ce
  hmumtazz

  committed
  6a846ce

  Copy full SHA for 6a846ce

Commits on Dec 12, 2024

• deleted unused dependcies, and test.py for serverless

  Show description for 6f17670
  hmumtazz

  committed
  6f17670

  Copy full SHA for 6f17670

Commits on Dec 13, 2024

• Fixed failing test

  Show description for 120169f
  hmumtazz

  committed
  120169f

  Copy full SHA for 120169f

• Fixed tests

  Show description for 91c9af0
  hmumtazz

  committed
  91c9af0

  Copy full SHA for 91c9af0

• Fixed tests

  Show description for 82e69f5
  hmumtazz

  committed
  82e69f5

  Copy full SHA for 82e69f5

Commits on Jan 7, 2025

• Addressed comments like making code less redudent, and combining methods

  Show description for c0114c3
  hmumtazz

  committed
  c0114c3

  Copy full SHA for c0114c3

• Addressed code comments, making code less redundent, and combining existing methods

  Show description for 197a134
  hmumtazz

  committed
  197a134

  Copy full SHA for 197a134

Commits on Jan 21, 2025

• Fixed methods at UT's, adressed comments

  Show description for 220e9ee
  hmumtazz

  committed
  220e9ee

  Copy full SHA for 220e9ee