1.0.1-GA to Main (#83) (#84)

* Develop to 1.0.0-GA (#83) * Issue #4 feat: enhance the ingest event to add obsrv and source meta * issue#223: feat: Updated API endpoints with verbs * issue#223: fix: Test case modification * issue #84 : fix: API swagger doc update * #99 fix: upgrade packages to fix vulnerabilities * Command-service Vulnerabilities fixes (#56) * Release 1.3.0 into Main (#53) * issue#223: feat: Updated API endpoints with verbs * issue#223: fix: Test case modification * issue #84 : fix: API swagger doc update * #99 fix: upgrade packages to fix vulnerabilities --------- Co-authored-by: Jerald <[email protected]> Co-authored-by: harishkumar gangula <[email protected]> Co-authored-by: Praveen <[email protected]> * command-service vulnerabilities fixes --------- Co-authored-by: Manjunath Davanam <[email protected]> Co-authored-by: Jerald <[email protected]> Co-authored-by: harishkumar gangula <[email protected]> Co-authored-by: Praveen <[email protected]> * #90 fix: Resolve API Issues (#52) * #90 fix: format error messages, code cleanup * #90 fix: validate extraction config during ingest * #90 fix: add test cases for extraction key validation * #90 fix: move error handler to helpers for standard handling * #90 fix: update telemetry audit event set function (#58) * Build and deployment (#57) * build api image * Update build_and_deploy.yaml * Update build_and_deploy.yaml * Update build_and_deploy.yaml * Update build_and_deploy.yaml * build api image * Update build_and_deploy.yaml * build api image * build api image * build api image * build and deploy api image * build and deploy api image * build and deploy api image * build and deploy of api service * build and deploy of api service * modify docker file * modify docker file * modify docker file * update build and deployment * update build and deployment * Update build_and_deploy.yaml * api service build and deployment * api service build and deployment * feat: obsrv api service build and deployment github actions configuration * #0 fix: update the tag condition in actions --------- Co-authored-by: ManojCKrishna <[email protected]> Co-authored-by: Manoj Krishna <[email protected]> Co-authored-by: Harish Kumar Gangula <[email protected]> Co-authored-by: Manjunath Davanam <[email protected]> Co-authored-by: Praveen <[email protected]> * #0 fix: add azure exhaust support (#44) * #0 fix: add azure exhaust support * #0 fix: update azure exhaust service without async * develop into release 1.3.0 (#59) * Issue #4 feat: enhance the ingest event to add obsrv and source meta * #90 fix: Resolve API Issues (#52) * #90 fix: format error messages, code cleanup * #90 fix: validate extraction config during ingest * #90 fix: add test cases for extraction key validation * #90 fix: move error handler to helpers for standard handling * #90 fix: update telemetry audit event set function (#58) * Build and deployment (#57) * build api image * Update build_and_deploy.yaml * Update build_and_deploy.yaml * Update build_and_deploy.yaml * Update build_and_deploy.yaml * build api image * Update build_and_deploy.yaml * build api image * build api image * build api image * build and deploy api image * build and deploy api image * build and deploy api image * build and deploy of api service * build and deploy of api service * modify docker file * modify docker file * modify docker file * update build and deployment * update build and deployment * Update build_and_deploy.yaml * api service build and deployment * api service build and deployment * feat: obsrv api service build and deployment github actions configuration * #0 fix: update the tag condition in actions --------- Co-authored-by: ManojCKrishna <[email protected]> Co-authored-by: Manoj Krishna <[email protected]> Co-authored-by: Harish Kumar Gangula <[email protected]> Co-authored-by: Manjunath Davanam <[email protected]> Co-authored-by: Praveen <[email protected]> * #0 fix: add azure exhaust support (#44) * #0 fix: add azure exhaust support * #0 fix: update azure exhaust service without async --------- Co-authored-by: shiva-rakshith <[email protected]> Co-authored-by: Praveen Veleneni <[email protected]> Co-authored-by: Manoj Krishna <[email protected]> Co-authored-by: ManojCKrishna <[email protected]> Co-authored-by: Manoj Krishna <[email protected]> Co-authored-by: Harish Kumar Gangula <[email protected]> Co-authored-by: Manjunath Davanam <[email protected]> * #126 Feat: Add Querying on Aggregated datasources (#60) * #126 feat: enable querying on aggregated datasources * #126 feat: Fix validation issues and update routes for aggregate queries * #126 fix: remove unused methods * #126 fix: remove aggregates from endpoint * #126 fix: add validation for granularity options * #126 fix: Remove unused routes and validators * #126 feat: add new property to datasources structure and update rollup querying * #305 Feat: Add and Update dataset status (#62) * #305 feat: add retire apis and update dataset status * #305 fix: remove unused import * #305 fix: add command service api call to restart jobs * #305 fix: fix command service payload * #305 fix: update api doc * #305 fix: clear unused code * #305 fix: clear unused code * #305 fix: undo change submit ingestion * #305 fix: update test case to use enums * #305 fix: add test env file for test cases * #305 fix: add enum for string in test case * Issue #305 feat: exclude system-events data source in validation (#63) * Issue #165 feat: generate AUDIT events (#71) * Issue #fix-344 fix: defect fix (#76) * Issue #193 Feat: postman collection for Obsrv APIs (#77) * #198 feat: add validation for denorm out fields, to avoid duplicates (#78) * #198 feat: add validation for denorm out fields, to avoid duplicate out fields * #198 fix: update error message for denorm fields duplicate * #198 feat: add test cases for validation of denorm validation * #198 fix: remove console statement * Tenant ingestion (#81) * #0 feat: Add tenant_ingest API endpoint * #0 fix: Update API ID in RoutesConfig.ts * #0 fix: Update datasetId format in IngestorService.ts * #0 fix: Swap datasetId and tenantId in IngestorService.ts (#82) * Changed status enum according to latest workflow (#80) * Issue #193 Fix: changed status enum according to latest workflow * Issue #193 Fix: changed status enum according to latest workflow --------- Co-authored-by: shiva-rakshith <[email protected]> Co-authored-by: Jerald <[email protected]> Co-authored-by: harishkumar gangula <[email protected]> Co-authored-by: Praveen <[email protected]> Co-authored-by: Sowmya N Dixit <[email protected]> Co-authored-by: Manoj Krishna <[email protected]> Co-authored-by: ManojCKrishna <[email protected]> Co-authored-by: Manoj Krishna <[email protected]> Co-authored-by: Ravi Mula <[email protected]> Co-authored-by: Ravinder Kumar <[email protected]> Co-authored-by: GayathriSrividya <[email protected]> * Develop to 1.0.1-GA (#86) * Issue #4 feat: enhance the ingest event to add obsrv and source meta * issue#223: feat: Updated API endpoints with verbs * issue#223: fix: Test case modification * issue #84 : fix: API swagger doc update * #99 fix: upgrade packages to fix vulnerabilities * Command-service Vulnerabilities fixes (#56) * Release 1.3.0 into Main (#53) * issue#223: feat: Updated API endpoints with verbs * issue#223: fix: Test case modification * issue #84 : fix: API swagger doc update * #99 fix: upgrade packages to fix vulnerabilities --------- Co-authored-by: Jerald <[email protected]> Co-authored-by: harishkumar gangula <[email protected]> Co-authored-by: Praveen <[email protected]> * command-service vulnerabilities fixes --------- Co-authored-by: Manjunath Davanam <[email protected]> Co-authored-by: Jerald <[email protected]> Co-authored-by: harishkumar gangula <[email protected]> Co-authored-by: Praveen <[email protected]> * #90 fix: Resolve API Issues (#52) * #90 fix: format error messages, code cleanup * #90 fix: validate extraction config during ingest * #90 fix: add test cases for extraction key validation * #90 fix: move error handler to helpers for standard handling * #90 fix: update telemetry audit event set function (#58) * Build and deployment (#57) * build api image * Update build_and_deploy.yaml * Update build_and_deploy.yaml * Update build_and_deploy.yaml * Update build_and_deploy.yaml * build api image * Update build_and_deploy.yaml * build api image * build api image * build api image * build and deploy api image * build and deploy api image * build and deploy api image * build and deploy of api service * build and deploy of api service * modify docker file * modify docker file * modify docker file * update build and deployment * update build and deployment * Update build_and_deploy.yaml * api service build and deployment * api service build and deployment * feat: obsrv api service build and deployment github actions configuration * #0 fix: update the tag condition in actions --------- Co-authored-by: ManojCKrishna <[email protected]> Co-authored-by: Manoj Krishna <[email protected]> Co-authored-by: Harish Kumar Gangula <[email protected]> Co-authored-by: Manjunath Davanam <[email protected]> Co-authored-by: Praveen <[email protected]> * #0 fix: add azure exhaust support (#44) * #0 fix: add azure exhaust support * #0 fix: update azure exhaust service without async * develop into release 1.3.0 (#59) * Issue #4 feat: enhance the ingest event to add obsrv and source meta * #90 fix: Resolve API Issues (#52) * #90 fix: format error messages, code cleanup * #90 fix: validate extraction config during ingest * #90 fix: add test cases for extraction key validation * #90 fix: move error handler to helpers for standard handling * #90 fix: update telemetry audit event set function (#58) * Build and deployment (#57) * build api image * Update build_and_deploy.yaml * Update build_and_deploy.yaml * Update build_and_deploy.yaml * Update build_and_deploy.yaml * build api image * Update build_and_deploy.yaml * build api image * build api image * build api image * build and deploy api image * build and deploy api image * build and deploy api image * build and deploy of api service * build and deploy of api service * modify docker file * modify docker file * modify docker file * update build and deployment * update build and deployment * Update build_and_deploy.yaml * api service build and deployment * api service build and deployment * feat: obsrv api service build and deployment github actions configuration * #0 fix: update the tag condition in actions --------- Co-authored-by: ManojCKrishna <[email protected]> Co-authored-by: Manoj Krishna <[email protected]> Co-authored-by: Harish Kumar Gangula <[email protected]> Co-authored-by: Manjunath Davanam <[email protected]> Co-authored-by: Praveen <[email protected]> * #0 fix: add azure exhaust support (#44) * #0 fix: add azure exhaust support * #0 fix: update azure exhaust service without async --------- Co-authored-by: shiva-rakshith <[email protected]> Co-authored-by: Praveen Veleneni <[email protected]> Co-authored-by: Manoj Krishna <[email protected]> Co-authored-by: ManojCKrishna <[email protected]> Co-authored-by: Manoj Krishna <[email protected]> Co-authored-by: Harish Kumar Gangula <[email protected]> Co-authored-by: Manjunath Davanam <[email protected]> * #126 Feat: Add Querying on Aggregated datasources (#60) * #126 feat: enable querying on aggregated datasources * #126 feat: Fix validation issues and update routes for aggregate queries * #126 fix: remove unused methods * #126 fix: remove aggregates from endpoint * #126 fix: add validation for granularity options * #126 fix: Remove unused routes and validators * #126 feat: add new property to datasources structure and update rollup querying * #305 Feat: Add and Update dataset status (#62) * #305 feat: add retire apis and update dataset status * #305 fix: remove unused import * #305 fix: add command service api call to restart jobs * #305 fix: fix command service payload * #305 fix: update api doc * #305 fix: clear unused code * #305 fix: clear unused code * #305 fix: undo change submit ingestion * #305 fix: update test case to use enums * #305 fix: add test env file for test cases * #305 fix: add enum for string in test case * Issue #305 feat: exclude system-events data source in validation (#63) * Issue #165 feat: generate AUDIT events (#71) * Issue #fix-344 fix: defect fix (#76) * Issue #193 Feat: postman collection for Obsrv APIs (#77) * #198 feat: add validation for denorm out fields, to avoid duplicates (#78) * #198 feat: add validation for denorm out fields, to avoid duplicate out fields * #198 fix: update error message for denorm fields duplicate * #198 feat: add test cases for validation of denorm validation * #198 fix: remove console statement * Tenant ingestion (#81) * #0 feat: Add tenant_ingest API endpoint * #0 fix: Update API ID in RoutesConfig.ts * #0 fix: Update datasetId format in IngestorService.ts * #0 fix: Swap datasetId and tenantId in IngestorService.ts (#82) * Changed status enum according to latest workflow (#80) * Issue #193 Fix: changed status enum according to latest workflow * Issue #193 Fix: changed status enum according to latest workflow * Issue #0000: fix: update Dockerfile (#85) --------- Co-authored-by: shiva-rakshith <[email protected]> Co-authored-by: Jerald <[email protected]> Co-authored-by: harishkumar gangula <[email protected]> Co-authored-by: Praveen <[email protected]> Co-authored-by: Manjunath Davanam <[email protected]> Co-authored-by: Sowmya N Dixit <[email protected]> Co-authored-by: Manoj Krishna <[email protected]> Co-authored-by: ManojCKrishna <[email protected]> Co-authored-by: Manoj Krishna <[email protected]> Co-authored-by: Ravinder Kumar <[email protected]> Co-authored-by: GayathriSrividya <[email protected]> Co-authored-by: Shreyas Bhaktharam <[email protected]> --------- Co-authored-by: Manjunath Davanam <[email protected]> Co-authored-by: shiva-rakshith <[email protected]> Co-authored-by: Jerald <[email protected]> Co-authored-by: harishkumar gangula <[email protected]> Co-authored-by: Praveen <[email protected]> Co-authored-by: Sowmya N Dixit <[email protected]> Co-authored-by: Manoj Krishna <[email protected]> Co-authored-by: ManojCKrishna <[email protected]> Co-authored-by: Manoj Krishna <[email protected]> Co-authored-by: Ravinder Kumar <[email protected]> Co-authored-by: GayathriSrividya <[email protected]> Co-authored-by: Shreyas Bhaktharam <[email protected]>
Sanketika-Obsrv · Jan 12, 2024 · 058ec85 · 058ec85
1 parent b5243a8
commit 058ec85
Show file tree

Hide file tree

Showing 12 changed files with 1,179 additions and 7 deletions.
diff --git a/api-service/postman-collection/Obsrv API Service.postman_collection.json b/api-service/postman-collection/Obsrv API Service.postman_collection.json
diff --git a/api-service/redoc-static.html b/api-service/redoc-static.html
diff --git a/api-service/src/configs/RoutesConfig.ts b/api-service/src/configs/RoutesConfig.ts
@@ -116,6 +116,12 @@ export const routesConfig = {
     path: "/obsrv/v1/data/create/:datasetId",
     validation_schema: "DataIngestionReq.json",
   },
+  tenant_ingest: {
+    api_id: "dataset.data.in",
+    method: "post",
+    path: "/data/tenant/in/:datasetId",
+    validation_schema: "DataIngestionReq.json",
+  },
   exhaust: {
     api_id: "obsrv.dataset.data.exhaust",
     method: "get",

diff --git a/api-service/src/helpers/Datasets.ts b/api-service/src/helpers/Datasets.ts
@@ -4,6 +4,8 @@ import { defaultConfig } from '../resources/schemas/DatasetConfigDefault'
 import { SchemaMerger } from '../generators/SchemaMerger'
 import { config } from '../configs/Config'
 import { DatasetStatus } from '../models/DatasetModels'
+import constants from "../resources/Constants.json";
+
 let schemaMerger = new SchemaMerger()
 export class Datasets {
     private id: string
@@ -47,10 +49,12 @@ export class Datasets {
     }
 
     public getValues() {
+        this.validateDenormConfig();
         return Object.assign(this.removeNullValues({ id: this.id, dataset_id: this.dataset_id, type: this.type, name: this.name, validation_config: this.validation_config, extraction_config: this.extraction_config, dedup_config: this.dedup_config, data_schema: this.data_schema, router_config: this.router_config, denorm_config: this.denorm_config, dataset_config: this.dataset_config, tags: this.tags, status: this.status, created_by: this.created_by, updated_by: this.updated_by, published_date: this.published_date }), { "updated_date": new Date })
     }
 
     public setValues() {
+        this.validateDenormConfig();
         return schemaMerger.mergeSchema(this.getDefaults(), this.getValues())
     }
 
@@ -69,4 +73,21 @@ export class Datasets {
             return {...defaultConfig.dataset}
         }
     }
+
+    private validateDenormConfig() {
+        if (this.denorm_config && _.has(this.denorm_config, 'denorm_fields')) {
+            let duplicatesExist = false;
+            let denormFields: any = _.get(this.denorm_config, 'denorm_fields', []);
+            denormFields = _.map(denormFields, (denormField: Record<string, string | number>) => _.get(denormField, 'denorm_out_field'));
+            denormFields.map(
+                (denormField: string | number) => {
+                    if(_.indexOf(denormFields, denormField) !== _.lastIndexOf(denormFields, denormField))
+                        duplicatesExist = true;
+                }
+            );
+            if(duplicatesExist) {
+                throw constants.DUPLICATE_DENORM_FIELD;
+            }
+        }
+    }
 }
diff --git a/api-service/src/resources/Constants.json b/api-service/src/resources/Constants.json
@@ -96,6 +96,11 @@
     "status": 404,
     "code": "NOT_FOUND"
   },
+  "DUPLICATE_DENORM_FIELD": {
+    "message": "Duplicate found for denorm output key",
+    "status": 400,
+    "code": "BAD_REQUEST"
+  },
   "INGESTION_SUBMITTED": "ingestion spec has been submitted successfully",
   "INGESTION_FAILED_ON_SAVE": "Failed to submit Ingestion Spec, record is not saved"
 }
diff --git a/api-service/src/routes/Router.ts b/api-service/src/routes/Router.ts
@@ -39,6 +39,7 @@ router.post([`${routesConfig.query.sql_query.path}`, `${routesConfig.query.sql_q
 
 /** Ingestor API */
 router.post(`${routesConfig.data_ingest.path}`, ResponseHandler.setApiId(routesConfig.data_ingest.api_id), telemetryAuditStart({ action: telemetryActions.ingestEvents, operationType: OperationType.CREATE }), onRequest({ entity: promEntities.data_in }), validationService.validateRequestBody, ingestorService.create);
+router.post(`${routesConfig.tenant_ingest.path}`, ResponseHandler.setApiId(routesConfig.tenant_ingest.api_id), telemetryAuditStart({ action: telemetryActions.ingestEvents, operationType: OperationType.CREATE }), onRequest({ entity: promEntities.data_in }), validationService.validateRequestBody, ingestorService.tenant);
 
 /** Dataset APIs */
 router.post(`${routesConfig.config.dataset.save.path}`, ResponseHandler.setApiId(routesConfig.config.dataset.save.api_id), telemetryAuditStart({ action: telemetryActions.createDataset, operationType: OperationType.CREATE }), validationService.validateRequestBody, datasetService.save);

diff --git a/api-service/src/services/IngestorService.ts b/api-service/src/services/IngestorService.ts
@@ -35,6 +35,20 @@ export class IngestorService {
             ResponseHandler.successResponse(req, res, { status: 200, data: { message: constants.DATASET.CREATED } });
         } catch (error: any) { this.errorHandler.handleError(req, res, next, error, false) }
     }
+
+    public tenant = async (req: Request, res: Response, next: NextFunction) => {
+        try {
+            let datasetId = this.getDatasetId(req);
+            const tenantId = _.get(req.headers, 'x-tenant-id', "default");
+            datasetId = `${tenantId}-${datasetId}`;
+            const validData = await this.validateData(req.body.data, datasetId);
+            req.body = { ...req.body.data, dataset: datasetId };
+            const topic = await this.getTopic(datasetId);
+            await this.kafkaConnector.execute(req, res, topic);
+            ResponseHandler.successResponse(req, res, { status: 200, data: { message: constants.DATASET.CREATED } });
+        } catch (error: any) { this.errorHandler.handleError(req, res, next, error, false) }
+    }
+
     public submitIngestion = async (req: Request, res: Response, next: NextFunction) => {
         try {
             await wrapperService.submitIngestion(req.body)

diff --git a/api-service/src/services/telemetry.ts b/api-service/src/services/telemetry.ts
@@ -12,7 +12,7 @@ export enum OperationType { CREATE = 1, UPDATE, PUBLISH, RETIRE, LIST, GET }
 
 const kafka = new Kafka({ clientId: telemetryTopic, brokers: brokerServers });
 const telemetryEventsProducer = kafka.producer();
-telemetryEventsProducer.connect();
+telemetryEventsProducer.connect().catch(err => console.error("Unable to connect to kafka", err.message));
 
 const getDefaults = () => {
     return {

diff --git a/api-service/src/test/DatasetTestService.spec.ts b/api-service/src/test/DatasetTestService.spec.ts
@@ -108,7 +108,7 @@ describe("Dataset create API", () => {
                 res.body.params.status.should.be.eq(constants.STATUS.FAILURE)
                 done();
             });
-    })
+    });
     it("should not insert record when given invalid schema", (done) => {
         chai
             .request(app)
@@ -123,7 +123,45 @@ describe("Dataset create API", () => {
                 res.body.params.status.should.be.eq(constants.STATUS.FAILURE)
                 done();
             });
-    })
+    });
+    it("should not insert the record when there's a duplicate denorm out field", (done) => {
+        chai.spy.on(dbConnector, "execute", () => {
+            return Promise.resolve([])
+        })
+        chai
+            .request(app)
+            .post(config.apiDatasetSaveEndPoint)
+            .send(TestDataset.DUPLICATE_DENORM_OUT_FIELD)
+            .end((err, res) => {
+                res.should.have.status(httpStatus.BAD_REQUEST);
+                res.body.should.be.a("object")
+                res.body.responseCode.should.be.eq(httpStatus["400_NAME"]);
+                res.body.should.have.property("result");
+                res.body.id.should.be.eq(routesConfig.config.dataset.save.api_id);
+                res.body.params.status.should.be.eq(constants.STATUS.FAILURE);
+                chai.spy.restore(dbConnector, "execute");
+                done();
+            });
+    });
+    it("should insert the record when there's no duplicate denorm out field", (done) => {
+        chai.spy.on(dbConnector, "execute", () => {
+            return Promise.resolve([])
+        })
+        chai
+            .request(app)
+            .post(config.apiDatasetSaveEndPoint)
+            .send(TestDataset.VALID_DENORM_OUT_FIELD)
+            .end((err, res) => {
+                res.should.have.status(httpStatus.OK);
+                res.body.should.be.a("object")
+                res.body.responseCode.should.be.eq(httpStatus["200_NAME"]);
+                res.body.should.have.property("result");
+                res.body.id.should.be.eq(routesConfig.config.dataset.save.api_id);
+                res.body.params.status.should.be.eq(constants.STATUS.SUCCESS);
+                chai.spy.restore(dbConnector, "execute");
+                done();
+            });
+    });
 })
 describe("Dataset update API", () => {
     beforeEach(() => {

diff --git a/api-service/src/test/Fixtures.ts b/api-service/src/test/Fixtures.ts
@@ -51,7 +51,9 @@ class TestDataset {
   public static VALID_LIST_REQUEST_ACTIVE_STATUS = { "filters": { "status": [ DatasetStatus.Live ] } };
   public static VALID_LIST_REQUEST_DISABLED_STATUS = { "filters": { "status": [ DatasetStatus.Retired ] } };
   public static MISSING_REQUIRED_FIELDS_UPDATE = { "name": "telemetry-raw", "data_schema": { "type": "object", "properties": { "eid": { "type": "string" }, "ver": { "type": "string" }, "syncts": { "type": "integer" }, "ets": { "type": "integer" }, "mid": { "type": "string" }, "actor": { "type": "object", "properties": { "id": { "type": "string" }, "type": { "type": "string" } }, "required": [ "id", "type" ] }, "edata": { "type": "object", "properties": { "type": { "type": "string" } }, "required": [ "type" ] }, "@timestamp": { "type": "string" }, "context": { "type": "object", "properties": { "pdata": { "type": "object", "properties": { "ver": { "type": "string" }, "id": { "type": "string" }, "pid": { "type": "string" } }, "required": [ "ver", "id", "pid" ] }, "did": { "type": "string" }, "env": { "type": "string" }, "channel": { "type": "string" } }, "required": [ "pdata", "did", "env", "channel" ] }, "@version": { "type": "string" }, "object": { "type": "object", "properties": { "id": { "type": "string" }, "type": { "type": "string" } }, "required": [ "id", "type" ] } }, "required": [ "eid", "ver", "syncts", "ets", "mid", "actor", "edata", "@timestamp", "context", "@version", "object" ] }, "router_config": { "topic": "router.topic" }, "status": DatasetStatus.Live, "published_date": "2023-03-14T04:46:33.459Z" };
-  public static VALID_RECORD = { "type": "master-dataset", "dataset_id": "3f8b2ba7-9c74-4d7f-8b38-2b0d460b999c", "id": "observations", "name": "telemetry-raw", " validation_config": { "validate": true, "mode": "Strict" }, "extraction_config": { "is_batch_event": false, "extraction_key": "", "dedup_config": { "drop_duplicates": true, "dedup_key": "id", "dedup_period": 3 } }, "dedup_config": { "drop_duplicates": true, "dedup_key": "id", "dedup_period": 3 }, "data_schema": { "type": "object", "properties": { "eid": { "type": "string" }, "ver": { "type": "string" }, "syncts": { "type": "integer" }, "ets": { "type": "integer" }, "mid": { "type": "string" }, "actor": { "type": "object", "properties": { "id": { "type": "string" }, "type": { "type": "string" } }, "required": [ "id", "type" ] }, "edata": { "type": "object", "properties": { "type": { "type": "string" } }, "required": [ "type" ] }, "@timestamp": { "type": "string" }, "context": { "type": "object", "properties": { "pdata": { "type": "object", "properties": { "ver": { "type": "string" }, "id": { "type": "string" }, "pid": { "type": "string" } }, "required": [ "ver", "id", "pid" ] }, "did": { "type": "string" }, "env": { "type": "string" }, "channel": { "type": "string" } }, "required": [ "pdata", "did", "env", "channel" ] }, "@version": { "type": "string" }, "object": { "type": "object", "properties": { "id": { "type": "string" }, "type": { "type": "string" } }, "required": [ "id", "type" ] } }, "required": [ "eid", "ver", "syncts", "ets", "mid", "actor", "edata", "@timestamp", "context", "@version", "object" ] }, "denorm_config": { "redis_db_host": "redis_host", "redis_db_port": "redis_port", "denorm_fields": { "denorm_key": "", "redis_db": 1, "denorm_out_field": "metadata" } }, "tags": [], "router_config": { "topic": "router.topic" }, "client_state": {}, "status": DatasetStatus.Live, "created_by": "SYSTEM", "updated_by": "SYSTEM", "created_date": "2023-03-13T07:46:06.410Z", "updated_date": "2023-03-14T04:46:33.459Z", "published_date": "2023-03-14T04:46:33.459Z" }
+  public static VALID_RECORD = { "type": "master-dataset", "dataset_id": "3f8b2ba7-9c74-4d7f-8b38-2b0d460b999c", "id": "observations", "name": "telemetry-raw", " validation_config": { "validate": true, "mode": "Strict" }, "extraction_config": { "is_batch_event": false, "extraction_key": "", "dedup_config": { "drop_duplicates": true, "dedup_key": "id", "dedup_period": 3 } }, "dedup_config": { "drop_duplicates": true, "dedup_key": "id", "dedup_period": 3 }, "data_schema": { "type": "object", "properties": { "eid": { "type": "string" }, "ver": { "type": "string" }, "syncts": { "type": "integer" }, "ets": { "type": "integer" }, "mid": { "type": "string" }, "actor": { "type": "object", "properties": { "id": { "type": "string" }, "type": { "type": "string" } }, "required": [ "id", "type" ] }, "edata": { "type": "object", "properties": { "type": { "type": "string" } }, "required": [ "type" ] }, "@timestamp": { "type": "string" }, "context": { "type": "object", "properties": { "pdata": { "type": "object", "properties": { "ver": { "type": "string" }, "id": { "type": "string" }, "pid": { "type": "string" } }, "required": [ "ver", "id", "pid" ] }, "did": { "type": "string" }, "env": { "type": "string" }, "channel": { "type": "string" } }, "required": [ "pdata", "did", "env", "channel" ] }, "@version": { "type": "string" }, "object": { "type": "object", "properties": { "id": { "type": "string" }, "type": { "type": "string" } }, "required": [ "id", "type" ] } }, "required": [ "eid", "ver", "syncts", "ets", "mid", "actor", "edata", "@timestamp", "context", "@version", "object" ] }, "denorm_config": { "redis_db_host": "redis_host", "redis_db_port": "redis_port", "denorm_fields": [{ "denorm_key": "", "redis_db": 1, "denorm_out_field": "metadata" }]}, "tags": [], "router_config": { "topic": "router.topic" }, "client_state": {}, "status": DatasetStatus.Live, "created_by": "SYSTEM", "updated_by": "SYSTEM", "created_date": "2023-03-13T07:46:06.410Z", "updated_date": "2023-03-14T04:46:33.459Z", "published_date": "2023-03-14T04:46:33.459Z" };
+  public static DUPLICATE_DENORM_OUT_FIELD = { "type": "master-dataset", "dataset_id": "3f8b2ba7-9c74-4d7f-8b38-2b0d460b999c", "id": "observations", "name": "telemetry-raw", " validation_config": { "validate": true, "mode": "Strict" }, "extraction_config": { "is_batch_event": false, "extraction_key": "", "dedup_config": { "drop_duplicates": true, "dedup_key": "id", "dedup_period": 3 } }, "dedup_config": { "drop_duplicates": true, "dedup_key": "id", "dedup_period": 3 }, "data_schema": { "type": "object", "properties": { "eid": { "type": "string" }, "ver": { "type": "string" }, "syncts": { "type": "integer" }, "ets": { "type": "integer" }, "mid": { "type": "string" }, "actor": { "type": "object", "properties": { "id": { "type": "string" }, "type": { "type": "string" } }, "required": [ "id", "type" ] }, "edata": { "type": "object", "properties": { "type": { "type": "string" } }, "required": [ "type" ] }, "@timestamp": { "type": "string" }, "context": { "type": "object", "properties": { "pdata": { "type": "object", "properties": { "ver": { "type": "string" }, "id": { "type": "string" }, "pid": { "type": "string" } }, "required": [ "ver", "id", "pid" ] }, "did": { "type": "string" }, "env": { "type": "string" }, "channel": { "type": "string" } }, "required": [ "pdata", "did", "env", "channel" ] }, "@version": { "type": "string" }, "object": { "type": "object", "properties": { "id": { "type": "string" }, "type": { "type": "string" } }, "required": [ "id", "type" ] } }, "required": [ "eid", "ver", "syncts", "ets", "mid", "actor", "edata", "@timestamp", "context", "@version", "object" ] }, "denorm_config": { "redis_db_host": "redis_host", "redis_db_port": "redis_port", "denorm_fields": [{ "denorm_key": "test", "redis_db": 1, "denorm_out_field": "metadata" }, { "denorm_key": "test", "redis_db": 1, "denorm_out_field": "metadata" }]}, "tags": [], "router_config": { "topic": "router.topic" }, "client_state": {}, "status": DatasetStatus.Live, "created_by": "SYSTEM", "updated_by": "SYSTEM", "created_date": "2023-03-13T07:46:06.410Z", "updated_date": "2023-03-14T04:46:33.459Z", "published_date": "2023-03-14T04:46:33.459Z" };
+  public static VALID_DENORM_OUT_FIELD = { "type": "master-dataset", "dataset_id": "3f8b2ba7-9c74-4d7f-8b38-2b0d460b91ac", "id": "observctions", "name": "telemetay-raw", " validation_config": { "validate": true, "mode": "Strict" }, "extraction_config": { "is_batch_event": false, "extraction_key": "", "dedup_config": { "drop_duplicates": true, "dedup_key": "id", "dedup_period": 3 } }, "dedup_config": { "drop_duplicates": true, "dedup_key": "id", "dedup_period": 3 }, "data_schema": { "type": "object", "properties": { "eid": { "type": "string" }, "ver": { "type": "string" }, "syncts": { "type": "integer" }, "ets": { "type": "integer" }, "mid": { "type": "string" }, "actor": { "type": "object", "properties": { "id": { "type": "string" }, "type": { "type": "string" } }, "required": [ "id", "type" ] }, "edata": { "type": "object", "properties": { "type": { "type": "string" } }, "required": [ "type" ] }, "@timestamp": { "type": "string" }, "context": { "type": "object", "properties": { "pdata": { "type": "object", "properties": { "ver": { "type": "string" }, "id": { "type": "string" }, "pid": { "type": "string" } }, "required": [ "ver", "id", "pid" ] }, "did": { "type": "string" }, "env": { "type": "string" }, "channel": { "type": "string" } }, "required": [ "pdata", "did", "env", "channel" ] }, "@version": { "type": "string" }, "object": { "type": "object", "properties": { "id": { "type": "string" }, "type": { "type": "string" } }, "required": [ "id", "type" ] } }, "required": [ "eid", "ver", "syncts", "ets", "mid", "actor", "edata", "@timestamp", "context", "@version", "object" ] }, "denorm_config": { "redis_db_host": "redis_host", "redis_db_port": "redis_port", "denorm_fields": [{ "denorm_key": "test", "redis_db": 1, "denorm_out_field": "metadata2" }, { "denorm_key": "test", "redis_db": 1, "denorm_out_field": "metadata" }]}, "tags": [], "router_config": { "topic": "router.topic" }, "client_state": {}, "status": DatasetStatus.Live, "created_by": "SYSTEM", "updated_by": "SYSTEM", "created_date": "2023-03-13T07:46:06.410Z", "updated_date": "2023-03-14T04:46:33.459Z", "published_date": "2023-03-14T04:46:33.459Z" };
 }
 
 class TestDataSource {

diff --git a/api-service/src/validators/RequestsValidator.ts b/api-service/src/validators/RequestsValidator.ts
@@ -53,6 +53,7 @@ export class RequestsValidator implements IValidator {
             routesConfig.query.native_query,
             routesConfig.query.sql_query,
             routesConfig.data_ingest,
+            routesConfig.tenant_ingest,
             routesConfig.config.dataset.save,
             routesConfig.config.datasource.save,
             routesConfig.config.dataset.list,

diff --git a/command-service/Dockerfile b/command-service/Dockerfile
@@ -3,11 +3,11 @@ COPY --from=ubuntu /usr/local/bin /usr/local/bin
 
 RUN apk update && apk add curl jq && curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" && chmod +x kubectl && mv kubectl /usr/local/bin/
 
-RUN apk add libcrypto3=3.1.4-r0
+RUN apk add libcrypto3=3.1.4-r3
 RUN apk upgrade
 WORKDIR /app
-COPY command-service/requirements.txt .
+COPY requirements.txt .
 RUN pip install --no-cache-dir -r requirements.txt
-COPY command-service/src ./src
+COPY src ./src
 WORKDIR /app/src
 CMD [ "uvicorn", "routes:app", "--host", "0.0.0.0" ]