Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: warehouse transformer #5205

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Conversation

achettyiitr
Copy link
Member

@achettyiitr achettyiitr commented Oct 17, 2024

Description

  • Warehouse transformation as a module.
  • It's basically ported from the warehouse module of the rudder-transformer from Javascript to GO.
  • More about it can be found in this doc.
  • Currently, there is a flag to enable warehouse transformations along with Processor using the flag Processor.enableWarehouseTransformations.
    • Once enabled, we make an API call to the warehouse transformer package.
    • Compare the response, in case we see any differences, we log it for debugging purposes.

Linear Ticket

  • Resolves WAR-187

Security

  • The code changed/added as part of this pull request won't create any security issues with how the software is being used.

@achettyiitr achettyiitr force-pushed the feat.warehouse-transformer branch from a941966 to 22b287b Compare October 17, 2024 07:49
@achettyiitr achettyiitr requested a review from lvrach October 17, 2024 07:49
@achettyiitr achettyiitr force-pushed the feat.warehouse-transformer branch from 22b287b to 5ae4838 Compare October 17, 2024 07:51
@achettyiitr achettyiitr force-pushed the feat.warehouse-transformer branch 3 times, most recently from 1a41ba4 to e414cca Compare October 18, 2024 04:45
Copy link

codecov bot commented Oct 18, 2024

Codecov Report

Attention: Patch coverage is 75.47387% with 427 lines in your changes missing coverage. Please review.

Project coverage is 74.83%. Comparing base (2541b1c) to head (546af2a).
Report is 3 commits behind head on master.

Files with missing lines Patch % Lines
warehouse/transformer/events.go 56.98% 154 Missing and 74 partials ⚠️
warehouse/transformer/set.go 63.86% 31 Missing and 12 partials ⚠️
processor/processor.go 29.16% 34 Missing ⚠️
warehouse/transformer/logger.go 5.71% 33 Missing ⚠️
warehouse/transformer/testhelper/testhelper.go 20.83% 19 Missing ⚠️
warehouse/transformer/safe.go 92.30% 11 Missing and 4 partials ⚠️
warehouse/transformer/idresolution.go 90.97% 8 Missing and 4 partials ⚠️
warehouse/transformer/testhelper/outputbuilder.go 75.00% 8 Missing and 4 partials ⚠️
warehouse/transformer/transformer.go 93.95% 8 Missing and 1 partial ⚠️
...rmer/internal/reservedkeywords/reservedkeywords.go 73.91% 4 Missing and 2 partials ⚠️
... and 5 more
Additional details and impacted files
@@            Coverage Diff            @@
##           master    #5205     +/-   ##
=========================================
  Coverage   74.82%   74.83%             
=========================================
  Files         440      458     +18     
  Lines       61668    63406   +1738     
=========================================
+ Hits        46145    47448   +1303     
- Misses      12987    13316    +329     
- Partials     2536     2642    +106     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@achettyiitr achettyiitr force-pushed the feat.warehouse-transformer branch 4 times, most recently from 404773f to c798625 Compare October 21, 2024 01:40
@achettyiitr achettyiitr requested a review from lvrach October 21, 2024 01:42
@achettyiitr achettyiitr force-pushed the feat.warehouse-transformer branch 2 times, most recently from 9f36c96 to 3f956b4 Compare October 21, 2024 05:47
@achettyiitr achettyiitr force-pushed the feat.warehouse-transformer branch 2 times, most recently from 11996b9 to 14aa043 Compare October 21, 2024 09:54
@achettyiitr achettyiitr requested a review from ktgowtham October 21, 2024 09:54
@github-actions github-actions bot added the Stale label Nov 11, 2024
@github-actions github-actions bot closed this Nov 18, 2024
@achettyiitr achettyiitr reopened this Dec 18, 2024
@github-actions github-actions bot removed the Stale label Dec 19, 2024
@github-actions github-actions bot added the Stale label Jan 9, 2025
@rudderlabs rudderlabs deleted a comment from github-actions bot Jan 14, 2025
@rudderlabs rudderlabs deleted a comment from github-actions bot Jan 14, 2025
@achettyiitr achettyiitr force-pushed the feat.warehouse-transformer branch from 14aa043 to 3491119 Compare January 14, 2025 14:26
@github-actions github-actions bot removed the Stale label Jan 15, 2025
@achettyiitr achettyiitr force-pushed the feat.warehouse-transformer branch 4 times, most recently from 558e1a4 to ca8ec5a Compare January 15, 2025 08:13
@achettyiitr achettyiitr marked this pull request as draft January 15, 2025 08:13
@achettyiitr achettyiitr requested review from ktgowtham and lvrach and removed request for ktgowtham and lvrach January 15, 2025 08:14
@achettyiitr achettyiitr force-pushed the feat.warehouse-transformer branch 9 times, most recently from 6934017 to dcc7800 Compare January 20, 2025 07:52
@achettyiitr achettyiitr marked this pull request as ready for review January 20, 2025 08:33
@achettyiitr achettyiitr force-pushed the feat.warehouse-transformer branch from dcc7800 to a527f87 Compare January 20, 2025 08:56
@achettyiitr achettyiitr force-pushed the feat.warehouse-transformer branch from a527f87 to 546af2a Compare January 20, 2025 09:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants