Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ingesting DBT Data is not working - OMD 1.5.2 #17781

Closed
mazenorabi opened this issue Sep 10, 2024 · 2 comments
Closed

Ingesting DBT Data is not working - OMD 1.5.2 #17781

mazenorabi opened this issue Sep 10, 2024 · 2 comments
Assignees
Labels
bug Something isn't working P0 Highest priority

Comments

@mazenorabi
Copy link

mazenorabi commented Sep 10, 2024

Affected module
Ingestion Framework

Describe the bug
Running dbt workflow externally to ingest dbt data return many errors as below and nothing happen:

nodes.`model.snowflake_dbt.sessions`.Nodes7.latest_version
[   Extra inputs are not permitted [type=extra_forbidden, input_value=None, input_type=NoneType]
     For further information visit https://errors.pydantic.dev/2.8/v/extra_forbidden
nodes.`model.snowflake_dbt.sessions`.Nodes7.deprecation_date
   Extra inputs are not permitted [type=extra_forbidden, input_value=None, input_type=NoneType]
     For further information visit https://errors.pydantic.dev/2.8/v/extra_forbidden]
 Traceback (most recent call last):
   File "/ingestion/main.py", line 114, in <module>
     main()
  File "/ingestion/main.py", line 108, in main
  workflow.raise_from_status()
 File "/usr/local/lib/python3.10/site-packages/metadata/workflow/workflow_status_mixin.py", line 134, in raise_from_status
raise err
File "/usr/local/lib/python3.10/site-packages/metadata/workflow/workflow_status_mixin.py", line 131, in raise_from_status     
self.raise_from_status_internal(raise_warnings)
File "/usr/local/lib/python3.10/site-packages/metadata/workflow/ingestion.py", line 150, in raise_from_status_internal
  raise WorkflowExecutionError(
 metadata.config.common.WorkflowExecutionError: dbt reported errors: dbt Summary: [0 Records, [0 Updated Records, 0 Warnings, 1 Errors, 0 Filtered]

To Reproduce

Use this yaml config to run DBT ingestion (run the ingestion framework as docker image on K8s):

source:
  type: dbt
  serviceName: {{ DATABASE_SERVICE_NAME }}
  sourceConfig:
    config:
      type: DBT
      dbtConfigSource:
        dbtConfigType: local
        dbtCatalogFilePath: {{ DBT_CATALOG_PATH }}
        dbtManifestFilePath: {{ DBT_MANIFEST_PATH }}
        dbtRunResultsFilePath: {{ DBT_RUN_RESULTS_PATH }}
      dbtUpdateDescriptions: true
      includeTags: true
      dbtClassificationName: {{ DBT_CLASSIFICATION_NAME }}
sink:
  type: metadata-rest
  config: {}
workflowConfig:
  loggerLevel: DEBUG
  openMetadataServerConfig:
    hostPort: https://openmetadata/api
    authProvider: openmetadata
    verifySSL: ignore
    securityConfig:
      jwtToken: ***

Expected behavior
to ingest DBT Data, see lineage and DBT tab on UI.

Version:

  • OS: K8s
  • Python version: 3.10
  • OpenMetadata version: 1.5.2 Upgraded from 1.4.7
  • OpenMetadata Ingestion package version: ingestion-base==1.5.2
  • DBT version: 1.8.5
  • Warehouse: Snowflake

Additional context
Add any other context about the problem here.

@mazenorabi mazenorabi changed the title Ingesting DBT Date file is not working Ingesting DBT Data is not working - OMD 1.5.2 Sep 10, 2024
@pmbrull pmbrull added the bug Something isn't working label Oct 4, 2024
@LeoLuo123
Copy link
Contributor

We also see this issue. Wondering when this can be fixed

@harshach harshach removed this from Release 1.7.0 Dec 14, 2024
@harshach harshach moved this to Integration in Release 1.6.2 Dec 14, 2024
@harshach harshach added the P0 Highest priority label Dec 14, 2024
@ulixius9
Copy link
Member

closing by yu-iskw/dbt-artifacts-parser#130

upgrading to dbt-artifacts-parser==0.8.0 fixes this issue

@github-project-automation github-project-automation bot moved this from Integration to Done in Release 1.6.2 Dec 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working P0 Highest priority
Projects
No open projects
Status: Done
Development

No branches or pull requests

7 participants