Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

singer-check-tap ignores missing properties #19

Open
lukaepti opened this issue Feb 17, 2021 · 0 comments
Open

singer-check-tap ignores missing properties #19

lukaepti opened this issue Feb 17, 2021 · 0 comments

Comments

@lukaepti
Copy link

lukaepti commented Feb 17, 2021

Problem:
singer-check-tap ignores RECORDs with missing properties, even though their SCHEMA does not allow said properties to be null

Expected behavior:
singer-check-tap declares output invalid, or at least warns about records being at odds with the schema

Example:

{"type": "SCHEMA", "stream": "stream", "schema": {"type": "object", "properties": {"id": {"type": "number"}}}, "key_properties": ["id"]}
{"type": "RECORD", "stream": "stream", "record": {}, "time_extracted": "2021-02-17T22:15:35.591070Z" } 

In this simple example, the schema declares id's type to be string (and only string). In fact, it declares it to be a key_property. The record, however, is empty - it doesn't contain id nor any other property. singer-check-tap considers this valid (returns success) as evidenced by this output:

Checking stdin for valid Singer-formatted data
The output is valid.
It contained 2 messages for 1 streams.

      1 schema messages
      1 record messages
      0 state messages

Details by stream:
+--------+---------+---------+
| stream | records | schemas |
+--------+---------+---------+
| stream | 1       | 1       |
+--------+---------+---------+
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant