Feature: Add Activity YML Codegen

[awoehrl]

Hi @bcodell,

I've tried my luck on a codegen macro for automating the generation of docs and tests for activities (#35). The macro takes a list of activities as input and outputs a model docs yaml file containing the docs and tests for all listed activities and their respective columns. It documents the available keys in the json object as well.

Things to watch out for:

• I'm using the node graph to get the config for a specific activity. This is used for parsing the data_types object as well as getting the stream name. Maybe there is an easier way to do this, but I couldn't find it in the existing code.
• I've tried to adhere to your docs format, but it should be checked for clarity as I'm not a native speaker

Example output:

version: 2
models:
  - name: customer__viewed_page
    columns:
      - name: activity_id
        description: "Unique identifier for the activity."
        data_type: STRING
        tests:
          - unique
          - not_null
      - name: customer_id
        description: "Identifier for the customer."
        data_type: STRING
      - name: anonymous_customer_id
        description: "Anonymous identifier for the customer."
        data_type: STRING
        tests:
          - not_null
      - name: activity
        description: "Type of activity performed by the customer."
        data_type: STRING
        tests:
          - not_null
      - name: ts
        description: "Timestamp of when the activity occurred."
        data_type: TIMESTAMP
        tests:
          - not_null
      - name: revenue_impact
        description: "Revenue impact of the activity, if applicable."
        data_type: INT64
      - name: link
        description: "Link associated with the activity, if applicable."
        data_type: STRING
      - name: feature_json
        description: > 
          JSON containing additional feature data related to the activity. Contains the following items:
            - article_id: STRING
      - name: activity_occurrence
        description: "Number of times the activity occurred."
        data_type: INT64
      - name: activity_repeated_at
        description: "Timestamp of when the activity was repeated, if applicable."
        data_type: TIMESTAMP
  - name: customer__bought_subscription
    columns:
      - name: activity_id
        description: "Unique identifier for the activity."
        data_type: STRING
        tests:
          - unique
          - not_null
      - name: customer_id
        description: "Identifier for the customer."
        data_type: STRING
      - name: anonymous_customer_id
        description: "Anonymous identifier for the customer."
        data_type: STRING
        tests:
          - not_null
      - name: activity
        description: "Type of activity performed by the customer."
        data_type: STRING
        tests:
          - not_null
      - name: ts
        description: "Timestamp of when the activity occurred."
        data_type: TIMESTAMP
        tests:
          - not_null
      - name: revenue_impact
        description: "Revenue impact of the activity, if applicable."
        data_type: INT64
      - name: link
        description: "Link associated with the activity, if applicable."
        data_type: STRING
      - name: feature_json
        description: > 
          JSON containing additional feature data related to the activity. Contains the following items:
            - product_id: STRING
            - product_category: STRING
            - product_name: STRING
            - product_price: FLOAT64
      - name: activity_occurrence
        description: "Number of times the activity occurred."
        data_type: INT64
      - name: activity_repeated_at

Let me know what you think!

Resolves #35

[bcodell]

Hey @awoehrl thanks for publishing this and nice work! I left some feedback - take a look at the comments. Some changes are necessary, but others suggested by my comments are certainly up for debate. Let me know what thoughts/feedback you have, and I'll take another look at any changes you make once you push them!

[bcodell]

Nice work! 🙌 🎉 🚀

[bcodell]

CI failed because the contributor's forked repo doesn't have access to secrets needed for specific CI steps. I'm merging as there are no code conflicts and the changes should not affect existing functionality.