Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DATA-3441 Update data export command #4596

Merged
merged 34 commits into from
Dec 18, 2024

Conversation

katiepeters
Copy link
Member

@katiepeters katiepeters commented Dec 3, 2024

This PR replaces our old data export command with two separate subcommands: data export binary and data export tabular. The logic powering data export binary is the same. data export tabular was essentially rewritten and now uses ExportTabularData.

This method:

  • Checks that the user is logged in.
  • Creates any missing directories in the specified --destination.
  • Utilizes ExportTabularData to query for a specific data source within an optional interval.
  • Writes the results to data.ndjson.

If there is an error, the export process will be attempted up to 5 times. If there is still an error, the data.ndjson file is removed.

Automated Testing:

  • Updated the existing tests for TestTabularDataByFilterAction to work with the new setup.
  • Added a new error test case to test for retries and removal of data.ndjson.

Manual Testing:
Ran export command locally after setting my base-url to my ExportTabularData deployed branch.

Queried for same data source as in DATA-3440

--part-id=cfc5404e-e269-425d-b1f9-ad7ce18790e9 --resource-name=globetrotter --resource-subtype=rdk:component:movement_sensor --method=Position
  • Successfully created data.ndjson file (30.7 MB).
  • Finished in ~1 min, 40 sec.

Also tested with interval start/end, just interval start, and just interval end. Confirmed that correct data ranges were returned ✅

@viambot viambot added the safe to test This pull request is marked safe to test from a trusted zone label Dec 3, 2024
@viambot viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Dec 3, 2024
@viambot viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Dec 3, 2024
@viambot viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Dec 6, 2024
@viambot viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Dec 10, 2024
@viambot viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Dec 10, 2024
@viambot viambot added the safe to test This pull request is marked safe to test from a trusted zone label Dec 13, 2024
@viambot viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Dec 13, 2024
Copy link
Member

@vijayvuyyuru vijayvuyyuru left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking pretty good, had a couple comments.

cli/app.go Outdated Show resolved Hide resolved
cli/data.go Outdated Show resolved Hide resolved
cli/data.go Outdated Show resolved Hide resolved
cli/data.go Show resolved Hide resolved
@viambot viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Dec 16, 2024
@katiepeters katiepeters requested a review from n0nick December 16, 2024 23:30
@viambot viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Dec 16, 2024
cli/data.go Outdated Show resolved Hide resolved
Copy link
Member

@n0nick n0nick left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚢

@viambot viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Dec 18, 2024
Copy link
Member

@vijayvuyyuru vijayvuyyuru left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔥

@katiepeters katiepeters requested review from a team, purplenicole730 and lia-viam and removed request for a team December 18, 2024 18:59
@katiepeters katiepeters merged commit 9fb1e11 into viamrobotics:main Dec 18, 2024
16 checks passed
@katiepeters katiepeters deleted the DATA-3441 branch December 18, 2024 19:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
safe to test This pull request is marked safe to test from a trusted zone
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants