first stab at a data backfill for StateFileArchivedIntake #5305

jnf · 2024-12-31T18:40:52Z

Link to pivotal/JIRA issue

FYST-1504

Is PM acceptance required? (delete one)

No - merge after code review approval

What was done?

Adds a rake task (and corresponding service object) to identify and create records for intakes/submissions from TY23 that are 'archiveable.'
Defining 'archiveable' as an intake + accepted efile submission for TY23.
- Not concerned w/ efile submissions in any other state, as, for the purposes of the larger project, we're creating this archived records for the explicit purpose of providing PDFs to clients who successfully submitted a return.
The rake task is batched at 10. That's super safe mode and we can tune it if we want. I chose such a low number as 'archiving' requires attaching a PDF blob to the archive record. It's still unclear to me whether this duplicates the actual file in S3 or just adds an additional reference. Need to figure out which it is and ensure that the later task of removing old intakes from the production table doesn't accidentally break this association.

How to test?

The service object is testable in a rails console. Creating an instance for a state exposes the core methods of #query_archiveable, #current_batch, and #archive_batch.
The rake task can be run locally, but you'll need some TY23 returns with an 'accepted' efile submission. jey has a database dump you can use with NY and AZ returns in this state.

github-actions · 2024-12-31T18:45:07Z

Heroku app: https://gyr-review-app-5305-3f92ceae63ac.herokuapp.com/
View logs: heroku logs --app gyr-review-app-5305 (optionally add --tail)

anisharamnani · 2025-01-02T17:40:30Z

app/services/state_file/ty23_archiver_service.rb

@@ -0,0 +1,74 @@
+module StateFile


at a high level, i think it would be helpful to write tests for this file.

anisharamnani · 2025-01-02T17:41:53Z

app/services/state_file/ty23_archiver_service.rb

+        next
+      end
+      Rails.logger.info("Archived #{archived_ids.count} #{data_source} intakes: [#{archived_ids.join(', ')}]")
+      @current_batch = nil # reset the batch


why is the @current_batch set to nil for this method?

ah, i think it is to exit the while loop in the rake task.

setting it to nil to represent that the batch has been processed; those intakes are no longer eligible for archiving because they've been archived :)

anisharamnani · 2025-01-02T17:42:45Z

app/services/state_file/ty23_archiver_service.rb

+        next
+      end
+      Rails.logger.info("Archived #{archived_ids.count} #{data_source} intakes: [#{archived_ids.join(', ')}]")
+      @current_batch = nil # reset the batch


ah, i think it is to exit the while loop in the rake task.

anisharamnani · 2025-01-02T17:47:46Z

app/services/state_file/ty23_archiver_service.rb

+        archive = StateFileArchivedIntake.new(record.without('source_intake_id'))
+        archive.submission_pdf.attach(data_source.find(record['source_intake_id']).submission_pdf.blob)
+        archive.save!
+        archived_ids << record['source_intake_id']


🤔 i am assuming that we don't want to archive and delete in the same rake task.

maybe it would be helpful to have a method that after the archive_batch is run, we verify that the archivable intakes now have an archivable record.

i don't if that makes sense or is easy.

also how were you thinking of recording the intake IDs for deleting? or were you thinking of running the query again and then deleting them?

not deleting in this task. collecting the ids for logging purposes only. we'll do the actual table truncation in a different effort.

anisharamnani · 2025-01-02T17:55:34Z

also, the annotate error should be resolved if you merge in FYST-1466-prior-year-access-mvp 🎉

anisharamnani · 2025-01-02T17:59:42Z

app/services/state_file/ty23_archiver_service.rb

+          #{tax_year} AS tax_year, '#{state_code}' AS state_code,
+          email_address, hashed_ssn, id AS source_intake_id
+        FROM
+          state_file_az_intakes


🤔 just realized this query is just for AZ. since this is a WIP PR, i'm sure you were gonna change this, but just pointing out for other reviewers.

yeah, fixed in the next commit.

…off date configurable; this will help in testing where we won't reliably have accepted returns with transition dates before the 'real' ty23 cutoff of may-ish 2024

…ver servicer

… intake record

…ailing address syncing

…iving a batch

jnf added the wip denotes a work in progress that isn't ready for formal review label Dec 31, 2024

anisharamnani reviewed Jan 2, 2025

View reviewed changes

jnf added 6 commits January 6, 2025 09:13

first stab at a data backfill for StateFileArchivedIntake

716db27

let's log to both stdout and the regular log file

a347164

update ty23 archiver to use keyword arguments and make the filing cut…

210a596

…off date configurable; this will help in testing where we won't reliably have accepted returns with transition dates before the 'real' ty23 cutoff of may-ish 2024

got my LIMIT clause attached to the wrong part of the query

97f1afe

update the rake task to use keyword args when instantiating the archi…

78a07b8

…ver servicer

abstract table name in the archiveable query

a941eb4

jnf force-pushed the jnf/FYST-1466-2 branch from 6143bf3 to a941eb4 Compare January 6, 2025 17:13

jnf and others added 5 commits January 6, 2025 10:50

i've made my peace with subqueries

fd60195

leaves a todo/hook for populating the mailing address on the archived…

9c490b0

… intake record

start of an automated test for the archiver service, setting up for m…

77e7a2a

…ailing address syncing

download & attach submission_pdf to StateFileArchivedIntake when arch…

30c22fa

…iving a batch

added additional fields

746eefc

mpidcock force-pushed the jnf/FYST-1466-2 branch from a92fec4 to 746eefc Compare January 8, 2025 04:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

first stab at a data backfill for StateFileArchivedIntake #5305

first stab at a data backfill for StateFileArchivedIntake #5305

jnf commented Dec 31, 2024 •

edited by jira bot

Loading

github-actions bot commented Dec 31, 2024

anisharamnani Jan 2, 2025

anisharamnani Jan 2, 2025

anisharamnani Jan 2, 2025

jnf Jan 6, 2025

anisharamnani Jan 2, 2025

anisharamnani Jan 2, 2025

anisharamnani Jan 2, 2025 •

edited

Loading

jnf Jan 6, 2025

anisharamnani commented Jan 2, 2025

anisharamnani Jan 2, 2025

jnf Jan 6, 2025

first stab at a data backfill for StateFileArchivedIntake #5305

Are you sure you want to change the base?

first stab at a data backfill for StateFileArchivedIntake #5305

Conversation

jnf commented Dec 31, 2024 • edited by jira bot Loading

Link to pivotal/JIRA issue

Is PM acceptance required? (delete one)

What was done?

How to test?

github-actions bot commented Dec 31, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

anisharamnani Jan 2, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

anisharamnani commented Jan 2, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jnf commented Dec 31, 2024 •

edited by jira bot

Loading

anisharamnani Jan 2, 2025 •

edited

Loading