PR #5 for silent failure work - Add files to create a job to poll the upload status from lighthouse and to create a cron job to send emails for failed uploads

[pmclaren19]

Summary

This pr is the third pr to merge some of the work from #20105. The first pr that included the db changes was previously merged #20318. The second pr that added the model, factory, constants, helpers and vcr cassettes was also previously merged #20346. The third pr added evss files and changes for document upload and was also previously merged #20380. The fourth pr added lighthouse files and changes for document upload and was also previously merged #20453.

PR Changes for the poll job:

• Make sure benefits_documents_use_lighthouse is enabled
• Make sure cst_synchronous_evidence_uploads is disabled
• cst_send_evidence_failure_emails can be disabled or enabled
• Make sure cst_send_evidence_submission_failure_emails is enabled
• Locally make sure your settings.local.yml has a vanotify section -> here
• Created upload status polling job for EvidenceSubmissions
• Created a upload_status_updater for updating the status and failure dates, acknowledgment dates, and deletion dates
• Wrote unit tests for upload_status_updater, update_documents_status_service and document_status_polling_service
• Updated Evidence Submission model to have scopes, and upload status state checker functions
• Added acknowledgement_date, failed_date, and error_message to EvidenceSubmissions model
• Created document_status_polling_service
• Created update_documents_status_service to handle document batches and send them to upload_status_updater
• Created factories for EvidenceSubmissions tests
• Created VCR cassettes for polling responses
• Updated app/sidekiq/lighthouse/evidence_submissions/document_upload.rb so that when an upload is successful we keep the upload_status as PENDING (minor bug existed where we were setting this to SUCCESS)
• Updated test spec/sidekiq/lighthouse/evidence_submissions/document_upload_spec.rb for regarding the upload_status

PR Changes for the cron job:

• Make sure benefits_documents_use_lighthouse is enabled
• Make sure cst_synchronous_evidence_uploads is disabled
• cst_send_evidence_failure_emails can be disabled or enabled
• Make sure cst_send_evidence_submission_failure_emails is enabled
• Locally make sure your settings.local.yml has a vanotify section -> here
• Add Lighthouse::EvidenceSubmissions::FailureNotificationEmailJob to lib/periodic_jobs.rb
• Created app/sidekiq/lighthouse/evidence_submissions/failure_notification_email_job.rb that is used to run the cron job to send emails for failed uploads
• Created spec/sidekiq/lighthouse/evidence_submissions/failure_notification_email_job_spec.rb to test it

Related issue(s)

• Link to ticket created in va.gov-team repo [CST] Merge feature branch for silent failure work into master va.gov-team#100037
• Link to draft pr that this is a part of [CST]: DRAFT PR for Feature branch updated es error handling [CST]: DRAFT PR for Feature branch updated es error handling [CST]: DRAFT PR for Feature branch updated es error handling [CST]: DRAFT PR for Feature branch updated es error handling  #20105

Testing done

• New code is covered by unit tests

How to test Polling Job

• Go to vets-website src/applications/claims-status/constants.js and set shouldUseMocks= false
• Make sure benefits_documents_use_lighthouse is enabled
• Make sure cst_synchronous_evidence_uploads is disabled
• cst_send_evidence_failure_emails can be disabled or enabled
• Make sure cst_send_evidence_submission_failure_emails is enabled
• Locally make sure your settings.local.yml has a vanotify section
• Update lib/lighthouse/benefits_claims/service.rb with an icn of a user in staging EX: @icn = '1012830712V627751' # icn for user 19
• Update lib/lighthouse/benefits_documents/configuration.rb to the participantId of the staging user that you are using ( you can get this from argo or ask someone with argo access to get this for you) EX: participantId: 600_073_191,
• Run vets-api and vets-website locally and upload a file
• Should see that 1 record was added/updated for evidence_submission table with a 'PENDING' status
• Then run the polling job by going to rails console and running Lighthouse::EvidenceSubmissions::EvidenceSubmissionDocumentUploadPollingJob.perform_async
• Should see that there is still 1 record in the evidence_submission table but the upload status has been updated to 'SUCCESS' status (NOTE: this manual test will always update it to SUCCESS, if you want to test the FAILED path manually in the dev env you'd have to manually change the lighthouse response)

How to test Cron Job

• Make sure benefits_documents_use_lighthouse is enabled
• Make sure cst_synchronous_evidence_uploads is disabled
• cst_send_evidence_failure_emails can be disabled or enabled
• Make sure cst_send_evidence_submission_failure_emails is enabled
• Locally make sure your settings.local.yml has a vanotify section
• Update lib/lighthouse/benefits_claims/service.rb with an icn of a user in staging EX: @icn = '1012830712V627751' # icn for user 19
• Update document_upload so that sidekiq retries are set to 0 for testing locally
• Update lib/lighthouse/benefits_documents/configuration.rb to the participantId is not being set to a specific staging user (only needed if you were testing previous scenario) --> this is how we will generate an error
• Update app/sidekiq/lighthouse/evidence_submissions/failure_notification_email_job.rb so that where notify_client.send_email() is called we replace recipient_identifier: { id_value: icn, id_type: 'ICN' } with email_address: 'YOUR_EMAIL',
• Run vets-api and vets-website locally and upload a file
• Should see that 1 record was added/updated for evidence_submission table and it has a FAILED status and tempalte_metadata also has a date_failed
• Then run the cron job by going to rails console and running Lighthouse::EvidenceSubmissions::FailureNotificationEmailJob.perform_async
• Should see that there is still 1 record in the evidence_submission table but now it has a va_notify_id and va_notify_date
• Should receive an email

Screenshots for Testing Cron Job

**Ensure that benefits_documents_use_lighthouse is enabled, cst_send_evidence_failure_emails is enabled or disabled, cst_synchronous_evidence_uploads is disabled, and cst_send_evidence_submission_failure_emails is enabled. Follow above instruction to make the file upload fail. When you upload a file, an evidence submission record is added and updated (it will have a FAILED upload_status and a date_failed). **

** Test that once you run the cron job in rails console an email is sent for failed upload and the evidence submission record is updated to have a va_notify_id and va_notify_date **

Screenshots for Testing Poll Job

Ensure that benefits_documents_use_lighthouse is enabled, cst_send_evidence_failure_emails is enabled or disabled, cst_synchronous_evidence_uploads is disabled, and cst_send_evidence_submission_failure_emails is enabled. Follow above instruction to make the file upload be successful. When you upload a file, an evidence submission record is added and updated (it will have a PENDING upload_status).

** Test that once you run the polling job in rails console an evidence submission record is updated to have a SUCCESS upload status and a delete_date **

What areas of the site does it impact?

Claim Status Tool

Acceptance criteria

• I fixed|updated|added unit tests and integration tests for each feature (if applicable).
• No error nor warning in the console.
• Events are being sent to the appropriate logging solution
• Documentation has been updated (link to documentation)
• No sensitive information (i.e. PII/credentials/internal URLs/etc.) is captured in logging, hardcoded, or specs
• Feature/bug has a monitor built into Datadog (if applicable)
• If app impacted requires authentication, did you login to a local build and verify all authenticated routes work as expected
• I added a screenshot of the developed feature

Requested Feedback

(OPTIONAL)What should the reviewers know in addition to the above. Is there anything specific you wish the reviewer to assist with. Do you have any concerns with this PR, why?

[github-actions]

1 Error

🚫

This PR changes 513 LoC (not counting whitespace/newlines). In order to ensure each PR receives the proper attention it deserves, those exceeding 500 will not be reviewed, nor will they be allowed to merge. Please break this PR up into smaller ones. If you have reason to believe that this PR should be granted an exception, please see the Submitting pull requests for approval - FAQ. File Summary Files .github/CODEOWNERS (+6/-0) app/sidekiq/lighthouse/evidence_submissions/document_upload.rb (+2/-3) app/sidekiq/lighthouse/evidence_submissions/evidence_submission_document_upload_polling_job.rb (+49/-0) app/sidekiq/lighthouse/evidence_submissions/failure_notification_email_job.rb (+67/-0) lib/lighthouse/benefits_documents/documents_status_polling_service.rb (+21/-0) lib/lighthouse/benefits_documents/update_documents_status_service.rb (+28/-0) lib/lighthouse/benefits_documents/upload_status_updater.rb (+51/-0) lib/periodic_jobs.rb (+2/-0) spec/factories/lighthouse/benefits_documents/evidence_submission.rb (+0/-4) spec/lib/lighthouse/benefits_documents/update_documents_status_service_spec.rb (+61/-0) spec/lib/lighthouse/benefits_documents/upload_status_updater_spec.rb (+76/-0) spec/sidekiq/lighthouse/evidence_submissions/document_upload_spec.rb (+1/-1) spec/sidekiq/lighthouse/evidence_submissions/evidence_submission_document_upload_polling_job_spec.rb (+67/-0) spec/sidekiq/lighthouse/evidence_submissions/failure_notification_email_job_spec.rb (+74/-0) Note: We exclude files matching the following when considering PR size: *.csv, *.json, *.tsv, *.txt, *.md, Gemfile.lock, app/swagger, modules/mobile/docs, spec/fixtures/, spec/support/vcr_cassettes/, modules/mobile/spec/support/vcr_cassettes/, db/seeds, modules/vaos/app/docs, modules/meb_api/app/docs, modules/appeals_api/app/swagger/, *.bru, *.pdf Big PRs are difficult to review, often become stale, and cause delays.

Generated by 🚫 Danger

[jerekshoe]

Good call