Skip to content

Commit

Permalink
fix: Implement logic to remove data after merge completion
Browse files Browse the repository at this point in the history
  • Loading branch information
esloch committed Feb 26, 2024
1 parent 2fab90c commit 30e00b3
Showing 1 changed file with 3 additions and 2 deletions.
5 changes: 3 additions & 2 deletions scripts/fetch_rxivx_data.sh
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,9 @@ if makim scheduler.download-rxivr --server "${server}" --begin "${latest_date}"
if python "${path_root}/scripts/merge_arxiv_data.py" "${path_root}/data/rxivx/${server}/final/${server}_full_data.json" "${path_root}/data/rxivx/${server}/downloaded/${output_filename}"; then
echo "[INFO]: Merge in the full database completed successfully."

# If both download and merge were successful, consider handling of files as needed
echo "[INFO]: Handling of files post-merge can be placed here."
# If both download and merge were successful, delete the most recent file
echo "[INFO]: Deleting the most old file: ${most_recent_file}"
rm "${path_root}/data/rxivx/${server}/downloaded/${most_recent_file}"
else
echo "[ERROR]: Merge process failed."
exit 1
Expand Down

0 comments on commit 30e00b3

Please sign in to comment.