Image file pruning on local versus remote

[timchurches]

Apologies if this has been discussed before (I did look) or is covered in the documentation. We have our local indi-allsky uploading images to AWS S3 buck and a remote web-only indi-allsky server serving images directly from that bucket. All working nicely. The local system is using SyncAPI to update the remote but is set to send empty image files, and to upload to S3 before the SyncAPI update.

My question is: if the local indi-allsky is set to prune image files older than x days, are the same prune (delete) commands sent via SyncAPI to the remote server as well? I am hoping they are not, since S3 storage is essentially unbounded and very cheap, whereas the storage on the local indi-allsky server is only as big as the SD card (128GB in our case). Secondary question: does the remote web-only server honour its settings for image file pruning, which ideally would be much longer than and different to those for the local system?

[aaronwmorris]

The image pruning is not replicated to the remote side. The expiration of images occurs in the capture process and this does not run on the remote side. Effectively, this means images are retained indefinitely on the remote (and S3).

There is a CLI program within indi-allsky which you can run on the remote system if you want to manually prune entries.

source virtualenv/indi-allsky/bin/activate

./misc/expire_images.py

However, even this script does not delete images or videos from S3. That process is still manual. I prepared some functions to delete images from S3, but it requires 1 API call to delete 1 file. Its fairly easy to browse the S3 repository an manually delete by date.

[timchurches]

OK, many thanks for the comprehensive answer, and manual expiration and file running on the remote system and S3 is fine for us and what I hoped you would say. I think the AWS CLI allows bucket object deletion using date fllters, but that can wait... and manually pruning the S3 bucket now and then is fine anyway, as you say.

[Dan-Judd]

Thanks Aaron for this answer. I too was looking for the same info.

I edited the expire_images.py file running on my remote server and changed from 10 and 365 days to 3 and 30 days.

    self._image_days = 3
    self._image_raw_days = 3
    self._image_fits_days = 3
    self._video_days = 30

When I run expire_images.py it reports that the days for deletion are still 10 and 365. Not sure what I am missing?

[aaronwmorris]

The script takes command line arguments, so the real defaults are near the bottom. You can execute the script like this:

./misc/expire_images.py --days 3 --raw 3 --fits 3 --timelapse_days 30

If you want to put it in a cronjob, start the script with an explicit path to the virtualenv python:

0 3 0 0 0 /home/username/indi-allsky/virtualenv/indi-allsky/bin/python3 /home/username/indi-allsky/misc/expire_images.py --days 3 --raw 3 --fits 3 --timelapse_days 30 >/dev/null 2>&1