Also:

• View a full list of options and config settings by running auto-archiver --help
• Quicker startup of auto-archiver, by only installing the enabled modules
• New/additional modules can be enabled/disabled on the fly using command line arguments such as --extractor=new_temporary_extractor
• You can now run auto-archiver first time without needing to create an orchestration.yaml file first, auto-archiver will set up a basic one so you can get started in no time 🚀
• You can choose to update your config to file/rewrite your latest command line args to it (so you don't have to keep passing them) by setting the -s or --store flag on the command line
• All config settings are checked/validated at startup, along with module dependencies, to make sure you have setup modules correctly. Developers: you can create your own 'validators' to check values, and have the program check against them. Just drop them into the validators.py file (uses argparse type under the hood). Fixes improve config parsing to accommodate type casting and validation rules #130
• Improved errors/warnings if there are any config/dependency issues and more helpful guidance when getting started (plus: colour output)
  *You can now log directly to files using the logging: file option. Set the logging level as well using logging: level
• Fixed up unit tests + added a few new ones
• Set your own custom module folder with module_paths=/my/own/modules/ to allow you to easily extend auto archiver with new modules. Simply create a new module, place it in that folder then pass the folder path on the command line/save it in your orchestration.yaml
• Update yt-dlp to latest version and factor out obsolete code from the bluesky.py dropin (no longer needed, as we can call the yt-dlp extractor directly
• Create an 'authentication' mechanism for storing site authentication information (username/password, api keys, cookies etc.) in the config file, which can then be used by all extractors. Set your authentication details in one place, and all modules will be able to use them without having to set multiple additional configs!
• Fixes using the screenshot enricher with facebook (click the 'accept cookies' and close the 'login' modals to take a proper screenshot). Screenshot enricher can now also use your cookies to login to sites to take screenshots (cookies from a cookiejar file or directly from your browser).
• New CSV feeder, which allows you to input CSV files of URLs (more info here)

Developers:

• Removes ArchivingContext completely, and now passes contextual information around using the metadata metadata.set_context('key', 'val') or metadata.get_context('key', 'val')
• Easily get authentication information in your module using self.auth_for_site(url) - this returns a dict with the different authentication options available depending on what the user has set (e.g. username/password, cookiejar, api_key, cookie).
• Add your own modules without touching the auto-archiver project structure. Pass --module_paths /my/modules/folder/ on the command line, or add module_paths: /my/modules/folder/ to your config file.
• Easily leverage the power of yt-dlp by using it to extract information for a website without having to create an entirely new extractor. Subclass the 'GenericDropin from the generic_extractor module, and add the two methods extract_post and create_metadata, which both have access to yt-dlp's 'InfoExtractor' under the hood. An example of how to do this can be found here
• Allow modules to have 'dependencies' of either a) other modules b) 3rd party python packages c) binary packages, by settings them in the 'dependencies' field in the manifest.py. (Note: fixes No hashing algorithm defined for use in HtmlFormatter when HashEnricher not included #156 )

Other Fixes:

• Fix a bug sending items to the database (Fixes Database.failed() method called without all required params #173 )