Introduce extension points for share storage

[tdikland]

Currently the Delta Sharing server depends on a postgres database to store information about the defined shares, schemas, and tables. For a variety of reasons users may want to use a different store. My proposed solution if to introduce a ShareStore trait (open to naming suggestions) and let the app state depend on a concrete implementation defined in a library or user supplied.

[tdikland]

I'd be interested to work on this.

[roeap]

This makes a lot of sense!

In fact I am hoping, that we can introduce extension points for various components of the sharing server - including the share store.

Regarding the naming - while I don't really have a better name, conceptually this should mirror catalogues, such that we can for instance populate the shares and their children from unity catalogue ...

I don't really remember the UC APIs, but we may want to have the option to add some more descriptive metadata on top of this. Main application would be to be able to populate UIs o.a. with some information consumers may be interested in around the shared datasets.

[tdikland]

take

[tdikland]

Thanks for the feedback! Let me put up a draft and discuss further from there.

[roeap]

Not sure if this makes any sense at all, but I just had a thought 😆.

What if we more or less mirror the CatalogList, SchemaProvider, and TableProvider from datafusion - of course adjusting some types to out specific needs...

and then add some sort of metadata trait on top of this. This would allow us to re-use the StorageCatalog from delta-rs directly and potentially extend this to other catalogues that are implemented for datafusion.

Then again, just a quick thought ...

[tdikland]

I've put up a draft for now. Let me have a think about your suggestions and see how they would fit in here:)