Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve ability to support custom arguments for Fuseki Servers #2753

Open
rvesse opened this issue Oct 2, 2024 · 5 comments · May be fixed by #2754 or #2773
Open

Improve ability to support custom arguments for Fuseki Servers #2753

rvesse opened this issue Oct 2, 2024 · 5 comments · May be fixed by #2754 or #2773
Labels
enhancement Incrementally add new feature Fuseki

Comments

@rvesse
Copy link
Member

rvesse commented Oct 2, 2024

Version

5.1.0

Feature

The Fuseki Modules mechanism makes it possible to add extensions to Fuseki but modules have to rely upon external configuration sources e.g. System Properties, Environment Variables, Fuseki Configuration file etc. because there isn't a meaningful way to pass configuration via the command line.

FusekiMain is built on Jena's general CLI framework which means we can in principal extend FusekiMain and add additional arguments via the constructor, or use the static FusekiMain.addArgModule() method to register global extensions.

However, it's hard to do anything meaningful with these custom arguments, processModulesAndArgs() currently populates a private ServerConfig class which is then used to actually populate a FusekiServer.builder() via a private static applyServerArgs() which cannot be modified. Thus even if a custom argument is introduced the only way for it to pass configuration to a module would be by placing it into System Properties, static variable or another JVM wide store that the module can later read back.

As a practical example of this we're looking to introduce support for specifying Fuseki configuration via a YAML configuration file for which we'd like to add a --yaml-config <yaml-file> argument and then have a FusekiModule do a prepare() step which reads in the <yaml-file> and modifies the FusekiServer.Builder accordingly. Right now we can't do that cleanly because there's no way to get the value of <yaml-file> passed into the builder for later access without resorting to passing it out-of-band.

It would be better if we were able to carry extra configuration information via ServerConfig and override applyServerArgs() so that we can apply that to the builder this would offer users who want to extend FusekiMain much more flexibility.

Are you interested in contributing a solution yourself?

Yes

@rvesse rvesse added the enhancement Incrementally add new feature label Oct 2, 2024
rvesse added a commit to rvesse/jena that referenced this issue Oct 2, 2024
Improves how Fuseki arguments are customised, in particular makes the
ServerConfig class public adding the ability to store extra
configuration on it.

It also changes the previously private static applyServerArgs() method
into a protected instance method so users can extend FusekiMain and
override it to use that to customise the FusekiServer.Builder further.

Unit tests are added that validate that custom arguments can now be
passed through CLI inputs and used to influence the Server being built.
rvesse added a commit to rvesse/jena that referenced this issue Oct 2, 2024
Improves how Fuseki arguments are customised, in particular makes the
ServerConfig class public adding the ability to store extra
configuration on it.

It also changes the previously private static applyServerArgs() method
into a protected instance method so users can extend FusekiMain and
override it to use that to customise the FusekiServer.Builder further.

Unit tests are added that validate that custom arguments can now be
passed through CLI inputs and used to influence the Server being built.
rvesse added a commit to rvesse/jena that referenced this issue Oct 2, 2024
Improves how Fuseki arguments are customised, in particular makes the
ServerConfig class public adding the ability to store extra
configuration on it.

It also changes the previously private static applyServerArgs() method
into a protected instance method so users can extend FusekiMain and
override it to use that to customise the FusekiServer.Builder further.

Unit tests are added that validate that custom arguments can now be
passed through CLI inputs and used to influence the Server being built.
@rvesse rvesse linked a pull request Oct 2, 2024 that will close this issue
4 tasks
@afs
Copy link
Member

afs commented Oct 7, 2024

General comment - when (if!) there is a switch to Fuseki Main + UI + Admin for the standalone server (jena-fuseki-fulljar and the jars file in the download), there maybe other set of arguments that are optional. Maybe there should be an extension point for this.

If that comes along, PR #2754 would be fitted into a formal lifecycle for the command line.

Do you think adding a new interface FusekiModule would fit your customization start flow? (FusekiModule collects several interfaces already)

@afs afs added the Fuseki label Oct 9, 2024
@rvesse
Copy link
Member Author

rvesse commented Oct 10, 2024

@afs Yeah when we started on this work I had it in my head that we'd previously discussed an extension point for adding custom arguments into Fuseki but then I found what was there didn't really work for the reasons described here.

Ideally making the arguments a more formal integration point would be useful. However not sure what that looks like in practise because there's some separation of concerns issues. Currently arguments are used to populate a ServerConfig which is then applied to the builder, and it's only when the builder has build() called that modules actually kick in currently. So it seems like this would need to be an independent extension point with its own interface that gets called prior to argument processing.

I'd almost be tempted to move argument processing concerns more directly into the FusekiServer.Builder thought then that conflates concerns between CLI and API usage. Or possibly remove ServerConfig entirely and just have a builder as an instance variable of FusekiMain and let it populate it directly from its processModulesAndArgs() method. This would enable the extension style approach I showed in the PR, or more API driven extension points where you could register interfaces/functions that worked on the builder instance and used that

I'm not in a rush to get this into 5.2.0, we have a workaround for now which is basically to pre-process the arguments and effectively just overwrite the value of the --conf argument if it has a .yaml extension with our generated .ttl file.

Let me have a play around in a separate PR for comparison purposes, this will likely be Tues next week at the earliest before I can get to this.

@afs
Copy link
Member

afs commented Oct 10, 2024

The builder is available in FusekiModule.prepare but builder is limited as it has a fixed contract. Some custom arguments may be complete separate from building.

The extensions would ideally not subclass FusekiMain or any Fuseki cmd/builder class because multiple sets of argument don't "play nice" by subclassing.

It needs a "prepare args" via some provided ArgModules with a lifecycle "setup - parse - respond to args - ....".

rvesse added a commit to rvesse/jena that referenced this issue Oct 14, 2024
Refactors how the FusekiMain CLI can be customised to make it more
flexible and interface driven.  CLI extensions have to be statically
registered prior to invoking FusekiMain but are now able to both
customise the arguments available and to directly affect the
FusekiServer.Builder without needing to sub-class FusekiMain
@rvesse rvesse linked a pull request Oct 14, 2024 that will close this issue
4 tasks
rvesse added a commit to rvesse/jena that referenced this issue Oct 14, 2024
Refactors how the FusekiMain CLI can be customised to make it more
flexible and interface driven.  CLI extensions have to be statically
registered prior to invoking FusekiMain but are now able to both
customise the arguments available and to directly affect the
FusekiServer.Builder without needing to sub-class FusekiMain
@afs
Copy link
Member

afs commented Nov 13, 2024

@rvesse -- I'm working on a PR with material taken from #2773 that focuses on just a customised command line.

I think there needs to more steps in the customisation process

  • add custom arguments
  • process arguments to get any flags and arguments - this happens after the standard arguments have filled ServerConfig
  • after the server configuration has been setup before building starts - this can react to the configuration model before the build process uses it.

ServerConfig becomes public (it is just a mutable record - may rename as ServerArgs) and now has both the file name from --config and also a model slot initialized to null. At the start of building and there no model (the normal case), then the filename is read as RDF. A customizer can get the file name, and itself set the configuration model.

  • The configuration file name can be checked before use (e.g. either an allow-list or a deny-list implemented, or string for a customiser to interpret.
  • --config=custom.yaml can be made to work, as well as --yaml-config.

afs added a commit to afs/jena that referenced this issue Nov 13, 2024
afs added a commit to afs/jena that referenced this issue Nov 14, 2024
afs added a commit to afs/jena that referenced this issue Nov 14, 2024
afs added a commit to afs/jena that referenced this issue Nov 14, 2024
@rvesse
Copy link
Member Author

rvesse commented Nov 14, 2024

@afs You're much more familiar with this area of rhetoric code so if you have a cleaner approach then my initial attempt happy to test that once it's ready

afs added a commit to afs/jena that referenced this issue Nov 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Incrementally add new feature Fuseki
Projects
None yet
2 participants