Consolidation and normalization of x-maturity parameters within the servers block

[RichardBruskiewich]

The original use of x-maturity properties was simply to tag the four distinct environments - development, testing, staging and production - for deployment of online resources like KP and ARA systems. Along the way, a few issues have arisen:

1. Deployment of multiple servers of a given x-maturity environment. In Translator community discussions, the purpose of such multiple deployments appears to simply provide redundancy for reliability of resource access, and all such servers so deployed are considered fungible (i.e. functionally equivalent). However, at present, for some resources, such redundancy is expressed in the servers block of the SmartAPI specification, as a duplication of x-maturity entries for each server.
2. Community web services (TRAPI) and semantic (Biolink) release versions are being iteratively developed and deployed sequentially across x-maturity environments. Thus, as of February 2023, the 'production' servers might implement TRAPI 1.3 and Bioink 2.4.8, but 'development' (or staging, or testing) servers might be deployed with TRAPI 1.4 and Biolink 3.1.1. Thus, tracking standard versions by x-maturity is desirable. A Git issue was posted to propose indexing of Biolink release by x-maturity; however, it was subsequently pointed out that TRAPI versioning may have the same imperatives.
3. The evolution of SRI Testing facilities promoted the specification of the info.x-trapi.test_data_location property in the Translator SmartAPI Registry originally as a single JSON file KP test edge data specification or ARA configuration file file. However, the need for differential testing across x-maturity environment resulted an extension of the value of this property to distinctly x-maturity indexed files. This results in x-maturity tags being redundantly used in the info block.

This issue is being posted in the spirit of harmonizing and simplifying metadata related to x-maturity environments. In particular, it is generally proposed to move all parameters of service deployment - e.g. TRAPI version, Biolink version and test_data_location - that may be heterogeneous across x-maturity environments, to a DRY indexing of such parameters by x-maturity, within the servers block where x-maturity environment management is nominal.

Dealing with item 1 above would ideally use a list of server endpoints under the servers block 'url' key, but this is not actually allowed for under OpenAPI. That said, there seems to be the potentially interesting loophole of using Server Variables (which we exploit below to reduce ARAX server redundancy).

I don't know how many degrees of freedom we have here to revise the (Translator) SmartAPI specification, but here are some basic ideas normalizing x-trapi and x-translator metadata into the servers block, given the above realities:

servers:
  - description: ARAX TRAPI 1.3 endpoint - production
    # multiple servers targeting the same x-maturity
    # environment, should be specified using server variables.
    url:  https://arax.{host}.io/api/arax/v1.3
    variables:
       host:
         enum:
           - 'transltr'
           - 'ncats'
         default: 'transltr'
         description: various fungible production server path identifiers
    x-maturity: production
    x-trapi:
       trapi-version: 1.3
       test_data_location: 
       # may be a list of data files, as per the recent specification change
       # however, there is no need for the JSON object format of this property (except that we lose the 'default' option)
       - https://arax.transltr.io/api/arax/v1.3/production_test_data_file_1.json
       - https://arax.transltr.io/api/arax/v1.3/production_test_data_file_2.json
    x-translator:
      - biolink-version: 2.2.11
  - description: ARAX TRAPI 1.4 endpoint - development
    url:  https://arax.ncats.io/{server_id}/api/arax/v1.4
    variables:
       server_id:
         enum:
           - 'test'
           - 'beta'
           - 'devEd'
         default: 'test'
         description: various fungible development server path identifiers
    x-maturity: development
    x-trapi:
       # This environment transitioning to the next generation of TRAPI
       trapi-version: 1.4
       test_data_location: 
       # may be a list of data files, as per the recent
       # extension to the test_data_location property specification
       - https://arax.ncats.io/test/api/arax/v1.4/development_test_data_file_1.json
       - https://arax.ncats.io/test/api/arax/v1.4/development_test_data_file_2.json
    x-translator:
      - biolink-version: 3.1.1

[EvanDietzMorris]

This approach makes a lot of sense to me. Different maturity levels of deployment can and will implement different biolink and trapi versions, and have different versions of data being served (hence different test files), so there must be some way to represent these differences. Having these values spread across the spec seems messy, and as mentioned comes close to violating the DRY principle. The servers block seems like a natural place to include all of the attributes that may vary from server to server.

With this solution I don't see a need to have the test_data_location be a dictionary. A list or single URL should do, as all of the other relevant information could be found in the same server block entry.

I don't see the need for multiple servers per maturity level. If the goal is simple redundancy or load balancing it seems like those things could be better implemented on the back end, with the service still exposed at one public endpoint. Perhaps that decision is outside of the scope of this issue though.

[RichardBruskiewich]

Thanks @EvanDietzMorris for the feedback.

On the issue of multiple servers, the main example I know of so far is ARAX - which has multiple entries for some x-maturity environments (although there may be others) and the main rationale given overall was redundancy of service: all servers are equivalent but one or the other may be offline at a given time.

Anyhow, I guess this issue - of multiple services - can be punted to the wider Architecture team. The solution I indicated above - the user of OpenAPI server variables - is actually probably the 'best practices' solution to server redundancy and ought to be promoted within Translator.