feat(sdk): Add Input Parameter support for configmap, secrets, node selectors, tolerations, pull secrets

[HumairAK]

Description of your changes:

Related:

• [kubernetes_platform] Error calling kubernetes.use_secret_as_env and use_secret_as_volume with secret name as parameter #10914
• [feature] Add ability to specify node affinity & toleration using KFP V2 #9682
• [sdk] Unable to pass env variables as pipeline parameters #11395
• [feature] Make set_env_variable compatible with the channels [KFP V2] #10111
• [backend] Cannot inject environment variables and volumes into components with the KFP CLI #9924
• [feature] Support parameter inputs for V2 Kubernetes_platform Spec #10534

These will be marked resolved once the backend pr is merged after this.

This change provides sdk support for the following kubernetes platform
fields:

• configmap name for configmap as env & volume
• secret name for configmap as env & volume
• image pull secret name

It also adds support for tolerations, node selectors as input json for
the entire toleration & node selector spec. We opt for this solution
because if a user were to try to use as input parameters individual key
value pairs, the input parameter count would increase drastically.

The implementation borrows the InputParamSpec from the pipeline spec
package. This is copied over instead of imported due to cyclic
dependency issues with proto imports and the current project structure.

Additionally, pvc_mount is updated to accomodate this new spec, which
makes it more in line & consistent with how input parameters are handled
elsewhere in the spec.

Finally, all old name parameter fields like configmap names, secret
names, etc. are deprecated, as the runtime value proto message covers
this via it's constant field.

Checklist:

• You have signed off your commits
• The title for your pull request (PR) should follow our title convention. Learn more about the pull request title convention used in this repository.

[HumairAK]

cc @chensun please review the proto changes here when you get the chance so we are aligned here, as some fields are also being deprecated

[HumairAK]

Please also note that this is a more generic implementation of how we were handling inputs already for pvc mounts in the k8s platform spec here, but extended to various other fields

One thing also worth noting is that we probably don't need the entire InputParameterSpec as it is defined in the pipelinespec, for example TaskFinalStatus might not make much sense here, but for simplicity and maintainability's sake I figured it's easier to just keep them fully in sync. In the future we should consider just updating the structure of our proto files so we can directly import these instead of copying them over.

[HumairAK]

CI failures are due to flaky tests I believe. Will retry them a couple of times to confirm.
Actually I don't think so, investigating.

[HumairAK]

The tests failures are a redherring, it's because of the compiled proto packages that are not utilizing the new code
See this test PR that references the k8s go packages locally, the tests here pass, the only difference being this commit

[chensun]

The order of such change should be:

1. proto change, adding new field while mark the old one deprecated.
2. backend change, support both new field and old field so it's forward and backward compatible at the same time.
3. SDK change, switching to use new field.

[chensun]

Here's a backend reference for this proto: https://github.com/kubeflow/pipelines/blob/94eca2102c8ee0df590566b1e115275f0654b6f6/backend/src/v2/driver/driver.go#L37C56-L37C74

If SDK changes goes live, users will hit errors from backend as the backend won't recognize the new field yet, and users will be stuck with no (backend) upgrade path.

[HumairAK]

@chensun can you elaborate on your comment, I don't think I fully understand.

SDK change, switching to use new field.

@chensun the changes are additive, and user is not forced to switch to the new fields, as the old ones are still applicable. Only if they want to use the parameterization, then they need to opt for the new field.

I certainly don't want to wait for a backend release with the support for the new fields before we can cut the sdk release with the new changes., since backend releases happen at a much slower cadence.

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from chensun. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• kubernetes_platform/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment