Design & Discussion

[vishal-biyani]

60 Second pitch

To build homebrew for Kubernetes clusters that can easily install & manage applications and stacks running on Kubernetes easily and without knowing all underlying details

Jobs to be done

• Make it easy for the user to install a stack on a K8S cluster. This includes:
  • The user does not have to worry about what is the source of the install (YAML, Helm Chart or Operator, etc.)
  • The user does not have to think about configuring stuff after installing something. For ex. when you install a storage driver - let's say Ceph, there should be no need to configure the StorageClass separately.

For the initial prototype, we can use some examples such as installing

• Ceph (Either we check this or official docs: https://blog.kasten.io/kubernetes-backup-and-dr-using-rook-1-4-and-ceph-3-0 )
• Prometheus Stack

Future features (Not part of the prototype but JTBD)

• A UI that shows stacks that can be installed backed by a recommendation score for that target cluster (Which is provided by a SaaS service)
• A probability score which says that this will be X% successful on your cluster or something to that effect.

Out of Scope

• We assume that the K8S cluster is already configured and we don't do any cluster provisioning etc.

Design/Rough thoughts

CLI Experience

• To search a kbrew formula
  $ kbrew search

• To get information of a specific formula

$ kbrew info

• To install a specific formula
  $ kbrew install

• To upgrade one
  $ kbrew upgrade

How the HomeBrew works

If you look at the file https://github.com/Homebrew/homebrew-cask/blob/master/Casks/alfred.rb you will notice some things:

• This is only metadata that homebrew needs - such as name, description etc.
• The actual Alfred is in a different URL - in some software, this is a Github URL, in some other cases it is a binary from a website. The point is all hidden from the user.

[PrasadG193]

The applications could be a mix of helm charts and YAML manifests. E.g Helm chart for Operators + yaml manifests for CRs. Such collection can be treated as a single application. At server side, the application manifests can be converted into a single YAML manifest format by rendering helm chart and combining that with standalone YAMLs (where multiple resources manifest separated by ---) and sent back to the client. The client can apply the received manifest to create K8s resources.

Drawbacks:

1. Main drawback with this approach is client won't be able to figure out dependencies among resources listing in single yaml. E.g if the Operator needs to be running and ready before creating CR, we'll have to wait for that condition before creating CR.
   Workaround: ...

[vishal-biyani]

I think these are already well understood problems overall. Let's look at existing implementations:

1. Chart hooks are a way to do certain things in release lifecycle - it could be that a secret must be created before a Chart is installed or something more complex like taking a DB backup (https://helm.sh/docs/topics/charts_hooks/)

2. Operator has an much more elaborate scheme of things such as:

   a.Starts with an object called CluserServiceVersion (CSV) - which has Metadata, Install Strategy, CRDs etc.
   b. An operator has a "subscription" and "CatalogSource". You can actually get an operator from multiple subscription channels (Alpha vs Beta etc.)

Now we don't have to make it complex and same as those once. We need to think what is minimal set of "types" we need to make kbrew work.

So for example we need a way to define more than one step and formulate dependency on them. For ex. first step could be install an operator and second step could be create a CR for example.

What was the workaround you had in mind?

[vishal-biyani]

Hey @PrasadG193 I was looking at Homebrew Formulae and I picked up a non trivial formula to analyse. At it's core the formula is a Ruby script. I am picking up this Elastic Formula and adding some observations from that and wherever possible some points related to kbrew there. Formula here: https://github.com/Homebrew/homebrew-core/blob/cbc59187f9/Formula/elasticsearch.rb

• I think like 1-7 is fairly straight metadata

• Line 9-12 have specific SHA for specific versions of MacOS. Think of this as a manifest which is compatible with specific version of Kubernetes due to a specific API being only available in a specific group in K8S. And now extrapolate this to all three major artefacts we handle - i.e. manifests, Helm Charts and Operators.

• Line 19,20 depends_on is very relevant to us too. For example we should be able to call out specific dependencies and act accordingly

• Line 22,23 is a nice example of formulae following certain convention to assign the name to installation/service

• The install block starting on line 26 has lot of details - but at it's core it is doing bunch of work around setting up Elastic related components on your machine

• Line 64, post_install hooks are great for any post install activity - very relevant to us too!

• The plist on line 86 is again a great example of so many customisations you can do for a installation.

• Line 124 has tests which affirm installation went smooth!

So for installing Elastic and so many other things on Mac - the Brew package manager had to derive a set of vocabulary and constructs/objects. We need to think in this direction and come up with something on problem we are trying to solve. Let's discuss tomorrow!

[PrasadG193]

The preinstall-install-postinstall model would probably work for most of the K8s applications for us. But we will also think about pre/post upgrade/rollback/delete scenarios.

[vishal-biyani]

Let's double clicking on install part of it, for now let's keep the things such as post and pre install activities, metadata etc. out of discussion for MVP at least.

1. YAML

• Kbrew needs to understand all objects - at least names & types present in YAML for us to do some things effectively later during/post install

2. Helm

• We use Helm API to install Helm chart - so we don't have to replicate in Helm functionality. Once Helm returns - we can always add additional pre/post things.

3. Operator (Which could be a YAML or Helm Chart)
   Combination of (1) or (2) in most Operators.

So we need to add a source and once we add a source, we can search all apps and install.

So if we had a YAML similar to below, we could point to an operator, or a Helm chart or a Manifest and simply install with kbrew. In future this YAML will reside in Kbrew formula repo similar to homebrew:

apiVersion: v1
kind: kbrew
app:
  type: helm (helm|manifest)
  url: 
  sha256:
  version:

[PrasadG193]

@vishal-biyani additionally we can have source repository as a base directory/repo for all apps and then we can search through all the apps in the repo

apiVersion: v1
kind: kbrew
repositories:
- type: helm (helm|manifest)
  url: 
  apps:
  - name: NAME
    url: 
    sha256:
    version:

[vishal-biyani]

Hey @PrasadG193

So based on discussion here is what I think would be a good rough starting point

Think of the whole model as a pre step, install step and post step:

pre-install: ["cert-manager", "zookeeper"]
install/app: kafka
post-install: ["kafka-cluster-cr"]

And based on that a rough initial draft YAML:

apiVersion: v1
kind: kbrew
app:
  repositories: # Should be one
    type: manifest (Could be GKE/OperatorHub tomorrow)
    url: https://github.com/banzaicloud/kafka-operator/releases/download/v0.14.0/kafka-operator.crds.yaml
    # Operatorhub URLs - is there any stadard logic to build URL for a operator - right now it does not look like
    type: helm
    url: charts.bitnami.com
  name: kafka
  version: 1.12
  sha: 
  pre_install:
    app: ["zookeeper", "cert-manager"]
    step: "configmap.yaml" #Can we make this inline?
  # Could be a kbrew app or a simple object creation such as configmap
  post_install:
  # Could be a kbrew app or a a simple object creation such as configmap
    app: 
    step: "kafka-cluster-cr.yaml" #Can we make this inline?

[vishal-biyani]

From @PrasadG193: Between various phases of a kbrew app install such as pre-i/install/post-install - there is a need to share information. For ex. when installing Bitnami MySQL - the password is generated and that may be needed in the post phase.

[PrasadG193]

@vishal-biyani in the install hooks, we need to preserve the order of apps and steps while execution. So we'll need to change struct of hooks to list of hookObject, e.g

  pre_install:
    - apps: ["zookeeper", "cert-manager"]
    - steps: ["kubectl create configmap"]
    - apps: ["mysql"]
    - steps: ["echo password:", "kubectl get secret -jsonpath='data.rootPasword'"]

[PrasadG193]

We need to think about how do we pass configuration at runtime for both - helm and raw apps. I have the following high level thoughts in mind to address that -

1. For passing helm chart options for a helm app, we can introduce --options flag in kbrew through which users can pass key=value pairs which will be passed to helm install command
2. For raw apps, it is the responsibility of recipe author to read variable values from env vars. That way, users can set the required env var before installing kbrew raw app

[sahil-lakhwani]

From @PrasadG193: Between various phases of a kbrew app install such as pre-i/install/post-install - there is a need to share information. For ex. when installing Bitnami MySQL - the password is generated and that may be needed in the post phase.

And to extend this, there could also be some mechanisms to pass information between recipes.

[RealHarshThakur]

Based on the needs like depends_on and passing information between steps/recipes it is worth considering to use HCL as all of this is supported by the configuration language. Although, we can accomplish the same with YAML, we would have to do a lot of groundwork which is built-in by HCL. If this leads into end users asking for YAML, there are tools to help with it while primarily using HCL in the backend .