Databricks integration

[ericfeunekes]

Documenting the discussion that @benjyw and I had regarding integration with Databricks.

We considered using Databricks's dbx library to manage deployment configuration. But this library is just a wrapper around the REST API; ultimately we should get better performance and usability if we create custom targets that map to the Databricks REST API. This also has the benefit of allowing for support of additional languages, like Scala, which I don't think dbx currently supports.

Until Databricks supports pex files, this integration would work by packaging task scripts as individual wheel files, and injecting them into jobs using init scripts.