This is the script for a talk I gave at Kube London, May 30th 2023. It wasn’t recorded, and I didn’t use slides – just a browser with tabs open at the pages linked in the script.
I’m going to talk today about what I think is promising new work from a colleague of mine in the Flux project, Stefan Prodan. It’s called Timoni. As always let’s start with –
What’s the problem to be solved?
- In Kubernetes, you can’t really get away YAML and the complexity of Kubernetes API objects, which combine several concerns, and you usually only care about one at a time, and …
- maybe you can make it easier to reuse chunks?
That’s kind of what Helm set out to do. Timoni is a Kubernetes-specific package manager, like Helm.
This is how Stefan describes Timoni: https://timoni.sh/. I would describe Timoni as “Helm but with the benefit of hindsight”.
Why not just use Helm? What’s wrong with it anyway. For my money, these are the main things:
- string templating
- index files are an awful hangover
- “values” just means you end up with an ad-hoc API as complicated as the underlying declarations, but somewhat particular to the chart.
Timoni uses CUE rather than string templates, which eliminates the first problem; and relies on OCI, avoiding the second and bringing in some other benefits. I think Timoni could still fall into the third trap, which I’ll talk about later.
How does Timoni do better than Helm?
In Timoni, rather than text templates you write CUE programs.
CUE programs can look like templates anyway – as in, you can spell out a value and just interpolate other bits in – and the CUE tooling helps you get from YAMLs to programs. There’s a whole tutorial about it.
So there’s a nice story there for going from raw YAMLs to a Timoni module.
Anyway the point is they are still programs, and you can split them up into files and directories, factor values into definitions, and use abstractions.
Let’s take a little tour of a Timoni module, to demonstrate that.
This is a Timoni module (it’s also a CUE module): https://github.com/stefanprodan/timoni/blob/main/examples/podinfo/timoni.cue
You can see it imports definitions from somewhere (it’s a
subdirectory), and that there’s some reference to “values”
(parameters, basically). The values have a schema – that’s what this
line values: templates.#Config says. They will be supplied by the
consumer of the module, when they create an instance or use it in a
bundle.
Here’s another module definition that looks slightly different:
https://github.com/stefanprodan/timoni/blob/main/examples/redis/timoni.cue#L38. In
this one, there’s two apply lines. Timoni applies the first set of
things, waits for them to be ready, then applies the next. Here it’s
used to make sure the Redis cluster will come up correctly.
Here’s the values config for the Redis module: https://github.com/stefanprodan/timoni/blob/main/examples/redis/templates/config.cue#L11
You can see it looks a bit like a values.yaml file in a Helm chart,
and that’s pretty much what it is.
Down the bottom of this file there’s the bit that drives the templating: https://github.com/stefanprodan/timoni/blob/main/examples/redis/templates/config.cue#L72
This spells out the actual objects to include in the configuration, to be applied to the cluster.
Here’s a template which shows how values are used: https://github.com/stefanprodan/timoni/blob/main/examples/redis/templates/replica.service.cue
There’s a pattern here: the “template” includes a field config,
which is assigned by the instance value; fields within config are then
referenced elsewhere in the template. But, in theory, you could arrive
at the objects to apply however you want (including inlining them).
How do you use Timoni?
We just looked at what a module looks like. As a consumer of modules, there are two levels: instances, and bundles (of instances). The instance level is imperative: “please apply this module, at this version, with these values”. The bundles level is declarative: “here is a file with the modules and versions and values to be applied, please make it so.”
A bundle file is also CUE, so you can use definitions and all that inside.
The timoni command-line tool will apply modules and bundles, and
tell you the diff with what’s there, tell you what’s been installed,
and remove configurations.
So, Timoni is a format for writing CUE modules, with a tool for applying those into a Kubernetes cluster conveniently.
What was the OCI thing?
This is a bit of an excursion, but I think it’s worth explaining why OCI recommends itself as the packaging and distribution machinery for Timoni (and for Helm, for that matter).
OCI, if the abbreviation is opaque to you, is the Open Container Initiative, which you can think of as a standards vehicle for Docker images. One of the OCI specifications, about how to package together a bunch of files, is called “image-spec”. The other specifications are about how to run images – “runtime-spec”, and how to make them accessible on the internet – “distribution-spec”.
The image-spec defines a manifest format which provides a bit of metadata, including a type, and a list of layer digests. Each layer is a tarball. The distribution-spec gives protocols for publishing and accessing images.
I mention these because the image-spec and distribution-spec together have some nice properties, independent of the runtime spec:
- layers are content addressed – you can get a layer from anywhere and so long as it validates it’s guaranteed to be the right one.
- an image has a layered filesystem – this is part of image spec rather than artifact (but you can use it for your own artifacts!)
On that latter one: you can use layers to mix and match chunks of filesystem, which is very handy if you have a large bit of filesystem that doesn’t change much, and a handful of files that do. Timoni doesn’t use this yet! But it will do, which will be nice, as modules will surely share definitions like the Kubernetes API types.
The content-addressability and layering, and being unencumbered by other considerations, make tools for handling and hosting OCI images and artifacts simple to implement and operate. And, in the world of hyperscalers, there’s pretty much always an implementation of the hosting bit around for you to use, like ECR on AWS, which will take care of things like scaling and access control so you don’t have to.
Using only OCI for distribution also means that verification is simple: the Timoni tooling can use Cosign to sign modules on push, and verify on pull. (*this is still in the works!)
So, Timoni gives you a format for writing CUE so it’s easy to consume as a unit, but it also hooks into tooling for conveniently publishing and consuming those units.
Why not just CUE?
CUE’s job is done when it’s produced a configuration – Timoni takes that further, and gives you help with applying it to the cluster, seeing what’s changed, and pruning objects that are no longer in the configuration.
That said, there is at least one CUE operator, that takes CUE programs and runs them, and integrates with Flux. Why not just use that?
One argument is from accessibility, which is why I keep saying “conveniently”. Having conventions like Timoni modules and bundles makes it much easier to be a consumer without making it much harder to be an author.
What are the limitations of Timoni?
I’m not sure it’s easy to compose your own specialisations with stock configurations, which ought to be a strength of using CUE. For example, if you want to use some vendor’s WordPress module, but put your own resource limits in all the pods – I don’t think you can do that. You’re relying on the author to make that part of the values.
Without composition of that kind, Timoni falls into the same trap as Helm’s “values”. You only get control over what’s exposed as values, and in the limit, authors are under pressure to expose all the complexity of the underlying Kubernetes API – but instead of being the Kubernetes API, it’s an informal, accreted, semi-conventional (the worst kind of conventional) API. Here’s an example: https://github.com/stefanprodan/timoni/tree/main/examples/podinfo#general-values.
But: at least it is typed! That means you can validate it statically, generate documentation from it, and even mechanically determine whether a new values schema is backward-compatible with the old one.
And: you don’t have to write a module like that, so long as you follow the right format.
And: it’s version 0.8! I’m sure it will evolve, once it’s got a toehold in the ecosystem.