Knative

I got around to updating my Knative Tutorial from Knative v0.16.0 to the latest Knative Serving v0.18.0 release and Knative Eventing v0.18.1 release.

In this short blog post, I want to outline a couple of minor issues I encountered during my upgrade. Note that I skipped v0.17 altogether, some of these changes might have happened in that release.

Istio Installation

The biggest change I encountered is how Istio is installed for Knative. In previous releases, I simply pointed to the yaml files in the latest Istio version in third_party folder of Knative Serving.

I finally got around to updating my Knative Tutorial from Knative v0.14.0 to the latest Knative v0.16.0 release. Since I skipped v0.15.0, I’m not sure which changes are due to v0.15.0 vs. v0.16.0. Regardless, there have been some notable changes that I want to outline in this blog post. This is not meant to be an exhaustive list. Feel free to let me know in the comments if there are other notable changes that I should be aware of.

Motivation

When I started working from home in mid-March, I was totally obsessed with COVID-19 news. I was constantly checking number of cases and news from the UK (where I currently live) and from Cyprus (where I’m originally from). It took me a couple of weeks to realize how unproductive this was. I started limiting myself to check for news once a day. This definitely helped me to regain sanity and productivity but it was manual.

In this post, I want to talk about an event-driven image processing pipeline that I built recently using Knative Eventing. Along the way, I’ll tell you about event sources, custom events and other components provided by Knative that simply development of event-driven architectures.

Requirements

Let’s first talk about the basic requirements I had for the image processing pipeline:

1. Users upload files to an input bucket and get processed images in an output bucket.
2. Uploaded images are filtered (eg. no adult or violent images) before sending through the pipeline.
3. Pipeline can contain any number of processing services that can be added or removed as needed. For the initial pipeline, I decided to go with 3 services: resizer, watermarker, and labeler. The resizer will resize large images. The watermarker will add a watermark to resized images and the labeler will extract information about images (labels) and save it.

Requirement #3 is especially important. I wanted to be able to add services to the pipeline as I need them or create multiple pipelines with different services chained together.

If you ever used Knative on Google Cloud, you must have heard of Knative-GCP project. As the name suggests, Knative-GCP project provides a number of sources such as CloudPubSubSource, CloudStorageSource, CloudSchedulerSource and more to help reading various Google Cloud sources into your Knative cluster.

I recently updated my Knative Tutorial to use the latest Knative Eventing release v0.14.2 and its corresponding Knative-GCP release v0.14.0. I ran into a weird authentication problem that I want to outline here.

Knative Eventing docs are a little confusing when it comes to different event delivery methods it supports. It talks about event brokers and triggers and it also talks about sources, services, channels, and subscriptions. What to use and when? It’s not clear. Let’s break it down.

Delivery methods

There are 3 distinct methods in Knative:

1. Simple delivery
2. Complex delivery with optional reply
3. Broker and Trigger delivery

Broker and Trigger delivery is what you should care about most of the time. However, the simple and complex delivery have been in Knative for a while and still good to know for what’s happening under the covers.

It’s hard to keep with Knative releases with a release every 6 weeks. I finally managed to update my Knative Tutorial for the latest Knative v0.12.0. In this blog post, I want to outline some of the differences I’ve observed.

Knative Serving

Knative Serving has been pretty stable in the recent releases and Knative Serving v0.12.0 is no exception. I didn’t need to update my tutorial specifically for this release.

Knative Eventing

Knative Eventing v0.12.0 changed the default yaml for Knative Eventing bundles. Now, they are under eventing.yaml (previously, it was release.yaml) and this is the yaml you need to point to install eventing. This makes sense as it’s more consistent with Knative Serving and its serving.yaml.

The default Knative Installation instructions for Google Kubernete Engine (GKE) is problematic (see bug 2266). In this post, I want to outline what the problem is, tell you what I do, and also provide you the scripts that work for me until a proper solution is implemented either in gcloud or Knative.

The problem

The default Knative Installation instructions tell you to create a GKE cluster as follows:

gcloud beta container clusters create $CLUSTER_NAME \
 --addons=HorizontalPodAutoscaling,HttpLoadBalancing,Istio \
 --machine-type=n1-standard-4 \
 --cluster-version=latest --zone=$CLUSTER_ZONE \
 --enable-stackdriver-kubernetes --enable-ip-alias \
 --enable-autoscaling --min-nodes=1 --max-nodes=10 \
 --enable-autorepair \
 --scopes cloud-platform

Notice the Istio add-on. This command creates a Kubernetes cluster with Istio already installed. This is good because Istio is a dependency of Knative but keep reading.

In my previous post, I talked about Knative v0.9.0 and some of the eventing changes in the latest release. I’ve been playing with Knative v0.9.0 since then to read Google Cloud Pub/Sub messages using PullSubscription and I ran into a rather fundamental issue that baffled me for a while. I’d like to outline the problem and the solution here, just in case it’s useful to others.

Knative Services as eventing sinks

In my PullSubscription, I could define Kubernetes Services as event sinks as follows:

Knative has been evolving pretty quickly. There’s a new release roughly every 6 weeks with significant changes in each release. Knative v0.7.0 was all about changes in Knative Serving (my post). Knative v0.8.0 was about deprecation of Knative Build in favor of Tekton Pipelines (my other post).

Knative Serving v0.9.0 and Eventing v0.9.0 have been released a little over a week ago. In Serving, there’s a v1 API and a number of improvements on autoscaling and cold starts. In Eventing, the way events are read changed quite a bit. I want to outline some of these changes here.