Google Kubernetes Engine

Motivation

When I started working from home in mid-March, I was totally obsessed with COVID-19 news. I was constantly checking number of cases and news from the UK (where I currently live) and from Cyprus (where I’m originally from). It took me a couple of weeks to realize how unproductive this was. I started limiting myself to check for news once a day. This definitely helped me to regain sanity and productivity but it was manual.

In this post, I want to talk about an event-driven image processing pipeline that I built recently using Knative Eventing. Along the way, I’ll tell you about event sources, custom events and other components provided by Knative that simply development of event-driven architectures.

Requirements

Let’s first talk about the basic requirements I had for the image processing pipeline:

1. Users upload files to an input bucket and get processed images in an output bucket.
2. Uploaded images are filtered (eg. no adult or violent images) before sending through the pipeline.
3. Pipeline can contain any number of processing services that can be added or removed as needed. For the initial pipeline, I decided to go with 3 services: resizer, watermarker, and labeler. The resizer will resize large images. The watermarker will add a watermark to resized images and the labeler will extract information about images (labels) and save it.

Requirement #3 is especially important. I wanted to be able to add services to the pipeline as I need them or create multiple pipelines with different services chained together.

If you ever used Knative on Google Cloud, you must have heard of Knative-GCP project. As the name suggests, Knative-GCP project provides a number of sources such as CloudPubSubSource, CloudStorageSource, CloudSchedulerSource and more to help reading various Google Cloud sources into your Knative cluster.

I recently updated my Knative Tutorial to use the latest Knative Eventing release v0.14.2 and its corresponding Knative-GCP release v0.14.0. I ran into a weird authentication problem that I want to outline here.