Mete Atamel

Challenges in connecting services

When I think about my recent projects, I probably spent half of my time coding new services and the other half in connecting services. Service A calling Service B, or Service C calling an external service and using the result to feed into another Service D.

Connecting services is one of those things that ‘should be easy’ but in reality, it takes a lot of time and effort. You need to figure out a common connection format for services to use, make the connection, parse the results, and pass the results on. I’m not even mentioning error handling, retries and all those production readiness type features that you ultimately need to do.

My colleague Felipe Hoffa recently published a blog post titled Get started with BigQuery and dbt, the easy way. More specifically, he showed how to install dbt in Google Cloud Shell, configure it and manually run it to create a temporary dataset in BigQuery. This is great for testing dbt + BigQuery but how do you run this kind of setup in production?

dbt documentation states that Running dbt in production simply means setting up a system to run a dbt job on a schedule, rather than running dbt commands manually from the command line.

I finally got around to updating my Knative Tutorial from Knative v0.14.0 to the latest Knative v0.16.0 release. Since I skipped v0.15.0, I’m not sure which changes are due to v0.15.0 vs. v0.16.0. Regardless, there have been some notable changes that I want to outline in this blog post. This is not meant to be an exhaustive list. Feel free to let me know in the comments if there are other notable changes that I should be aware of.

.NET for Google Cloud Functions (Alpha)

I spoke at many .NET conferences over the last 3-4 years and one of the top requests I always received was: When will .NET be supported on Cloud Functions?

Unfortunately, I didn’t have a good answer for a while. That all changed last month with the following tweet from Jon Skeet from our C# team:

I'm thrilled that .NET support is coming to Google Cloud Functions, along with the .NET Functions Framework. Sign up for the public alpha at https://t.co/nASIACFCrg

Read More →

.NET Core 3.1 updates on Google Cloud

.NET Core 3.1 was released on December 3rd, 2019 and is a LTS release, supported for three years.

In Google Cloud, you could already deploy .NET Core 3.1 containers in Cloud Run (see cloud-run-dotnetcore-31) and also in App Engine flexible environment with a custom runtime.

We recently extended .NET Core 3.1 support in a couple of ways:

1. Cloud Shell now supports .NET Core 3.1.
2. App Engine flexible environment runtime now supports .NET Core 3.1.

.NET Core 3.1 in Cloud Shell

Inside Cloud Shell, you can see the latest 3.1.301 version:

Motivation

When I started working from home in mid-March, I was totally obsessed with COVID-19 news. I was constantly checking number of cases and news from the UK (where I currently live) and from Cyprus (where I’m originally from). It took me a couple of weeks to realize how unproductive this was. I started limiting myself to check for news once a day. This definitely helped me to regain sanity and productivity but it was manual.

In this post, I want to talk about an event-driven image processing pipeline that I built recently using Knative Eventing. Along the way, I’ll tell you about event sources, custom events and other components provided by Knative that simply development of event-driven architectures.

Requirements

Let’s first talk about the basic requirements I had for the image processing pipeline:

1. Users upload files to an input bucket and get processed images in an output bucket.
2. Uploaded images are filtered (eg. no adult or violent images) before sending through the pipeline.
3. Pipeline can contain any number of processing services that can be added or removed as needed. For the initial pipeline, I decided to go with 3 services: resizer, watermarker, and labeler. The resizer will resize large images. The watermarker will add a watermark to resized images and the labeler will extract information about images (labels) and save it.

Requirement #3 is especially important. I wanted to be able to add services to the pipeline as I need them or create multiple pipelines with different services chained together.

If you ever used Knative on Google Cloud, you must have heard of Knative-GCP project. As the name suggests, Knative-GCP project provides a number of sources such as CloudPubSubSource, CloudStorageSource, CloudSchedulerSource and more to help reading various Google Cloud sources into your Knative cluster.

I recently updated my Knative Tutorial to use the latest Knative Eventing release v0.14.2 and its corresponding Knative-GCP release v0.14.0. I ran into a weird authentication problem that I want to outline here.

Knative Eventing docs are a little confusing when it comes to different event delivery methods it supports. It talks about event brokers and triggers and it also talks about sources, services, channels, and subscriptions. What to use and when? It’s not clear. Let’s break it down.

Delivery methods

There are 3 distinct methods in Knative:

1. Simple delivery
2. Complex delivery with optional reply
3. Broker and Trigger delivery

Broker and Trigger delivery is what you should care about most of the time. However, the simple and complex delivery have been in Knative for a while and still good to know for what’s happening under the covers.

In part 3 of the blog series, I talked about how we transformed our Windows-only .NET Framework app to a containerized multi-platform .NET Core app.

This removed our dependency on Windows and enabled us to deploy to Linux-based platforms such as App Engine (Flex). On the other hand, the app still ran on VMs, it was billed per second even if nobody used it, deployments were slow and most importantly, it was a single monolith that was deployed and scaled as a single unit.