How To on Atamel.Dev

AsyncAPI gets a new version 3.0 and new operations

atamel@gmail.com (Mete Atamel) — Mon, 13 May 2024 00:00:00 +0000

Almost one year ago, I talked about AsyncAPI 2.6 and how confusing its publish and subscribe operations can be in my Understanding AsyncAPI’s publish & subscribe semantics with an example post.

Since then, a new 3.0 version of AsyncAPI has been released with breaking changes and a totally new send and receive operations.

In this blog post, I want to revisit the example from last year and show how to rewrite it for AsyncAPI 3.0 with the new send and receive operations.

Making API calls exactly once when using Workflows

atamel@gmail.com (Mete Atamel) — Fri, 03 May 2024 00:00:00 +0000

One challenge with any distributed system, including Workflows, is ensuring that requests sent from one service to another are processed exactly once, when needed; for example, when placing a customer order in a shipping queue, withdrawing funds from a bank account, or processing a payment.

In this blog post, we’ll provide an example of a website invoking Workflows, and Workflows in turn invoking a Cloud Function. We’ll show how to make sure both Workflows and the Cloud Function logic only runs once. We’ll also talk about how to invoke Workflows exactly once when using HTTP callbacks, Pub/Sub messages, or Cloud Tasks.

Orchestrate Vertex AI’s PaLM and Gemini APIs with Workflows

atamel@gmail.com (Mete Atamel) — Thu, 22 Feb 2024 00:00:00 +0000

Everyone is excited about generative AI (gen AI) nowadays and rightfully so. You might be generating text with PaLM 2 or Gemini Pro, generating images with ImageGen 2, translating code from language to another with Codey, or describing images and videos with Gemini Pro Vision.

No matter how you’re using gen AI, at the end of the day, you’re calling an endpoint either with an SDK or a library or via a REST API. Workflows, my go-to service to orchestrate and automate other services, is more relevant than ever when it comes to gen AI.

Announcing Workflows execution steps history

atamel@gmail.com (Mete Atamel) — Sat, 20 Jan 2024 00:00:00 +0000

As you orchestrate more services with Workflows, the workflow gets more complicated with more steps, jumps, iterations, parallel branches. When the workflow execution inevitably fails at some point, you need to debug and figure out which step failed and why. So far, you only had an execution summary with inputs/outputs and logs to rely on in your execution debugging. While this was good enough for basic workflows, it didn’t provide step level debugging information.

Deploy and manage Kubernetes applications with Workflows

atamel@gmail.com (Mete Atamel) — Fri, 01 Dec 2023 00:00:00 +0000

Workflows is a versatile service in orchestrating and automating a wide range of use cases: microservices, business processes, Data and ML pipelines, IT operations, and more. It can also be used to automate deployment of containerized applications on Kubernetes Engine (GKE) and this got even easier with the newly released (in preview) Kubernetes API Connector.

The new Kubernetes API connector enables access to GKE services from Workflows and this in turn enables Kubernetes based resource management or orchestration, scheduled Kubernetes jobs, and more.

Introducing a new Eventarc destination - internal HTTP endpoint in a VPC network

atamel@gmail.com (Mete Atamel) — Mon, 13 Nov 2023 00:00:00 +0000

Introduction

Eventarc helps users build event-driven architectures without having to implement, customize, or maintain the underlying infrastructure.

Eventarc has added support (in public preview) for delivering events to internal HTTP endpoints in a Virtual Private Cloud (VPC) network. Customers, especially large enterprises, often run compute (typically GKE or GCE) on VPC-private IPs, often behind internal load balancers. This launch will enable these services to consume Eventarc events.

Internal HTTP endpoints can be an internal IP address or fully qualified DNS name (FQDN) for any HTTP endpoint in the VPC network. Examples of destinations that can be targeted via the internal HTTP endpoints include Compute Engine VMs with internal IPs, services fronted by an L7 Internal Load Balancer, Private Service Connect endpoints, Google Kubernetes Engine Ingress, Google Kubernetes Engine Services, Cloud Run behind an internal Application Load Balancer and any destinations registered with Cloud DNS via DNS record.

What languages are supported in WebAssembly outside the browser?

atamel@gmail.com (Mete Atamel) — Thu, 07 Sep 2023 00:00:00 +0000

What languages are supported in WebAssembly running outside the browser? This is a question I often hear people ask. It’s has a complicated answer because:

WebAssembly outside the browser needs WASI and not all languages have WASI support in their toolchain.
Even if WASI is supported well in a language, WASI has its own limitations that you need to take into account.

In short, you can’t take any code written in any language and expect to compile and run it as a Wasm+Wasi module right now. Documentation on what’s supported is patchy or misleading at times. Unfortunately, you often need to try things out before knowing what works and what doesn’t.

Adding HTTP around Wasm with Wagi

atamel@gmail.com (Mete Atamel) — Tue, 18 Jul 2023 00:00:00 +0000

In my previous posts, I talked about how you can run WebAssembly (Wasm) outside the browser with Wasi and run it in a Docker container with runwasi. The Wasi specification allows Wasm modules access to things like the filesystem and environment variables (and I showed how in this blog post) but networking and threading are not implemented yet. This is severely limiting if you want to run HTTP based microservices on Wasm for example.

Buffer workflow executions with a Cloud Tasks queue

atamel@gmail.com (Mete Atamel) — Wed, 12 Jul 2023 00:00:00 +0000

Introduction

In my previous post, I talked about how you can use a parent workflow to execute child workflows in parallel for faster overall processing time and easier detection of errors. Another useful pattern is to use a Cloud Tasks queue to create Workflows executions and that’s the topic of this post.

When your application experiences a sudden surge of traffic, it’s natural to want to handle the increased load by creating a high number of concurrent workflow executions. However, Google Cloud’s Workflows enforces quotas to prevent abuse and ensure fair resource allocation. These quotas limit the maximum number of concurrent workflow executions per region, per project, for example, Workflows currently enforces a maximum of 2000 concurrent executions by default. Once this limit is reached, any new executions beyond the quota will fail with an HTTP 429 error.

Workflows executing other parallel workflows: A practical guide

atamel@gmail.com (Mete Atamel) — Sat, 08 Jul 2023 00:00:00 +0000

Introduction

There are numerous scenarios where you might want to execute tasks in parallel. One common use case involves dividing data into batches, processing each batch in parallel, and combining the results in the end. This approach not only enhances the speed of the overall processing but it also allows for easier error detection in smaller tasks.

On the other hand, setting up parallel tasks, monitoring them, handling errors in each task, and combining the results in the end is not trivial. Thankfully, Google Cloud’s Workflows can help. In this post, we will explore how you can use a parent workflow to set up and execute parallel child workflows.

Running Wasm in a container

atamel@gmail.com (Mete Atamel) — Thu, 29 Jun 2023 00:00:00 +0000

Docker recently announced experimental support for running Wasm modules (see Announcing Docker+Wasm Technical Preview 2). In this blog post, I explain what this means and how to run a Wasm module in Docker.

Why run Wasm in a container?

In my Exploring WebAssembly outside the browser post, I mentioned how Wasm is faster, smaller, more secure, and more portable than a container. You might be wondering: Why take something faster, smaller, more secure, and more portable and run it in a container?

Compile Rust & Go to a Wasm+Wasi module and run in a Wasm runtime

atamel@gmail.com (Mete Atamel) — Mon, 26 Jun 2023 00:00:00 +0000

In my Exploring WebAssembly outside the browser post, I talked about how WebAssembly System Interface (WASI) enables Wasm modules to run outside the browser and interact with the host in a limited set of use cases that Wasi supports (see Wasi proposals).

WASI

In this blog post, let’s look into details of how to compile code to a Wasm+Wasi module and then run it in a Wasm runtime. Notice that I use Wasm+Wasi module deliberately (instead of just Wasm) because some languages have Wasm support and can run perfectly fine in the browser but they have no or limited Wasi support to run outside the browser.

New Batch connector for Workflows

atamel@gmail.com (Mete Atamel) — Tue, 30 May 2023 00:00:00 +0000

Batch and Workflows

Workflows just released a new connector for Batch that greatly simplifies how to create and run Batch jobs from Workflows. Let’s take a look how you can use the new Batch connector of Workflows.

Recap: Batch and Workflows

Batch is a fully managed service to schedule, queue, and execute batch jobs on Google’s infrastructure. These batch jobs run on Compute Engine VM instances but they are managed by Batch service, so you don’t have to provision and manage VM instances yourself.

Google Cloud Pub/Sub + AsyncAPI

atamel@gmail.com (Mete Atamel) — Thu, 25 May 2023 00:00:00 +0000

Cloud Pub/Sub and AsyncAPI

I’ve been covering different aspects of AsyncAPI in my recent blog posts. In this final post of my AsyncAPI blog post series, I want to talk about how to document Google Cloud’s Pub/Sub using AsyncAPI.

AsyncAPI has pretty good support for Google Pub/Sub, thanks to contributions from Jeremy Whitlock, an engineer from Google, and the flexibility baked in AsyncAPI spec. Jeremy also has a nice blog post on this topic that you can read for more details.

CloudEvents + AsyncAPI

atamel@gmail.com (Mete Atamel) — Tue, 23 May 2023 00:00:00 +0000

CloudEvents and AsyncAPI

I’ve been recently talking about CloudEvents and AsyncAPI,two of my favorite open-source specifications for event-driven architectures. In this blog post, I want to talk about how you can use CloudEvents and AsyncAPI together. More specifically, I’ll show you how to document CloudEvents enabled services using AsyncAPI, thanks to the flexibility and openness of both projects.

Recap: CloudEvents and AsyncAPI

Let’s first do a quick recap CloudEvents and AsyncAPI.

Understanding AsyncAPI's publish & subscribe semantics with an example

atamel@gmail.com (Mete Atamel) — Thu, 18 May 2023 00:00:00 +0000

In AsyncAPI, a channel can have a publish and subscribe operation. This can be confusing, depending on which perspective you’re considering (server vs. user) and what you’re comparing against (eg. WebSocket).

In this blog post, I want to go through an example to show you how to construct your AsyncAPI file with the right publish and subscribe semantics. As a bonus, I also show you how to refactor your AsyncAPI files with common configuration.

AsyncAPI Tools

atamel@gmail.com (Mete Atamel) — Tue, 16 May 2023 00:00:00 +0000

In my previous post, I talked about basic of AsyncAPI. In this post, I want to get into more details on some tools around AsyncAPI.

More specially, we’ll install AsyncAPI CLI and Generator, generate a sample AsyncAPI definition, visualize it in AsyncAPI Studio, and generate code from it. You’ll see how useful AsyncAPI can be in documenting and maintaining your event-driven architectures.

Install AsyncAPI tools

First, let’s install some of the AsyncAPI tools.

AsyncAPI Basics

atamel@gmail.com (Mete Atamel) — Fri, 12 May 2023 00:00:00 +0000

AsyncAPI Logo

Recently, I’ve been looking into AsyncAPI, an open-source specification and tools to document and maintain event-driven architectures (EDAs).

In this blog post, I want summarize the basics of AsyncAPI and point to some useful links to learn more. In future blog posts, I’ll get into more details of AsyncAPI.

AsyncAPI: Why? What?

AsyncAPI is an open source initiative with the goal of making event-driven APIs as easy as REST APIs. Fundamentally, it is a specification to define asynchronous APIs, similar to what OpenAPI (aka Swagger) does for REST APIs.

Buffer HTTP requests with Cloud Tasks

atamel@gmail.com (Mete Atamel) — Thu, 04 May 2023 00:00:00 +0000

Cloud Tasks is a fully-managed service that manages the execution, dispatch, and asynchronous delivery of a large number of tasks to App Engine or any arbitrary HTTP endpoint. You can also use a Cloud Tasks queue to buffer requests between services for more robust intra-service communication.

Cloud Tasks introduces two new features, the new queue-level routing configuration and BufferTask API. Together, they enable creating HTTP tasks and adding to a queue without needing the tasks client library.

Workflows gets an updated JSON Schema

atamel@gmail.com (Mete Atamel) — Tue, 11 Apr 2023 00:00:00 +0000

If you use Workflows, you’ve been crafting your Workflows definitions in YAML (or JSON). You’re probably painfully aware of the limited support you get in your IDE with syntax validation or auto-completion with these YAML definitions. This was due to Workflow’s schema being out of date, as I covered in my previous post last year.

I’m happy to report that our team recently updated the Workflows schema with the latest syntax. With a new schema, you get a much improved syntax validation and auto-completion of Workflows definition files in your favorite IDE.

Extending Cloud Code with custom templates

atamel@gmail.com (Mete Atamel) — Sat, 18 Mar 2023 00:00:00 +0000

Cloud Code is a set of IDE plugins for popular IDEs that make it easier to create, deploy and integrate applications with Google Cloud. Cloud Code provides an excellent extension mechanism through custom templates. In this post, I show you how you can create and use your own custom templates to add some features beyond those supported natively in Cloud Code, such as .NET functions, event triggered functions and more.

How to use Google Cloud Serverless tech to iterate quickly in a startup environment

atamel@gmail.com (Mete Atamel) — Thu, 15 Dec 2022 00:00:00 +0000

In a startup, you need to get to the MVP fast, gather feedback from early adopters, and iterate in a quick cycle. Anything that takes time away from developing and iterating on features delays the launch and that’s a serious problem when time-to-market is crucial.

Google Cloud offers products that can help you to build and run your backend services on a fully managed serverless platform, saving time and freeing you from the burden of provisioning and managing infrastructure needed to run those services. Less time on infrastructure means more time implementing and iterating on your services. Additionally, most serverless products have a pay-as-you-go pricing model with little upfront cost at the beginning. Pricing increases in line with the scale of your services and your startup, so you only pay for what you need, when you need it

Introducing Cloud Functions support in Cloud Code

atamel@gmail.com (Mete Atamel) — Mon, 12 Dec 2022 00:00:00 +0000

Cloud Code now supports Cloud Functions

Cloud Code has been providing IDE support for the development cycle of Kubernetes and Cloud Run applications for a while now. I’m happy to report that the Dec 2022 version (1.21.0) of Cloud Code now supports Cloud Functions!

In this first release of Cloud Functions support, you can:

Use the Cloud Functions Explorer to view your project’s Cloud Functions properties and source code.
Download your Cloud Functions to edit your code locally, then configure your local workspace to deploy those changes directly from Cloud Code.
Invoke your HTTP-triggered functions from VS Code.
Use the Cloud Code Logs Viewer to view logs from your Cloud Functions.

Let’s take a closer look.

Workflows patterns and best practices - Part 3

atamel@gmail.com (Mete Atamel) — Mon, 05 Dec 2022 00:00:00 +0000

This is a three-part series of posts, in which we summarize Workflows and service orchestration patterns. In this third and final post, we talk about managing workflow life cycles and the benefits of using Firestore with Workflows.

If you’re not careful, the workflow definitions you create with YAML or JSON can get out of hand pretty quickly. While it is possible to use subworkflows to define snippets of a workflow that can be reused from multiple workflows, Workflows does not support importing these subworkflows. Thankfully, there are other tools, such as Terraform, that can help.

Workflows patterns and best practices - Part 2

atamel@gmail.com (Mete Atamel) — Mon, 28 Nov 2022 00:00:00 +0000

This is part 2 of a three-part series of posts, in which we summarize Workflows and service orchestration patterns. You can apply these patterns to better take advantage of Workflows and service orchestration on Google Cloud.

In the first post, we introduced some general tips and tricks, as well as patterns for event-driven orchestrations, parallel steps, and connectors. This second post covers more advanced patterns.

Let’s dive in!

Design for resiliency with retries and the saga pattern It’s easy to put together a workflow that chains a series of services, especially if you assume that those services will never fail. This is a common distributed systems fallacy, however, because of course a service will fail at some point. The workflow step calling that service will fail, and then the whole workflow will fail. This is not what you want to see in a resilient architecture. Thankfully, Workflows has building blocks to handle both transient and permanent service failures.

Workflows patterns and best practices - Part 1

atamel@gmail.com (Mete Atamel) — Tue, 22 Nov 2022 00:00:00 +0000

For the last couple of years, we’ve been using Workflows, Google Cloud’s service orchestrator, to bring order to our serverless microservices architectures. As we used and gained more experience with Workflows and service orchestration, we shared what he had learned in conference talks, blog posts, samples, and tutorials. Along the way, some common patterns and best practices emerged.

To help you take better advantage of Workflows and service orchestration on Google Cloud, we’ve summarized these proven patterns and best practices in a three-part series of blog posts.

.NET 7 on Cloud Run

atamel@gmail.com (Mete Atamel) — Fri, 11 Nov 2022 00:00:00 +0000

.NET 7 on Cloud Run

.NET 7 was released a few days ago with new features and performance improvements and it’s already supported on Cloud Run on Google Cloud!

In this short blog post, I show you how to deploy a .NET 7 web app to Cloud Run.

Create a .NET 7 web app

First, make sure you’re on .NET 7:

dotnet --version
7.0.100

Create a simple web app:

Executing commands (gcloud, kubectl) from Workflows

atamel@gmail.com (Mete Atamel) — Mon, 17 Oct 2022 00:00:00 +0000

In a previous post, I showed how to manage the lifecycle of a virtual machine using Workflows and the Compute Engine connector. This works well when there’s a connector for the resource you’re trying to manage. When there is no connector, you can try to use the API of the resource from Workflows, if there’s one. Alternatively, you can also use my favorite command line tool to manage the resource: gcloud. Or if you’re managing a Kubernetes cluster, maybe you want to call kubectl instead.

Workflows that pause and wait for human approvals from Google Sheets

atamel@gmail.com (Mete Atamel) — Mon, 10 Oct 2022 00:00:00 +0000

I’ve been writing a series of posts to showcase Google Workspace and Google Cloud Workflows integration.

In my first post, I showed an IT automation use case in which a Google Sheets spreadsheet triggers a workflow to create virtual machines in Google Cloud. In the second post, I showed how to feed a Google Sheets spreadsheet with data from BigQuery using a workflow.

In this third and final post of the series, I show how to design a workflow that pauses and waits for human approvals from Google Sheets.

.NET 6 on Cloud Functions (2nd gen)

atamel@gmail.com (Mete Atamel) — Mon, 03 Oct 2022 00:00:00 +0000

Back in August, we announced the 2nd generation of Cloud Functions with longer request processing times, larger instances, new event sources with Eventarc, and more.

A few weeks ago, .NET 6 support (public preview) was silently added to Cloud Functions.

Let’s see how to deploy some .NET 6 functions to Cloud Functions 2nd gen.

Functions Framework for .NET

Functions Framework for .NET is the easiest way to create .NET functions for consuming HTTP or CloudEvent requests.

Writing to Google Sheets from Workflows

atamel@gmail.com (Mete Atamel) — Mon, 26 Sep 2022 00:00:00 +0000

In my previous post, I showed how to trigger a workflow in Google Cloud from a Google Sheets spreadsheet using Apps Script. In this post, I show how to do the reverse: write to Google Sheets from a workflow in Google Cloud.

Use case

Imagine you have some dataset in BigQuery. Periodically, you want to query and extract a subset of the dataset and save it to a Google Sheets spreadsheet. You can implement such a process with Workflows quite easily.

Multi-environment service orchestrations

atamel@gmail.com (Mete Atamel) — Tue, 20 Sep 2022 00:00:00 +0000

In a previous post, I showed how to use a GitOps approach to manage the deployment lifecycle of a service orchestration. This approach makes it easy to deploy changes to a workflow in a staging environment, run tests against it, and gradually roll out these changes to the production environment.

While GitOps helps to manage the deployment lifecycle, it’s not enough. Sometimes, you need to make changes to the workflow before deploying to different environments. You need to design workflows with multiple environments in mind.

GitOps your service orchestrations

atamel@gmail.com (Mete Atamel) — Fri, 16 Sep 2022 00:00:00 +0000

GitOps takes DevOps best practices used for application development (such as version control and CI/CD) and applies them to infrastructure automation. In GitOps, the Git repository serves as the source of truth and the CD pipeline is responsible for building, testing, and deploying the application code and the underlying infrastructure.

Nowadays, an application is not just code running on infrastructure that you own and operate. It is usually a set of first-party and third-party microservices working together in an event-driven architecture or with a central service orchestrator such as Workflows.

Triggering Workflows from Google Sheets

atamel@gmail.com (Mete Atamel) — Fri, 09 Sep 2022 00:00:00 +0000

Is it possible to integrate Google Workspace tools such as Calendar, Sheets, and Forms with Workflows? For example, can you trigger a workflow from a Google Form or a Sheet? Turns out, this is not only possible but also easier than you might think. Let me show you how with a sample use case.

Use case

Imagine you are an administrator in charge of allocating virtual machines (VM) in your cloud infrastructure to users. You want to capture user requests with the specifications for the VMs, have an approval step for the request, and then create the VM with an automated process.

Route Datadog monitoring alerts to Google Cloud with Eventarc

atamel@gmail.com (Mete Atamel) — Wed, 24 Aug 2022 00:00:00 +0000

A few weeks ago, we announced third-party event sources for Eventarc, with the first cohort of third-party providers by our ecosystem partners. This blog post describes how to listen for events from one of those third-party providers, Datadog, and route them to a service in Google Cloud via Eventarc.

Datadog is a monitoring platform for cloud applications. It brings together end-to-end traces, metrics, and logs to make your applications and infrastructure observable. In Datadog, you can create monitors that track metrics, events, logs, integration availability, and network endpoints, and you can set alert thresholds on those monitors. Once these thresholds are reached, Datadog notifies your teams or services via email, Slack, and now a Google Cloud service via Eventarc.

Creating Workflows that pause and wait for events

atamel@gmail.com (Mete Atamel) — Fri, 05 Aug 2022 00:00:00 +0000

In Workflows, it’s easy to chain various services together into an automated workflow. For some use cases, you might need to pause workflow execution and wait for some input. This input could be a human approval or an external service calling back with data needed to complete the workflow.

With Workflows callbacks, a workflow can create an HTTP endpoint and pause execution until it receives an HTTP callback to that endpoint. This is very useful for creating human-in-the-middle type workflows. In a previous blog post, Guillaume Laforge showed how to build an automated translation workflow with human validation using callbacks.

Taking screenshots of web pages with Cloud Run jobs, Workflows, and Eventarc

atamel@gmail.com (Mete Atamel) — Thu, 16 Jun 2022 00:00:00 +0000

At Google Cloud I/O, we announced the public preview of Cloud Run jobs. Unlike Cloud Run services that run continuously to respond to web requests or events, Cloud Run jobs run code that performs some work and quits when the work is done. Cloud Run jobs are a good fit for administrative tasks such as database migration, scheduled work like nightly reports, or doing batch data transformation.

In this post, I show you a fully serverless, event-driven application to take screenshots of web pages, powered by Cloud Run jobs, Workflows, and Eventarc.

Worklows state management with Firestore

atamel@gmail.com (Mete Atamel) — Fri, 08 Apr 2022 00:00:00 +0000

Workflows Firestore

In Workflows, sometimes, you need to store some state, a key/value pair, in a step in one execution and later read that state in another step in another execution. There’s no intrinsic key/value store in Workflows. However, you can use Firestore as a key/value store and that’s what I want to show you here.

If you want to skip to see some samples, check out workflow.yaml.

Introducing Eventarc triggers for Workflows

atamel@gmail.com (Mete Atamel) — Fri, 01 Apr 2022 00:00:00 +0000

We’re happy to announce that you can now create Eventarc triggers to directly target Workflows destinations. Available as a preview feature, it simplifies event-driven orchestrations by enabling you to route Eventarc events to Workflows without having an intermediary service.

Integrating Eventarc and Workflows In a previous post, we talked about how to integrate Eventarc and Workflows. Since there was no direct integration between the two services, we had to deploy an intermediary Cloud Run service to receive events from Eventarc and then use the Workflows API to kick off a Workflows execution. Let’s take a look at a concrete example on how this worked.

Creating Eventarc triggers with Terraform

atamel@gmail.com (Mete Atamel) — Fri, 18 Mar 2022 00:00:00 +0000

Terraform is increasingly the preferred tool for building, changing, and versioning infrastructure in Google Cloud and across clouds. In an earlier post, I showed how to create Eventarc triggers using Google Cloud Console or via the command line with gcloud. In this post, I show how to create the same triggers with the google_eventarc_trigger Terraform resource.

See eventarc-samples/terraform on GitHub for the prerequisites and main.tf for full Terraform configuration.

Define a Cloud Run service as an event sink Before you can create a trigger, you need to create a Cloud Run service as an event sink for the trigger. You can use Terraform’s google_cloud_run_service resource to define a Cloud Run service:

Applying a path pattern when filtering in Eventarc

atamel@gmail.com (Mete Atamel) — Wed, 02 Mar 2022 00:00:00 +0000

Eventarc

You can now apply a path pattern when filtering in Eventarc. This is especially useful when you need to filter on resource names beyond exact match. Path pattern syntax allows you to define a regex-like expression that matches events as broadly as you like.

Let’s take a look at a concrete example.

Without path patterns

Let’s say you want to listen for new file creations in a Cloud Storage bucket with an AuditLog trigger.

Building APIs with Cloud Functions and API Gateway

atamel@gmail.com (Mete Atamel) — Thu, 10 Feb 2022 00:00:00 +0000

Building APIs with Cloud Run

If I want to build an API, I usually use Cloud Run. In Cloud Run, you run a container and in that container, you run a web server that handles a base URL in this format:

https://<service-name>-<hash>-<region>.a.run.app

You can then have the web server handle any path under that base URL such as:

https://<service-name>-<hash>-<region>.a.run.app/hello
https://<service-name>-<hash>-<region>.a.run.app/bye

Building APIs with Cloud Functions

In Cloud Functions, you only have access to a function (no web server) and that function can only handle the base path:

Long-running containers with Workflows and Compute Engine

atamel@gmail.com (Mete Atamel) — Wed, 09 Feb 2022 00:00:00 +0000

Sometimes, you need to run a piece of code for hours, days, or even weeks. Cloud Functions and Cloud Run are my default choices to run code. However, they both have limitations on how long a function or container can run. This rules out the idea of executing long-running code in a serverless way.

Thanks to Workflows and Compute Engine, you can have an almost serverless experience with long running code.

Implementing the saga pattern in Workflows

atamel@gmail.com (Mete Atamel) — Sat, 05 Feb 2022 00:00:00 +0000

It’s common to have a separate database for each service in microservices-based architectures. This pattern ensures that the independently designed and deployed microservices remain independent in the data layer as well. But it also introduces a new problem: How do you implement transactions (single units of work, usually made up of multiple operations) that span multiple microservices each with their own local database?

In a traditional monolith architecture, you can rely on ACID transactions (atomicity, consistency, isolation, durability) against a single database. In a microservices architecture, ensuring data consistency across multiple service-specific databases becomes more challenging. You cannot simply rely on local transactions. You need a cross-service transaction strategy. That’s where the saga pattern comes into play.

.NET 6 support in Google Cloud Buildpacks and Cloud Run

atamel@gmail.com (Mete Atamel) — Fri, 04 Feb 2022 00:00:00 +0000

I really like the idea of containers, a reproducible context that our apps can rely on. However, I really dislike having to create a Dockerfile to define my container images. I always need to copy & paste it from somewhere and it takes me some time to get it right.

That’s why I like using Google Cloud Buildpacks to build containers without having to worry about a Dockerfile.

I’m happy to announce that, as of yesterday, Google Cloud Buildpacks now supports .NET 6 (the latest LTS version) apps as well!

Auto-completion for Workflows JSON and YAML on Visual Studio Code

atamel@gmail.com (Mete Atamel) — Fri, 28 Jan 2022 00:00:00 +0000

If you’re like me, you probably use VS Code to author your Workflows JSON or YAML. You also probably expect some kind syntax validation or auto-completion as you work on your workflow. Unfortunately, there’s no VS Code extension for Workflows and Cloud Code for VS Code does not support Workflows.

However, there’s a way to get partial auto-completion for Workflows in VS Code.

VS Code and JSON Schema

VS Code has the ability to display auto-complete suggestions for JSON and YAML files out of the box. It uses JSON Schema Store which hosts JSON Schemas for popular configuration files in JSON and YAML.

Introducing the new Eventarc UI, Cloud Run for Anthos destinations

atamel@gmail.com (Mete Atamel) — Sat, 15 Jan 2022 00:00:00 +0000

December was a busy month for the Eventarc team, who landed a number of features at the end of the year. Let’s take a closer look at some of these new capabilities.

Cloud Storage trigger is GA Back in September, we announced the public preview of Cloud Storage triggers as the preferred way of routing Cloud Storage events to Cloud Run targets. They are now generally available. For more details, see the documentation on how to create a Cloud Storage trigger and check out my previous blog post on Cloud Storage triggers.

Cross-region and cross-project event routing with Eventarc and Pub/Sub

atamel@gmail.com (Mete Atamel) — Thu, 09 Dec 2021 00:00:00 +0000

With event-driven architectures, it’s quite common to read events from a source in one region or project and route them to a destination in another region or another project. Let’s take a look at how you can implement cross-region and cross-project event routing in Google Cloud.

Cross-region event routing is straightforward in Google Cloud, whether you’re using Pub/Sub directly or Eventarc. Pub/Sub routes messages globally. When applications hosted in any region publish messages to a topic, subscribers from any region can pull from that topic. Eventarc enables you to route events across regions by creating a trigger in the region of the event’s source and specifying a destination in a different region. For more details, take a look at my previous blog post on Eventarc locations.

A closer look at locations in Eventarc

atamel@gmail.com (Mete Atamel) — Tue, 26 Oct 2021 00:00:00 +0000

New locations in Eventarc Back in August, we announced more Eventarc locations (17 new regions, as well as 6 new dual-region and multi-region locations to be precise). This takes the total number of locations in Eventarc to more than 30. You can see the full list in the Eventarc locations page or by running gcloud eventarc locations list .

What does location mean in Eventarc? An Eventarc location usually refers to the single region that the Eventarc trigger gets created in. However, depending on the trigger type, the location can be more than a single region:

Trying out source-based deployment in Cloud Run

atamel@gmail.com (Mete Atamel) — Tue, 19 Oct 2021 00:00:00 +0000

Until recently, this is how you deployed code to Cloud Run:

Define your container-based app with a Dockerfile. Build the container image and push it to the Container Registry (typically with Cloud Build). Deploy the container image to Cloud Run. Back in December, we announced the beta release of source-based deployments for Cloud Run. This combines steps 2 and 3 above into a single command. Perhaps more importantly, it also eliminates the need for a Dockerfile for supported language versions.

Analyzing Twitter sentiment with new Workflows processing capabilities

atamel@gmail.com (Mete Atamel) — Sat, 09 Oct 2021 00:00:00 +0000

The Workflows team recently announced the general availability of iteration syntax and connectors!

Iteration syntax supports easier creation and better readability of workflows that process many items. You can use a for loop to iterate through a collection of data in a list or map, and keep track of the current index. If you have a specific range of numeric values to iterate through, you can also use range-based iteration.

Introducing the new Cloud Storage trigger in Eventarc

atamel@gmail.com (Mete Atamel) — Tue, 21 Sep 2021 00:00:00 +0000

Eventarc now supports a new Cloud Storage trigger to receive events from Cloud Storage buckets!

Wait a minute. Didn’t Eventarc already support receiving Cloud Storage events? You’re absolutely right! Eventarc has long supported Cloud Storage events via the Cloud Audit Logs trigger. However, the new Cloud Storage trigger has a number of advantages and it’s now the preferred way of receiving Cloud Storage events. Let’s take a look at the details.

Get notified when an expensive BigQuery job executes using Eventarc and SendGrid

atamel@gmail.com (Mete Atamel) — Tue, 08 Jun 2021 00:00:00 +0000

Events supported by Eventarc

Last week, I put together a list of events supported by Eventarc in our eventarc-samples repo. Thanks to our docs team, this list is now part of our official docs under reference section.

After looking at the full list, I started thinking about some use cases enabled by these events. I want to talk about one of those use cases today: How to get notified when an expensive BigQuery job executes?

Deploying multi-YAML Workflows definitions with Terraform

atamel@gmail.com (Mete Atamel) — Thu, 13 May 2021 00:00:00 +0000

I’m a big fan of using Workflows to orchestrate and automate services running on Google Cloud and beyond. In Workflows, you can define a workflow in a YAML or JSON file and deploy it using gcloud or using Google Cloud Console. These approaches work but a more declarative and arguably better approach is to use Terraform.

Let’s see how to use Terraform to define and deploy workflows and explore options for keeping Terraform configuration files more manageable.

Introducing new connectors for Workflows

atamel@gmail.com (Mete Atamel) — Tue, 27 Apr 2021 00:00:00 +0000

Workflows is a service to orchestrate not only Google Cloud services, such as Cloud Functions, Cloud Run, or machine learning APIs, but also external services. As you might expect from an orchestrator, Workflows allows you to define the flow of your business logic, as steps, in a YAML or JSON definition language, and provides an execution API and UI to trigger workflow executions. You can read more about the benefits of Workflows in our previous article.

Choosing the right orchestrator in Google Cloud

atamel@gmail.com (Mete Atamel) — Thu, 22 Apr 2021 00:00:00 +0000

What is orchestration? Orchestration often refers to the automated configuration, coordination, and management of computer systems and services.

In the context of service-oriented architectures, orchestration can range from simply executing a single service at a specific time and day, to a more sophisticated approach of automating and monitoring multiple services over longer periods of time, with the ability to react and handle failures as they crop up.

In the data engineering context, orchestration is central to coordinating the services and workflows that prepare, ingest, and transform data. It can go beyond data processing and also involve a workflow to train a machine learning (ML) model from the data.

Three ways of receiving events in Cloud Run

atamel@gmail.com (Mete Atamel) — Fri, 12 Mar 2021 00:00:00 +0000

Cloud Run and Eventarc is a great combination for building event-driven services with different event routing options. There are two trigger types (Audit Logs and Pub/Sub) to choose from in Eventarc. Eventarc uses Pub/Sub as its underlying transport layer and provides convenience and standardization on top of it. If you wanted to, you could skip Eventarc and read messages directly from Pub/Sub in Cloud Run.

This blog post details three ways of receiving events in Cloud Run and provides a decision framework on how to choose.

Demystifying event filters in Eventarc

atamel@gmail.com (Mete Atamel) — Tue, 02 Mar 2021 00:00:00 +0000

Eventarc enables you to read events from Google Cloud sources (via its Audit Logs integration) and custom sources (via its Pub/Sub integration) and then route them to Cloud Run services.

https://storage.googleapis.com/gweb-cloudblog-publish/images/eventarc-architecture.max-2000x2000.png The event routing rules are defined with a trigger. In a trigger, you specify the right event filters such as service name, method name, resource (which effectively defines the event source) and the target of the events (which can only be a Cloud Run service as of today).

Orchestrating the Pic-a-Daily serverless app with Workflows

atamel@gmail.com (Mete Atamel) — Sat, 13 Feb 2021 00:00:00 +0000

Over the past year, we (Mete and Guillaume) have developed a picture sharing application, named Pic-a-Daily, to showcase Google Cloud serverless technologies such as Cloud Functions, App Engine, and Cloud Run. Into the mix, we’ve thrown a pinch of Pub/Sub for interservice communication, a zest of Firestore for storing picture metadata, and a touch of machine learning for a little bit of magic.

https://storage.googleapis.com/gweb-cloudblog-publish/images/1_Shqfx7L.max-1400x1400.png We also created a hands-on workshop to build the application, and slides with explanations of the technologies used. The workshop consists of codelabs that you can complete at your own pace. All the code is open source and available in a GitHub repository.

Eventarc brings eventing to Cloud Run and is now GA

atamel@gmail.com (Mete Atamel) — Fri, 29 Jan 2021 00:00:00 +0000

Back in October, we announced the public preview of Eventarc, as new eventing functionality that lets developers route events to Cloud Run services. In a previous post, we outlined more benefits of Eventarc: a unified eventing experience in Google Cloud, centralized event routing, consistency with eventing format, libraries and an ambitious long term vision.

Today, we’re happy to announce that Eventarc is now generally available. Developers can focus on writing code to handle events, while Eventarc takes care of the details of event ingestion, delivery, security, observability, and error handling.

Eventarc - A unified eventing experience in Google Cloud

atamel@gmail.com (Mete Atamel) — Sat, 16 Jan 2021 00:00:00 +0000

I recently talked about orchestration versus choreography in connecting microservices and introduced Workflows for use cases that can benefit from a central orchestrator. I also mentioned Eventarc and Pub/Sub in the choreography camp for more loosely coupled event-driven architectures.

In this blog post, I talk more about the unified eventing experience by Eventarc.

What is Eventarc? We announced Eventarc back in October as a new eventing functionality that enables you to send events to Cloud Run from more than 60 Google Cloud sources. It works by reading Audit Logs from various sources and sending them to Cloud Run services as events in CloudEvents format. It can also read events from Pub/Sub topics for custom applications.

Better service orchestration with Workflows

atamel@gmail.com (Mete Atamel) — Wed, 02 Dec 2020 00:00:00 +0000

Going from a single monolithic application to a set of small, independent microservices has clear benefits. Microservices enable reusability, make it easier to change and scale apps on demand. At the same time, they introduce new challenges. No longer is there a single monolith with all the business logic neatly contained and services communicating with simple method calls. In the microservices world, communication has to go over the wire with REST or some kind of eventing mechanism and you need to find a way to get independent microservices to work toward a common goal.

Introducing Eventarc in Pic-a-Daily Serverless Workshop

atamel@gmail.com (Mete Atamel) — Mon, 30 Nov 2020 00:00:00 +0000

Pic-a-Daily Serverless Workshop

As you might know, Guillaume Larforge and I have a Pic-a-Daily Serverless Workshop. In this workshop, we build a picture sharing application using Google Cloud serverless technologies such as Cloud Functions, App Engine, Cloud Run and more.

Pic-a-Daily Serverless Workshop

We recently added a new service to the workshop. In this blog post, I want to talk about the new service. I also want to talk about Eventarc and how it helped us to get events to the new service.

.NET 5.0 on Google Cloud

atamel@gmail.com (Mete Atamel) — Fri, 20 Nov 2020 00:00:00 +0000

.NET 5.0 was released just a few days ago with many new features, improvements, C# 9 support, F# 5 support, and more. .NET 5.0 is the first release of the unified .NET vision that was announced last year. Going forward, there will be just one .NET targeting Windows, Linux, macOS, and more.

Google Cloud already has support for different versions of .NET. You can run traditional Windows based .NET apps on Windows Servers in Compute Engine or on Windows Containers in Google Kubernetes Engine (GKE). For modern Linux based containerized .NET apps, there’s more choice with App Engine (Flex), GKE and my favorite Cloud Run. Not to mention, the .NET Core 3.1 support in Cloud Functions is currently in preview for serverless .NET functions.

Knative v0.18.0 update

atamel@gmail.com (Mete Atamel) — Mon, 12 Oct 2020 00:00:00 +0000

I got around to updating my Knative Tutorial from Knative v0.16.0 to the latest Knative Serving v0.18.0 release and Knative Eventing v0.18.1 release.

In this short blog post, I want to outline a couple of minor issues I encountered during my upgrade. Note that I skipped v0.17 altogether, some of these changes might have happened in that release.

Istio Installation

The biggest change I encountered is how Istio is installed for Knative. In previous releases, I simply pointed to the yaml files in the latest Istio version in third_party folder of Knative Serving.

Events for Cloud Run for Anthos >= Knative Eventing on Kubernetes

atamel@gmail.com (Mete Atamel) — Thu, 08 Oct 2020 00:00:00 +0000

Introduction

We recently announced a new feature, Events for Cloud Run for Anthos, to build event driven systems on Google Kubernetes Engine (GKE). In the announcement, we also stated that the solution is based on open-source Knative.

In this blog post, I want to further explain the relationship between this new feature and Knative. I also want to convince you that our solution is an easier way to deploy Knative compliant event consuming services on Google Cloud.

Cloud Run for Anthos brings eventing to your Kubernetes microservices

atamel@gmail.com (Mete Atamel) — Thu, 24 Sep 2020 00:00:00 +0000

Building microservices on Google Kubernetes Engine (GKE) provides you with maximum flexibility to build your applications, while still benefiting from the scale and toolset that Google Cloud has to offer. But with great flexibility comes great responsibility. Orchestrating microservices can be difficult, requiring non-trivial implementation, customization, and maintenance of messaging systems.

Cloud Run for Anthos now includes an events feature that allows you to easily build event-driven systems on Google Cloud. Now in beta, Cloud Run for Anthos’ event feature assumes responsibility for the implementation and management of eventing infrastructure, so you don’t have to.

A first look at serverless orchestration with Workflows

atamel@gmail.com (Mete Atamel) — Tue, 08 Sep 2020 00:00:00 +0000

Challenges in connecting services

When I think about my recent projects, I probably spent half of my time coding new services and the other half in connecting services. Service A calling Service B, or Service C calling an external service and using the result to feed into another Service D.

Connecting services is one of those things that ‘should be easy’ but in reality, it takes a lot of time and effort. You need to figure out a common connection format for services to use, make the connection, parse the results, and pass the results on. I’m not even mentioning error handling, retries and all those production readiness type features that you ultimately need to do.

Scheduled serverless dbt + BigQuery service

atamel@gmail.com (Mete Atamel) — Wed, 29 Jul 2020 00:00:00 +0000

My colleague Felipe Hoffa recently published a blog post titled Get started with BigQuery and dbt, the easy way. More specifically, he showed how to install dbt in Google Cloud Shell, configure it and manually run it to create a temporary dataset in BigQuery. This is great for testing dbt + BigQuery but how do you run this kind of setup in production?

dbt documentation states that Running dbt in production simply means setting up a system to run a dbt job on a schedule, rather than running dbt commands manually from the command line.

Knative v0.16.0 update

atamel@gmail.com (Mete Atamel) — Wed, 22 Jul 2020 00:00:00 +0000

I finally got around to updating my Knative Tutorial from Knative v0.14.0 to the latest Knative v0.16.0 release. Since I skipped v0.15.0, I’m not sure which changes are due to v0.15.0 vs. v0.16.0. Regardless, there have been some notable changes that I want to outline in this blog post. This is not meant to be an exhaustive list. Feel free to let me know in the comments if there are other notable changes that I should be aware of.

Knative v0.12.0 update

atamel@gmail.com (Mete Atamel) — Mon, 27 Jan 2020 15:07:35 +0000

It’s hard to keep with Knative releases with a release every 6 weeks. I finally managed to update my Knative Tutorial for the latest Knative v0.12.0. In this blog post, I want to outline some of the differences I’ve observed.

Knative Serving

Knative Serving has been pretty stable in the recent releases and Knative Serving v0.12.0 is no exception. I didn’t need to update my tutorial specifically for this release.

Knative Eventing

Knative Eventing v0.12.0 changed the default yaml for Knative Eventing bundles. Now, they are under eventing.yaml (previously, it was release.yaml) and this is the yaml you need to point to install eventing. This makes sense as it’s more consistent with Knative Serving and its serving.yaml.

How to properly install Knative on GKE

atamel@gmail.com (Mete Atamel) — Mon, 13 Jan 2020 16:50:06 +0000

The default Knative Installation instructions for Google Kubernete Engine (GKE) is problematic (see bug 2266). In this post, I want to outline what the problem is, tell you what I do, and also provide you the scripts that work for me until a proper solution is implemented either in gcloud or Knative.

The problem

The default Knative Installation instructions tell you to create a GKE cluster as follows:

gcloud beta container clusters create $CLUSTER_NAME \
 --addons=HorizontalPodAutoscaling,HttpLoadBalancing,Istio \
 --machine-type=n1-standard-4 \
 --cluster-version=latest --zone=$CLUSTER_ZONE \
 --enable-stackdriver-kubernetes --enable-ip-alias \
 --enable-autoscaling --min-nodes=1 --max-nodes=10 \
 --enable-autorepair \
 --scopes cloud-platform

Notice the Istio add-on. This command creates a Kubernetes cluster with Istio already installed. This is good because Istio is a dependency of Knative but keep reading.

How to deploy a Windows container on Google Kubernetes Engine

atamel@gmail.com (Mete Atamel) — Sat, 28 Sep 2019 00:00:00 +0000

Many people who run Windows containers want to use a container management platform like Kubernetes for resiliency and scalability. In a previous post, we showed you how to run an IIS site inside a Windows container deployed to Windows Server 2019 running on Compute Engine. That’s a good start, but you can now also run Windows containers on Google Kubernetes Engine (GKE).

Support for Windows containers in Kubernetes was announced earlier in the year with version 1.14, followed by GKE announcement on the same. You can sign up for early access and start testing out Windows containers on GKE.

How to deploy a Windows container on Google Compute Engine

atamel@gmail.com (Mete Atamel) — Sat, 21 Sep 2019 00:00:00 +0000

Last year, we published a blog post and demonstrated how to deploy a Windows container running Windows Server 2016 on Google Compute Engine. Since then, there have been a number of important developments. First, Microsoft announced the availability of Windows Server 2019. Second, Kubernetes 1.14 was released with support for Windows nodes and Windows containers.

Supporting Windows workloads and helping you modernize your apps using containers and Kubernetes is one of our top priorities at Google Cloud. Soon after the Microsoft and Kubernetes announcements, we added support for Windows Server 2019 in Compute Engine and Windows containers in Google Kubernetes Engine (GKE).

Google Home meets .NET containers using Dialogflow

atamel@gmail.com (Mete Atamel) — Fri, 13 Jul 2018 00:00:00 +0000

I use my Google Home all the time to check the weather before leaving home, set up alarms, listen to music, but I never considered writing an app for it. What does it take to write an app for the Google Home assistant? And can we make it smarter by leveraging Google Cloud? Those were the questions that my colleague Chris Bacon, and I were thinking about when we decided to build a demo for a conference talk.

How to run Windows Containers on Compute Engine

atamel@gmail.com (Mete Atamel) — Tue, 03 Apr 2018 00:00:00 +0000

Container virtualization is a rapidly evolving technology that can simplify how you deploy and manage distributed applications. When people discuss containers, they usually mean Linux-based containers. This makes sense, because native Linux kernel features like cgroups introduced the idea of resource isolation, eventually leading to containers as we know them today.

For a long time, you could only containerize Linux processes, but Microsoft introduced support for Windows-based containers in Windows Server 2016 and Windows 10. With this, you can now take an existing Windows application, containerize it using Docker, and run it as an isolated container on Windows. Microsoft supports two flavors of Windows containers: Windows Server and Hyper-V. You can build Windows containers on either the microsoft/windowsservercore and microsoft/nanoserver base images. You can read more about Windows containers in the Microsoft Windows containers documentation.

Scheduling Dataflow pipelines using App Engine Cron Service or Cloud Functions

atamel@gmail.com (Mete Atamel) — Tue, 12 Apr 2016 00:00:00 +0000

Google Cloud Dataflow provides a unified programming model for batch and stream data processing along with a managed service to execute parallel data processing pipelines on Google Cloud Platform.

Once a Dataflow pipeline is created, it can be tested locally using DirectPipelineRunner, and if everything looks good, it can be manually executed as a job in Dataflow Service by triggering DataflowPipelineRunner or BlockingDataflowPipelineRunner with Apache Maven or Dataflow Eclipse Plugin. You can monitor the progress of your submitted job with Dataflow Monitoring Interface from Cloud Platform Console or Dataflow Command-line Interface from gcloud.