How To

The Workflows team recently announced the general availability of iteration syntax and connectors!

Iteration syntax supports easier creation and better readability of workflows that process many items. You can use a for loop to iterate through a collection of data in a list or map, and keep track of the current index. If you have a specific range of numeric values to iterate through, you can also use range-based iteration.

Eventarc now supports a new Cloud Storage trigger to receive events from Cloud Storage buckets!

Wait a minute. Didn’t Eventarc already support receiving Cloud Storage events? You’re absolutely right! Eventarc has long supported Cloud Storage events via the Cloud Audit Logs trigger. However, the new Cloud Storage trigger has a number of advantages and it’s now the preferred way of receiving Cloud Storage events. Let’s take a look at the details.

Events supported by Eventarc

Last week, I put together a list of events supported by Eventarc in our eventarc-samples repo. Thanks to our docs team, this list is now part of our official docs under reference section.

After looking at the full list, I started thinking about some use cases enabled by these events. I want to talk about one of those use cases today: How to get notified when an expensive BigQuery job executes?

I’m a big fan of using Workflows to orchestrate and automate services running on Google Cloud and beyond. In Workflows, you can define a workflow in a YAML or JSON file and deploy it using gcloud or using Google Cloud Console. These approaches work but a more declarative and arguably better approach is to use Terraform.

Let’s see how to use Terraform to define and deploy workflows and explore options for keeping Terraform configuration files more manageable.

Workflows is a service to orchestrate not only Google Cloud services, such as Cloud Functions, Cloud Run, or machine learning APIs, but also external services. As you might expect from an orchestrator, Workflows allows you to define the flow of your business logic, as steps, in a YAML or JSON definition language, and provides an execution API and UI to trigger workflow executions. You can read more about the benefits of Workflows in our previous article.

What is orchestration? Orchestration often refers to the automated configuration, coordination, and management of computer systems and services.

In the context of service-oriented architectures, orchestration can range from simply executing a single service at a specific time and day, to a more sophisticated approach of automating and monitoring multiple services over longer periods of time, with the ability to react and handle failures as they crop up.

In the data engineering context, orchestration is central to coordinating the services and workflows that prepare, ingest, and transform data. It can go beyond data processing and also involve a workflow to train a machine learning (ML) model from the data.

Cloud Run and Eventarc is a great combination for building event-driven services with different event routing options. There are two trigger types (Audit Logs and Pub/Sub) to choose from in Eventarc. Eventarc uses Pub/Sub as its underlying transport layer and provides convenience and standardization on top of it. If you wanted to, you could skip Eventarc and read messages directly from Pub/Sub in Cloud Run.

This blog post details three ways of receiving events in Cloud Run and provides a decision framework on how to choose.

Eventarc enables you to read events from Google Cloud sources (via its Audit Logs integration) and custom sources (via its Pub/Sub integration) and then route them to Cloud Run services.

https://storage.googleapis.com/gweb-cloudblog-publish/images/eventarc-architecture.max-2000x2000.png The event routing rules are defined with a trigger. In a trigger, you specify the right event filters such as service name, method name, resource (which effectively defines the event source) and the target of the events (which can only be a Cloud Run service as of today).

Over the past year, we (Mete and Guillaume) have developed a picture sharing application, named Pic-a-Daily, to showcase Google Cloud serverless technologies such as Cloud Functions, App Engine, and Cloud Run. Into the mix, we’ve thrown a pinch of Pub/Sub for interservice communication, a zest of Firestore for storing picture metadata, and a touch of machine learning for a little bit of magic.

https://storage.googleapis.com/gweb-cloudblog-publish/images/1_Shqfx7L.max-1400x1400.png We also created a hands-on workshop to build the application, and slides with explanations of the technologies used. The workshop consists of codelabs that you can complete at your own pace. All the code is open source and available in a GitHub repository.

Back in October, we announced the public preview of Eventarc, as new eventing functionality that lets developers route events to Cloud Run services. In a previous post, we outlined more benefits of Eventarc: a unified eventing experience in Google Cloud, centralized event routing, consistency with eventing format, libraries and an ambitious long term vision.

Today, we’re happy to announce that Eventarc is now generally available. Developers can focus on writing code to handle events, while Eventarc takes care of the details of event ingestion, delivery, security, observability, and error handling.