Workflows

In Workflows, sometimes, you need to store some state, a key/value pair, in a step in one execution and later read that state in another step in another execution. There’s no intrinsic key/value store in Workflows. However, you can use Firestore as a key/value store and that’s what I want to show you here. If you want to skip to see some samples, check out workflow.yaml. If you want to learn more about it, keep reading. Read More →

We’re happy to announce that you can now create Eventarc triggers to directly target Workflows destinations. Available as a preview feature, it simplifies event-driven orchestrations by enabling you to route Eventarc events to Workflows without having an intermediary service. Integrating Eventarc and Workflows In a previous post, we talked about how to integrate Eventarc and Workflows. Since there was no direct integration between the two services, we had to deploy an intermediary Cloud Run service to receive events from Eventarc and then use the Workflows API to kick off a Workflows execution. Read More ↗︎

Sometimes, you need to run a piece of code for hours, days, or even weeks. Cloud Functions and Cloud Run are my default choices to run code. However, they both have limitations on how long a function or container can run. This rules out the idea of executing long-running code in a serverless way. Thanks to Workflows and Compute Engine, you can have an almost serverless experience with long running code. Read More ↗︎

It’s common to have a separate database for each service in microservices-based architectures. This pattern ensures that the independently designed and deployed microservices remain independent in the data layer as well. But it also introduces a new problem: How do you implement transactions (single units of work, usually made up of multiple operations) that span multiple microservices each with their own local database? In a traditional monolith architecture, you can rely on ACID transactions (atomicity, consistency, isolation, durability) against a single database. Read More ↗︎

If you’re like me, you probably use VS Code to author your Workflows JSON or YAML. You also probably expect some kind syntax validation or auto-completion as you work on your workflow. Unfortunately, there’s no VS Code extension for Workflows and Cloud Code for VS Code does not support Workflows. However, there’s a way to get partial auto-completion for Workflows in VS Code. VS Code and JSON Schema VS Code has the ability to display auto-complete suggestions for JSON and YAML files out of the box. Read More →

The Workflows team recently announced the general availability of iteration syntax and connectors! Iteration syntax supports easier creation and better readability of workflows that process many items. You can use a for loop to iterate through a collection of data in a list or map, and keep track of the current index. If you have a specific range of numeric values to iterate through, you can also use range-based iteration. Read More ↗︎

I’m a big fan of using Workflows to orchestrate and automate services running on Google Cloud and beyond. In Workflows, you can define a workflow in a YAML or JSON file and deploy it using gcloud or using Google Cloud Console. These approaches work but a more declarative and arguably better approach is to use Terraform. Let’s see how to use Terraform to define and deploy workflows and explore options for keeping Terraform configuration files more manageable. Read More ↗︎

Workflows is a service to orchestrate not only Google Cloud services, such as Cloud Functions, Cloud Run, or machine learning APIs, but also external services. As you might expect from an orchestrator, Workflows allows you to define the flow of your business logic, as steps, in a YAML or JSON definition language, and provides an execution API and UI to trigger workflow executions. You can read more about the benefits of Workflows in our previous article. Read More ↗︎

What is orchestration? Orchestration often refers to the automated configuration, coordination, and management of computer systems and services. In the context of service-oriented architectures, orchestration can range from simply executing a single service at a specific time and day, to a more sophisticated approach of automating and monitoring multiple services over longer periods of time, with the ability to react and handle failures as they crop up. In the data engineering context, orchestration is central to coordinating the services and workflows that prepare, ingest, and transform data. Read More ↗︎

Over the past year, we (Mete and Guillaume) have developed a picture sharing application, named Pic-a-Daily, to showcase Google Cloud serverless technologies such as Cloud Functions, App Engine, and Cloud Run. Into the mix, we’ve thrown a pinch of Pub/Sub for interservice communication, a zest of Firestore for storing picture metadata, and a touch of machine learning for a little bit of magic. https://storage.googleapis.com/gweb-cloudblog-publish/images/1_Shqfx7L.max-1400x1400.png We also created a hands-on workshop to build the application, and slides with explanations of the technologies used. Read More ↗︎