Workflows

Everyone is excited about generative AI (gen AI) nowadays and rightfully so. You might be generating text with PaLM 2 or Gemini Pro, generating images with ImageGen 2, translating code from language to another with Codey, or describing images and videos with Gemini Pro Vision. No matter how you’re using gen AI, at the end of the day, you’re calling an endpoint either with an SDK or a library or via a REST API. Read More ↗︎

As you orchestrate more services with Workflows, the workflow gets more complicated with more steps, jumps, iterations, parallel branches. When the workflow execution inevitably fails at some point, you need to debug and figure out which step failed and why. So far, you only had an execution summary with inputs/outputs and logs to rely on in your execution debugging. While this was good enough for basic workflows, it didn’t provide step level debugging information. Read More ↗︎

Workflows is a versatile service in orchestrating and automating a wide range of use cases: microservices, business processes, Data and ML pipelines, IT operations, and more. It can also be used to automate deployment of containerized applications on Kubernetes Engine (GKE) and this got even easier with the newly released (in preview) Kubernetes API Connector. The new Kubernetes API connector enables access to GKE services from Workflows and this in turn enables Kubernetes based resource management or orchestration, scheduled Kubernetes jobs, and more. Read More ↗︎

Introduction Eventarc helps users build event-driven architectures without having to implement, customize, or maintain the underlying infrastructure. Eventarc has added support (in public preview) for delivering events to internal HTTP endpoints in a Virtual Private Cloud (VPC) network. Customers, especially large enterprises, often run compute (typically GKE or GCE) on VPC-private IPs, often behind internal load balancers. This launch will enable these services to consume Eventarc events. Internal HTTP endpoints can be an internal IP address or fully qualified DNS name (FQDN) for any HTTP endpoint in the VPC network. Read More →

Introduction In my previous post, I talked about how you can use a parent workflow to execute child workflows in parallel for faster overall processing time and easier detection of errors. Another useful pattern is to use a Cloud Tasks queue to create Workflows executions and that’s the topic of this post. When your application experiences a sudden surge of traffic, it’s natural to want to handle the increased load by creating a high number of concurrent workflow executions. Read More →

Introduction There are numerous scenarios where you might want to execute tasks in parallel. One common use case involves dividing data into batches, processing each batch in parallel, and combining the results in the end. This approach not only enhances the speed of the overall processing but it also allows for easier error detection in smaller tasks. On the other hand, setting up parallel tasks, monitoring them, handling errors in each task, and combining the results in the end is not trivial. Read More →

Workflows just released a new connector for Batch that greatly simplifies how to create and run Batch jobs from Workflows. Let’s take a look how you can use the new Batch connector of Workflows. Recap: Batch and Workflows Batch is a fully managed service to schedule, queue, and execute batch jobs on Google’s infrastructure. These batch jobs run on Compute Engine VM instances but they are managed by Batch service, so you don’t have to provision and manage VM instances yourself. Read More →

If you use Workflows, you’ve been crafting your Workflows definitions in YAML (or JSON). You’re probably painfully aware of the limited support you get in your IDE with syntax validation or auto-completion with these YAML definitions. This was due to Workflow’s schema being out of date, as I covered in my previous post last year. I’m happy to report that our team recently updated the Workflows schema with the latest syntax. Read More →

This is a three-part series of posts, in which we summarize Workflows and service orchestration patterns. In this third and final post, we talk about managing workflow life cycles and the benefits of using Firestore with Workflows. If you’re not careful, the workflow definitions you create with YAML or JSON can get out of hand pretty quickly. While it is possible to use subworkflows to define snippets of a workflow that can be reused from multiple workflows, Workflows does not support importing these subworkflows. Read More ↗︎

This is part 2 of a three-part series of posts, in which we summarize Workflows and service orchestration patterns. You can apply these patterns to better take advantage of Workflows and service orchestration on Google Cloud. In the first post, we introduced some general tips and tricks, as well as patterns for event-driven orchestrations, parallel steps, and connectors. This second post covers more advanced patterns. Let’s dive in! Design for resiliency with retries and the saga pattern It’s easy to put together a workflow that chains a series of services, especially if you assume that those services will never fail. Read More ↗︎