Google Cloud Platform

Introduction

Eventarc helps users build event-driven architectures without having to implement, customize, or maintain the underlying infrastructure.

Eventarc has added support (in public preview) for delivering events to internal HTTP endpoints in a Virtual Private Cloud (VPC) network. Customers, especially large enterprises, often run compute (typically GKE or GCE) on VPC-private IPs, often behind internal load balancers. This launch will enable these services to consume Eventarc events.

Internal HTTP endpoints can be an internal IP address or fully qualified DNS name (FQDN) for any HTTP endpoint in the VPC network. Examples of destinations that can be targeted via the internal HTTP endpoints include Compute Engine VMs with internal IPs, services fronted by an L7 Internal Load Balancer, Private Service Connect endpoints, Google Kubernetes Engine Ingress, Google Kubernetes Engine Services, Cloud Run behind an internal Application Load Balancer and any destinations registered with Cloud DNS via DNS record.

Introduction

In my previous post, I talked about how you can use a parent workflow to execute child workflows in parallel for faster overall processing time and easier detection of errors. Another useful pattern is to use a Cloud Tasks queue to create Workflows executions and that’s the topic of this post.

When your application experiences a sudden surge of traffic, it’s natural to want to handle the increased load by creating a high number of concurrent workflow executions. However, Google Cloud’s Workflows enforces quotas to prevent abuse and ensure fair resource allocation. These quotas limit the maximum number of concurrent workflow executions per region, per project, for example, Workflows currently enforces a maximum of 2000 concurrent executions by default. Once this limit is reached, any new executions beyond the quota will fail with an HTTP 429 error.

Introduction

There are numerous scenarios where you might want to execute tasks in parallel. One common use case involves dividing data into batches, processing each batch in parallel, and combining the results in the end. This approach not only enhances the speed of the overall processing but it also allows for easier error detection in smaller tasks.

On the other hand, setting up parallel tasks, monitoring them, handling errors in each task, and combining the results in the end is not trivial. Thankfully, Google Cloud’s Workflows can help. In this post, we will explore how you can use a parent workflow to set up and execute parallel child workflows.

If you’re looking to upskill in Generative AI, there’s a Generative AI Learning Path in Google Cloud Skills Boost. It currently consists of 10 courses and provides a good foundation on the theory behind Generative AI.

As I went through these courses myself, I took notes, as I learn best when I write things down. In part 1 of the blog series, I shared my notes for courses 1 to 6. In this part 2 of the blog series, I continue sharing my notes for courses 7 to 10.

If you’re looking to upskill in Generative AI (GenAI), there’s a Generative AI Learning Path in Google Cloud Skills Boost. It currently consists of 10 courses and provides a good foundation on the theory behind Generative AI and what tools and services Google provides in GenAI. The best part is that it’s completely free!

As I went through these courses myself, I took notes, as I learn best when I write things down. In this part 1 of the blog series, I want to share my notes for courses 1 to 6, in case you want to quickly read summaries of these courses.

Workflows just released a new connector for Batch that greatly simplifies how to create and run Batch jobs from Workflows. Let’s take a look how you can use the new Batch connector of Workflows.

Recap: Batch and Workflows

Batch is a fully managed service to schedule, queue, and execute batch jobs on Google’s infrastructure. These batch jobs run on Compute Engine VM instances but they are managed by Batch service, so you don’t have to provision and manage VM instances yourself.

Cloud Tasks is a fully-managed service that manages the execution, dispatch, and asynchronous delivery of a large number of tasks to App Engine or any arbitrary HTTP endpoint. You can also use a Cloud Tasks queue to buffer requests between services for more robust intra-service communication.  

Cloud Tasks introduces two new features, the new queue-level routing configuration and BufferTask API. Together, they enable creating HTTP tasks and adding to a queue without needing the tasks client library.

If you use Workflows, you’ve been crafting your Workflows definitions in YAML (or JSON). You’re probably painfully aware of the limited support you get in your IDE with syntax validation or auto-completion with these YAML definitions. This was due to Workflow’s schema being out of date, as I covered in my previous post last year.

I’m happy to report that our team recently updated the Workflows schema with the latest syntax. With a new schema, you get a much improved syntax validation and auto-completion of Workflows definition files in your favorite IDE. 

Cloud Code is a set of IDE plugins for popular IDEs that make it easier to create, deploy and integrate applications with Google Cloud. Cloud Code provides an excellent extension mechanism through custom templates. In this post, I show you how you can create and use your own custom templates to add some features beyond those supported natively in Cloud Code, such as .NET functions, event triggered functions and more. 

In a startup, you need to get to the MVP fast, gather feedback from early adopters, and iterate in a quick cycle. Anything that takes time away from developing and iterating on features delays the launch and that’s a serious problem when time-to-market is crucial.

Google Cloud offers products that can help you to build and run your backend services on a fully managed serverless platform, saving time and freeing you from the burden of provisioning and managing infrastructure needed to run those services. Less time on infrastructure means more time implementing and iterating on your services. Additionally, most serverless products have a pay-as-you-go pricing model with little upfront cost at the beginning. Pricing increases in line with the scale of your services and your startup, so you only pay for what you need, when you need it