Sometimes, you need to run a piece of code for hours, days, or even weeks. Cloud Functions and Cloud Run are my default choices to run code. However, they both have limitations on how long a function or container can run. This rules out the idea of executing long-running code in a serverless way.
Thanks to Workflows and Compute Engine, you can have an almost serverless experience with long running code.
Here’s the idea:
Containerize the long-running task, so it can run anywhere.
Plan to run the container on a Compute Engine VM with no time limitations.
Use Workflows to automate VM creation, running the container on the VM, and VM deletion.
With this approach, you simply execute the workflow and get back the result of the long-running task. The underlying lifecycle of the VM and running of the container are all abstracted away. This is almost serverless!
Let’s look at a concrete example.