In Workflows, it’s easy to chain various services together into an automated workflow. For some use cases, you might need to pause workflow execution and wait for some input. This input could be a human approval or an external service calling back with data needed to complete the workflow.
With Workflows callbacks, a workflow can create an HTTP endpoint and pause execution until it receives an HTTP callback to that endpoint. This is very useful for creating human-in-the-middle type workflows. In a previous blog post, Guillaume Laforge showed how to build an automated translation workflow with human validation using callbacks.
Callbacks are great, but someone (or some service) has to know the callback endpoint and make an HTTP call to that endpoint. And, many services send or generate events, instead of calling HTTP endpoints. Wouldn’t it be nice if you could pause a workflow execution and resume when a specific event is received? For example, you could use this capability to create workflows that pause and wait for a new message from a Pub/Sub topic or a new file to be created in a Cloud Storage bucket.
Event callbacks
While Workflows does not provide event callbacks out of the box, it’s possible to have a workflow execution wait for an event by using callbacks, Firestore, and Eventarc.
The idea is to use Firestore to store callback details from your original workflow, use Eventarc to listen for events from various event sources, and use a second event-triggered workflow to receive events from Eventarc and call back to callback endpoints in your original workflow.
Architecture
Here is the setup in more detail:
- A
callback-event-sampleworkflow (yaml) creates a callback for an event source that it is interested in waiting events from. - It stores the callback information for the event source in a document in Firestore.
- It carries on with its workflow and at some point, starts waiting for an event.
- In the meantime,
callback-event-listener(yaml) is ready to be triggered by events from a Pub/Sub topic and a Cloud Storage bucket with Eventarc. - At some point, Eventarc receives an event and triggers the event listener.
- Event listener finds the document for the event source in Firestore.
- Event listener calls back all the callback URLs registered with that event
source.
callback-event-sampleworkflow receives the event via its callback endpoint and stops waiting. - It deletes the callback URL from Firestore and continues with its execution.
Detailed set up instructions along with YAML workflow definitions are on GitHub.
Test
To test event callbacks, first execute the sample workflow:
gcloud workflows run callback-event-sample
Once it starts, the workflow will pause and wait, which you can confirm by its running state on Google Cloud Console:
To test Pub/Sub callbacks, send a Pub/Sub message:
TOPIC=topic-callback
gcloud pubsub topics publish $TOPIC --message="Hello World"
In the workflow logs, you should see that the workflow started and stopped waiting for the event:
Started waiting for an event from source topic-callback
Stopped waiting for an event from source topic-callback
To test Cloud Storage events, upload a new file to Cloud Storage:
BUCKET=$PROJECT_ID-bucket-callback
echo "Hello World" > random.txt
gsutil cp random.txt gs://$BUCKET/random.txt
You should see that the workflow started and stopped waiting for the event:
Started waiting for an event from source $PROJECT_ID-bucket-callback
Stopped waiting for an event from source $PROJECT_ID-bucket-callback
At this point, the workflow should stop executing and you should also see the received Pub/Sub and Cloud Storage events in the output:
This blog post covered how to create workflows that wait for Pub/Sub and Cloud Storage events by using callbacks, Firestore, and Eventarc. You can extend the sample to listen for any events that Eventarc supports. Feel free to reach out to me on Twitter @meteatamel for any questions or feedback.