In Workflows, sometimes, you need to store some state, a key/value pair, in a step in one execution and later read that state in another step in another execution. There’s no intrinsic key/value store in Workflows. However, you can use Firestore as a key/value store and that’s what I want to show you here.
If you want to skip to see some samples, check out workflow.yaml.
If you want to learn more about it, keep reading.
Firestore setup for Workflows
In Firestore, you typically have a single collection and multiple documents in that collection. To use Firestore as a key/value store for Workflows, one idea is to use the workflow name as the collection name and use a single document to store all the key/value pairs.
your-workflow-name
|
├── key-value-store
|
├── key1=value1
├── key2=true
├── key3=1
├── key4=1.5
Put value
Here’s a subworkflow to save a key/value pair on Firestore:
firestore_put:
params: [key, value, valueType: "string"]
steps:
- init:
assign:
- database_root: ${"projects/" + sys.get_env("GOOGLE_CLOUD_PROJECT_ID") + "/databases/(default)/documents/" + sys.get_env("GOOGLE_CLOUD_WORKFLOW_ID") + "/"}
- doc_name: ${database_root + "key-value-store"}
- store:
call: googleapis.firestore.v1.projects.databases.documents.patch
args:
name: ${doc_name}
updateMask:
fieldPaths: [${key}]
body:
fields:
${key}:
${valueType + "Value"}: ${value}
In Firestore, you need to specify the type of the variable being saved. The subworkflow assumes the string value type but you can also pass in some other basic types such as boolean, integer, double.
Get value
Here’s the subworkflow to retrieve the value for a given key:
firestore_get:
params: [key, valueType: "string"]
steps:
- init:
assign:
- database_root: ${"projects/" + sys.get_env("GOOGLE_CLOUD_PROJECT_ID") + "/databases/(default)/documents/" + sys.get_env("GOOGLE_CLOUD_WORKFLOW_ID") + "/"}
- doc_name: ${database_root + "key-value-store"}
- get:
call: googleapis.firestore.v1.projects.databases.documents.get
args:
name: ${doc_name}
mask:
fieldPaths: [${key}]
result: getResult
- return_value:
switch:
- condition: ${not("fields" in getResult)}
return: null
- condition: true
return: ${getResult.fields[key][valueType + "Value"]}
Note that if the key does not exist, the subworkflow simply returns null. I thought this is easier than throwing a KeyNotFound error and let the user handle it.
Cleanup
You also probably want to clear the keys once in a while by deleting the document. Here’s a subworflow for that:
firestore_clear:
steps:
- init:
assign:
- database_root: ${"projects/" + sys.get_env("GOOGLE_CLOUD_PROJECT_ID") + "/databases/(default)/documents/" + sys.get_env("GOOGLE_CLOUD_WORKFLOW_ID") + "/"}
- doc_name: ${database_root + "key-value-store"}
- drop:
call: googleapis.firestore.v1.projects.databases.documents.delete
args:
name: ${doc_name}
Examples
Now that subworkflows are in place, this is how you can store key/value pairs for different types
and retrieve results. Notice the optional clear_keys step in the end:
main:
steps:
- put_string_value:
call: firestore_put
args:
key: "key1"
value: "value1"
- get_string_value:
call: firestore_get
args:
key: "key1"
result: string_value
- put_boolean_value:
call: firestore_put
args:
key: "key2"
value: true
valueType: "boolean"
- get_boolean_value:
call: firestore_get
args:
key: "key2"
valueType: "boolean"
result: boolean_value
- put_integer_value:
call: firestore_put
args:
key: "key3"
value: 1
valueType: "integer"
- get_integer_value:
call: firestore_get
args:
key: "key3"
valueType: "integer"
result: integer_value
- put_double_value:
call: firestore_put
args:
key: "key4"
value: 1.5
valueType: "double"
- get_double_value:
call: firestore_get
args:
key: "key4"
valueType: "double"
result: double_value
- get_nonexisting_key:
call: firestore_get
args:
key: "nonexisting"
result: nonexisting_value
# - clear_keys:
# call: firestore_clear
- return_values:
return:
string_value: ${string_value}
boolean_value: ${boolean_value}
integer_value: ${integer_value}
double_value: ${double_value}
nonexisting_value: ${nonexisting_value}
Even though Workflows does not provide an intrinsic key/value store, Firestore is quite easy to use instead once you figure out the right API calls.
What do you think of this solution? If you have ideas to improve it, please reach out to me on Twitter @meteatamel.