Google Cloud Pub/Sub + AsyncAPI

Cloud Pub/Sub and AsyncAPI

I’ve been covering different aspects of AsyncAPI in my recent blog posts. In this final post of my AsyncAPI blog post series, I want to talk about how to document Google Cloud’s Pub/Sub using AsyncAPI.

AsyncAPI has pretty good support for Google Pub/Sub, thanks to contributions from Jeremy Whitlock, an engineer from Google, and the flexibility baked in AsyncAPI spec. Jeremy also has a nice blog post on this topic that you can read for more details.

Recap: Google Cloud Pub/Sub and AsyncAPI

Let’s first do a quick recap Google Cloud’s Pub/Sub and AsyncAPI.

Pub/Sub is a managed service in Google Cloud for asynchronous and scalable messaging. It has everything you expect in a messaging service such as producers, consumers, topics in a fully managed and globally available service.

AsyncAPI is an open source specification to define asynchronous APIs, similar to what OpenAPI (aka Swagger) does for REST APIs.

If your event-driven service in Google Cloud sends or receives messages using Pub/Sub, you can use AsyncAPI to document that service and its messaging schema, thanks to flexibility of AsyncAPI with its bindings.

Bindings in AsyncAPI

Bindings in AsyncAPI is a mechanism to extend AsyncAPI beyond its spec. One can have server bindings, channel bindings, operation bindings and message bindings. That’s how AsyncAPI supports different messaging systems such as Google Cloud’s Pub/Sub, Apache Kafka, Solace, etc.

Google Cloud Pub/Sub Bindings documentation for AsyncAPI is the source of truth.

Let’s go through an example to see how you can define an application using Google Cloud Pub/Sub with AsyncAPI.

The full example is in google-pubsub-sample.yaml.

Pub/Sub Endpoints → AsyncAPI Servers

First, you need to provide some info about your Pub/Sub application.

asyncapi: 2.6.0
info:
    title: Google Cloud Pub/Sub Topology
    description: AsyncAPI definition for Google Cloud Pub/Sub
    version: 0.0.1

Pub/Sub provides a globally available endpoint but it also have regional endpoints. You need to define those endpoints as server in AsyncAPI. Here’s how you can have the global endpoint as default but also restrict to region-specific endpoints:

servers:
  cloudPubSub:
    url: '{cloudPubSubEndpoint}.googleapis.com'
    description: The API for Cloud Pub/Sub.
    protocol: googlepubsub
    variables:
      cloudPubSubEndpoint:
        description: The Cloud Pub/Sub endpoint region.
        # Default to the global endpoint
        default: pubsub
        # Or, restrict to only the following region-specific endpoints.
        # enum:
        # - us-central1
        # - us-central2

Pub/Sub Topics → AsyncAPI Channels

Next, you need to define some channels to publish messages to or receive messages from. Channels in AsyncAPI map to topics in Pub/Sub and topics in Pub/Sub can have Avro, Protobuf, or no-schemas defined in Pub/Sub.

For example, here’s a channel with an Avro schema backed topic:

    topic-avro-schema:
        bindings:
            googlepubsub:
                topic: projects/your-project/topics/topic-avro-schema
                schemaSettings:
                    encoding: json
                    name: projects/your-project/schemas/message-avro
        publish:
            message:
                $ref: '#/components/messages/messageAvro'
        subscribe:
            message:
                $ref: '#/components/messages/messageAvro'

Notice how bindings is being used to provide Pub/Sub specific configuration.

Similarly, this is a channel with a Protobuf schema backed topic:

    topic-proto-schema:
        bindings:
            googlepubsub:
                topic: projects/your-project/topics/topic-proto-schema
                messageRetentionDuration: 86400s
                messageStoragePolicy:
                  allowedPersistenceRegions:
                  - us-central1
                  - us-central2
                  - us-east1
                  - us-east4
                  - us-east5
                  - us-east7
                  - us-south1
                  - us-west1
                  - us-west2
                  - us-west3
                  - us-west4
                schemaSettings:
                    encoding: binary
                    name: projects/your-project/schemas/message-proto
        publish:
            message:
                $ref: '#/components/messages/messageProto'
        subscribe:
            message:
                $ref: '#/components/messages/messageProto'

Notice that there are additional Pub/Sub settings like messageRetentionDuration or messageStoragePolicy that you can set.

Finally, you can also have a channel with no Pub/Sub schema:

    topic-no-schema:
        bindings:
            googlepubsub:
                topic: projects/your-project/topics/topic-no-schema
        publish:
            message:
                $ref: '#/components/messages/messageNoSchema'
        subscribe:
            message:
                $ref: '#/components/messages/messageNoSchema'

Pub/Sub Schema → AsyncAPI Message Payload

Finally, you need to define the message schemas in AsyncAPI.

For Avro messages, you refer to the schema definition in Pub/Sub. You also define the payload inline for tooling and documentation:

components:
    messages:
        messageAvro:
            bindings:
                googlepubsub:
                    schema:
                        name: projects/your-project/schemas/message-avro
                        type: avro
            contentType: application/json
            name: MessageAvro
            payload:
                fields:
                    - name: message
                      type: string
                name: Message
                type: record
            schemaFormat: application/vnd.apache.avro+yaml;version=1.9.0

For Protobuf messages, you still refer to the schema definition in Pub/Sub but you cannot define the payload, as Protobuf is not supported in AsyncAPI yet:

        messageProto:
            bindings:
                googlepubsub:
                    schema:
                        name: projects/your-project/schemas/message-proto
                        type: protobuf
            contentType: application/octet-stream
            name: MessageProto
            # Proto is currently not supported in AsyncAPI, we're simply
            # indicating that there's a payload here.
            payload: true

Finally, message without schema can be defined as follows:

        messageNoSchema:
            contentType: application/octet-stream
            description: A message having no schema
            payload: true

That’s it! I’m pleased to see how flexible AsyncAPI is in helping to document all sorts of event-driven services, including Google Cloud’s Pub/Sub.

This wraps up my 5-part blog series in AsyncAPI. Here’s the whole series for your reference:

AsyncAPI  Pub/Sub  Event-driven  How To 

See also