In my previous post, I talked about Knative v0.9.0 and some of the eventing changes in the latest release. I’ve been playing with Knative v0.9.0 since then to read Google Cloud Pub/Sub messages using PullSubscription and I ran into a rather fundamental issue that baffled me for a while. I’d like to outline the problem and the solution here, just in case it’s useful to others.
Knative Services as eventing sinks
In my PullSubscription, I could define Kubernetes Services as event sinks as follows:
apiVersion: pubsub.cloud.run/v1alpha1
kind: PullSubscription
metadata:
name: testing-source-event-display
spec:
topic: testing
sink:
apiVersion: v1
kind: Service
name: event-display
This worked fine and allowed me to read Pub/Sub messages in a Kubernetes Service. As the next step, I wanted to use Knative Services as event sinks because Knative Services are much easier to deal with. In PullSubscription, I defined my sink as follows:
apiVersion: pubsub.cloud.run/v1alpha1
kind: PullSubscription
metadata:
name: testing-source-event-display
spec:
topic: testing
sink:
apiVersion: serving.knative.dev/v1alpha1
kind: Service
name: event-display
I could apply this definition and see that pods related to PullSubscription were created:
$ kubectl get pods
NAME READY STATUS
cre-pull-e7072664-ee7a-11e9-8ae9-42010a84006f-7db6dcf4b9-5ztgx 1/1 Running 2 7m12s
pubsub-s-testing-source-event-display-pullsubscription-cretw8mw 0/1 Completed 0 7m12s
However, when I checked the logs of the pods, I saw some errors:
$ kubectl logs cre-pull-e7072664-ee7a-11e9-8ae9-42010a84006f-7db6dcf4b9-5ztgx
{"level":"info","ts":1571054849.2031848,"logger":"fallback","caller":"pubsub/transport.go:231","msg":"got an event!"}
{"level":"warn","ts":1571054849.2170587,"logger":"fallback","caller":"pubsub/transport.go:248","msg":"pubsub receiver return err","error":"Post [http://event-display.default.svc.cluster.local](http://event-display.default.svc.cluster.local): dial tcp: lookup event-display.default.svc.cluster.local on 10.0.16.10:53: no such host"}
PullSubscription gets the message but it cannot pass it to the Knative Service. The default.svc.cluster.local implied that Knative expected my service to be cluster local (internal). This wasn’t the case before. I tried to turn my Knative service a cluster local service as I explain in my tutorial and that didn’t work either.
Cluster local issue
At this point, I was stuck. After researching, I realized that I wasn’t alone. There is a bug (1973) where other users also report similar problems. Thanks to Knative Engineering team and comments in the bug, I figured out that Knative now requires an extra Istio cluster local gateway added to your cluster in order to have Knative Services as event sinks in PullSubscription.
Installing cluster local gateway
There’s an Updating your install to use cluster local gateway page that describes how to add local gateway to your Knative cluster. There’s also a third_party folder in Knative Service with istio-knative-extras.yaml that you can point to install the cluster local gateway.
In my case, I was using a GKE cluster with Istio add-on with the following versions:
- GKE: 1.14.6-gke.1
- Istio: 1.1.13-gke.0
The third_party folder in Knative had Istio versions 1.2.7 and 1.3.2 as of today. Not an exact match with my Istio version but I went with 1.2.7:
$ kubectl apply -f [https://raw.githubusercontent.com/knative/serving/master/third\_party/](https://raw.githubusercontent.com/knative/serving/master/third_party/)istio-1.2.7/istio-knative-extras.yaml
serviceaccount/cluster-local-gateway-service-account created
serviceaccount/istio-multi configured
clusterrole.rbac.authorization.k8s.io/istio-reader configured
clusterrolebinding.rbac.authorization.k8s.io/istio-multi configured
service/cluster-local-gateway created
deployment.apps/cluster-local-gateway created
After this I was able to use Knative Services as events sinks.
This is still not ideal because I’m not sure if it’ll continue to work with different versions of Istio of as GKE add-on gets upgraded. However, it’ll work as a temporary solution until the we come up with a more complete installation instructions for Knative Eventing on GKE.
I outlined all the setup needed to receive Pub/Sub messages in Knative Eventing in my tutorial. Check it out here.