7.observability.markdown

Let's Build a Connector - Part 7 - Observability

In this tutorial, we're going to take a break from implementing new query features, and implement some features related to observability. That is, features allowing us to observe our running connector's behavior in production. The NDC specification defines three observability features which are supported by the Rust and TypeScript SDKs: health checks, metrics and tracing. We'll look at each of these in turn.

Note: if you are using a different SDK, or perhaps not using an SDK at all, then the approach taken to these features may be different, but the basic ideas will apply. Consult the specification for the low-level details of these features if necessary.

Health Checks

The simplest observability feature is the service health endpoint, which performs some basic sanity checks on the state of the connector. This endpoint can be polled in order to set up alerts in case of failures.

For example, running the docker compose up command will build and start the connector, and the Dockerfile is configured to poll the health endpoint to determine the service health of the connector.

To implement the health check endpoint using the TypeScript SDK, we only have to implement the healthCheck function. In our case, we'll issue a very simple request to the database, to make sure we can connect:

async function healthCheck(configuration: Configuration, state: State): Promise<undefined> {
    try {
        await state.db.all("SELECT 1");
    } catch (x) {
        throw new ConnectorError(503, "Service Unavailable");
    }
}

Metrics

Next, we'll add some metrics which will be reported to the metrics endpoint.

Metrics can be polled directly (for example, try watch 'curl http://localhost:8080/metrics | tail -10' in your terminal to see the raw data), or picked up by a Prometheus collector. The Docker compose project is configured with Prometheus, so you can also run docker compose up and open http://localhost:9090.

We add metrics by modifying our tryInitState function, which receives the metrics Registry as an argument:

async function tryInitState(configuration: Configuration, registry: Registry): Promise<State> {
    // ...

    const query_count = new Counter({
        name: 'query_count',
        help: 'Number of queries executed since the connector was started',
        labelNames: ["table"]
    });
    registry.registerMetric(query_count);
    const metrics = { query_count };

    return { db, metrics };
}

Here, we add one metric, a counter of the number of query requests served so far. It is labeled by the name of the table which was queried.

In our query function, we can add the following line to increment the (labeled) counter:

state.metrics.query_count.labels(request.collection).inc();

Tracing

The final observability feature is tracing, which is implemented using the OpenTelemetry specification. The TypeScript SDK takes care of setting this all up for us, so we just need to instrument the key functions in our query function.

For example, we can modify the fetch_rows function to wrap the call to db.all(..) in the withActiveSpan function, which will send a new trace span to the trace reporter corresponding to the time spent in that function:

const spanAttributes = { sql };
    const tracer = opentelemetry.trace.getTracer("ndc-learn");

    return withActiveSpan(tracer, "run SQL", async () => {
        const rows = await state.db.all(sql, ...parameters);
        return rows.map((row) => postprocess_fields(query, collection_relationships, row))
    }, spanAttributes);

Note that we can also attach attributes to the span, so here we attach the generated SQL, so that we can use it for debugging.

Note: in general, don't attach any attributes which can leak sensitive user data. Here, for example, we attach the SQL, but not the parameters which might contain sensitive user IDs or other information. Consider the possible users of your connector and how it might be deployed when you decide what is appropriate to log here. For connectors which will be published to the connector hub, err on the side of logging fewer attributes if they might possibly contain sensitive data.

If you run docker compose up, and issue a query (for example, by opening http://localhost:3000 and using Graphiql), traces will now be generated and sent to Jaeger, which is also running in a Docker container. To view your traces, you can open Jaeger at http://localhost:4002.

That's all for this tutorial. Hopefully this helps you to add some useful observability features to your connector.