Merge branch 'better-messages'

hasura · Oct 24, 2024 · e1312f3 · e1312f3
2 parents 7fcca94 + 53048a0
commit e1312f3
Show file tree

Hide file tree

Showing 16 changed files with 222 additions and 231 deletions.
diff --git a/.github/workflows/ndc-nodejs-lambda-connector.yaml b/.github/workflows/ndc-nodejs-lambda-connector.yaml
@@ -6,7 +6,7 @@ on:
       - test-ci/**
   push:
     branches:
-      - 'main'
+      - "main"
       - test-ci/**
     tags:
       - v**

diff --git a/README.md b/README.md
@@ -4,7 +4,7 @@
 [![License](https://img.shields.io/badge/license-Apache--2.0-purple.svg?style=flat)](https://github.com/hasura/ndc-duckdb/blob/main/LICENSE.txt)
 [![Status](https://img.shields.io/badge/status-alpha-yellow.svg?style=flat)](https://github.com/hasura/ndc-duckdb/blob/main/README.md)
 
-This DuckDuckAPI connector allows you to easily build a high-performing connector to expose existing API services, where reads happen against DuckDB and writes happen directly to the upstream API servce. This is ideal to make the API data accessible to LLMs via PromptQL, 
+This DuckDuckAPI connector allows you to easily build a high-performing connector to expose existing API services, where reads happen against DuckDB and writes happen directly to the upstream API servce. This is ideal to make the API data accessible to LLMs via PromptQL,
 
 1. Create a DuckDB schema and write a loading script to load data from an API into DuckDB
 2. Implement functions to wrap over upstream API endpoints, particularly for write operations
@@ -19,6 +19,7 @@ Ofcourse, the tradeoff is that the data will only be eventually consistent becau
 3. Run: `NODE_OPTIONS="--max-old-space-size=4096" HASURA_SERVICE_TOKEN_SECRET=secrettoken HASURA_CONNECTOR_PORT=9094 npx ts-node ./src/index.ts serve --configuration=.`
 4. Remove `duck.db` and `duck.db.wal` from `.gitignore` if you'd like
 5. Create a new DDN project using the duckduckapi connector:
+
 ```bash
 ddn supergraph init new-project
 ddn connector-link add myapi --configure-host http://local.hasura.dev:9094 --configure-connector-token secrettoken
@@ -55,24 +56,25 @@ Below, you'll find a matrix of all supported features for the DuckDB connector:
 
 | Feature                         | Supported | Notes |
 | ------------------------------- | --------- | ----- |
-| Native Queries + Logical Models | ❌     |       |
-| Simple Object Query             | ✅     |       |
-| Filter / Search                 | ✅     |       |
-| Simple Aggregation              | ❌     |       |
-| Sort                            | ✅     |       |
-| Paginate                        | ✅     |       |
-| Table Relationships             | ✅     |       |
-| Views                           | ❌     |       |
-| Distinct                        | ❌     |       |
-| Remote Relationships            | ✅     |       |
-| Custom Fields                   | ❌     |       |
-| Mutations                       | ❌     |       |
+| Native Queries + Logical Models | ❌        |       |
+| Simple Object Query             | ✅        |       |
+| Filter / Search                 | ✅        |       |
+| Simple Aggregation              | ❌        |       |
+| Sort                            | ✅        |       |
+| Paginate                        | ✅        |       |
+| Table Relationships             | ✅        |       |
+| Views                           | ❌        |       |
+| Distinct                        | ❌        |       |
+| Remote Relationships            | ✅        |       |
+| Custom Fields                   | ❌        |       |
+| Mutations                       | ❌        |       |
 
 ## Functions features
 
 Any functions exported from `functions.ts` are made available as NDC functions/procedures to use in your Hasura metadata and expose as GraphQL fields in queries or mutation.
 
 #### Queries
+
 If you write a function that performs a read-only operation, you should mark it with the `@readonly` JSDoc tag, and it will be exposed as an NDC function, which will ultimately show up as a GraphQL query field in Hasura.
 
 ```typescript
@@ -83,74 +85,77 @@ export function add(x: number, y: number): number {
 ```
 
 #### Mutations
+
 Functions without the `@readonly` JSDoc tag are exposed as NDC procedures, which will ultimately show up as a GraphQL mutation field in Hasura.
 
 Arguments to the function end up being field arguments in GraphQL and the return value is what the field will return when queried. Every function must return a value; `void`, `null` or `undefined` is not supported.
 
 ```typescript
 /** @readonly */
 export function hello(name: string, year: number): string {
-  return `Hello ${name}, welcome to ${year}`
+  return `Hello ${name}, welcome to ${year}`;
 }
 ```
 
 #### Async functions
+
 Async functions are supported:
 
 ```typescript
 type HttpStatusResponse = {
-  code: number
-  description: string
-}
+  code: number;
+  description: string;
+};
 
 export async function test(): Promise<string> {
-  const result = await fetch("http://httpstat.us/200")
-  const responseBody = await result.json() as HttpStatusResponse;
+  const result = await fetch("http://httpstat.us/200");
+  const responseBody = (await result.json()) as HttpStatusResponse;
   return responseBody.description;
 }
 ```
 
 #### Multiple functions files
+
 If you'd like to split your functions across multiple files, do so, then simply re-export them from `functions.ts` like so:
 
 ```typescript
-export * from "./another-file-1"
-export * from "./another-file-2"
+export * from "./another-file-1";
+export * from "./another-file-2";
 ```
 
 ### Supported types
+
 The basic scalar types supported are:
 
-* `string` (NDC scalar type: `String`)
-* `number` (NDC scalar type: `Float`)
-* `boolean` (NDC scalar type: `Boolean`)
-* `bigint` (NDC scalar type: `BigInt`, represented as a string in JSON)
-* `Date` (NDC scalar type: `DateTime`, represented as an [ISO formatted](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Date/toISOString) string in JSON)
+- `string` (NDC scalar type: `String`)
+- `number` (NDC scalar type: `Float`)
+- `boolean` (NDC scalar type: `Boolean`)
+- `bigint` (NDC scalar type: `BigInt`, represented as a string in JSON)
+- `Date` (NDC scalar type: `DateTime`, represented as an [ISO formatted](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Date/toISOString) string in JSON)
 
 You can also import `JSONValue` from the SDK and use it to accept and return arbitrary JSON. Note that the value must be serializable to JSON.
 
 ```typescript
-import * as sdk from "@hasura/ndc-lambda-sdk"
+import * as sdk from "@hasura/ndc-lambda-sdk";
 
 export function myFunc(json: sdk.JSONValue): sdk.JSONValue {
   const propValue =
-    json.value instanceof Object && "prop" in json.value && typeof json.value.prop === "string"
+    json.value instanceof Object &&
+    "prop" in json.value &&
+    typeof json.value.prop === "string"
       ? json.value.prop
       : "default value";
-  return new sdk.JSONValue({prop: propValue});
+  return new sdk.JSONValue({ prop: propValue });
 }
 ```
 
 `null`, `undefined` and optional arguments/properties are supported:
 
 ```typescript
 export function myFunc(name: string | null, age?: number): string {
-  const greeting = name != null
-    ? `hello ${name}`
-    : "hello stranger";
-  const ageStatement = age !== undefined
-    ? `you are ${age}`
-    : "I don't know your age";
+  const greeting = name != null ? `hello ${name}` : "hello stranger";
+  const ageStatement =
+    age !== undefined ? `you are ${age}` : "I don't know your age";
 
   return `${greeting}, ${ageStatement}`;
 }
@@ -162,21 +167,21 @@ Object types and interfaces are supported. The types of the properties defined o
 
 ```typescript
 type FullName = {
-  title: string
-  firstName: string
-  surname: string
-}
+  title: string;
+  firstName: string;
+  surname: string;
+};
 
 interface Greeting {
-  polite: string
-  casual: string
+  polite: string;
+  casual: string;
 }
 
 export function greet(name: FullName): Greeting {
   return {
     polite: `Hello ${name.title} ${name.surname}`,
-    casual: `G'day ${name.firstName}`
-  }
+    casual: `G'day ${name.firstName}`,
+  };
 }
 ```
 
@@ -192,7 +197,7 @@ Anonymous types are supported, but will be automatically named after the first p
 
 ```typescript
 export function greet(
-  name: { firstName: string, surname: string } // This type will be automatically named greet_name
+  name: { firstName: string; surname: string }, // This type will be automatically named greet_name
 ): string {
   return `Hello ${name.firstName} ${name.surname}`;
 }

diff --git a/connector-definition/template/functions.ts b/connector-definition/template/functions.ts
@@ -11,13 +11,18 @@ export let loaderStatus: string = "stopped";
  * $ddn.jobs.sample-loader.init
  */
 export async function __dda_loader_init(headers: JSONValue): Promise<string> {
-  const oauthServices = headers.value as any;
-  const { access_token } = getTokensFromHeader(headers, "google-calendar");
+  let access_token: string | null = null;
+  try {
+    access_token = getTokensFromHeader(headers, "google-calendar").access_token;
+  } catch (error) {
+    loaderStatus = `Error in getting the google-calendar oauth credentials: ${error}. Login to google-calendar?`;
+    return loaderStatus;
+  }
 
   if (!access_token) {
     console.log(headers.value);
     loaderStatus =
-      "google-calendar key not found in oauth services. Login to google-calendar?";
+      "google-calendar access token not found in oauth services. Login to google-calendar?";
     return loaderStatus;
   }
 
@@ -29,8 +34,10 @@ export async function __dda_loader_init(headers: JSONValue): Promise<string> {
 
   if (!result) {
     loaderStatus = result + ". Have you logged in to google-calendar?";
+    return loaderStatus;
   }
 
+  console.log("Initializing sync manager");
   syncManager.initialize();
   loaderStatus = "running";
   process.on("SIGINT", async () => {
@@ -50,6 +57,7 @@ export async function __dda_loader_init(headers: JSONValue): Promise<string> {
  *  @readonly
  * */
 export function __dda_loader_status(): string {
+  console.log(loaderStatus);
   return loaderStatus;
 }
 

diff --git a/connector-definition/template/index.ts b/connector-definition/template/index.ts
@@ -1,49 +1,10 @@
 import { start } from "@hasura/ndc-duckduckapi";
 import { makeConnector, duckduckapi } from "@hasura/ndc-duckduckapi";
 import * as path from "path";
+import { readFileSync } from "fs";
 
 const calendar: duckduckapi = {
-  dbSchema: `
-    CREATE TABLE IF NOT EXISTS calendar_events (
-  id VARCHAR PRIMARY KEY,
-  summary VARCHAR,
-  description VARCHAR,
-  start_time TIMESTAMP,  -- Will store both date-only and datetime values
-  end_time TIMESTAMP,    -- Will store both date-only and datetime values
-  created_at TIMESTAMP,
-  updated_at TIMESTAMP,
-  creator_email VARCHAR,
-  organizer_email VARCHAR,
-  status VARCHAR,
-  location VARCHAR,
-  recurring_event_id VARCHAR,
-  recurrence JSON, 
-  transparency VARCHAR,
-  visibility VARCHAR,
-  ical_uid VARCHAR,
-  attendees JSON,
-  reminders JSON,
-  conference_data JSON,
-  color_id VARCHAR,
-  original_start_time TIMESTAMP,
-  extended_properties JSON,
-  attachments JSON,
-  html_link VARCHAR,
-  meeting_type VARCHAR,
-  sequence INTEGER,
-  event_type VARCHAR,
-  calendar_id VARCHAR,
-  sync_status VARCHAR,
-  last_synced TIMESTAMP,
-  is_all_day BOOLEAN   -- New field to distinguish all-day events
-);
-
-CREATE TABLE IF NOT EXISTS sync_state (
-  calendar_id VARCHAR PRIMARY KEY,
-  sync_token VARCHAR,
-  last_sync TIMESTAMP
-);
-    `,
+  dbSchema: readFileSync(path.join(__dirname, "schema.sql"), "utf-8"),
   functionsFilePath: path.resolve(__dirname, "./functions.ts"),
 };
 

diff --git a/docs/architecture.md b/docs/architecture.md
@@ -1,6 +1,7 @@
 # General Architecture of the DuckDB Connector
 
 ## Query Engine
+
 The query engine's job is to take a `QueryRequest`, which contains information about the query a user would like to run, translate it to DuckDB SQL (using the SQLite dialect), execute it against the database, and return the results as a `QueryResponse`.
 
 One place in particular that uses the Query Engine is the `/query` endpoint (defined in the `ndc-hub` repository).
@@ -12,29 +13,30 @@ API:
 ```typescript
 export async function plan_queries(
   configuration: Configuration,
-  query: QueryRequest
-): Promise<SQLQuery[]>
+  query: QueryRequest,
+): Promise<SQLQuery[]>;
 ```
 
 ```typescript
 async function perform_query(
   state: State,
-  query_plans: SQLQuery[]
-): Promise<QueryResponse>
+  query_plans: SQLQuery[],
+): Promise<QueryResponse>;
 ```
 
 Note that the response from this function should be in the format of an ndc-spec [QueryResponse](https://hasura.github.io/ndc-spec/reference/types.html#queryresponse) represented as JSON.
 
 ### Query Planning
+
 The query plan is essentially side-effect free - we use information from the request as well as the information about the metadata to translate the query request into a SQL statement to run against the database.
 
 This process is currently found in the [src/handlers](/src/handlers/query.ts) directory in the query.ts file. The API is the following function:
 
 ```typescript
 export async function plan_queries(
   configuration: Configuration,
-  query: QueryRequest
-): Promise<SQLQuery[]>
+  query: QueryRequest,
+): Promise<SQLQuery[]>;
 ```
 
 The `plan_queries` function returns a `SQLQuery[]` which functions as an execution plan.
@@ -50,10 +52,10 @@ export type SQLQuery = {
 };
 ```
 
-The incoming `QueryRequest` is used to construct a SQL statement by performing a recursive post-order walk over the QueryRequest to construct the SQL statement to be executed. The different pieces of the query are computed and then the query is constructed. 
+The incoming `QueryRequest` is used to construct a SQL statement by performing a recursive post-order walk over the QueryRequest to construct the SQL statement to be executed. The different pieces of the query are computed and then the query is constructed.
 
 ```typescript
-  sql = wrap_rows(`
+sql = wrap_rows(`
   SELECT
   JSON_OBJECT(${collect_rows.join(",")}) as r
   FROM ${from_sql}
@@ -65,6 +67,7 @@ The incoming `QueryRequest` is used to construct a SQL statement by performing a
 ```
 
 ### Query Execution
+
 The query execution creates a connection to the database and executes the query plan against the DuckDB or MotherDuck hosted database. It then returns the results from the query back to the caller of the function.
 
 ```typescript
@@ -82,7 +85,7 @@ async function do_all(con: any, query: SQLQuery): Promise<any[]> {
 
 async function perform_query(
   state: State,
-  query_plans: SQLQuery[]
+  query_plans: SQLQuery[],
 ): Promise<QueryResponse> {
   const con = state.client.connect();
   const response: RowSet[] = [];
@@ -100,10 +103,11 @@ async function perform_query(
 Here are a few ideas I have about working with this connector.
 
 ### KISS (Keep it simple stupid!)
-Robust and full-featured connector implementations should preferably be written in Rust for performance purposes. For Community Connectors it is preferred to try to keep things simple where possible, all we are doing is constructing a SQL query from the QueryRequest, which ultimately is manipulating a string. 
 
+Robust and full-featured connector implementations should preferably be written in Rust for performance purposes. For Community Connectors it is preferred to try to keep things simple where possible, all we are doing is constructing a SQL query from the QueryRequest, which ultimately is manipulating a string.
 
 ### Consider the generated SQL
+
 When working on a feature or fixing a bug, consider the generated SQL first. What does it currently look like? What should it look like?
 
 Construct what the desired query should be and make sure you can run it, and then work towards altering the SQL construction to match the hand-crafted query.
diff --git a/docs/code-of-conduct.md b/docs/code-of-conduct.md
@@ -71,4 +71,4 @@ members of the project's leadership.
 This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
 available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html
 
-[homepage]: https://www.contributor-covenant.org
+[homepage]: https://www.contributor-covenant.org
diff --git a/docs/contributing.md b/docs/contributing.md
@@ -36,4 +36,4 @@ For all contributions, a CLA (Contributor License Agreement) needs to be signed
 - The first line should be a summary of the changes, not exceeding 50 characters, followed by an optional body which has more details about the changes. Refer to [this link](https://github.com/erlang/otp/wiki/writing-good-commit-messages) for more information on writing good commit messages.
 - Use the imperative present tense: "add/fix/change", not "added/fixed/changed" nor "adds/fixes/changes".
 - Don't capitalize the first letter of the summary line.
-- Don't add a period/dot (.) at the end of the summary line.
+- Don't add a period/dot (.) at the end of the summary line.
-Original file line number
+Diff line change
@@ Expand Up / @@ -6,7 +6,7 @@ on: @@
           - test-ci/**
       push:
         branches:
-          - 'main'
+          - "main"
           - test-ci/**
         tags:
           - v**
@@ Expand Down @@