Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error in Get Started with Hasura DDN and ClickHouse Guide #10669

Open
bitjson opened this issue Feb 5, 2025 · 14 comments
Open

Error in Get Started with Hasura DDN and ClickHouse Guide #10669

bitjson opened this issue Feb 5, 2025 · 14 comments
Labels
k/v3-bug Bug affecting Hasura v3 (DDN)

Comments

@bitjson
Copy link

bitjson commented Feb 5, 2025

I'm following this tutorial exactly: https://hasura.io/docs/3.0/how-to-build-with-ddn/with-clickhouse

The ddn connector introspect command is failing with this error:

❯ ddn connector introspect my_ch

8:39PM INF Using Subgraph config file "app/subgraph.yaml" found in context.
8:39PM INF Using localEnvFile ".env" found in context.
Error: reqwest::Error { kind: Decode, source: Error("invalid type: integer `0`, expected a boolean", line: 52, column: 21) }
8:39PM ERR introspect failed with error: exit status 1

Versions:

  • ddn --version: DDN CLI Version: v2.20.1 (latest)
  • clickhouse/clickhouse-server:latest is sha256:e80057e6e7f35356252afa13a69a42dee47a43652682b67d786ea46611d3a339

In case it's useful:

❯ ddn connector introspect my_ch --log-level DEBUG

8:46PM DBG global config directory is not pre-set, defaulting
8:46PM DBG global config directory set as '/Users/u/.ddn'
8:46PM DBG global config file set as '/Users/u/.ddn/config.yaml'
8:46PM DBG global config file exists, verifying contents
8:46PM DBG global config is not pre-set, reading from current env
8:46PM DBG global config: uuid: 890324d9-e988-4b3e-b1f9-bff28a0a966d
8:46PM DBG global config: enableTelemetry: true
8:46PM DBG global config: showUpdateNotification: true
8:46PM DBG global config: cliEnvironment: default
8:46PM DBG execution id: 15458465-94f3-4f58-b864-2c629dbee6f9
8:46PM DBG Supergraph config not supplied by flag. Attempting to find a SubgraphConfig config...
8:46PM INF Using Subgraph config file "app/subgraph.yaml" found in context.
8:46PM DBG Using Subgraph config "/Users/u/my-project/app/subgraph.yaml"
8:46PM INF Using localEnvFile ".env" found in context.
8:46PM DBG loading plugin clickhouse
8:46PM DBG finding plugin manifest files in directory /Users/u/.ddn/plugins/index/plugins
8:46PM DBG loading plugins
8:46PM DBG plugin already installed for connector hasura/clickhouse:v1.0.5. proceeding to execution
8:46PM DBG plugin installation successful for connector hasura/clickhouse:v1.0.5
8:46PM DBG Executing the command Command="/bin/bash -c hasura-clickhouse update" Dir= Env vars=[...]
Error: reqwest::Error { kind: Decode, source: Error("invalid type: integer `0`, expected a boolean", line: 52, column: 21) }
8:46PM DBG telemetry: beamed
8:46PM ERR introspect failed with error: exit status 1
@bitjson bitjson added the k/v3-bug Bug affecting Hasura v3 (DDN) label Feb 5, 2025
@bitjson
Copy link
Author

bitjson commented Feb 5, 2025

Also, ddn console --local isn't connecting:

❯ ddn console --local
10:08PM INF Opening https://console.hasura.io/local?url=http://localhost:3280&promptQLUrl=

Trying to connect to Engine..., Please ensure that your Engine is running at http://localhost:3280:

Image

I also tried switching it to the other ports listed by ddn run docker-start3000, 8080, 4317, and 4318:

 => CACHED [engine 1/2] FROM ghcr.io/hasura/v3-engine:latest@sha256:0b8cd  0.0s
 => [engine 2/2] COPY ./build /md/                                         0.0s
 => [app_my_ch internal] load .dockerignore                                0.0s
 => => transferring context: 2B                                            0.0s
 => [engine] exporting to image                                            0.0s
 => => exporting layers                                                    0.0s
 => => writing image sha256:8500df292d0dadceaa2e4e15a6d4f96a0d896a3f84eac  0.0s
 => => naming to docker.io/library/my-project-engine                       0.0s
 => [app_my_ch internal] load build context                                0.8s
 => => transferring context: 83.80MB                                       0.8s
 => CACHED [app_my_ch 1/2] FROM ghcr.io/hasura/ndc-clickhouse:v1.0.5@sha2  0.0s
 => [engine] resolving provenance for metadata file                        0.0s
 => [app_my_ch 2/2] COPY ./ /etc/connector                                 0.4s
 => [app_my_ch] exporting to image                                         0.2s
 => => exporting layers                                                    0.2s
 => => writing image sha256:d8e2c42adb8d1ebbe94b9aba6765d329f39f8464992f1  0.0s
 => => naming to docker.io/library/my-project-app_my_ch                    0.0s
 => [app_my_ch] resolving provenance for metadata file                     0.0s
[+] Running 6/6
 ! app_my_ch The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested 0.1s 
 ✔ engine                                                                                                                                                   Built0.0s 
 ✔ Network my-project_default                                                                                                                               Created0.1s 
 ✔ Container my-project-otel-collector-1                                                                                                                    Created0.0s 
 ✔ Container my-project-app_my_ch-1                                                                                                                         Created0.0s 
 ✔ Container my-project-engine-1                                                                                                                            Created0.0s 
Attaching to app_my_ch-1, engine-1, otel-collector-1
engine-1          | starting server on [::]:3000
otel-collector-1  | 2025-02-05T03:07:49.363Z	info	[email protected]/service.go:115	Setting up own telemetry...
otel-collector-1  | 2025-02-05T03:07:49.363Z	info	[email protected]/telemetry.go:96	Serving metrics	{"address": ":8888", "level": "Normal"}
otel-collector-1  | 2025-02-05T03:07:49.363Z	info	[email protected]/service.go:193	Starting otelcol...	{"Version": "0.104.0", "NumCPU": 10}
otel-collector-1  | 2025-02-05T03:07:49.364Z	info	extensions/extensions.go:34	Starting extensions...
otel-collector-1  | 2025-02-05T03:07:49.364Z	info	[email protected]/otlp.go:102	Starting GRPC server	{"kind": "receiver", "name": "otlp", "data_type": "traces", "endpoint": "0.0.0.0:4317"}
otel-collector-1  | 2025-02-05T03:07:49.364Z	info	[email protected]/otlp.go:152	Starting HTTP server	{"kind": "receiver", "name": "otlp", "data_type": "traces", "endpoint": "0.0.0.0:4318"}
otel-collector-1  | 2025-02-05T03:07:49.364Z	info	[email protected]/service.go:219	Everything is ready. Begin running and processing data.
app_my_ch-1       | Starting server on [::]:8080

(However, https://localhost:3280 is accessible and working:)
Image

@seanparkross
Copy link
Contributor

Hi @bitjson
Can verify. I have escalated to the team on this connector. Thanks. Will have an update asap.

@bitjson
Copy link
Author

bitjson commented Feb 5, 2025

@seanparkross great thanks! Any insight on why console --local might not be connecting, or what sort of trouble shooting steps I should try?

@BenoitRanque
Copy link
Contributor

@bitjson I found an issue that may or may not be related, can you try adding a .dockerignore file with the following content under app/connector/my_ch?

*.yaml
clickhouse-data
.hasura-connector

Explanation: the clickhouse connector compose file from the tutorial binds data to a directory clickhouse-data
Under .hasura-connector/Dockerfile.my_ch, we can see the configuration is copied into the container when built:

FROM ghcr.io/hasura/ndc-clickhouse:v1.0.5
COPY ./ /etc/connector

I ran into permission issues, because the directory for the database is owned by the database container, but we also try to copy it into the connector container.

The .dockerignore prevents us from copying those files into the container.
The only file we really need to copy into the container is the configuration.json file, and any native query files if they exist.

Please let me know if the above helps. For me it allows running the introspect command without errors

@bitjson
Copy link
Author

bitjson commented Feb 6, 2025

@BenoitRanque no, still receiving the same error. Trying from scratch:

 ddn supergraph init my-project && cd my-project
ddn connector init my_ch -i # select ClickHouse
# Connection string: http://local.hasura.dev:8123
# Username: default_user
# Password: default_password
touch app/connector/my_ch/compose.clickhouse.yaml

compose.clickhouse.yaml:

services:
  clickhouse:
    image: clickhouse/clickhouse-server
    container_name: clickhouse-server
    ports:
      - "8123:8123"
      - "9000:9000"
    volumes:
      - ./clickhouse-data:/var/lib/clickhouse
    environment:
      CLICKHOUSE_USER: "default_user"
      CLICKHOUSE_PASSWORD: "default_password"
      CLICKHOUSE_DB: "default"

New step: Added app/connector/my_ch/.dockerignore:

*.yaml
clickhouse-data
.hasura-connector

Verified the my_ch compose stack didn't exist yet, then ran: docker compose -f app/connector/my_ch/compose.clickhouse.yaml up

Filled DB:

curl -u default_user:default_password -X POST \
    --data "CREATE TABLE users (user_id UInt32, name String, age UInt8) ENGINE = MergeTree() ORDER BY user_id;" \
    http://localhost:8123
curl -u default_user:default_password -X POST \
    --data "CREATE TABLE users (user_id UInt32, name String, age UInt8) ENGINE = MergeTree() ORDER BY user_id;" \
    http://localhost:8123
curl -u default_user:default_password -X POST \
    --data "SELECT * FROM users;" \
    http://localhost:8123

Still seeing the error:

❯ ddn connector introspect my_ch                  
10:30AM INF Using Subgraph config file "app/subgraph.yaml" found in context.
10:30AM INF Using localEnvFile ".env" found in context.
Error: reqwest::Error { kind: Decode, source: Error("invalid type: integer `0`, expected a boolean", line: 52, column: 21) }
10:30AM ERR introspect failed with error: exit status 1

Notably, whatever the issue is with ddn connector introspect my_ch in this context, I'm actually not seeing it when targeting a locally-running ClickHouse cluster, and I've been able to start building out my ddn schema, adding relationships, etc.

I also realized that ddn console --local wasn't connecting due to Brave browser's tracker blocking 🤦‍♂️

Image

You might want to mention excluding http://localhost:3280/ from any browser blocking systems in these getting started guides as a troubleshooting step. (And I opened a separate issue about offline access: #10671)

@seanparkross
Copy link
Contributor

Seems like a change in ClickHouse 25.1 has caused this https://clickhouse.com/docs/en/whats-new/changelog

We're working on a fix. Sorry about the inconvenience!

@BenoitRanque
Copy link
Contributor

@bitjson my bad, able to reproduce after pulling clickhouse/clickhouse-server:latest

Turns out this is caused by a behavior change in clickhouse 25.1. I've created an issue in their repository

While this may be fixed in the future, we'd rather not wait or be dependent on that fix, so we're applying a workaround.

@BenoitRanque
Copy link
Contributor

@bitjson another note: because this is new in clickhouse 25.1, you can workaround the problem by using clickhouse/clickhouse-server:24.12

@bitjson
Copy link
Author

bitjson commented Feb 6, 2025

@seanparkross @BenoitRanque ah, good to know, thanks!

I'm trying to connect to an existing database that's already on 25.1 (so I may have to wait for the workaround) – I'm not really certain if the ClickHouse connector is successfully connecting or if this same issue is causing queries to fail: internal error: ndc_unexpected: ndc_client error: request to connector failed with status code N/A: error decoding response body

Image

Is this failure due to the same breaking change in ClickHouse?

And general new user question: is there some way to get better visibility into what is happening in the connector?

It's logged only one line: 2025-02-06 15:38:07 Starting server on [::]:8080 – can I enable more logging to see if it's successfully connecting to the database and/or log all errors?

@BenoitRanque
Copy link
Contributor

@bitjson We've just released the fix, please update and let us know if it helps :)

That error should only have affected introspection, but the above looks like a runtime error. Can't hurt to try, but I'm expecting that error to persist after the update. Can you confirm that is the case? Have you been able to get local clickhouse to work?

My guess is this may be a connection error, perhaps you need to enable an IP?
Are you able to access the database over http? Something like this (with your actual creds of course):

curl -u default_user:default_password -X POST \
    --data "SELECT 'it works!';" \
    http://localhost:8123

@bitjson
Copy link
Author

bitjson commented Feb 7, 2025

@BenoitRanque yes, the fix works, thanks!

And the runtime issue was a network configuration error, now working as expected. Thanks!

@bitjson bitjson closed this as completed Feb 7, 2025
@bitjson
Copy link
Author

bitjson commented Feb 7, 2025

Actually, I think I was still testing against the v24 database. I'm still seeing the error with the following setup:

  • DDN CLI Version: v2.22.0
  • clickhouse/clickhouse-server@sha256:e9908bce0c245498813965e8b48080a5ebc545de218c897699cb72f70a0f2daa
  • ghcr.io/hasura/v3-engine@sha256:bb9739ae80de4f4b9ff03825c6043a56aac4fc57c06b2174f936199ee0aea532
  • ghcr.io/hasura/ndc-clickhouse:v1.1.0 (sha256:ebcae5e31916fac6ac82a1b38da87f5b106d6e47a1589a249322b537949ef665)
❯ ddn connector introspect clickhouse_connector
3:17PM INF Using Subgraph config file "app/subgraph.yaml" found in context.
3:17PM INF Using localEnvFile ".env" found in context.
Error: reqwest::Error { kind: Decode, source: Error("invalid type: integer `0`, expected a boolean", line: 52, column: 21) }
3:17PM ERR introspect failed with error: exit status 1

@bitjson bitjson reopened this Feb 7, 2025
@BenoitRanque
Copy link
Contributor

@bitjson you probably need to upgrade the connector and connector plugin versions, if you have not created a new project.
Specifically you'll need to upgrade to v1.1.0 in

  • connector.yaml
  • .hasura-connector/Dockerfile.my_ch
  • .hasura-connector/connector-metadata.yaml (there are two here, one being the one you missed, where the cli plugin version is set)

To shed some light on the issue: hasura uses a plugin architecture for connectors, where each connector provides an executable to use for, amongst other things, introspection. That executable version is set in .hasura-connector/connector-metadata.yaml.

I can confirm that on my machine, upgrading the version numbers in those files fixes the issue, so it if persists after that we'll have to keep digging

@bitjson
Copy link
Author

bitjson commented Feb 7, 2025

@BenoitRanque Ah good to know! After ensuring I've updated every other instance of 1.0.5 to 1.0.1 across all config files, I'm now seeing a different error 😅

❯ ddn connector introspect clickhouse_connector
3:47PM INF Using Subgraph config file "app/subgraph.yaml" found in context.
3:47PM INF Using localEnvFile ".env" found in context.
Error: reqwest::Error { kind: Decode, source: Error("invalid unicode code point", line: 1, column: 13692) }
3:47PM ERR introspect failed with error: exit status 1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
k/v3-bug Bug affecting Hasura v3 (DDN)
Projects
None yet
Development

No branches or pull requests

3 participants