Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DE-6473] Add feature for trino-client helper rest api #95

Open
wants to merge 11 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
314 changes: 314 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -208,3 +208,317 @@ Explore victoriaMetrics go client <https://github.com/VictoriaMetrics/metrics>


Handle routing errors properly instead of returning HTTP500 in all cases, eg: sql transaction request must return HTTP400 instead of HTTP500


# RZP-Trino REST Client

## Overview

This module exposes an API to interact with Trino. It retrieves data using the Trino Go client, processes it, and returns it in JSON format to the client.

### Features:
- **Trino Integration:** Connects to a Trino instance and executes SQL queries.
- **Data Processing:** Applies business logic to the data before sending it to the client.
- **JSON API:** Communicates with clients using JSON format.
- **Request/Response Limitation:** Throws an error if response data exceeds 5,000 records.
- **Configurable:** Uses TOML files for different environments (development, staging, production).

## Base URL

The base URL for the API is: `http://rzp-trino-client:8000`

### 1. `GET /v1/health`

**Description:**
Checks the health of the trino client API service.

**Response:**

```json
{
"status": "string", // Service health status
}
```

### 2. `POST /v1/query`

**Description:**
Executes a SQL query against Trino.

**Request:**

```json
{
"sql": "string" // SQL query to be executed on trino
}
```

**Reponse:**

```json
{
"status": "string", // "Success/ Error/ Running"
"columns": [
{
"name": "string", // Column name
"type": "string" // Column type
}
],
"data": [
[
{
"data": "string" // Data for each cell
}
]
],
"error": {
"message": "string", // Error message (if any)
"errorCode": "integer",// Error code (if any)
"errorName": "string", // Error name (if any)
"errorType": "string" // Error type (if any)
}
}
```

**Reponse(Success example):**

```json
{
"status": "Success",
"columns": [
{
"name": "nationkey",
"type": "BIGINT"
},
{
"name": "name",
"type": "VARCHAR"
},
{
"name": "regionkey",
"type": "BIGINT"
},
{
"name": "comment",
"type": "VARCHAR"
}
],
"data": [
[
{
"data": 0
},
{
"data": "ALGERIA"
},
{
"data": 0
},
{
"data": " haggle. carefully final deposits detect slyly agai"
}
],
[
{
"data": 1
},
{
"data": "ARGENTINA"
},
{
"data": 1
},
{
"data": "al foxes promise slyly according to the regular accounts. bold requests alon"
}
],
[
{
"data": 2
},
{
"data": "BRAZIL"
},
{
"data": 1
},
{
"data": "y alongside of the pending deposits. carefully special packages are about the ironic forges. slyly special "
}
],
[
{
"data": 3
},
{
"data": "CANADA"
},
{
"data": 1
},
{
"data": "eas hang ironic, silent packages. slyly regular packages are furiously over the tithes. fluffily bold"
}
],
[
{
"data": 4
},
{
"data": "EGYPT"
},
{
"data": 4
},
{
"data": "y above the carefully unusual theodolites. final dugouts are quickly across the furiously regular d"
}
],
[
{
"data": 5
},
{
"data": "ETHIOPIA"
},
{
"data": 0
},
{
"data": "ven packages wake quickly. regu"
}
],
[
{
"data": 6
},
{
"data": "FRANCE"
},
{
"data": 3
},
{
"data": "refully final requests. regular, ironi"
}
],
[
{
"data": 7
},
{
"data": "GERMANY"
},
{
"data": 3
},
{
"data": "l platelets. regular accounts x-ray: unusual, regular acco"
}
],
[
{
"data": 8
},
{
"data": "INDIA"
},
{
"data": 2
},
{
"data": "ss excuses cajole slyly across the packages. deposits print aroun"
}
],
[
{
"data": 9
},
{
"data": "INDONESIA"
},
{
"data": 2
},
{
"data": " slyly express asymptotes. regular deposits haggle slyly. carefully ironic hockey players sleep blithely. carefull"
}
]
]
}
```

**Reponse(error example):**

```json
{
"status": "Error",
"error": {
"message": "Unable to query trino: trino: query failed (200 OK): \"USER_ERROR: line 1:31: mismatched input '10'. Expecting: ',', '.', 'AS', 'CROSS', 'EXCEPT', 'FETCH', 'FOR', 'FULL', 'GROUP', 'HAVING', 'INNER', 'INTERSECT', 'JOIN', 'LEFT', 'LIMIT', 'MATCH_RECOGNIZE', 'NATURAL', 'OFFSET', 'ORDER', 'RIGHT', 'TABLESAMPLE', 'UNION', 'WHERE', 'WINDOW', <EOF>, <identifier>\"",
"errorCode": 500,
"errorName": "",
"errorType": ""
}
}
```
## Project Structure

```plaintext
trino-api/
├── build/
│ ├── docker/
│ ├── dev/
│ │ ├── Dockerfile.api
│ ├── entrypoint.sh
├── cmd/
│ ├── api/
│ ├── main.go
├── config/
│ ├── default.toml
│ ├── dev_docker.toml
│ ├── stage.toml
│ ├── prod.toml
├── deployment/
│ ├── dev/
│ ├── docker-compose.yml
├── internal/
│ ├── app/
│ │ ├── handler/
│ │ │ ├── handler.go
│ │ │ ├── handler_test.go
│ │ ├── process/
│ │ │ ├── processor.go
│ │ ├── routes/
│ │ │ ├── routes.go
│ │ ├── app.go
│ │
│ ├── config/
│ │ ├── config.go
│ ├── model/
│ │ ├── model.go
│ ├── utils/
│ │ ├── response.go
│ ├── services/
│ │ ├── trino/
│ │ ├── client.go
│ └── boot/
│ ├── boot.go
├── pkg/
│ ├── config/
│ ├── config.go
│ ├── config_test.go
├── go.mod
├── go.sum
├── Makefile
├── README.md
3 changes: 3 additions & 0 deletions trino-api/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
/bin/
.idea
.vscode
Empty file added trino-api/bin/.gitkeep
Empty file.
41 changes: 41 additions & 0 deletions trino-api/build/docker/dev/Dockerfile.api
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
# Use a lightweight Go image
FROM golang:1.21.5-alpine AS builder

ENV CGO_ENABLED 0

RUN mkdir /src

WORKDIR /src

RUN apk add --update --no-cache --repository https://dl-4.alpinelinux.org/alpine/latest-stable/community/ make

ADD Makefile /src

COPY go.mod .
COPY go.sum .

RUN go mod download

COPY . /src/

RUN make go-build-api

# stage 2

FROM alpine:latest

COPY --from=builder /src/bin/api /app/
COPY --from=builder /src/config/ /app/config/
COPY build/docker/entrypoint.sh /app/

ENV WORKDIR=/app
ENV DUMB_INIT_SETSID=0
WORKDIR /app

RUN apk add --update --no-cache dumb-init su-exec ca-certificates curl

EXPOSE 8000

RUN chmod +x entrypoint.sh

ENTRYPOINT ["./api"]
28 changes: 28 additions & 0 deletions trino-api/build/docker/entrypoint.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
#!/usr/bin/dumb-init /bin/sh
set -euo pipefail

start_application()
{
su-exec appuser $WORKDIR/"$appName" &

# Get pid for app
APP_PID=$!

# wait returns immediately after signal is caught,
# hence double wait is required in shutdown_application
# refer : http://veithen.io/2014/11/16/sigterm-propagation.html
wait "$APP_PID"
}

shutdown_application()
{
kill -s SIGTERM "$APP_PID"
trap - SIGTERM SIGINT
wait "$APP_PID"
EXIT_STATUS=$?
return ${EXIT_STATUS}
}

appName="$1"
trap shutdown_application SIGTERM SIGINT
start_application
Loading
Loading