secrets.md

Secrets

Learn more about the LangStream project here.

Secrets

A place to hold secrets. Each label:value in this file is used as a reference in configuration and pipeline manifests. Their values carry on to a step’s environment where it is applied. Secret values can be modified directly in secrets.yaml, or you can pass secrets as environment variables.

Manifest

An example secrets.yaml manifest contains the credentials necessary to connect to Astra and OpenAI.

The :- characters designate a default value. For example, provider: "${OPEN_AI_PROVIDER:-openai}" designates openai as the default.

For finding these credentials, see Credentials.

secrets:
  - id: astra
    data:
      clientId: ${ASTRA_CLIENT_ID:-}
      secret: ${ASTRA_SECRET:-}
      token: ${ASTRA_TOKEN:-}
      database: ${ASTRA_DATABASE:-}
      # uncomment this and link to a file containing the secure connect bundle
      # secureBundle: "<file:secure-connect-bundle.zip>"
      secureBundle: ${ASTRA_SECURE_BUNDLE:-}
      environment: ${ASTRA_ENVIRONMENT:-PROD}
  - id: open-ai
    data:
      access-key: "${OPEN_AI_ACCESS_KEY:-}"
      url: "${OPEN_AI_URL:-}"
      provider: "${OPEN_AI_PROVIDER:-openai}"
      embeddings-model: "${OPEN_AI_EMBEDDINGS_MODEL:-text-embedding-ada-002}"
      chat-completions-model: "${OPEN_AI_CHAT_COMPLETIONS_MODEL:-gpt-3.5-turbo}"
      text-completions-model: "${OPEN_AI_TEXT_COMPLETIONS_MODEL:-gpt-3.5-turbo-instruct}"
 

Pass secrets as environment variables

Secret values can be modified directly in secrets.yaml, or you can pass your secrets as environment variables. The secrets.yaml resolves these environment variables.

export ASTRA_CLIENT_ID=...
export ASTRA_SECRET=...
export ASTRA_DATABASE=...
export ASTRA_TOKEN=...

When you go to production, you should create a dedicated secrets.yaml file for each environment.

Credentials

Where do you find credentials for these items? Here's a little help:

Secret	Location	Notes and Example Value
kafka
username	ssl.properties	KAFKA_USERNAME=langstream-tenant
password	ssl.properties	KAFKA_PASSWORD=token:eyXxx...
tenant	ssl.properties	KAFKA_USERNAME=langstream-tenant
bootstrap.servers	ssl.properties	KAFKA_BOOTSTRAP_SERVERS=kafka-gcp-useast1.streaming.datastax.com:9093
open-ai
access-key	OpenAI Access Key	access-key: xxx
url	OpenAI Azure URL	OPEN_AI_URL=https://company-openai-dev.openai.azure.com/
provider	OpenAI Azure	OPEN_AI_PROVIDER=openai
embeddings-model	OpenAI Azure	OPEN_AI_EMBEDDINGS_MODEL=text-embedding-ada-002 export
chat-completions-model	OpenAI Azure	OPEN_AI_CHAT_COMPLETIONS_MODEL=gpt-35-turbo
text-completions-model	OpenAI Azure	OPEN_AI_TEXT_COMPLETIONS_MODEL=gpt-3.5-turbo-instruct
vertex-ai
url	Google Service Account	VERTEX_AI_URL=https://us-central1-aiplatform.googleapis.com
token	Vertex API token	VERTEX_AI_TOKEN=xxx
serviceAccountJSON	A JSON file downloaded from the Google console containing auth info.	VERTEX_AI_JSON=xxx
region	Google Service Account	VERTEX_AI_REGION=us-central1
project	Google Service Account	VERTEX_AI_PROJECT=myproject
chat-completions-model	Google Service Account	VERTEX_AI_CHAT_COMPLETIONS_MODEL=chat-bison
text-completions-model	Google Service Account	VERTEX_AI_TEXT_COMPLETIONS_MODEL=text-bison
hugging-face
access-key	hugging-face	access-key:
provider	Can be api or local	HUGGING_FACE_PROVIDER=api
embeddings-model	hugging-face	HUGGING_FACE_EMBEDDINGS_MODEL=multilingual-e5-small
embeddings-model-url	hugging-face	HUGGING_FACE_EMBEDDINGS_MODEL_URL=djl://ai.djl.huggingface.pytorch/intfloat/multilingual-e5-small
astra
clientID	Astra	ASTRA_CLIENT_ID=fnsNZtMgvgBHurHJjfSbgQwifnsNZtMgvgBHurHJjfSbgQwi ClientID is generated with token
secret	Astra	ASTRA_SECRET=xxxx Secret is generated with token
token	Astra	ASTRA_TOKEN=AstraCSxxxx
database	Astra	ASTRA_DATABASE=my-database The name of your Astra database
secureBundle	Base64-encoded secure connect bundle downloaded from Astra	ASTRA_SECURE_BUNDLE="file:secure-connect-bundle.zip"
environment	Astra	ASTRA_ENVIRONMENT=PROD
s3
bucket-name	Minio console	S3_BUCKET_NAME=langstream-code-storage
endpoint	Minio console	S3_ENDPOINT=http://minio.minio-dev.svc.cluster.local:9000
access-key	Minio console	S3_ACCESS_KEY=minioadmin
secret	Minio console	S3_SECRET=minioadmin
region	Minio console	S3_REGION=us-central1
google
client-id	Google Service Account	client-id: xxxx
github
client-id	Github	client-id: xxxx
pinecone
service	Pinecone console	PINECONE_SERVICE=pinecone
access-key	Pinecone console	PINECONE_ACCESS_KEY=xxxx
project-name	Pinecone console	PINECONE_PROJECT_NAME=b4ea705
environment	Pinecone console	PINECONE_ENVIRONMENT=asia-southeast1-gcp-free
index-name	Pinecone console	PINECONE_INDEX_NAME=my-pinecone-index

Please note that the example values provided are taken from the current content and may not accurately reflect the actual values that should be used for each secret.

Root	Node	Type	Description
secrets			The base node in the yaml, Holds the collection of secrets.
	name		The secret name used for display
	id		The id of the secret used for referencing its value
	data	<any key:value>	Object of applicable values, given the secret. Provide any combination of key:value that is applicable to the given secret. To retrieve the values use the format - secrets.<name>.<key> (don't include "data" when referencing secrets)