From b359848c8b09bc7eebdc8deee83c3f057a0f012d Mon Sep 17 00:00:00 2001 From: Gian Merlino Date: Thu, 23 Jan 2025 23:18:34 -0800 Subject: [PATCH] Clarify properties in durable storage docs. Three changes: - Add configs for Google Storage, which was previously missing. - Separate out the configs for the various durable storage types, to make it more clear how each one should be configured. - Replace the blockquote at the top with an infobox, which is more appropriate styling for that particular case. --- docs/operations/durable-storage.md | 45 ++++++++++++++++++++++++------ 1 file changed, 37 insertions(+), 8 deletions(-) diff --git a/docs/operations/durable-storage.md b/docs/operations/durable-storage.md index 6cdbaf9b3c0c..30b075cac638 100644 --- a/docs/operations/durable-storage.md +++ b/docs/operations/durable-storage.md @@ -25,7 +25,9 @@ sidebar_label: "Durable storage" You can use durable storage to improve querying from deep storage and SQL-based ingestion. -> Note that S3, Azure and Google are all supported as durable storage locations. +:::info + S3, Google Storage, and Azure Blob Storage are supported as durable storage locations. +::: Durable storage for queries from deep storage provides a location where you can write the results of deep storage queries to. Durable storage for SQL-based ingestion is used to temporarily house intermediate files, which can improve reliability. @@ -35,21 +37,48 @@ If the configured limit is too low, Druid may throw the error, `NotEnoughTempora ## Enable durable storage -To enable durable storage, you need to set the following common service properties: +To enable durable storage, you need to set the following common service properties. + +For S3: + +``` +druid.msq.intermediate.storage.enable=true +druid.msq.intermediate.storage.type=s3 + +# Remote storage location +druid.msq.intermediate.storage.bucket=YOUR_BUCKET +druid.msq.intermediate.storage.prefix=YOUR_PREFIX + +# Local temporary directory (on each Druid server) +druid.msq.intermediate.storage.tempDir=/path/to/your/temp/dir +``` + +For Google Storage: ``` druid.msq.intermediate.storage.enable=true +druid.msq.intermediate.storage.type=google + +# Remote storage location +druid.msq.intermediate.storage.bucket=YOUR_BUCKET +druid.msq.intermediate.storage.prefix=YOUR_PREFIX + +# Local temporary directory (on each Druid server) druid.msq.intermediate.storage.tempDir=/path/to/your/temp/dir +``` -# Include these configs if you're using S3 -# druid.msq.intermediate.storage.type=s3 -# druid.msq.intermediate.storage.bucket=YOUR_BUCKET +For Azure Blob Storage: -# Include these configs if you're using Azure Blob Storage -# druid.msq.intermediate.storage.type=azure -# druid.sq.intermediate.storage.container=YOUR_CONTAINER +``` +druid.msq.intermediate.storage.enable=true +druid.msq.intermediate.storage.type=azure +# Remote storage location +druid.msq.intermediate.storage.container=YOUR_CONTAINER druid.msq.intermediate.storage.prefix=YOUR_PREFIX + +# Local temporary directory (on each Druid server) +druid.msq.intermediate.storage.tempDir=/path/to/your/temp/dir ``` For detailed information about these and additional settings related to durable storage, see [Durable storage configurations](../multi-stage-query/reference.md#durable-storage-configurations).