Groomed documentation (#88)

Author: alxmrs

Configuration.md

@@ -268,10 +268,3 @@ to discover the kinds of requests that can be made.
 
 > **NOTE**: MARS data is stored on tape drives. It takes longer for multiple workers to request data than a single
 > worker. Thus, it's recommended _not_ to set a partition key when writing MARS data configurations.
-
-## Writing Efficient Data Requests
-
-TODO([#26](https://github.com/googlestaging/weather-tools/issues/26)). In the mean-time, please consult this ECMWF 
-documentation: 
-* [Web API Retrieval Efficiency](https://confluence.ecmwf.int/display/WEBAPI/Retrieval+efficiency)
-* [Era 5 daily data retrieval efficiency](https://confluence.ecmwf.int/display/WEBAPI/ERA5+daily+retrieval+efficiency)

Efficient-Requests.md

@@ -0,0 +1,6 @@
+# Writing Efficient Data Requests
+
+TODO([#26](https://github.com/googlestaging/weather-tools/issues/26)). In the mean-time, please consult this ECMWF
+documentation:
+* [Web API Retrieval Efficiency](https://confluence.ecmwf.int/display/WEBAPI/Retrieval+efficiency)
+* [Era 5 daily data retrieval efficiency](https://confluence.ecmwf.int/display/WEBAPI/ERA5+daily+retrieval+efficiency)

README.md

@@ -13,7 +13,7 @@ across Alphabet.
 The first tool created was the weather downloader (`weather-dl`). This makes it easier to ingest data from the European
 Center for Medium Range Forecasts (ECMWF). `weather-dl` enables users to describe very specifically what data they'd
 like to ingest from ECMWF's catalogs. It also offers them control over how to parallelize requests, empowering users to
-[retrieve data efficiently](Configuration.html#writing-efficient-data-requests). Downloads are driven from a
+[retrieve data efficiently](Efficient-Requests.md). Downloads are driven from a
 [configuration file](Configuration.md), which can be reviewed (and version-controlled) independently of pipeline or
 analysis code.
 
@@ -92,6 +92,7 @@ _Steps_:
    ```shell
    weather-mv --uris "./local_run/**.nc" \ # or  --uris "./split_data/**.nc" \
       --output_table "$PROJECT.$DATASET_ID.$TABLE_ID" \
+      --temp_location "gs://$BUCKET/tmp" \  # Needed for batch writes to BigQuery
       --direct_num_workers 2
    ```
 
@@ -111,6 +112,8 @@ our [guide](CONTRIBUTING.md) to get started.
 
 ## License
 
+This is not an official Google product.
+
 ```
 Copyright 2021 Google LLC
 

docs/Efficient-Requests.md

@@ -0,0 +1 @@
+../Efficient-Requests.md

docs/_static/custom.css

@@ -1,3 +1,11 @@
 p {
     text-align: justify;
-}
+}
+
+body {
+   min-width: 250px;
+}
+
+div.body {
+    min-width: 250px;
+}

docs/index.md

@@ -16,6 +16,7 @@
    weather_sp/README
    Runners
    Configuration
+   Efficient-Requests
    CONTRIBUTING
    modules
 ```

weather_dl/README.md

@@ -35,11 +35,13 @@ _Common options_:
 
 * `-f, --force-download`: Force redownload of partitions that were previously downloaded.
 * `-d, --dry-run`: Run pipeline steps without _actually_ downloading or writing to cloud storage.
-* `-m, --manifest-location MANIFEST_LOCATION`:  Location of the manifest. Either a Firestore collection URI
-  ('fs://<my-collection>?projectId=<my-project-id>'), a GCS bucket URI, or 'noop://<name>' for an in-memory location.
 * `-l, --local-run`: Run locally and download to local hard drive. The data and manifest directory is set by default
   to '<$CWD>/local_run'. The runner will be set to `DirectRunner`. The only other relevant option is the config
   and `--direct_num_workers`
+* `-m, --manifest-location MANIFEST_LOCATION`:  Location of the manifest. Either a Firestore collection URI
+  ('fs://<my-collection>?projectId=<my-project-id>'), a GCS bucket URI, or 'noop://<name>' for an in-memory location.
+* `-n, --num-requests-per-key`: Number of concurrent requests to make per API key. Default: make an educated guess per
+  client & config. Please see the client documentation for more details.
 
 Invoke with `-h` or `--help` to see the full range of options.
 
@@ -67,6 +69,17 @@ weather-dl configs/mars_example_config.cfg \
            --job_name $JOB_NAME
 ```
 
+Using the DataflowRunner and specifying 3 requests per license
+
+```bash
+weather-dl configs/mars_example_config.cfg \
+           -n 3 \
+           --runner DataflowRunner \
+           --project $PROJECT \
+           --temp_location gs://$BUCKET/tmp  \
+           --job_name $JOB_NAME
+```
+
 For a full list of how to configure the Dataflow pipeline, please review
 [this table](https://cloud.google.com/dataflow/docs/reference/pipeline-options).
 

weather_mv/README.md

@@ -11,9 +11,6 @@ Weather Mover loads weather data from cloud storage into [Google BigQuery](https
 * **Parallel Upload**: Each file will be processed in parallel. With Dataflow autoscaling, even large datasets can be
   processed in a reasonable amount of time.
 
-> Note: Data is written into BigQuery using streaming inserts. It may take [up to 90 minutes](https://cloud.google.com/bigquery/streaming-data-into-bigquery#dataavailability)
-> for buffers to persist into storage. However, weather data will be available for querying immediately.
-
 ## Usage
 
 ```
@@ -48,6 +45,7 @@ _Usage Examples_:
 ```bash
 weather-mv --uris "gs://your-bucket/*.nc" \
            --output_table $PROJECT.$DATASET_ID.$TABLE_ID \
+           --temp_location "gs://$BUCKET/tmp" \  # Needed for batch writes to BigQuery
            --direct_num_workers 2
 ```
 
@@ -57,6 +55,7 @@ Upload only a subset of variables:
 weather-mv --uris "gs://your-bucket/*.nc" \
            --output_table $PROJECT.$DATASET_ID.$TABLE_ID \
            --variables u10 v10 t
+           --temp_location "gs://$BUCKET/tmp" \
            --direct_num_workers 2
 ```
 
@@ -66,6 +65,7 @@ Upload all variables, but for a specific geographic region (for example, the con
 weather-mv --uris "gs://your-bucket/*.nc" \
            --output_table $PROJECT.$DATASET_ID.$TABLE_ID \
            --area 49.34 -124.68 24.74 -66.95 \
+           --temp_location "gs://$BUCKET/tmp" \
            --direct_num_workers 2
 ```
 
@@ -77,7 +77,7 @@ weather-mv --uris "gs://your-bucket/*.nc" \
            --runner DataflowRunner \
            --project $PROJECT \
            --region  $REGION \
-           --temp_location gs://$BUCKET/tmp \
+           --temp_location "gs://$BUCKET/tmp" \
            --job_name $JOB_NAME 
 ```
 
@@ -93,6 +93,10 @@ streaming ingestion, use the `--topic` flag (see above). Objects that don't matc
 ingestion. It's worth noting: when setting up PubSub, **make sure to create a topic for GCS `OBJECT_FINALIZE` events
 only.**
 
+Data is written into BigQuery using streaming inserts. It may
+take [up to 90 minutes](https://cloud.google.com/bigquery/streaming-data-into-bigquery#dataavailability)
+for buffers to persist into storage. However, weather data will be available for querying immediately.
+
 > Note: It's recommended that you specify variables to ingest (`-v, --variables`) instead of inferring the schema for
 > streaming pipelines. Not all variables will be distributed with every file, especially when they are in Grib format.