|
| 1 | + |
| 2 | +## `dhusget` script |
| 3 | + |
| 4 | +`dhusget.sh` is a simple demo script illustrating how to use OData and OpenSearch APIs to query and download the products from any Data Hub Service. It allows: |
| 5 | + |
| 6 | +1. Search products over a pre-defined AOI |
| 7 | +2. Filter the products by ingestion time, sensing time |
| 8 | +3. Filter the products by mission (Sentinel-1, Sentinel-2, Sentinel-3), instrument and product type |
| 9 | +4. Save the list of results in CSV and XML files |
| 10 | +5. Download the products |
| 11 | +6. Download the manifest files only |
| 12 | +7. Perform the MD5 integrity check of the downloaded products |
| 13 | +8. Order and Download offline products from the Long Term Archive (LTA) |
| 14 | + |
| 15 | + |
| 16 | +It requires the installation of `wget`. |
| 17 | + |
| 18 | +Usage: |
| 19 | + |
| 20 | +#### dhusget.sh [LOGIN OPTIONS]... [SEARCH QUERY OPTIONS]... [SEARCH RESULT OPTIONS]... [DOWNLOAD OPTIONS]... |
| 21 | + |
| 22 | +LOGIN OPTIONS: |
| 23 | + |
| 24 | +- `-d <DHuS URL>` : URL of the Data Hub Service to be polled |
| 25 | +- `-u <username>` : data hub username |
| 26 | +- `-p <password>` : data hub password provided after registration; |
| 27 | + |
| 28 | +SEARCH QUERY OPTIONS: |
| 29 | + |
| 30 | +- `-m <mission name>` : sentinel mission name |
| 31 | +- `-i <instrument name>` : instrument name |
| 32 | +- `-t <time in hours>` : search for products ingested in the last `<time in hours>` (integer) from the time of execution of the script. (e.g. '=-t 24=' to search for products ingested in the last 24 Hours) |
| 33 | +- `-s <ingestion_date_FROM>` : search for products ingested after the date and time specified by `<ingestion_date_FROM>`. The date format is `ISO 8601:YYYY-MM-DDThh:mm:ss.cccZ` (e.g. `-s 2016-10-02T06:00:00.000Z`) |
| 34 | +- `-e <ingestion_date_TO>` : search for products ingested before the date specified by `<ingestion_date_TO>`. The date format is `ISO 8601: YYYY-MM-DDThh:mm:ss.cccZ` (e.g. `-e 2016-10-10T12:00:00.000Z`) |
| 35 | +- `-S <sensing_date_FROM>` : search for products with sensing date greater than the date and time specified by `<sensing_date_FROM>`. The date format is ISO 8601: `YYYY-MM-DDThh:mm:ss.cccZ` (e.g. `-S 2016-10-02T06:00:00.000Z`) |
| 36 | +- `-E <sensing_date_TO>` : search for products with sensing date less than the date and time specified by `<sensing_date_TO>`. The date format is ISO 8601: `YYYY-MM-DDThh:mm:ss.cccZ` (e.g. `-E 2016-10-10T12:00:00.000Z`) |
| 37 | +- `-f <file>` : search for products ingested after the date and time provided through the input `<file>`. The file is updated at the end of the script execution with the ingestion date of the last successful downloaded product. |
| 38 | +- `-c <coordinates i.e.: lon1,lat1:lon2,lat2>` : coordinates of two opposite vertices of the rectangular area of interest |
| 39 | +- `-T <product type>` : product type of the product to search (available values are: SLC, GRD, OCN, RAW, S2MSI1C) |
| 40 | +- `-F <free OpenSearch query>` : free text OpenSearch query. The query must be written enclosed by single apexes `'<query>'` (e.g. `-F 'platformname:Sentinel-1 AND producttype:SLC'` ). Note: the free text OpenSearch query can be combined with the other possible specified search options |
| 41 | + |
| 42 | +SEARCH RESULT OPTIONS: |
| 43 | + |
| 44 | +- `-l <results>` : maximum number of results per page [1,2,3,4,..]; default value = 25 |
| 45 | + |
| 46 | +- `-P <page>` : page number [1,2,3,4,..]; default value = 1 |
| 47 | + |
| 48 | +- `-q <XMLfile>` : write the OpenSearch query results in a specified XML file. Default file is './OSquery-result.xml' |
| 49 | + |
| 50 | +- `-C <CSVfile>` : write the list of product results in a specified CSV file. Default file is './products-list.csv' |
| 51 | + |
| 52 | +DOWNLOAD OPTIONS: |
| 53 | + |
| 54 | +- `-o <option>` : what to download; possible options are: |
| 55 | + - `'manifest'` to download the manifest of all products returned from the search or |
| 56 | + - `'product'` to download all products returned from the search |
| 57 | + - `'all'` to download both |
| 58 | + |
| 59 | + |
| 60 | +- `-O <folder>` : save the Product ZIP files in a specified folder |
| 61 | +- `-N <1...n>` : set number of wget download retries. Default value is 5. Fatal errors like 'connection refused' or 'not found' (404) are not retried |
| 62 | +- `-R <file>` : write in the specified file the list of products that have failed the MD5 integrity check. By default the list is written in `./failed_MD5_check_list.txt.` The format of the output file is compatible with option -r |
| 63 | +- `-D` : if specified, remove the products that have failed the MD5 integrity check from disk. By deafult products are not removed |
| 64 | +- `-r <file>` : download the products listed in an input file written according to the following format: |
| 65 | + - one product per line |
| 66 | + - <space><one character><space><UUID><space><one character><space><filename> |
| 67 | + |
| 68 | +- `-L <lock folder>` : by default only one instance of dhusget can be executed at a time. This is ensured by the creation of a temporary lock folder `$HOME/dhusget_tmp/lock` which is removed a the end of each run. In order to run more than one dhusget instance at a time is sufficient to assign different lock folders using the -L option (e.g. '-L foldername') to each dhusget instance |
| 69 | +- `-n <1...n>` : number of concurrent downloads (either products or manifest files). Default value is 2; this value doesn't override the quota limit set on the server side for the user. |
| 70 | +- `-w <1...n>` : minutes to wait until retrying the download of an offline product. Default value is 10 |
| 71 | +- `-W <1...n>` : number of download retries for offline products. Default value is 40 |
| 72 | + |
| 73 | +**NOTE:** dhusget functionality is guaranteed in Linux environment. Linux emulators can cause failures. |
0 commit comments