tools for accessing the DES data archive at FNAL
Right now this package is in its alpha state and so all installation is directly from the main branch!
Install the package and dependencies directly from GitHub with pip
pip install git+https://github.com/fermitools/htgettoken.git
pip install numpy # you may need this to make sure fitsio installs ok
pip install git+https://github.com/des-science/des-archive-access.gitInstall the dependencies with conda and then install the tool with pip
git clone https://github.com/des-science/des-archive-access.git
cd des-archive-access
conda install --file requirements.txt --file requirements-dev.txt
pip install --no-deps --no-build-isolation .
cd -You will need at least Python 3.8.
In order to download files, you need to generate an OIDC token. To make the token, do the following.
- Run the command line tool
des-archive-access-make-token. - You will be redirected to a CILogon website.
- Login with your FNAL SERVICES account. You may be offered other identity providers, but you have to use the FNAL one!
The token will be stored in the ~/.des_archive_access/ directory in your home area. Make the sure the permissions on this directory are 700 via chmod 700 ~/.des_archive_access/. You can change this location by setting the environment variable DES_ARCHIVE_ACCESS_DIR.
Before you can query the file archive metadata, you need to download the metadata database (roughly 8GB) via the des-archive-access-download-metadata
command. This command will put the data in your home area. If you'd like to specify a different path to the DB, set it via the environment variable DES_ARCHIVE_ACCESS_DB like this
export DES_ARCHIVE_ACCESS_DB=/my/metadata.dbThen you can use the des-archive-access command to interact with the metadata.
$ des-archive-access --help
Usage: des-archive-access [OPTIONS] COMMAND [ARGS]...
DES archive access CLI
Execute `des-archive-access` at the command line to run queries in an
interactive SQL shell.
Alternatively, use the options below to execute queries directly.
Options:
-l, --loadsql TEXT Load a SQL command from a file and execute it.
-c, --command TEXT Load a SQL command from the command line and execute it.
--help Show this message and exit.
Commands:
sql Execute a QUERY.
sqlrepl Alternative way of staring the SQL shell.
$ des-archive-access -c "select band, tilename, ccdnum, filename from y6a2_image limit 10;"
found 10 rows in 0.003719 seconds (2688.656410 rows/s)
BAND TILENAME CCDNUM FILENAME
r 17 D00792065_r_c17_r4056p01_bkg.fits
g 40 D00791804_g_c40_r4056p01_bkg.fits
z 43 D00701336_z_c43_r3518p01_bkg.fits
Y 49 D00701878_Y_c49_r3518p01_bkg.fits
Y 3 D00701839_Y_c03_r3518p01_bkg.fits
Y 7 D00702301_Y_c07_r3518p01_bkg.fits
i 14 D00700055_i_c14_r3517p01_immasked.fits
i 44 D00700055_i_c44_r3517p01_immasked.fits
i 15 D00700055_i_c15_r3517p01_immasked.fits
i 62 D00699602_i_c62_r3517p01_bkg.fitsIf no flag is given, the des-archive-access command loads you into SQL shell. You can execute commands against the database via
$ des-archive-access
> select band, tilename, ccdnum, filename from y6a2_image limit 10; > blah.fits
found 10 rows in 0.015860 seconds (630.504337 rows/s)
> select band, tilename, ccdnum, filename from y6a2_image limit 10;
found 10 rows in 0.000135 seconds (73973.615520 rows/s)
BAND TILENAME CCDNUM FILENAME
r 17 D00792065_r_c17_r4056p01_bkg.fits
g 40 D00791804_g_c40_r4056p01_bkg.fits
z 43 D00701336_z_c43_r3518p01_bkg.fits
Y 49 D00701878_Y_c49_r3518p01_bkg.fits
Y 3 D00701839_Y_c03_r3518p01_bkg.fits
Y 7 D00702301_Y_c07_r3518p01_bkg.fits
i 14 D00700055_i_c14_r3517p01_immasked.fits
i 44 D00700055_i_c44_r3517p01_immasked.fits
i 15 D00700055_i_c15_r3517p01_immasked.fits
i 62 D00699602_i_c62_r3517p01_bkg.fits
> :qdes-archive-access supports writing query results to disk via the same syntax as easyaccess
$ des-archive-access -c "select band, tilename, ccdnum, filename from y6a2_image limit 10; > blah.fits"
found 10 rows in 0.006988 seconds (1431.014671 rows/s)
$ ls blah.fits
blah.fitsThis functionality works in both the SQL shell and at the command line.
You can use the des-archive-access-download command to download files from the archive.
$ des-archive-access-download --help
usage: des-archive-access-download [-h] [-l LIST] [-a ARCHIVE] [-d DESDATA] [-f] [--debug] [--no-refresh-token] [file]
Download files from the DES archive at FNAL.
positional arguments:
file file to download
options:
-h, --help show this help message and exit
-l LIST, --list LIST download all files in a list
-a ARCHIVE, --archive ARCHIVE
HTTPS address of the FNAL archive
-d DESDATA, --desdata DESDATA
The destination DESDATA directory.
-f, --force Force the download even if data already exists
--debug Print the 'curl' command and stderr to help debug connection and download issues.
--no-refresh-token Do not attempt to automatically refresh the OIDC token.
$ des-archive-access-download OPS/finalcut/Y6A1/20181129-r4056/D00797980/p01/red/immask/D00797980_r_c27_r4056p01_immasked.fits.fz
/Users/beckermr/DESDATA/OPS/finalcut/Y6A1/20181129-r4056/D00797980/p01/red/immask/D00797980_r_c27_r4056p01_immasked.fits.fzYou must set the DESDATA environment variable. Files will be downloaded to this location at the same relative path as the location in the archive.
des-archive-accesscurrently only supports writing SQL queries in FITS binary format and in only a single file.- All of the
easyaccessfeatures for introspecting tables and columns are not yet implemented.
You can use the des-archive-access-sync-tile-data command to sync data from NCSA to FNAL. This command downloads all of the relevant Y6 data for a given coadd tile and band to a local DESDATA directory.
$ des-archive-access-sync-tile-data --help
usage: des-archive-access-sync-tile-data [-h] --tilename TILENAME --band BAND [-d DESDATA]
Sync all data for a given coadd tile from NCSA to FNAL.
options:
-h, --help show this help message and exit
--tilename TILENAME tile to process
--band BAND band to process
-d DESDATA, --desdata DESDATA
The destination DESDATA directory.
$ des-archive-access-sync-tile-data --tilename DES2041-5248 --band rTo move data into dcache at FNAL, you need to first setup your local certificate / voms proxy. Then you can use the following commands
$ cd ${DESDATA}
$ ifdh cp -r OPS_Taiga ${PNFS_DES}/DESDM_ARCHIVE/OPS_Taigawhere PNFS_DES is the path to the DES dcache persistent disk storage.