Error in creating the climatology

[Sougata18]

I am trying to generate the ic,bc and clim files using the example_config.jl. My model domain falls within the Aranbian sea region. Now when I try to execute the example_config.jl ( name changed to arabiansea_config.jl, as mentioned in the tutorial ), it gives me an error:

syntax error, unexpected $end, expecting SCAN_ATTR or SCAN_DATASET or SCAN_ERROR
context: ^
ERROR: LoadError: NetCDF error: Opening path https://raslekar:[email protected]/thredds/dodsC/cmems_mod_glo_phy_anfc_0.083deg_P1D-m: NetCDF: Malformed or inaccessible DAP2 DDS or DAP4 DMR response (NetCDF error code: -72)
Stacktrace:
  [1] nc_open(path::String, mode::UInt16)
    @ NCDatasets ~/.julia/packages/NCDatasets/ivzVX/src/netcdf_c.jl:274
  [2] NCDatasets.NCDataset(filename::String, mode::String; format::Symbol, share::Bool, diskless::Bool, persist::Bool, memory::Nothing, maskingvalue::Missing, attrib::Vector{Any})
    @ NCDatasets ~/.julia/packages/NCDatasets/ivzVX/src/dataset.jl:211
  [3] NCDataset (repeats 2 times)
    @ ~/.julia/packages/NCDatasets/ivzVX/src/dataset.jl:170 [inlined]
  [4] download(dsopendap::ROMS.OPENDAP{NCDatasets.NCDataset}, variablename::Symbol; longitude::Tuple{Float64, Float64}, latitude::Tuple{Float64, Float64}, time::Vector{DateTime})
    @ ROMS ~/.julia/dev/ROMS/src/opendap.jl:32
  [5] load(dsopendap::ROMS.OPENDAP{NCDatasets.NCDataset}, variablename::Symbol; kwargs::Base.Pairs{Symbol, Any, Tuple{Symbol, Symbol, Symbol}, NamedTuple{(:time, :longitude, :latitude), Tuple{Vector{DateTime}, Tuple{Float64, Float64}, Tuple{Float64, Float64}}}})
    @ ROMS ~/.julia/dev/ROMS/src/opendap.jl:97
  [6] #load#39
    @ ~/.julia/dev/ROMS/src/CMEMS.jl:9 [inlined]
  [7] interp_clim(domain::ROMS.Grid{Float64, BitMatrix}, clim_filename::String, dataset::ROMS.OPENDAP{NCDatasets.NCDataset}, timerange::Vector{DateTime}; padding::Float64, missing_value::Float64)
    @ ROMS ~/.julia/dev/ROMS/src/interp_clim.jl:33
  [8] interp_clim(domain::ROMS.Grid{Float64, BitMatrix}, clim_filename::String, dataset::ROMS.OPENDAP{NCDatasets.NCDataset}, timerange::Vector{DateTime})
    @ ROMS ~/.julia/dev/ROMS/src/interp_clim.jl:10
  [9] top-level scope
    @ ~/ROMS.jl/test/arabiansea_config.jl:128
 [10] include(fname::String)
    @ Base.MainInclude ./client.jl:476
 [11] top-level scope
    @ REPL[7]:1
in expression starting at /home/dass/ROMS.jl/test/arabiansea_config.jl:128

Could you please guide me what changes I might need to do for a domain outside Mediterranean Sea.

Below is the code which I am using for the ic, bc and clim file generation:

---

using ROMS
using Downloads: download

# name of the domain
domain_name = "arabiansea";


#bathymetry
# longitude from 40°E to 70°E and latitude from 5°S to 37°N
bath_name = expanduser("~/ROMS.jl_data/bath/gebco_30sec_1.nc")

#if !isfile(bath_name)
#    mkpath(dirname(bath_name))
#    download("https://dox.ulg.ac.be/index.php/s/piwSaFP3nhM8jSD/download",bath_name)
#end

# range of longitude
xr = [40, 70]

# range of latitude
yr = [-5, 37]

# reduce bathymetry in x and y direction
red = (4, 4)

# maximum normalized topographic variations
rmax = 0.4;

# minimal depth
hmin = 7; # m

# name of folders and files

# grid file
modeldir = expanduser("~/ROMS.jl_data/grd") #("~/ROMS-implementation-test")

# This file corresponds to GRDNAME in roms.in
grid_fname = joinpath(modeldir,domain_name * ".nc")

basedir = expanduser("~/ROMS.jl_data")

# GCM interpolated on model grid (CLMNAME)
clim_filename =  joinpath(basedir,"arabiansea_clim2020.nc")

# initial conditions (ININAME in roms.in)
ic_filename =  joinpath(basedir,"arabiansea_ic2020.nc")

# boundary conditions (BRYNAME in roms.in)
bc_filename =  joinpath(basedir,"arabiansea_bc2020.nc")

# model specific parameters
opt = (
    Tcline = 20,   # m
    theta_s = 8.0,   # surface refinement
    theta_b = 4.0, # bottom refinement
    nlevels = 15,  # number of vertical levels
    Vtransform  = 2,
    Vstretching = 4,
)

# ECMWF from 2018-12-01 to 2020-01-01 is available at
# https://dox.ulg.ac.be/index.php/s/tbzNV9Z9UPtG5et/download
#ecmwf_fname = expanduser("~/Data/Atmosphere/ecmwf_operational_archive_2018-12-01T00:00:00_2020-01-01T00:00:00.nc")

# ECMWF from 2020-05-31 00:00:00  to 2020-06-29 23:00:00
ecmwf_fname = expanduser("~/ROMS.jl_data/atm_forcing/adaptor.mars.internal-1707969756.9824717-7220-13-9dcca4c7-a0e6-446d-8c15-42d95b28b7d4.nc")

#if !isfile(ecmwf_fname)
#    mkpath(dirname(ecmwf_fname))
#    download("https://dox.ulg.ac.be/index.php/s/8NJsCfk53fDFtbz/download",ecmwf_fname)
#end


# CMEMS credentials
# Replace ENV["CMEMS_USERNAME"] by "your_username" and likewise ENV["CMEMS_PASSWORD"]

cmems_username = "raslekar"
cmems_password = "Animeislife1999"

# change time range
# t0 start time
# t1 end time

t0 = DateTime(2020,5,31);
t1 = DateTime(2020,6,29);

# setup dir

mkpath(basedir);
mkpath(modeldir);

ROMS.generate_grid(grid_fname,bath_name,xr,yr,red,opt,hmin,rmax);

mkpath(basedir);
domain = ROMS.Grid(grid_fname,opt);

@info "domain size $(size(domain.mask))"

outdir = joinpath(basedir,"OGCM")
mkpath(outdir)

# Locate the dataset at https://marine.copernicus.eu/

# Example:
# https://resources.marine.copernicus.eu/product-detail/MEDSEA_MULTIYEAR_PHY_006_004/INFORMATION
# Then go to:
# Data access -> select data set -> Download options -> Subsetter -> View Script

service_id = "GLOBAL_MULTIYEAR_PHY_001_030" #"MEDSEA_MULTIYEAR_PHY_006_004-TDS"
motu_server = "https://my.cmems-du.eu/motu-web/Motu"

mapping = Dict(
    # var  product_id
    :sea_surface_height_above_geoid => ("zos","cmems_mod_glo_phy_anfc_0.083deg_P1D-m"),
    :sea_water_potential_temperature => ("thetao", "cmems_mod_glo_phy-thetao_anfc_0.083deg_PT6H-i"),
    :sea_water_salinity => ("so","cmems_mod_glo_phy-so_anfc_0.083deg_PT6H-i"),
    :eastward_sea_water_velocity => ("uo", "cmems_mod_glo_phy-cur_anfc_0.083deg_PT6H-i"),
    :northward_sea_water_velocity => ("vo", "cmems_mod_glo_phy-cur_anfc_0.083deg_PT6H-i"),
)

dataset = ROMS.CMEMS_opendap(cmems_username,cmems_password,mapping,outdir)


# take one extra day
#tr = [t0-Dates.Day(1), t1+Dates.Day(1)]
tr = [DateTime(2020,5,30), DateTime(2020,6,30)]
ROMS.interp_clim(domain,clim_filename,dataset,tr)

ROMS.extract_ic(domain,clim_filename,ic_filename, t0);
ROMS.extract_bc(domain,clim_filename,bc_filename)

# Prepare atmospheric forcings

filename_prefix = joinpath(basedir,"arabiansea2020_")
domain_name = "Arabian Sea Region"
Vnames = ["sustr","svstr","shflux","swflux","swrad","Uwind","Vwind",
    "lwrad","lwrad_down","latent","sensible","cloud","rain","Pair","Tair","Qair"]

# forcing_filenames corresponds to FRCNAME in roms.in
forcing_filenames = ROMS.prepare_ecmwf(ecmwf_fname,Vnames,filename_prefix,domain_name)


# nudging coefficient

tscale = 7; # days
alpha = 0.3;
halo = 2;
Niter = 50
max_tscale = 5e5

nudge_filename = joinpath(basedir,"roms_nud_$(tscale)_$(Niter).nc")
tracer_NudgeCoef = ROMS.nudgecoef(domain,nudge_filename,alpha,Niter,
          halo,tscale; max_tscale = max_tscale)

Thank you :)

[Alexander-Barth]

Have you seen the instructions here?

https://alexander-barth.github.io/ROMS.jl/dev/#Generate-initial,-boundary-conditions-and-forcing-files

[Sougata18]

Thanks!
I have checked the following and updated -

service_id = GLOBAL_MULTIYEAR_PHY_001_030

mapping = Dict(
   # var  product_id
   :sea_surface_height_above_geoid => ("zos","cmems_mod_glo_phy_my_0.083deg_P1D-m"),
   :sea_water_potential_temperature => ("thetao", "cmems_mod_glo_phy_my_0.083deg_P1D-m"),
   :sea_water_salinity => ("so","cmems_mod_glo_phy_my_0.083deg_P1D-m"),
   :eastward_sea_water_velocity => ("uo", "cmems_mod_glo_phy_my_0.083deg_P1D-m"),
   :northward_sea_water_velocity => ("vo", "cmems_mod_glo_phy_my_0.083deg_P1D-m"),
) 

but I'm not sure whether the motu_server = "https://my.cmems-du.eu/motu-web/Motu" is correct or not.

Also I am noticing this on the site:

In such case, could you please tell me how can I set the motu_server for a region like Arabian sea ?

[Alexander-Barth]

Can you download the data via the python "Copernicus Marine Toolbox"? (That would be a first step).

Also does this approach work for the dataset in question?
https://github.com/JuliaGeo/ZarrDatasets.jl

[Sougata18]

Can you download the data via the python "Copernicus Marine Toolbox"? (That would be a first step).

Yes I was able to successfully download the data using "Copernicus Marine Toolbox". But how can I use the directly downloaded dataset for the function interp_clim in ROMS.jl ?

---

Also does this approach work for the dataset in question?
https://github.com/JuliaGeo/ZarrDatasets.jl

When I trying with the ZarrDatasets package, with the dataset of my interest it is showing the zarr url as well as the dataset details (ds) correctly.
But when I am trying to download a nc version of the subset
( this part of the example script -

NCDataset("$(dataset_id)_selection.nc","c") do ds_nc
    write(ds_nc,ds_sub)

it is showing something like this -
ERROR: LoadError: InexactError: trunc(Int16, -1.07367629e9)'

The dataset (med-cmcc-ssh-rean-d) used in your example https://github.com/JuliaGeo/ZarrDatasets.jl.git is a single variable dataset. But the one I am trying to download is a multi variable dataset (cmems_mod_glo_phy_my_0.083deg_P1D-m). I am wondering if the error has something to do with that ?

Thanks!

[Sougata18]

Could you find anything regarding this ?

[Alexander-Barth]

There is now a function ROMS.CMEMS_zarr which can use the new CMEMS data services

ROMS.jl/test/example_config.jl

Lines 103 to 112 in 26ec58a

	mapping = Dict(
	# var dataset_id
	:sea_surface_height_above_geoid => "med-cmcc-ssh-rean-d",
	:sea_water_potential_temperature => "med-cmcc-tem-rean-d",
	:sea_water_salinity => "med-cmcc-sal-rean-d",
	:eastward_sea_water_velocity => "med-cmcc-cur-rean-d",
	:northward_sea_water_velocity => "med-cmcc-cur-rean-d",
	)
	dataset = ROMS.CMEMS_zarr(product_id,mapping,outdir, time_shift = 12*60*60)

(The function ROMS.CMEMS_opendap has been removed, as CMEMS stopped their opendap server)