Add KDVH migration package

[Lun4m]

This PR tackles the first part of issue #14.
I adapted some scripts originally written by Ketil for ODA to dump tables from KDVH and import them into LARD. Almost happy with the code...

Things I'm still not 100% sure about:

1. I'm using the KDVH proxy to dump the tables, but I couldn't find T_MONTH_INTERPOLATED and T_DIURNAL_INTERPOLATED, probably I need to connect to KDVH directly?
2. Is blobData used for non-scalar parameters (KLOBS for example)?
3. Do we need to dump observations that have both NULL data and flags? Related to ->
4. in data_functions.go we do a lot of postprocessing that I haven't touched since I don't really understand it. My main concern is with the insertion of "null" values (i.e. -32767, etc.), are these used somewhere downstream? Otherwise I might simply drop them, since our schema can handle real NULLs.

@ketilt, do you have any insight regarding these points? Or comments about the code, if you have time to take a look at it?

Still left to do:

• remove CorrKDVH field, since original and corrected have the same value
• Double check that the derived controlinfo and useinfo are correct

[ketilt]

1. It makes sense that they are not available in the proxy. Do not worry about dumping them. The tables are not in active use by anything. I can make a dump of them.
2. Yes
3. Related to what? There should be no reason to dump nulls. They likely exist only because of the table format of KDVH (empty cells in a row where something is not null)
4. This is a kvalobs feature. The missing indicator exists in two versions: -32767 means expected value missing and -32766 means value was found but was censored in QC. These numbers will not feature in the KDVH data, but they are all over the queues and all over ODA. If you've found a lossless way to handle this differently, it's probably better to do that instead

[ketilt]

Flag definitions:
https://dokit.met.no/klima/bjornn/kvalobs/flagg_i_kvalobs_versjon_8.7.1

[Lun4m]

3. Related to what? There should be no reason to dump nulls.
   They likely exist only because of the table format of KDVH (empty cells in a row where something is not null)

Related to point 4, sorry for the confusion! I was specifically asking since in makeDataPage it's fine if both are null:

lard/kdvh_importer/data_functions.go

Lines 28 to 39 in 648d5f8

	if kdvh.flagsAreInvalid() {
	useinfo = []byte("9999900900000000")
	} else {
	useinfo = []byte(kdvh.Flags + "00900000000")
	}
	if !nullData {
	controlinfo = []byte("0000000000000000")
	} else {
	controlinfo = []byte("0000003000000000")
	floatval = -32767
	}

But I'm with you, it doesn't make sense to dump them 👌

[ketilt]

3. Related to what? There should be no reason to dump nulls.
   They likely exist only because of the table format of KDVH (empty cells in a row where something is not null)

Related to point 4, sorry for the confusion! I was specifically asking since in makeDataPage it's fine if both are null:

lard/kdvh_importer/data_functions.go

Lines 28 to 39 in 648d5f8

	if kdvh.flagsAreInvalid() {
	useinfo = []byte("9999900900000000")
	} else {
	useinfo = []byte(kdvh.Flags + "00900000000")
	}
	if !nullData {
	controlinfo = []byte("0000000000000000")
	} else {
	controlinfo = []byte("0000003000000000")
	floatval = -32767
	}

But I'm with you, it doesn't make sense to dump them 👌

Hmm, could this be in the case of blob data?

[ketilt]

This is where the reverse-engineered flags would be stored? Or the original 5 digits of useinfo that KDVH has kept?

[Lun4m]

Both, the controlinfo is completely reverse-engineered, while the 5-digit KDVH flags are stored in useinfo with the 11 trailing digits set to their default values (line 242 in kdvh/import_functions.go). They should both be following what's defined in the excel document you shared with me.

[ketilt]

Where does this column appear? It might be ship names, but it might also be the initials of the observer for older data

[Lun4m]

T_MDATA (basically the only table I've been testing against). I'll add this info to the code!

[Lun4m]

Now that I think about it, if T_MDATA (or any other table) contains data from moving stations we probably should not store it in the same table we store static stations, since they require additional metadata (position?). But we haven't decided on the table structure yet.

[ketilt]

Non-moving installations like oil platforms also have call signs, and they make up most of that table, I think. We do also have ship data, and there are paramid for MLON and MLAT which contains timeseries of the coordinates of these stations. That is how it was solved in KDVH. So if you see MLAT and MLON, that means there could be ship data in a table. (The table T_CDCV_DATA also contains data from buoys, though I don't remember if they move around a lot.)

[intarga]

I'm not sure I like it either. Would it be better to generate all the rows up front and use CopyFromRows?

[Lun4m]

Mmm, do you mean decoupling LardObs to separate data and flags timeseries, instead of having them together? That sounds reasonable

[intarga]

That's also a good idea, but I meant have methods on the timeseries type that generate [][]any (all the rows at once). Then you could drop the interface and just do pgx.CopyFromRows(ts.new_method()). Not sure what the performance implications would be, depends a bit on Go's runtime and pgx's implementation.

[intarga]

@Lun4m Do you think there would be a point in removing the indices while importing?

I would think it's maybe redundant with COPY, but this doc page suggests it: https://www.postgresql.org/docs/current/populate.html#POPULATE-RM-INDEXES

In any case we should probably run ANALYZE after importing

[Lun4m]

I don't have an opinion about it, but there aren't any drawbacks at the moment, since we don't have users, right? I don't see any reason why we shouldn't do it 👍

On another note, do you think KDVH and Kvalobs flags should be stored in the same table?
Kvalobs has more fields (cfailed and corrected, as seen in the kvdata table).
Just wondering how we plan to handle queries that target imported data.
Is there any use case where someone might want to query old flags?

[intarga]

On another note, do you think KDVH and Kvalobs flags should be stored in the same table?

Hmm, I'm not sure... I think it makes sense to store them in the same table, they would have been migrated to the same table in ODA. It's really a question of what Frost needs, so I would maybe ask Jo if and how Frost handles this. Louise would be the best person to ask if she wasn't on permisjon

[intarga]

@Lun4m Are you filtering open and closed data? I don't remember seeing anything about that when reading the code, but perhaps I missed it

[Lun4m]

Good point, yeah, I forgot I need to filter those out for the time being

[ketilt]

@Lun4m Are you filtering open and closed data? I don't remember seeing anything about that when reading the code, but perhaps I missed it

I can provide a list of which stnr / tables are defined as closed

[intarga]

I can provide a list of which stnr / tables are defined as closed

Thanks, but I don't think that's necessary, we can fetch it from stinfosys