Data entry via ODM2-Admin, or to populate ODM2 DB used by ODM2-Admin

[emiliom]

@miguelcleon, I forget what mechanism you have developed to load data (and metadata) into the ODM2 database behind ODM2-Admin. I know you preserve sensor log files, so I assume you have a log-file loader for a set of pre-defined sensors. Besides that, does ODM2-Admin have the ability to do manual, form-drive data entry? If so, is it being used heavily? Or do you use your own Excel (or csv) templates as intermediates between miscellaneous data spreadsheets and ODM2? Thanks!

[miguelcleon]

Data can be loaded via command line using: https://github.com/miguelcleon/ODM2-Admin/blob/master/ODM2CZOData/management/commands/ProcessDataLoggerFile.py

our via the web interface using processdata logger file:
https://github.com/miguelcleon/ODM2-Admin/blob/master/ODM2CZOData/models.py#L929

[emiliom]

Are these logger files vendor specific, the output of specific instruments or vendor data logging software? Or are they essentially intermediate files you have defined to follow a set of conventions that makes them ready to load with your tools? From my quick inspection of the code you linked to, the latter seems to be the case. Thanks.

[miguelcleon]

some of the files are directly downloaded from campbell scientific data loggers and processed, other files are intermediate files with modest modifications for ingestion.

[emiliom]

How do you enter metadata like sampling feature details, related features, method descriptions, people and organizations, etc? Through the log files, or through a manual upload into the tables? Thanks.

[miguelcleon]

I have been doing a mixture of manual entry via the web app, Django ORM scripts and SQL scripts.

[miguelcleon]

This file is a sample Django ORM Script https://github.com/miguelcleon/ODM2-Admin/blob/master/ODM2CZOData/NewProcessinglevelForTimeSeries.py

[emiliom]

Thanks! Very helpful.

[emiliom]

• A sample script for loading soil profile data: https://github.com/miguelcleon/ODM2-Admin/blob/master/soilsIngestionExample/ImportMStoneData.py. The csv file that's loaded, EnzBiomass.csv, is in the same folder.
• Note the use of Django "commands" functionality (equivalent of shell scripts, sort of). Can be run at the shell or called by the ODM2-Admin UI.
• Also, use Jupyter notebooks hosted on HydroShare/NCSA JupyterHub to facilitate running and tweaking such scripts by occasional or less capable users. See Miguel's samples on his HydroShare resources. See HydroShare Web Apps and the hydroshare-jupyterhub github repo for general info.

[miguelcleon]

• The sample script doesn't actually use commands, my suggestion is that a new command should be created following the format of this command https://github.com/miguelcleon/ODM2-Admin/blob/master/odm2admin/management/commands/ProcessDataLoggerFile.py
  The sample script does use some helper functions I created which are here: https://github.com/miguelcleon/ODM2-Admin/blob/master/odm2admin/modelHelpers.py

• I created another github repo for Jupyter notebooks I've been working on https://github.com/miguelcleon/ODM2API-ipython-notebooks

[miguelcleon]

The Jupyter notebooks can be run from the hydroshare Jupyter hub http://hydroroger.ncsa.illinois.edu/

[emiliom]

Thanks for the additional info & clarifications. I was just adding information on this dormant "issue", to maintain it as a broadly useful resource on data loading topics in ODM2-Admin.