Prepare new release, 2.0.20-alpha (or 2.0.20-beta?)

[emiliom]

To-do's before issuing release, ideally:

1. @lsetiawan and @emiliom need to fix problem (SQLite doesn't work in a multiple server with PostgreSQL #127) with running multiple instances of wofpy that include SQLite and Postgresql backends at the same time. Unless it gets way too complicated, then we'll leave it for a future release! (the combinations MySQL+SQLite and MySQL+PostgreSQL do work). Will table until after this release. See my comment on #127
2. @ocefpaf completes enhancements of wof/wofpy_config.py, as described in #72: *wofpy_config.py will take two additional arguments (it already takes the copy-to file path): the "example" to be copied; and the deployment type (test vs "production" or test vs "server"). This also involves copying files into a "base" folder that could be hard-wired to be named wofpy, or be user-selectable (@lsetiawan can explain). Being addressed in Enhancement to wof/wofpy_config.py #132 (but not completed yet)
3. @lsetiawan and @emiliom will try to address/fix odm2-timeseries-dao problems with affiliations/organization; at a minimum, to ensure the EnviroDIY database in its original form (allowing for Null affiliations.organizationid entries) doesn't result in errors, even if we don't manage to fully fix the dao to capture and return Sources (organization, contact) information. Is fixed by Update DAO and WOF Timeseries Models to handle Affiliations better #130 and Update handling of contactInformation #131

I'd like to target the release for mid next week (~ May 31) .... With conda and hopefully PyPI packages to follow right after.

[emiliom]

And Don and I will work on finalizing this new installation and setup documentation

[emiliom]

@lsetiawan:

• Let's table task 1 until after this next release. See my comment on #127
• For task 3, let's test your current code on the EnviroDIY database, after changing some affiliations.organizationid entries back to Null. (BTW, I'm not able to connect to the cloud DB; I'm not VPN'd at this moment. Maybe you can make the manual change yourself?). The goal here is to make sure the wofpy odm2-timeseries dao works with the original EnviroDIY database, and fails gracefully (no fatal errors) with the affiliation/organization stuff.

[emiliom]

Task 3 ... Is fixed by #131

Yes! And I can confirm this on the MySQL LBR & postgresql EnviroDIY amazon cloud endpoints. Thanks!

@lsetiawan, I think you are done with updates for the next release 😃. Now it's mostly on @ocefpaf, plus me (with some input from you) on the new, draft documentation.

[ocefpaf]

Now it's mostly on @ocefpaf, plus me (with some input from you) on the new, draft documentation.

I am back on the laptop today. I'll review the docs today and prepare the GitHub release later today or early tomorrow, if that is OK with @emiliom.

[emiliom]

I'll review the docs today and prepare the GitHub release later today or early tomorrow, if that is OK with @emiliom.

I'm ok with that. Thanks.

[emiliom]

Starting draft here of the list of enhancements and bug fixes included in this new release:

• greatly improved installation and setup, to make it easier and clearer to users
• new, interim installation and setup documentation
• removed pinned dependence to old sqlalchemy version (resulted in problems with odm2 timeseries dao handling SQLite); and to odm2api in setup.py
• added a CLI interface to deploy wofpy: wofpy_config.py
• reorganization of examples (dao's)
• added sample "production_configs" files
• Travis-ci and Appveyor are back online, with a subset of priority target tests
• performed extensive ODM2 timeseries DAO use case testing (SQLite LBR, MySQL LBR, PostgreSQL EnviroDIY)
  • many bug fixes to ensure DAO works across those 3 RDBMS
  • bug fixes: "Sources" now being properly read and exposed in WaterML 1.1 GetValues response
  • end date time now read from either action or timeseriesvalues, as appropriate
• code cleanups
  • py3k & pep8 code cleanup (though not supporting py3k yet)
  • added some docstrings
  • removed code duplication: "templates", see Fix path to templates #122
• Note: most testing focused on WOF/WaterML 1.1 only! Double-checked that WOF/WaterML 1.0 and WaterML 2 works, but did not go further than that.

Add references to issues and PR's?

I don't think we need that but if you want I can add them above.

[ocefpaf]

@emiliom I am editing above to make things simple, is that OK?

[emiliom]

@emiliom I am editing above to make things simple, is that OK?

Sounds great, thanks. I was just adding whatever I could remember or look up easily.

[lsetiawan]

PR #136 completes task 2.

@ocefpaf completes enhancements of wof/wofpy_config.py, as described in #72: *wofpy_config.py will take two additional arguments (it already takes the copy-to file path): the "example" to be copied; and the deployment type (test vs "production" or test vs "server"). This also involves copying files into a "base" folder that could be hard-wired to be named wofpy, or be user-selectable (@lsetiawan can explain). Being addressed in #132 (but not completed yet)

The PR has been merged. So I think now we are finally ready for new release?

[emiliom]

Great! One last thing, before issuing the release: @lsetiawan, can you test wofpy_config.py, to confirm that it works as expected?

@ocefpaf remind us here what your suggested tag naming convention is, now that we have versioneer, #138 ?

For reference (mostly to myself): while #72 is now closed, there are a lot of useful comments there that we'll want to reference when enhancing documentation, after the release is issued. I don't want to lose track of that.

[ocefpaf]

@ocefpaf remind us here what your suggested tag naming convention is, now that we have versioneer, #138 ?

I would go for v2.1.0a0. (Actually I would drop the alpha status too 😄)

[emiliom]

I would go for v2.1.0a0. (Actually I would drop the alpha status too 😄)

Ok. And it looks like we no longer need to maintain a version string in a file (setup.cfg, setup.py, whatever), from what I can tell?? Everything flows from the tag assigned on the release, right?

Let's plan issuing the release on Monday.

[ocefpaf]

Ok. And it looks like we no longer need to maintain a version string in a file (setup.cfg, setup.py, whatever), from what I can tell?? Everything flows from the tag assigned on the release, right?

Yep. All we need is a tag.

[emiliom]

#139 Some final discussions on wofpy_config.py, before the new release

[emiliom]

@ocefpaf, remind us where the extras_require settings/options in https://github.com/ODM2/WOFpy/blob/master/setup.py#L52 are used? Is it pip install?

[emiliom]

@ocefpaf, one more question before we issue the release; about the odm2api dependency.

I see that odm2api is listed in requirements-dev.txt and not in requirements.txt. @lsetiawan reminded me that for the conda package, you'll include dependencies from both -- can you confirm this?

Finally, in setup.py#L58 still has this line:

 dependency_links=[
    'git+https://github.com/ODM2/[email protected]#egg=odm2api-0.1.0' # noqa

Is that being used anywhere? In requirements-dev.txt there's no version pinning on odm2api, and @lsetiawan has confirmed that wofpy works with the latest odm2api conda package.

[lsetiawan]

Comments for #128 (comment):

reorganization of examples (dao's)

I don't remember us doing this... Originally we planned to split working vs not working, but didn't end up doing that.

[ocefpaf]

I see that odm2api is listed in requirements-dev.txt and not in requirements.txt. @lsetiawan reminded me that for the conda package, you'll include dependencies from both -- can you confirm this?

The reason why this is still kind of messy is because odm2api, and many other dependencies, are actually never used in wofpy code but rather in aplications built using wofpy.

In the software development realm the dependencies are defined in the app and not in the software used as the backbone for the app.

Some time ago @lsetiawan and I reached an understanding that such dependencies would be listed in the requirements-dev.txt instead (or the "extra" requirements.), so user know they are needed to develop an app with wofpy, but not in requirements.txt to avoid confusion like: dependency X is never used in the code.

For the conda package we do specify everything the end user of the odm2 channel expected to see. That is the flexibility that conda grants us.

Finally, in setup.py#L58 still has this line:

 dependency_links=[
    'git+https://github.com/ODM2/[email protected]#egg=odm2api-0.1.0' # noqa

That should be remove. (I actually thought I already removed it in the past 😕 not sure if I did or if it made it back.)

Is that being used anywhere? In requirements-dev.txt there's no version pinning on odm2api, and @lsetiawan has confirmed that wofpy works with the latest odm2api conda package.

Unless we know of an incompatibilty I advice against pinning. It hides issues and makes it harder to move the newer versions later on.

[emiliom]

Re: odm2api dependencies: Thanks for the explanations (or reminders!), and the PR follow-up. @lsetiawan and I had already confirmed that there was no odm2api version dependency.

[emiliom]

Guys, thanks for your work on PR #143!

@lsetiawan, you got a little too eager closing issues. I was deliberately leaving open some of the ones we have addressed, to review them next week and see if there are any nuggets of useful discussions for future use. But maybe I'll just be grateful next week when I see them all closed and decide it was never worth spending extra time on them ... 🤔

[lsetiawan]

@lsetiawan, you got a little too eager closing issues. I was deliberately leaving open some of the ones we have addressed, to review them next week and see if there are any nuggets of useful discussions for future use. But maybe I'll just be grateful next week when I see them all closed and decide it was never worth spending extra time on them ...

My bad ... really wanted to start fresh, some of the issues are so old, and I have no idea what's going on in them. Now at least all of us are pinged in them, and the issues can be reopened if you think they are worth spending time on. Thanks.

[ocefpaf]

But maybe I'll just be grateful next week when I see them all closed and decide it was never worth spending extra time on them ... 🤔

In @lsetiawan defense that was my suggestion. In software development the person that opened the issue is the issue "owner." S/he will either re-open it or the issue will re-surface at some point. Etiher way I don't think it is worth looking into >2-year old issues when there are plenty of new ones for us to take care of 😄

[emiliom]

Looks like we're ready for the release, right? (Except for me not having reviewed and edited Don's installation & setup documentation ... I guess that'll have to wait)

If so, I'll go ahead and get it out today.

[ocefpaf]

I am biased b/c I am not using WOFpy on production and it is "easy" to install when you are the one writing the installation script 😬, but @lsetiawan's looks good to me.

[lsetiawan]

@emiliom We are definitely ready for a release. I don't see anything else atm that are problematic. Thanks.

[emiliom]

@ocefpaf, we need your help!! @lsetiawan and I are drafting the new release. It's pretty much (saved it in "draft" state, which you can probably see). BUT github is complaining about the release tag we're trying to use. I picked a string that's identical to your suggestion (v2.1.0a0), except we're calling it "beta": v2.1.0b0. github complains in prominent red letters saying:

Invalid tag name We can’t create a tag with this name. Take a look at the suggestions in the sidebar for example tag names.

The suggestions in the sidebar don't follow your pattern for alpha/beta suffixes; the closest would be something like v2.1.0-beta.0

What should we use??

[ocefpaf]

You can use v2.1.0-beta.0 and versionner will convert that to pep440 automatically.

[emiliom]

Thanks for your quick reply!! The release is now out. And turns out the tag string complaint was my fault! Your original string pattern (v2.1.0b0) was totally fine, but I had a blank character at the beginning ☹️

I assume versioneer is already automatically churning to create the conda and pip packages, w/o you needing to do anything? It'll be cool to see it in action.

Guys: don't close this issue yet. I'll close it myself early next week.

THANKS FOR ALL THE GREAT WORK THAT WENT INTO THIS RELEASE!!

[ocefpaf]

I assume versioneer is already automatically churning to create the conda and pip packages, w/o you needing to do anything? It'll be cool to see it in action.

Nope. versioneer propagates the tag to version string to avoid inconsistencies from human maintained version strings, like what happened in odm2api. I still need to manually create both PyPI and conda packages as usual.

[ocefpaf]

PyPI: https://pypi.python.org/pypi/WOFpy/2.1.0b0
conda-forge: ODM2/conda-recipes-ODM2#62

[lsetiawan]

@ocefpaf Thanks for all the knowledge that you shared with me through this development 😃 Can't wait to pick at your brain some more on other things! You're the best!

[emiliom]

@ocefpaf:

conda-forge: ODM2/conda-recipes-ODM2#62

The package is going into our ODM2 channel (https://anaconda.org/odm2/wofpy), isn't it? I just don't remember if you're also adding it to conda-forge, or what.

[ocefpaf]

The package is going into our ODM2 channel (https://anaconda.org/odm2/wofpy), isn't it?

Yep. Everything goes into the odm2 channel as you requested when we started. If you want to move them to conda-forge let me know.

[ocefpaf]

@ocefpaf Thanks for all the knowledge that you shared with me through this development 😃 Can't wait to pick at your brain some more on other things! You're the best!

I can say the same 😉

[emiliom]

Yep. Everything goes into the odm2 channel as you requested when we started. If you want to move them to conda-forge let me know.

Great. We're good with just the odm2 channel for now. FYI, I've prepared an email announcement to our collaborators, but I'm holding off on sending it until the conda package is ready. I don't know how long it takes, and I'm seeing a CircleCI failure at ODM2/conda-recipes-ODM2#62 ...

[ocefpaf]

I don't know how long it takes

Neither do I 😄 conda is always a moving target and the latest "advancements" broke our workflow. I am trying to fix it. In theory pinning the conda to 4.2.13 should work, but that failed here and I am investigating it...

[ocefpaf]

I believe we can close this issue, right?

[emiliom]

I believe we can close this issue, right?

Let's leave it open, mostly for me once I have time to come back to revisit the documentation. I've been swamped with other work. It'll get better in ~10 days.

So, basically, everyone can ignore this issue.

[emiliom]

Closing (cleaning up), finally.