Sensors by location #297

majesticio · 2023-09-29T14:28:06Z

closes #276

…) resolves #281 * Added ability to increase the working memory for a given endpoint resolves #281

* included order_by in v1 and v2 measurements (#263) Co-authored-by: Gabriel Fosse <[email protected]> * manufacturers resource (#273) * manufacturers resource resolves #252 --------- Co-authored-by: Gabriel Fosse <[email protected]> * Adding `v3/instruments` resource (#271) * instruments resource resolves #270 --------- Co-authored-by: Gabriel Fosse <[email protected]> * Added timer * Added owner router --------- Co-authored-by: Gabriel Fosse <[email protected]> Co-authored-by: Gabriel Fosse <[email protected]> Co-authored-by: Russ Biggs <[email protected]>

* order by classes * refactor query builder to handle sort and order by statements * add sort and order by to v3 endpoints resolves #163 --------- Co-authored-by: Gabriel Fosse <[email protected]>

* Moved the tr.start out of if block Fixes bug #286 --------- Co-authored-by: Christian Parker <[email protected]>

* removed count fields --------- Co-authored-by: Gabriel Fosse <[email protected]>

majesticio · 2023-09-29T14:31:13Z

This feature will timeout occasionally. Open for discussion on optimization or reducing summary metrics etc to make it faster.

caparker · 2024-02-08T15:05:59Z

@russbiggs @majesticio You want me to take a look at this one or just leave it be for now?

majesticio · 2024-02-08T15:22:29Z

@russbiggs @majesticio You want me to take a look at this one or just leave it be for now?

If you can take a look at this, that would be great.

caparker · 2024-02-09T17:22:59Z

Here is what I am thinking. Our current Sensor model looks basically like this

   id: int
    name: str
    parameter: ParameterBase
    datetime_first: DatetimeObject
    datetime_last: DatetimeObject
    coverage: Coverage
    latest: Latest
    summary: Summary

With Coverage and Summary being the more difficult things to calculate for an entire sensor. Coverage can be calculated with the sensors_rollup table so that is pretty simple. But the full summary object cannot. So I propose a two stage solution here.

First we use the sensor_rollups for coverage and summary for the time being. Due to the way that the sensor rollup table is maintained the count, avg and sd value are just estimates so its not a perfect solution. Also we cant get distribution related values from sensors rollup so the summary object will be a little sparse.
Then we finish up the daily, annual and total summary methods. We have all the methods and tables built and we just need to operationalize them. Once that is done we will have daily, annual and total summaries that are similar to the hourly data summaries and we can just pull directly from one of those for the summary and coverage objects.

So for now I will leave the Sensor model as it is and just update the query to be more efficient.

Rewrote the queries to speed them up a litte and added the measurements method

caparker · 2024-02-09T23:11:49Z

@majesticio it would be great if you could take a quick look at this and test it out. It should be good but its always better to get some fresh eyes on it.

For the measurements I just finished what was started and that was pulling in the measurements method from the locations endpoint. I think that is working well but if we find its too slow I think we can speed it up but writing a slightly "lighter" query.

majesticio · 2024-02-09T23:43:03Z

@majesticio it would be great if you could take a quick look at this and test it out. It should be good but its always better to get some fresh eyes on it.

For the measurements I just finished what was started and that was pulling in the measurements method from the locations endpoint. I think that is working well but if we find its too slow I think we can speed it up but writing a slightly "lighter" query.

Nice! I am indeed getting data from the /sensors endpoint. I will continue to poke at it but so far so good!

* Provide numeric value for dates for easier analysis later (#332) * Feature/ratelimiter update (#330) * updated request_is_limited algorithm * Strip out numbers from paths This is so we can better aggregate by path in cloudwatch * Provide numeric value for dates for easier analysis later --------- Co-authored-by: Russ Biggs <[email protected]> * Sensors by location (#297) * Added ability to increase the working memory for a given endpoint (#282) resolves #281 * Added ability to increase the working memory for a given endpoint resolves #281 * update staging deploy to deploy on release branches (#284) * Added timer (#280) * included order_by in v1 and v2 measurements (#263) Co-authored-by: Gabriel Fosse <[email protected]> * manufacturers resource (#273) * manufacturers resource resolves #252 --------- Co-authored-by: Gabriel Fosse <[email protected]> * Adding `v3/instruments` resource (#271) * instruments resource resolves #270 --------- Co-authored-by: Gabriel Fosse <[email protected]> * Added timer * Added owner router --------- Co-authored-by: Gabriel Fosse <[email protected]> Co-authored-by: Gabriel Fosse <[email protected]> Co-authored-by: Russ Biggs <[email protected]> * Add sort and order by query params to v3 endpoints (#285) * order by classes * refactor query builder to handle sort and order by statements * add sort and order by to v3 endpoints resolves #163 --------- Co-authored-by: Gabriel Fosse <[email protected]> * Feature/working memory query 281 (#287) * Moved the tr.start out of if block Fixes bug #286 --------- Co-authored-by: Christian Parker <[email protected]> * fix missing start variable and return empty string for order (#288) * Remove count fields (#289) * removed count fields --------- Co-authored-by: Gabriel Fosse <[email protected]> * fix failing unit test (#292) * rate limiter logging fix (#293) * location sensors query * location sensors path query * Cleaned up the sensors endpoints Rewrote the queries to speed them up a litte and added the measurements method * Removed old method that I had renamed --------- Co-authored-by: Christian Parker <[email protected]> Co-authored-by: Russ Biggs <[email protected]> Co-authored-by: Gabriel Fosse <[email protected]> * add deprecation warning (#333) * upgrdate to actions/checkout@v4 --------- Co-authored-by: Christian Parker <[email protected]> Co-authored-by: Gabriel Fosse <[email protected]> Co-authored-by: Gabriel Fosse <[email protected]>

* hot-fix resolve auth issues * add redis in startup event hook * Release 20230927 (#291) * Added ability to increase the working memory for a given endpoint (#282) resolves #281 * Added ability to increase the working memory for a given endpoint resolves #281 * update staging deploy to deploy on release branches (#284) * Added timer (#280) * included order_by in v1 and v2 measurements (#263) Co-authored-by: Gabriel Fosse <[email protected]> * manufacturers resource (#273) * manufacturers resource resolves #252 --------- Co-authored-by: Gabriel Fosse <[email protected]> * Adding `v3/instruments` resource (#271) * instruments resource resolves #270 --------- Co-authored-by: Gabriel Fosse <[email protected]> * Added timer * Added owner router --------- Co-authored-by: Gabriel Fosse <[email protected]> Co-authored-by: Gabriel Fosse <[email protected]> Co-authored-by: Russ Biggs <[email protected]> * Add sort and order by query params to v3 endpoints (#285) * order by classes * refactor query builder to handle sort and order by statements * add sort and order by to v3 endpoints resolves #163 --------- Co-authored-by: Gabriel Fosse <[email protected]> * Feature/working memory query 281 (#287) * Moved the tr.start out of if block Fixes bug #286 --------- Co-authored-by: Christian Parker <[email protected]> * fix missing start variable and return empty string for order (#288) * Remove count fields (#289) * removed count fields --------- Co-authored-by: Gabriel Fosse <[email protected]> * fix failing unit test (#292) * rate limiter logging fix (#293) * remove summary count fields from v3 countries (#294) * removed locations_count from instruments.py query --------- Co-authored-by: Christian Parker <[email protected]> Co-authored-by: Gabriel Fosse <[email protected]> Co-authored-by: Gabriel Fosse <[email protected]> * revert to old startup and shutdown events (#296) * add redis client to app state (#299) * fixed parameters query param for locations, countries, and providers (#306) (#308) * fixed parameters query param for locations, countries and providers * Made requested changes to implement ParametersQuery --------- Co-authored-by: Gabriel Fosse <[email protected]> Co-authored-by: Gabriel Fosse <[email protected]> * add tiles parameter query (#310) * Update test_v3_queries.py file to use the array overlap operator instead of the equality operator. (#312) Co-authored-by: Gabriel Fosse <[email protected]> * Fix/statement timeout (#316) * Fix/v2 measurements source (#314) * fix v2 measurements to query measurements table instead of hourly * realias to match query params * measurements geom fix * Added sensors_id to speed up sorting * update statement_timeout to milliseconds --------- Co-authored-by: Russ Biggs <[email protected]> * set statement_timeout default config value --------- Co-authored-by: Christian Parker <[email protected]> * Release 20231222 (#318) * Fix/v2 measurements source (#314) * fix v2 measurements to query measurements table instead of hourly * realias to match query params * measurements geom fix * Added sensors_id to speed up sorting * update statement_timeout to milliseconds --------- Co-authored-by: Russ Biggs <[email protected]> * ratelimit headers (#315) * ratelimit headers * dynamic ratelimt-policy values * Added wait_for method (#317) --------- Co-authored-by: Christian Parker <[email protected]> * Release 20240104 (#320) * Fix/v2 measurements source (#314) * fix v2 measurements to query measurements table instead of hourly * realias to match query params * measurements geom fix * Added sensors_id to speed up sorting * update statement_timeout to milliseconds --------- Co-authored-by: Russ Biggs <[email protected]> * ratelimit headers (#315) * ratelimit headers * dynamic ratelimt-policy values * Fix/statement timeout (#316) * Fix/v2 measurements source (#314) * fix v2 measurements to query measurements table instead of hourly * realias to match query params * measurements geom fix * Added sensors_id to speed up sorting * update statement_timeout to milliseconds --------- Co-authored-by: Russ Biggs <[email protected]> * set statement_timeout default config value --------- Co-authored-by: Christian Parker <[email protected]> * Added wait_for method (#317) * update default date for date_form --------- Co-authored-by: Christian Parker <[email protected]> * Release 20240109 (#324) * add hint to 408 timeout error (#322) * Fix/bad timeout value (#323) --------- Co-authored-by: Christian Parker <[email protected]> * Bad timeout value passed to db class (#321) The averages endpoint was passing a config value but unassigned and it was therefor being passed as the timeout value. I added a check in the db to make sure we are protected from that as well as updated the averages endpoint to explicitly name the config argument. fixes #321 --------- Co-authored-by: Christian Parker <[email protected]> --------- Co-authored-by: Christian Parker <[email protected]> * Readme redis update (#327) * readme updates * readme typo * Release 20240207 (#331) * Feature/ratelimiter update (#330) * updated request_is_limited algorithm * Strip out numbers from paths This is so we can better aggregate by path in cloudwatch --------- Co-authored-by: Christian Parker <[email protected]> * update key setting for ratelimiting * Release 20240210 (#334) * Provide numeric value for dates for easier analysis later (#332) * Feature/ratelimiter update (#330) * updated request_is_limited algorithm * Strip out numbers from paths This is so we can better aggregate by path in cloudwatch * Provide numeric value for dates for easier analysis later --------- Co-authored-by: Russ Biggs <[email protected]> * Sensors by location (#297) * Added ability to increase the working memory for a given endpoint (#282) resolves #281 * Added ability to increase the working memory for a given endpoint resolves #281 * update staging deploy to deploy on release branches (#284) * Added timer (#280) * included order_by in v1 and v2 measurements (#263) Co-authored-by: Gabriel Fosse <[email protected]> * manufacturers resource (#273) * manufacturers resource resolves #252 --------- Co-authored-by: Gabriel Fosse <[email protected]> * Adding `v3/instruments` resource (#271) * instruments resource resolves #270 --------- Co-authored-by: Gabriel Fosse <[email protected]> * Added timer * Added owner router --------- Co-authored-by: Gabriel Fosse <[email protected]> Co-authored-by: Gabriel Fosse <[email protected]> Co-authored-by: Russ Biggs <[email protected]> * Add sort and order by query params to v3 endpoints (#285) * order by classes * refactor query builder to handle sort and order by statements * add sort and order by to v3 endpoints resolves #163 --------- Co-authored-by: Gabriel Fosse <[email protected]> * Feature/working memory query 281 (#287) * Moved the tr.start out of if block Fixes bug #286 --------- Co-authored-by: Christian Parker <[email protected]> * fix missing start variable and return empty string for order (#288) * Remove count fields (#289) * removed count fields --------- Co-authored-by: Gabriel Fosse <[email protected]> * fix failing unit test (#292) * rate limiter logging fix (#293) * location sensors query * location sensors path query * Cleaned up the sensors endpoints Rewrote the queries to speed them up a litte and added the measurements method * Removed old method that I had renamed --------- Co-authored-by: Christian Parker <[email protected]> Co-authored-by: Russ Biggs <[email protected]> Co-authored-by: Gabriel Fosse <[email protected]> * add deprecation warning (#333) * upgrdate to actions/checkout@v4 --------- Co-authored-by: Christian Parker <[email protected]> Co-authored-by: Gabriel Fosse <[email protected]> Co-authored-by: Gabriel Fosse <[email protected]> * Release 20240214 (#338) * Feature/merge trends and measurements (#336) * Fix/statement timeout (#316) * Fix/v2 measurements source (#314) * fix v2 measurements to query measurements table instead of hourly * realias to match query params * measurements geom fix * Added sensors_id to speed up sorting * update statement_timeout to milliseconds --------- Co-authored-by: Russ Biggs <[email protected]> * set statement_timeout default config value --------- Co-authored-by: Christian Parker <[email protected]> * Release 20231222 (#318) * Fix/v2 measurements source (#314) * fix v2 measurements to query measurements table instead of hourly * realias to match query params * measurements geom fix * Added sensors_id to speed up sorting * update statement_timeout to milliseconds --------- Co-authored-by: Russ Biggs <[email protected]> * ratelimit headers (#315) * ratelimit headers * dynamic ratelimt-policy values * Added wait_for method (#317) --------- Co-authored-by: Christian Parker <[email protected]> * Added the trends methods (dow, hod, moy) to the measurements --------- Co-authored-by: Russ Biggs <[email protected]> * Feature/add license to location (#337) * Fix/statement timeout (#316) * Fix/v2 measurements source (#314) * fix v2 measurements to query measurements table instead of hourly * realias to match query params * measurements geom fix * Added sensors_id to speed up sorting * update statement_timeout to milliseconds --------- Co-authored-by: Russ Biggs <[email protected]> * set statement_timeout default config value --------- Co-authored-by: Christian Parker <[email protected]> * Release 20231222 (#318) * Fix/v2 measurements source (#314) * fix v2 measurements to query measurements table instead of hourly * realias to match query params * measurements geom fix * Added sensors_id to speed up sorting * update statement_timeout to milliseconds --------- Co-authored-by: Russ Biggs <[email protected]> * ratelimit headers (#315) * ratelimit headers * dynamic ratelimt-policy values * Added wait_for method (#317) --------- Co-authored-by: Christian Parker <[email protected]> * Added the licsense list object to the locations --------- Co-authored-by: Russ Biggs <[email protected]> --------- Co-authored-by: Christian Parker <[email protected]> * Release 20240220 (#343) * Feature/auth api (#340) * auth api updates * password reset (#341) * Feature/password email (#342) * password reset * password email * Feature/merge trends and measurements (#339) * Fix/statement timeout (#316) * Fix/v2 measurements source (#314) * fix v2 measurements to query measurements table instead of hourly * realias to match query params * measurements geom fix * Added sensors_id to speed up sorting * update statement_timeout to milliseconds --------- Co-authored-by: Russ Biggs <[email protected]> * set statement_timeout default config value --------- Co-authored-by: Christian Parker <[email protected]> * Release 20231222 (#318) * Fix/v2 measurements source (#314) * fix v2 measurements to query measurements table instead of hourly * realias to match query params * measurements geom fix * Added sensors_id to speed up sorting * update statement_timeout to milliseconds --------- Co-authored-by: Russ Biggs <[email protected]> * ratelimit headers (#315) * ratelimit headers * dynamic ratelimt-policy values * Added wait_for method (#317) --------- Co-authored-by: Christian Parker <[email protected]> * Added the trends methods (dow, hod, moy) to the measurements * Updated the trends and raw methods Added better period calculating for both * Added validation check for dates Still has error related to serializing a datetime object * Added work around for the value error + datetime issue Also fixed syntax errors in the bbox validation --------- Co-authored-by: Russ Biggs <[email protected]> * add placeholder for explorer_api_key --------- Co-authored-by: Christian Parker <[email protected]> * Removed casting of datetime objects * Added summary back * Updated to use new `as_timestamptz` functions * update prod ratelimiting * v3 auth updates * Restrict access to purpleair until we can remove data * Changed tests to fit api * add token register endpoint (#345) (#346) * Allow the latest value to be empty (#351) Fixes #349 --------- Co-authored-by: Russ Biggs <[email protected]> Co-authored-by: Gabriel Fosse <[email protected]> Co-authored-by: Gabriel Fosse <[email protected]>

caparker and others added 11 commits September 26, 2023 21:40

Added ability to increase the working memory for a given endpoint (#282…

d94a3bc

…) resolves #281 * Added ability to increase the working memory for a given endpoint resolves #281

update staging deploy to deploy on release branches (#284)

c23796a

Add sort and order by query params to v3 endpoints (#285)

75a2efd

* order by classes * refactor query builder to handle sort and order by statements * add sort and order by to v3 endpoints resolves #163 --------- Co-authored-by: Gabriel Fosse <[email protected]>

Feature/working memory query 281 (#287)

fd2ec69

* Moved the tr.start out of if block Fixes bug #286 --------- Co-authored-by: Christian Parker <[email protected]>

fix missing start variable and return empty string for order (#288)

8f39d78

Remove count fields (#289)

a36f711

* removed count fields --------- Co-authored-by: Gabriel Fosse <[email protected]>

fix failing unit test (#292)

979dd3f

rate limiter logging fix (#293)

43bf202

location sensors query

5411b79

location sensors path query

1cab52b

majesticio marked this pull request as draft September 29, 2023 14:28

Cleaned up the sensors endpoints

3ff9563

Rewrote the queries to speed them up a litte and added the measurements method

Removed old method that I had renamed

206d122

caparker marked this pull request as ready for review February 9, 2024 23:42

russbiggs changed the base branch from release-20230927 to release-20240210 February 10, 2024 13:23

Merge branch 'release-20240210' into sensors-by-location

04eecd7

russbiggs merged commit 68737f8 into release-20240210 Feb 10, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sensors by location #297

Sensors by location #297

majesticio commented Sep 29, 2023

majesticio commented Sep 29, 2023

caparker commented Feb 8, 2024

majesticio commented Feb 8, 2024

caparker commented Feb 9, 2024

caparker commented Feb 9, 2024

majesticio commented Feb 9, 2024

Sensors by location #297

Sensors by location #297

Conversation

majesticio commented Sep 29, 2023

majesticio commented Sep 29, 2023

caparker commented Feb 8, 2024

majesticio commented Feb 8, 2024

caparker commented Feb 9, 2024

caparker commented Feb 9, 2024

majesticio commented Feb 9, 2024