Replacement REST API #1055

bsherwin · 2025-01-22T16:42:31Z

Type of PR

Documentation changes
Code changes

Purpose

This is a NodeJS Express based REST API that will replace the existing REST API. In the readme, you can find information on how to run locally, publish to Azure using bicep and publish to Azure with Terraform.

Does this introduce a breaking change? If yes, details on what can break

No. This is an independent utility to use as you wish for ingesting data into your solution.

Author pre-publish checklist

No PII in logs
Made corresponding changes to the documentation

Issues Closed or Referenced

References [829] Add new REST API for data generation. #986

…ples/modern-data-warehouse-dataops into brsherwi/datagenerator

naga-nandyala

While the application and the approach of updating sensor information is working, the data (sensor_locations and sensors ) you are using does not seem to be complete.

sensor_locaiton - master data with list of roadsegments (dimension). All I see is unique road segment id (with same description)
sensors - actual sensor information (fact info)

not sure if my understanding of the data is in correct

sql_stmt = """
select roadsegmentid, roadsegmentdescription ,count(1) from {sensor_location_sdf} group by 1,2 order by 2 desc
"""

+-------------+-----------------------------------------------------+--------+
|roadsegmentid|roadsegmentdescription                               |count(1)|
+-------------+-----------------------------------------------------+--------+
|4650         |Wythe Place between Sunnyside Avenue and Tennis Court|1       |
|3984         |Wythe Place between Sunnyside Avenue and Tennis Court|1       |
|5982         |Wythe Place between Sunnyside Avenue and Tennis Court|1       |
|3318         |Wythe Place between Sunnyside Avenue and Tennis Court|1       |
|1986         |Wythe Place between Sunnyside Avenue and Tennis Court|1       |
|2319         |Wythe Place between Sunnyside Avenue and Tennis Court|1       |
|987          |Wythe Place between Sunnyside Avenue and Tennis Court|1       |
|4983         |Wythe Place between Sunnyside Avenue and Tennis Court|1       |
|321          |Wythe Place between Sunnyside Avenue and Tennis Court|1       |
|1653         |Wythe Place between Sunnyside Avenue and Tennis Court|1       |
|4317         |Wythe Place between Sunnyside Avenue and Tennis Court|1       |
|2652         |Wythe Place between Sunnyside Avenue and Tennis Court|1       |
|3651         |Wythe Place between Sunnyside Avenue and Tennis Court|1       |
|5316         |Wythe Place between Sunnyside Avenue and Tennis Court|1       |
|1320         |Wythe Place between Sunnyside Avenue and Tennis Court|1       |
|654          |Wythe Place between Sunnyside Avenue and Tennis Court|1       |
|5649         |Wythe Place between Sunnyside Avenue and Tennis Court|1       |
|2985         |Wythe Place between Sunnyside Avenue and Tennis Court|1       |
|1796         |Wyona Street between Cropsey Avenue and Banker Street|1       |
|3794         |Wyona Street between Cropsey Avenue and Banker Street|1       |
+-------------+-----------------------------------------------------+--------+

isn't the data model
One RoadSegment has multiple Kerbside_ids. Each Kerbside having one parking sensor?

utilities/data-generator/readme.md

naga-nandyala · 2025-02-04T20:41:54Z

utilities/data-generator/.devcontainer/post_create.sh

@@ -0,0 +1,27 @@
+#!/usr/bin/env bash


Please change the permission of this script to have execute permission.

E.g Fabric post_create has execute permssions ... but this script does not..

╭─naga@naga-ms ~/dev/ext_gitrepos/brian_datagen/modern-data-warehouse-dataops/e2e_samples/fabric_dataops_sample/.devcontainer ‹brsherwi/datagenerator› ╰─$ ls -ltra total 24 -rwxr-xr-x 1 naga naga 689 Jan 23 10:29 post_create.sh -rw-r--r-- 1 naga naga 220 Jan 23 10:29 docker-compose.yaml -rw-r--r-- 1 naga naga 1335 Jan 23 10:29 devcontainer.json -rw-r--r-- 1 naga naga 1825 Jan 23 10:29 Dockerfile drwxr-xr-x 2 naga naga 4096 Jan 23 10:29 . drwxr-xr-x 10 naga naga 4096 Jan 23 10:29 .. ╭─naga@naga-ms ~/dev/ext_gitrepos/brian_datagen/modern-data-warehouse-dataops/e2e_samples/fabric_dataops_sample/.devcontainer ‹brsherwi/datagenerator› ╰─$ cd ../../../utilities/data-generator/.devcontainer ╭─naga@naga-ms ~/dev/ext_gitrepos/brian_datagen/modern-data-warehouse-dataops/utilities/data-generator/.devcontainer ‹brsherwi/datagenerator› ╰─$ ls -ltra total 24 -rw-r--r-- 1 naga naga 861 Feb 3 08:19 post_create.sh -rw-r--r-- 1 naga naga 1044 Feb 3 08:19 docker-compose.yml -rw-r--r-- 1 naga naga 847 Feb 3 08:19 devcontainer.json -rw-r--r-- 1 naga naga 1746 Feb 3 08:19 Dockerfile drwxr-xr-x 6 naga naga 4096 Feb 3 08:19 .. -rw-r--r-- 1 naga naga 0 Feb 5 07:29 .env drwxr-xr-x 2 naga naga 4096 Feb 5 07:29 . ╭─naga@naga-ms ~/dev/ext_gitrepos/brian_datagen/modern-data-warehouse-dataops/utilities/data-generator/.devcontainer ‹brsherwi/datagenerator›

Hence it is failing with below error

[179581 ms] Start: Run in container: /bin/sh -c ./.devcontainer/post_create.sh /bin/sh: 1: ./.devcontainer/post_create.sh: Permission denied [179646 ms] postCreateCommand from devcontainer.json failed with exit code 126. Skipping any further user-provided commands. Done. Press any key to close the terminal.

naga-nandyala · 2025-02-05T05:36:53Z

utilities/data-generator/readme.md

+
+## Terraform Deployment (do not run in devcontainer because it's not set up for Docker-in-Docker)
+
+- Execute `az login` and select your subscription if prompted


Hi Brian

There are too many manual terraform override steps. (tenant id, subscription id , locatoin, rg name retrieving the outputs.)
It will be nice to create a deploy.sh/deploy_infrastructure.sh file (or similar) which reads and .env file. (similar to how it is being done in fabric e2e sample).

Tenant ID is not used in the terraform. Subscription and location are really the only ones I would expect to change. These could be moved to an environment file, but not sure what value it adds for terraform given you modify environment variables or you modify the variables.tf. Please explain your reasoning for using an environment variable setup.

@bsherwin Genreally speaking, variable.tf is a declaration file where you declare your variables and optionally provide a default value or the validations that you want to perform. Same goes for providers.tf which should use variables.

The actual runtime values should be passed using .tfvars files. Or you can just provide the runtime values at the time of plan/apply as done here -

modern-data-warehouse-dataops/e2e_samples/fabric_dataops_sample/scripts/deploy_infrastructure.sh

Line 96 in 55a5f03

terraform apply \

.

and

modern-data-warehouse-dataops/e2e_samples/fabric_dataops_sample/infrastructure/terraform/providers.tf

Line 2 in 55a5f03

provider "azuread" {

bsherwin · 2025-02-05T20:08:32Z

While the application and the approach of updating sensor information is working, the data (sensor_locations and sensors ) you are using does not seem to be complete.

sensor_locaiton - master data with list of roadsegments (dimension). All I see is unique road segment id (with same description) sensors - actual sensor information (fact info)

not sure if my understanding of the data is in correct

sql_stmt = """ select roadsegmentid, roadsegmentdescription ,count(1) from {sensor_location_sdf} group by 1,2 order by 2 desc """

+-------------+-----------------------------------------------------+--------+
|roadsegmentid|roadsegmentdescription                               |count(1)|
+-------------+-----------------------------------------------------+--------+
|4650         |Wythe Place between Sunnyside Avenue and Tennis Court|1       |
|3984         |Wythe Place between Sunnyside Avenue and Tennis Court|1       |
|5982         |Wythe Place between Sunnyside Avenue and Tennis Court|1       |
|3318         |Wythe Place between Sunnyside Avenue and Tennis Court|1       |
|1986         |Wythe Place between Sunnyside Avenue and Tennis Court|1       |
|2319         |Wythe Place between Sunnyside Avenue and Tennis Court|1       |
|987          |Wythe Place between Sunnyside Avenue and Tennis Court|1       |
|4983         |Wythe Place between Sunnyside Avenue and Tennis Court|1       |
|321          |Wythe Place between Sunnyside Avenue and Tennis Court|1       |
|1653         |Wythe Place between Sunnyside Avenue and Tennis Court|1       |
|4317         |Wythe Place between Sunnyside Avenue and Tennis Court|1       |
|2652         |Wythe Place between Sunnyside Avenue and Tennis Court|1       |
|3651         |Wythe Place between Sunnyside Avenue and Tennis Court|1       |
|5316         |Wythe Place between Sunnyside Avenue and Tennis Court|1       |
|1320         |Wythe Place between Sunnyside Avenue and Tennis Court|1       |
|654          |Wythe Place between Sunnyside Avenue and Tennis Court|1       |
|5649         |Wythe Place between Sunnyside Avenue and Tennis Court|1       |
|2985         |Wythe Place between Sunnyside Avenue and Tennis Court|1       |
|1796         |Wyona Street between Cropsey Avenue and Banker Street|1       |
|3794         |Wyona Street between Cropsey Avenue and Banker Street|1       |
+-------------+-----------------------------------------------------+--------+

isn't the data model One RoadSegment has multiple Kerbside_ids. Each Kerbside having one parking sensor?

Fixed the data...there was inadvertantly a 1:1 roadsegmentid to kerbsideid. Should be more in line with what you expect.

initial commit of REST API

7d3d44f

bsherwin added e2e: parking-sensors-databricks capability: Testing Related to Testing capability e2e: fabric Related with E2E Fabric Sample labels Jan 22, 2025

bsherwin added this to the E2E - parking_sensor_fabric - V1 milestone Jan 22, 2025

bsherwin requested review from devlace, promisinganuj and naga-nandyala January 22, 2025 16:42

bsherwin self-assigned this Jan 22, 2025

bsherwin and others added 8 commits January 22, 2025 15:15

fix linter issues

936be2a

add MIT License notices to code

773d455

Added MIT License

a09ed77

lint yaml

8ba8775

added linters to devcontainer and fixed issues

b6a33f1

Merge branch 'brsherwi/datagenerator' of https://github.com/Azure-Sam…

b32c7a4

…ples/modern-data-warehouse-dataops into brsherwi/datagenerator

fix yaml

019d7eb

change node version

93d39c4

naga-nandyala reviewed Feb 5, 2025

View reviewed changes

fix env file issues; fix data

5cfd814

naga-nandyala self-requested a review February 5, 2025 22:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Replacement REST API #1055

Replacement REST API #1055

bsherwin commented Jan 22, 2025

naga-nandyala left a comment •

edited

Loading

naga-nandyala Feb 4, 2025

naga-nandyala Feb 5, 2025

bsherwin Feb 5, 2025

promisinganuj Feb 5, 2025 •

edited

Loading

bsherwin commented Feb 5, 2025


		## Terraform Deployment (do not run in devcontainer because it's not set up for Docker-in-Docker)

		- Execute `az login` and select your subscription if prompted

Replacement REST API #1055

Are you sure you want to change the base?

Replacement REST API #1055

Conversation

bsherwin commented Jan 22, 2025

Type of PR

Purpose

Does this introduce a breaking change? If yes, details on what can break

Author pre-publish checklist

Issues Closed or Referenced

naga-nandyala left a comment • edited Loading

Choose a reason for hiding this comment

naga-nandyala Feb 4, 2025

Choose a reason for hiding this comment

naga-nandyala Feb 5, 2025

Choose a reason for hiding this comment

bsherwin Feb 5, 2025

Choose a reason for hiding this comment

promisinganuj Feb 5, 2025 • edited Loading

Choose a reason for hiding this comment

bsherwin commented Feb 5, 2025

naga-nandyala left a comment •

edited

Loading

promisinganuj Feb 5, 2025 •

edited

Loading