[RSDK-8932] ota docs, local server #361

mattjperez · 2024-12-10T20:41:15Z

This PR does three things

adds basic documentation about the OTA feature
makes the micro-rdk-server build use OTA by default
an http2 static file server for development and testing

Makefile

acmorrow · 2024-12-18T16:36:03Z

Cargo.toml

@@ -8,6 +8,7 @@ members = [
  "micro-rdk-server",
  "micro-rdk-ffi",
  "examples/modular-drivers",
+  "etc/ota-server",


Let's call this "ota-dev-server" or "ota-test-server" to clarify that it isn't meant for production use.

Oh, forgot about this one. Please remember to rename this before merge.

yup, making those code changes now

acmorrow · 2024-12-18T16:36:56Z

OTA.md

+
+## Workflow
+
+In app.viam, add the following to the `services` array; you can alternatively add a `generic` service then edit it to match the following


missing .com, here and at least one other place.

acmorrow · 2024-12-18T16:37:29Z

OTA.md

+		}
+	}


Indentation looks off here

that was more annoying than expected. different editors displayed for whitespace differently and had different space/tab behavior. ended up just using github editor for it

acmorrow · 2024-12-18T16:38:52Z

OTA.md

+
+## Requirements
+
+- An esp32 wrover-e with 8MB of flash memory.


"or more"

I think WROVER-E as it is usually capitalized when I see it.

OTA.md

acmorrow · 2024-12-18T16:44:27Z

OTA.md

+
+### OTA Build
+The `ota` build consists of *only*:
+- the type-specific partition header, `esp_app_desc_t`


What does this file get used for? Does it need to be kept? Hosted?

added note that the output of this is what should be hosted, the download target of an ota update

acmorrow · 2024-12-18T16:45:22Z

etc/ota-server/Cargo.toml

+rust-version.workspace = true
+
+[dependencies]
+axum = {version = "0.7.9", features = ["http2"]}


New dependencies here should go up in the root Cargo.toml, sorted in, then pulled in via .workspace=true down here.

Ping on this? I think this and the json formatting are the only things left before merge.

etc/ota-server/src/main.rs

acmorrow · 2024-12-18T16:55:37Z

Makefile

-build-esp32-bin:
-	cargo +esp espflash save-image --package micro-rdk-server --merge --chip esp32 target/xtensa-esp32-espidf/micro-rdk-server-esp32.bin -T micro-rdk-server/esp32/partitions.csv -s 4mb  --bin micro-rdk-server-esp32 --target=xtensa-esp32-espidf  -Zbuild-std=std,panic_abort --release
+build-esp32-bin: build-esp32-ota
+	cargo +esp espflash save-image --package micro-rdk-server --features=ota --merge --chip esp32 target/xtensa-esp32-espidf/micro-rdk-server-esp32.bin -T micro-rdk-server/esp32/ota_8mb_partitions.csv -s 8mb --bin micro-rdk-server-esp32 --target=xtensa-esp32-espidf -Zbuild-std=std,panic_abort --release


These are getting really quite long, and there are some subtle differences between the two commands that are obscured by other minor variations. Could you please:

Go multi-line for these

Ensure the flag order between build-esp32-bin and build-esp32-ota is the same, and use the same form of each flag (-T vs --partition-table), and convert any short-form arguments (e.g. -s) into long form?

While you are here, could you add --skip-update-check as well?

build-esp32-ota: cargo +esp espflash save-image \ --package micro-rdk-server \ ... --partition-table micro-rdk-server/esp32/ota_8mb_partitions.csv \ --flash-size 8mb

etc.

acmorrow

Some new comments and notes on updates. Also looks like a few original comments not addressed quite yet.

acmorrow · 2024-12-19T14:19:23Z

Makefile

-build-esp32-bin:
-	cargo +esp espflash save-image --package micro-rdk-server --merge --chip esp32 target/xtensa-esp32-espidf/micro-rdk-server-esp32.bin -T micro-rdk-server/esp32/partitions.csv -s 4mb  --bin micro-rdk-server-esp32 --target=xtensa-esp32-espidf  -Zbuild-std=std,panic_abort --release
+build-esp32-bin: build-esp32-ota
+	cargo +esp espflash save-image \


These are looking much better. A few small things:

Order of --bin and --partition-table args is not consistent between the two lists.

Let's move any "only in one list" flags/args to the end of each. So for build-esp32-bin you already have --merge there, but --flash-size is in the middle.

Let's make the actual target file always the last argument.

So the order of the 'sections' is:

identical

common flag but different args (e.g. partition table)

unique (e.g. --merge and --flash-size)

file target

removed --flash-size as it doesn't appear to be necessary, both make build-* still work.

put it back, turns out it did influence the final build size for some reason. It appears like it may be adding padding with --flash-size for the merged binary to be the same size as the flash target. building without is only slightly larger than the ota app image by itself.

OTA.md

acmorrow · 2024-12-19T14:21:09Z

OTA.md

+
+## Full Build
+
+If a device is built with the above partition table, the `make build-esp32-bin` command creates a Merged Binary that includes


Above you say "Merged Image" but here "Merged Binary".

acmorrow · 2024-12-19T14:21:20Z

OTA.md

+- the bootloader
+- the partition table mapping (`partitions.csv`)
+- populated partitions (with partition headers) according to the mapping
+/


acmorrow · 2024-12-19T14:22:30Z

OTA.md

+You can confirm this by using `ls -l** in your build directory to compare the size of the binary to your partition table.
+
+## OTA Build
+The `ota` build produces an app image (described above), which internally consists of:


In the ## Full Build section above, you describe make build-esp32bin early, as the task to be performed, but this section doesn't have that. I thihnk it should

OTA.md

etc/ota-server/src/main.rs

acmorrow

LGTM mod renaming ota-server and hoisting the new crate dependencies up into the root Cargo.toml.

mattjperez · 2024-12-19T16:14:49Z

rendered here https://github.com/mattjperez/micro-rdk/blob/rsdk-8932-ota-docs/OTA.md

gvaradarajan

LGTM, optional suggestions

gvaradarajan · 2024-12-19T16:17:23Z

OTA.md

+
+The `version` field is equivalent to a `tag` and can be any arbitrary string of up to 128 characters. 
+After successfully applying the new firmware, this `version` will be stored in NVS. 
+This values is compared to that of the latest machine config from app.viam.com and will trigger the update process.


values -> value? Also maybe clarify that a detected change in the version is what prompts the update process

gvaradarajan · 2024-12-19T16:17:26Z

OTA.md

+  - for example, make a device capable of OTA.
+
+**This is not the build that should be hosted at the `url` in the service config.**
+You can confirm this by using `ls -l** in your build directory to compare the size of the binary to your partition table.


should we clarify this with the expected size difference?

gvaradarajan · 2024-12-19T16:19:06Z

OTA.md

+
+While not all blob storage platform support HTTP/2, many offer Content Delivery Network (CDN) solutions that do.
+
+We don't currently support tokens in the [OTA Service Config](#ota-service-config), so if permissions are required to access the endpoint they must be embedded in the URL itself.


tokens -> authentication tokens. embedded in the URL itself -> embedded in the URL as query params?

mattjperez requested a review from a team as a code owner December 10, 2024 20:41

mattjperez changed the title ~~[RSDK-8932] experimental ota docs~~ [RSDK-8932] ota docs, local server Dec 10, 2024

mattjperez added 7 commits December 11, 2024 11:44

first draft

c455f0a

update build commands to use ota

dbcd06d

add local ota server for testing

10ab511

small changes

af3e08b

undo changes

a6bdcc2

fix targetdir when serving from workspace root

040524a

update cargo lock

e3e06d3

mattjperez force-pushed the rsdk-8932-ota-docs branch from d4156ce to e3e06d3 Compare December 11, 2024 16:45

mattjperez commented Dec 11, 2024

View reviewed changes

Makefile Show resolved Hide resolved

Merge branch 'main' into rsdk-8932-ota-docs

3fe7f7e

acmorrow requested changes Dec 18, 2024

View reviewed changes

add doc suggestions, TODO update ota-server name,deps

2a02ece

acmorrow requested changes Dec 19, 2024

View reviewed changes

mattjperez and others added 3 commits December 19, 2024 10:17

more doc suggestions

2d2bee1

service json formatting

2b8a68d

fmt using github editor

5baea09

acmorrow approved these changes Dec 19, 2024

View reviewed changes

mattjperez added 2 commits December 19, 2024 10:39

update ota-server to ota-dev-server

b7840b7

additional links

18b1bb1

gvaradarajan approved these changes Dec 19, 2024

View reviewed changes

mattjperez force-pushed the rsdk-8932-ota-docs branch from 92026bc to f3f4143 Compare December 19, 2024 19:01

add suggestions, put --flash-size back

146b257

mattjperez force-pushed the rsdk-8932-ota-docs branch from f3f4143 to 146b257 Compare December 19, 2024 19:03

mattjperez merged commit 7038313 into viamrobotics:main Dec 19, 2024
6 checks passed

mattjperez deleted the rsdk-8932-ota-docs branch December 19, 2024 19:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[RSDK-8932] ota docs, local server #361

[RSDK-8932] ota docs, local server #361

mattjperez commented Dec 10, 2024 •

edited

Loading

acmorrow Dec 18, 2024

acmorrow Dec 19, 2024

mattjperez Dec 19, 2024

acmorrow Dec 18, 2024

mattjperez Dec 19, 2024

acmorrow Dec 18, 2024

mattjperez Dec 19, 2024 •

edited

Loading

acmorrow Dec 18, 2024

mattjperez Dec 19, 2024

acmorrow Dec 18, 2024

mattjperez Dec 19, 2024

acmorrow Dec 18, 2024

acmorrow Dec 19, 2024

acmorrow Dec 18, 2024

acmorrow left a comment

acmorrow Dec 19, 2024

mattjperez Dec 19, 2024

mattjperez Dec 19, 2024

acmorrow Dec 19, 2024

acmorrow Dec 19, 2024

acmorrow Dec 19, 2024

acmorrow left a comment

mattjperez commented Dec 19, 2024

gvaradarajan left a comment

gvaradarajan Dec 19, 2024

gvaradarajan Dec 19, 2024

gvaradarajan Dec 19, 2024


		## Workflow

		In app.viam, add the following to the `services` array; you can alternatively add a `generic` service then edit it to match the following


		## Requirements

		- An esp32 wrover-e with 8MB of flash memory.


		## Full Build

		If a device is built with the above partition table, the `make build-esp32-bin` command creates a Merged Binary that includes


		While not all blob storage platform support HTTP/2, many offer Content Delivery Network (CDN) solutions that do.

		We don't currently support tokens in the [OTA Service Config](#ota-service-config), so if permissions are required to access the endpoint they must be embedded in the URL itself.

[RSDK-8932] ota docs, local server #361

[RSDK-8932] ota docs, local server #361

Conversation

mattjperez commented Dec 10, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mattjperez Dec 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

acmorrow left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

acmorrow left a comment

Choose a reason for hiding this comment

mattjperez commented Dec 19, 2024

gvaradarajan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mattjperez commented Dec 10, 2024 •

edited

Loading

mattjperez Dec 19, 2024 •

edited

Loading