[APP-7566]: Add bluetooth provisioning as a default provisioning method which runs parallel to hotspot provisioning. #70

maxhorowitz · 2025-02-20T20:20:10Z

Summary

This PR encompasses everything needed for adding bluetooth provisioning as a default provisioning method in the viam-agent (and is designed to run in parallel to the existing WiFi hotspot provisioning method).

Setup

I want you to test as if you are a customer who has just received a machine in the mail. The machine will just have a viam-agent binary installation. It will not have WiFi credentials (and thus no internet connectivity).

Machine setup

Using a Linux machine/laptop (that you are comfortable getting your /opt/viam/*, /etc/viam/* and other files tampered with) , pull down this branch. Because you will need to remove connectivity to emulate the user perspective, please do not SSH into a machine over WiFi (and don't use Ethernet because it can interfere with provisioning). Instead, test directly on the device (keyboard + mouse) needed.

Uninstall Viam

First, you will need to remove any Viam-related stuff from your computer (to emulate the "newly-out-of-the-box" scenario our customers will be in). Please run the following from the repository root:
sudo ./uninstall.sh

Remove local networks

You will then need to remove local networks from your machine to emulate the "offline-ness" that our customers will be dealing with. I've been running nmcli con show and subsequently sudo nmcli con delete <name> (for every network that could "interfere" with the provisioning flow).

Preinstall the Viam Agent

Then, run the following (again from the repository root):

make
sudo ./bin/viam-agent-custom-aarch64 --install
sudo /opt/viam/bin/viam-agent --debug

I've been running the commands altogether as such:
make && sudo ./bin/viam-agent-custom-aarch64 --install && sudo /opt/viam/bin/viam-agent --debug

Phone setup

Download the LightBlue app (available on both Android and iPhone). This is what we will use to communicate with the bluetooth service that is being advertised from the Linux machine.

Testing Procedure

At this point, we are ready to test.

LightBlue

Pairing

In the app, use the search bar to find the name of the Linux machine that you are currently testing against. It should be there. Pair with your machine (one of the tests is to see if the pairing request is automatically accepted on the machine side, so I am hoping this part works!).

If it does not pair, check your screen for a six-digit pairing code and accept the request manually. This is a known limitation that I am working on fixing. The TLDR is it's complicated because of reliance on systemd bus signals which may be picked up and discarded as negligible elsewhere in the viam-agent.

Bluetooth service used to "transmit" credentials

Once paired, look at the bluetooth service and nested characteristics available to us. You should be able to take the provided UUIDs and map them to the logs on your viam-agent machine:

There is an encoding that will be helpful for you to know here. Characteristics whose last 4 characters of their first 8 character sequence (preceding the -) will always be the following:

xxxx1111-... is the encompassing service
xxxx2222-... is the write-only characteristic for SSID
xxxx3333-... is the write-only characteristic for passkey
xxxx4444-... is the write-only characteristic for part ID
xxxx5555-... is the write-only characteristic for part secret
xxxx6666-... is the write-only characteristic for app address
xxxx7777-... is the read-only characteristic for nearby available WiFi networks that the machine has detected

2025-02-25T20:41:24.245Z	DEBUG	viam-agent	networking/bluetooth.go:507	BlueZ version (5.66) meets the requirement (5.66 or later)
2025-02-25T20:41:24.246Z	DEBUG	viam-agent	networking/bluetooth.go:43	Bluetooth peripheral service UUID: 79ff1111-4f38-44b9-b3b5-78fb7e14757e
2025-02-25T20:41:24.246Z	DEBUG	viam-agent	networking/bluetooth.go:45	WiFi SSID can be written to the following bluetooth characteristic: db8f2222-baea-452f-bad0-62b440f98161
2025-02-25T20:41:24.246Z	DEBUG	viam-agent	networking/bluetooth.go:47	WiFi passkey can be written to the following bluetooth characteristic: 06af3333-4442-42e5-9e6a-1729ffa1378d
2025-02-25T20:41:24.246Z	DEBUG	viam-agent	networking/bluetooth.go:49	Robot part key ID can be written to the following bluetooth characteristic: 26d24444-161f-4474-a4a5-e83c563ac52b
2025-02-25T20:41:24.246Z	DEBUG	viam-agent	networking/bluetooth.go:51	Robot part key can be written to the following bluetooth characteristic: 12db5555-ca13-4a0b-879a-c76fb1542fe1
2025-02-25T20:41:24.246Z	DEBUG	viam-agent	networking/bluetooth.go:53	Viam app address can be written to the following bluetooth characteristic: 9eaa6666-14f0-47a9-b69f-a093f8358b18
2025-02-25T20:41:24.246Z	DEBUG	viam-agent	networking/bluetooth.go:55	Available WiFi networks can be read from the following bluetooth characteristic: a8ee7777-2496-485a-b0ea-dc63a1122f1

The LightBlue interface is pretty easy to follow. You may need to change LightBlue messages from hex or binary to utf-8 string. Once confirmed you're in utf-8 string mode, you can write individual messages to the SSID, passkey, part ID, part secret, and app address "characteristics" by clicking "write value." Similarly, you can read from the available WiFi networks "characteristic" by clicking "read value." Any value written will get "sucked in" to the viam-agent provisioning flow. You can check this in the viam-agent logs. Once all values are submitted, the provisioning loop should close out the BT connection, connect to WiFi, retrieve its cloud config, and should start up a viam-server (thus ending the provisioning loop).

Cases

Hotspot provisioning flow (i.e. check the captive portal flow still works)
Bluetooth provisioning flow (i.e. check the LightBlue flow works)

…ed interface for OS abstraction).

…irware differences.

… is yet to be implemented.

…ure of method.

…fix used elsewhere in the Agent.

…t sending information to a closed channel.

…d and rename waitForBLEValue to retryCallbackOnEmptyCharacteristicError.

…tils.go

…move custom types that are now redundant.

…unexported interface so that the BT functionality can be mocked from provisioning tests.

… methods that exist on its implementation from the calling code (in networkmanager.go).

… unimplemented and left for follow up PRs).

…for machine settings config updates).

…ember to revert this commit once the fix for rc0.14 is released).

…(and remember to revert this commit once the fix for rc0.14 is released)." This reverts commit a4c226a.

…rmed with manual test).

maxhorowitz · 2025-02-24T21:45:28Z

subsystems/networking/networkmanager.go

+			userInput, err := n.bluetoothService.waitForCredentials(ctx, true, true) // Background goroutine ultimately cancelled by context.
+			if err != nil {
+				n.logger.Errorw("failed to wait for user input of credentials", "err", err)
+				return
+			}
+			inputChan <- *userInput


Would people like it more if I pass the inputChan into the waitForCredentials call itself? Then, I could get rid of the goroutine here and make the waitForCredentials call do nonblocking listening. In that case I would rename it to listenForCredentials.

maxhorowitz · 2025-02-24T21:52:51Z

subsystems/networking/definitions.go

-	Signal    int32
-	Connected bool
-	LastError string
+	Type      string `json:"type,omitempty"`


json:"xxx" tags are shorthand on purpose to minimize bytes wasted as uninformative JSON keys.

Our coding standards are not to use shorthand for json. The json should reflect the exact spelling (though lower/snake cased) of the object to avoid confussion.

Also, changing this may break the existing mobile provisioning, as this would now get marshalled differently. If you need a new/more compact syntax, create a new struct.

…oning tests.

maxhorowitz · 2025-02-25T22:32:54Z

subsystems/networking/bluetooth.go

+)
+
+// bluetoothService provides an interface for retrieving cloud config and/or WiFi credentials for a robot over bluetooth.
+type bluetoothService interface {


The private interface lets us

protect all linux related BLE behaviors from the provisioning code path, and

allow us to make modular unit tests that validate state behaviors in our provisioning flow in the event of various (mocked) bluetooth errors

maxhorowitz · 2025-02-25T22:34:36Z

subsystems/networking/bluetooth.go

@@ -0,0 +1,630 @@
+package networking


Generally speaking having a difficult time enforcing behaviors via unit testing the functions/methods in this file because many of them are low level BT commands.

Same problem with wifi provisioning... unit tests for stuff that interacts with a complex system are just nearly impossible to create in a useful way. So don't worry too much about that. We just have to do human-in-the-loop testing.

subsystems/networking/networkmanager_test.go

maxhorowitz · 2025-02-26T15:15:19Z

subsystems/networking/networkmanager.go

 		return err
 	}
-	return errors.Join(err, err2)
+	n.connState.setProvisioning(false)


Moved this from top of function (it should only be false after clean shutdown, right?).

No, that's the exact opposite. You set state when you're TRYING to close down that state. If things fail halfway through, then it doesn't think it's in a WORKING provisioning mode, and will restart from the ground up.

For STARTUP, the opposite is true. Mark it at the end, when it's fully started/good.

Otterverse

I apologize in advance for how long this review is. I'm leaving a summary here at the top of (most) of the important things, so it's easier to discuss (and check off as you go) hopefully. But it will overlap a lot with the inline comments too.

As always, hit me up in slack to discuss things whenever. Plenty of this is non-obvious, as you're deep in the guts of the most complicated part of Agent here.

Background go routines/threads need to be used only when/where required, and must be fully managable. Remember that calls to start/stop can come in at any point in the flow, so everything must be able to exit cleanly and quickly. No need to background threads if you're just going to wait for them in the same fuction. Just handle things linearly to keep it simpler.
Need to be careful about data races, and protect any data that can be changed from another thread with locks. Note that even if it's not explicitly in a thread, outside calls from start/stop/update can happen at any point, so everything in those paths has to be race-safe.
BT characteristics (and their associated UUIDs) are a bunch of indivudual variables resulting in ~1/3 of bluetooth.go being very repetitive boilerplate. Should get them arranged into a simpler data structure, with a map to use directly, or a true structure with (generalized) getter/setter methods. E.g. calls should be able to look like val, err := bt.GetCharacteristic("ssid") or bt.SetCharacteristic("ssid", myVal) and be human readable. Look at networkState and connState for hints.
As discussed in slack, the 20 character limit for charcteristic could be a real problem. SSIDs and PSKs can be longer than that by themselves. And scan results may contains dozens of networks.
Need a new config setting to disable bluetooth.
If agent starts on a device without bluetooth (or otherwise unsupport version/etc.), it needs to detect this and quietly do nothing (beyond an initial log.) Likely best to integrate with above, and just have a "temporary" flag that disables it until a full restart.
BLE needs to integrate with the health checking. Any backgrounded routines need to report health status regularly using health.healthySleep() (add a new one on Networking{} to track ble health.)
Ideally, this should be abstracted just like the portal is. E.g. whenever start/stopPortal() is called, start/stopBLE() is called, and otherwise works the same. They should even use the same inputChan, so the outer provisioning code only has to listen to that one channel.
Network scans (and other info) are updated in real time. This is why they're managed in their own structures like n.connState(). BLE needs to use those and not just pass in a static list one time at startup.

Otterverse · 2025-02-26T16:58:55Z

subsystems/networking/definitions.go

-	Signal    int32
-	Connected bool
-	LastError string
+	Type      string `json:"type,omitempty"`


Our coding standards are not to use shorthand for json. The json should reflect the exact spelling (though lower/snake cased) of the object to avoid confussion.

Also, changing this may break the existing mobile provisioning, as this would now get marshalled differently. If you need a new/more compact syntax, create a new struct.

Otterverse · 2025-02-26T17:05:13Z

subsystems/networking/networking.go

+	webServer       *http.Server
+	grpcServer      *grpc.Server
+	portalData      *portalData
+	hotspotIsActive bool


This duplicates the functionality of connState.GetProvisioning() (which properly mutex locks things and records timestamps.)

Otterverse · 2025-02-26T17:09:31Z

subsystems/networking/networkmanager.go

-		err = errors.Join(err, n.deactivateConnection(n.Config().HotspotInterface, n.Config().HotspotSSID))
-		return errw.Wrap(err, "starting web/grpc portal")
+	// Simultaneously start both the hotspot captive portal and the bluetooth service provisioning methods.
+	wg := sync.WaitGroup{}


Suggested change

wg := sync.WaitGroup{}

var wg sync.WaitGroup

Empty vars should be declared with "var" when possible, not an empty struct

Otterverse · 2025-02-26T17:11:36Z

subsystems/networking/networkmanager.go

+		)
+		n.bluetoothIsActive = true
+	}, wg.Done)
+	wg.Wait()


I do not understand why you're backgrounding functions above just to wait here. WaitGroups are for async thread management. You're firing off async threads then waiting in the same function that just fired them.

Otterverse · 2025-02-26T17:14:24Z

subsystems/networking/networkmanager.go

+			bluetoothErr = err
+			return
+		}
+		goutils.ManagedGo( // Listen for user input asynchronously. How should we be handling errors here?


This is firing a new thread from within another backgrounded thread? This one doesn't look like it's being tracked/managed by a waitgroup, so it'll just be orphaned as far as I can tell.

Otterverse · 2025-02-26T22:03:19Z

subsystems/networking/bluetooth.go

+		return fmt.Errorf("failed to stop advertising: %w", err)
+	}
+	bsl.advActive = false


What happens if this fails and returns an error, but is still marked active?

Otterverse · 2025-02-26T22:31:03Z

subsystems/networking/bluetooth.go

+}
+
+// getBlueZVersion retrieves the installed BlueZ version and extracts the numeric value correctly.
+func getBlueZVersion() (float64, error) {


This should get/check the version via dbus (which is how the rest of this interacts) not via exec calls (if at all possible.) Userspace utilities may not actually be what's running, or may not even be installed. If version isn't available directly, then check that the properties needed exist perhaps. Should allow wider compatibility too.

Otterverse · 2025-02-26T22:32:29Z

subsystems/networking/bluetooth.go

+func checkOS() error {
+	if runtime.GOOS != "linux" {
+		return fmt.Errorf("this program requires Linux, detected: %s", runtime.GOOS)
+	}
+	return nil
+}


This likely shouldn't exist. Use build flags to guard code that simply can't run elsewhere at compile time, not checking at runtime.

Otterverse · 2025-02-26T22:33:51Z

subsystems/networking/bluetooth.go

+			}
+
+			// Not ready to return (do not have the minimum required set of credentials), so sleep and try again.
+			time.Sleep(time.Second)


This needs to integrate with healthchecks. Use the healthySleep functions there. May need to extend if the health of bluetooth is separate from the general subsystems loops.

Otterverse · 2025-02-26T22:44:24Z

subsystems/networking/networkmanager.go

+	goutils.ManagedGo(func() {
+		if n.bluetoothService == nil {
+			bt, err := newBluetoothService(
+				n.logger, fmt.Sprintf("%s.%s.%s", n.cfg.Manufacturer, n.cfg.Model, n.cfg.FragmentID), n.getVisibleNetworks())


the config values should be n.Config() otherwise you can hit a race.

Suggested change

n.logger, fmt.Sprintf("%s.%s.%s", n.cfg.Manufacturer, n.cfg.Model, n.cfg.FragmentID), n.getVisibleNetworks())

n.logger, fmt.Sprintf("%s.%s.%s", n.Config().Manufacturer, n.Config().Model, n.Config().FragmentID), n.getVisibleNetworks())

n.getVisibleNetworks() is a one-time call here, but the list of visible networks changes frequently, especially if someone is trying to get a new network to show up. This needs to be handled like portal does, and updated when things change, not passed in once here only at startup.

maxhorowitz added 30 commits February 13, 2025 15:29

Add bluetooth wifi provisioner interface only (still need to add nest…

f31130c

…ed interface for OS abstraction).

Add bluetooth service (low-level) interface for abstracting away OS/f…

68694d9

…irware differences.

WIP

7d101b0

Remove example.go file.

6fa786f

Improve inline commenting.

c5c1bc6

Update subsystems/provisioning/bluetooth/bluetooth_service.go

4b5fcc6

Update subsystems/provisioning/bluetooth/bluetooth_service.go

6edd5cd

Remove both interfaces and add inline TODOs where Linux functionality…

02dd56f

… is yet to be implemented.

Remove error check from enableAutoAcceptPairRequest due to asynch nat…

80d0929

…ure of method.

Use standard library errors and import custom errors package with pre…

6ac1d56

…fix used elsewhere in the Agent.

Pass context one layer lower in call stack to properly protect agains…

0356d47

…t sending information to a closed channel.

Remove unused helper method for custom error type.

f362c74

Add context cancellation in case of error in WaitForCredentials metho…

9a47333

…d and rename waitForBLEValue to retryCallbackOnEmptyCharacteristicError.

Update subsystems/provisioning/bluetooth/bluetooth_wifi_provisioner_u…

5923659

…tils.go

Make retryCallbackOnExpectedError fn much more general.

16516e3

Update subsystems/provisioning/bluetooth/bluetooth_wifi_provisioner.go

74bf5c3

Update subsystems/provisioning/bluetooth/bluetooth_wifi_provisioner.go

cf43f28

Update error message TODOs to be more specific.

12afb28

Move bluetooth wifi provisioner file into provisioning package and re…

8400c94

…move custom types that are now redundant.

Remove nested goroutines for single goroutine.

9377326

Removed some of the unneccessary layers of abstraction, and added an …

0aa9149

…unexported interface so that the BT functionality can be mocked from provisioning tests.

Add interface (for bluetoothService) to conceal all fields and helper…

ca55251

… methods that exist on its implementation from the calling code (in networkmanager.go).

Add nolint tags where necessary to get the code to pass lint (most is…

325f9b7

… unimplemented and left for follow up PRs).

Add calling code (rebased from the large commit including a refactor …

3d36b0a

…for machine settings config updates).

Add commits from other PRs.

e8d43e6

Fix linux-side bugs (moved from MacOS to Linux).

cc5045b

Add debugging log lines for provisioning defaults JSON error (and rem…

a4c226a

…ember to revert this commit once the fix for rc0.14 is released).

Revert "Add debugging log lines for provisioning defaults JSON error …

63b42ea

…(and remember to revert this commit once the fix for rc0.14 is released)." This reverts commit a4c226a.

Fix establishing available WiFi networks in BLE characteristic (confi…

2d84f18

…rmed with manual test).

Add suggestions from linter.

88f771d

Merge branch 'main' into interface

b56068e

maxhorowitz commented Feb 24, 2025

View reviewed changes

maxhorowitz changed the title ~~[APP-7566]: Bluetooth Provisioning~~ [APP-7566]: Add bluetooth provisioning as a default provisioning method which runs parallel to hotspot provisioning. Feb 24, 2025

maxhorowitz added 9 commits February 25, 2025 11:06

Pass write only channel into bluetooth service as interface requirement.

d5c3c4e

Merge branch 'interface' of github.com:viamrobotics/agent into interface

ad872d1

Add changes from linter.

4ed6186

Make waiting for credentials asynchronous.

83e1f29

Add logging of errors in waitForCredentials call.

35ad378

Add networkmanager_test.go file for StartProvisioning and StopProvisi…

a5d32b1

…oning tests.

Merge latest main.

48dba09

Fix typo in log lines.

6c4c5fe

Add passing unit tests.

f5f6fc9

maxhorowitz commented Feb 25, 2025

View reviewed changes

maxhorowitz requested review from ale7714, Otterverse and abe-winter February 25, 2025 22:33

maxhorowitz commented Feb 25, 2025

View reviewed changes

maxhorowitz commented Feb 26, 2025

View reviewed changes

subsystems/networking/networkmanager_test.go Show resolved Hide resolved

Update subsystems/networking/networkmanager_test.go

deb212c

maxhorowitz commented Feb 26, 2025

View reviewed changes

Otterverse requested changes Feb 26, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[APP-7566]: Add bluetooth provisioning as a default provisioning method which runs parallel to hotspot provisioning. #70

[APP-7566]: Add bluetooth provisioning as a default provisioning method which runs parallel to hotspot provisioning. #70

maxhorowitz commented Feb 20, 2025 •

edited

Loading

maxhorowitz Feb 24, 2025 •

edited

Loading

maxhorowitz Feb 24, 2025 •

edited

Loading

Otterverse Feb 26, 2025

maxhorowitz Feb 25, 2025

maxhorowitz Feb 25, 2025

Otterverse Feb 26, 2025

maxhorowitz Feb 26, 2025

Otterverse Feb 26, 2025

Otterverse left a comment

Otterverse Feb 26, 2025

Otterverse Feb 26, 2025

Otterverse Feb 26, 2025

Otterverse Feb 26, 2025

Otterverse Feb 26, 2025

Otterverse Feb 26, 2025

Otterverse Feb 26, 2025

Otterverse Feb 26, 2025

Otterverse Feb 26, 2025

Otterverse Feb 26, 2025

Otterverse Feb 26, 2025

	n.logger, fmt.Sprintf("%s.%s.%s", n.cfg.Manufacturer, n.cfg.Model, n.cfg.FragmentID), n.getVisibleNetworks())
	n.logger, fmt.Sprintf("%s.%s.%s", n.Config().Manufacturer, n.Config().Model, n.Config().FragmentID), n.getVisibleNetworks())

[APP-7566]: Add bluetooth provisioning as a default provisioning method which runs parallel to hotspot provisioning. #70

Are you sure you want to change the base?

[APP-7566]: Add bluetooth provisioning as a default provisioning method which runs parallel to hotspot provisioning. #70

Conversation

maxhorowitz commented Feb 20, 2025 • edited Loading

Summary

Setup

Machine setup

Uninstall Viam

Remove local networks

Preinstall the Viam Agent

Phone setup

Testing Procedure

LightBlue

Pairing

Bluetooth service used to "transmit" credentials

Cases

maxhorowitz Feb 24, 2025 • edited Loading

Choose a reason for hiding this comment

maxhorowitz Feb 24, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Otterverse left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

maxhorowitz commented Feb 20, 2025 •

edited

Loading

maxhorowitz Feb 24, 2025 •

edited

Loading

maxhorowitz Feb 24, 2025 •

edited

Loading