Use a precompiled Portduino version of firmware for python integration testing?

[geeksville]

Hi @ianmcorvidae and @thebentern,

I'm making this issue to discuss an idea. In the 'old days' we had a compiled C++ version of the firmware code that ran on linux using the portduino layer. We used that binary in the CI build of the python toolchain to do integration testing - because we could talk to it like a "real" device (because it was 95% the same). This was handy for integration test purposes.

I guess that somewhere over time that got deprecated/removed.

Now that there is slick codecoverage support that someone added (@ianmcorvidae ?) for python: what do you think about using that again?

The reason I ask is that in cleaning up my power measurement stuff for checkin I realized that the PowerStress stuff needs code coverage testing. I could learn how to use the python mocking framework, but I'm worried that even if I use that framework if I mock out the PowerStress class to pretend to be the real device I'm not really testing that much. Because PowerStress is mostly done on the device side.

Instead: (proposal)

• Use a precompiled linux device firmware (do ya'll already/still do a linux build of the device firmware?), fire it up.
• Run the python tests as integration tests but have them talk via TCP to the actual device firmware fired up previously.
• This would allow testing lots more of the python code - and also implicitly testing the device code as well. So it would be a true integration test.
• Testing all this code would then be seen by the code coverage tool.

Does this seem plausible? Would you like changes?

My current queue is:

• finishing the power stuff (done next few days?)
• stress testing nrf52 filesystem stuff to try and reproduce the reported rare corruption and make a robust fix (idk, a couple of weeksish?)
• do this third?

[ianmcorvidae]

I've thought similarly, so I definitely agree! There's still a SimRadio system built off portduino, used among other things by the simulator (Meshtasticator), as well as the actual linux-native meshtasticd that can talk to a radio using SPI. So I think it's quite within reach. I'm not who added the code coverage testing -- I think it's nice for knowing what's being tested to some extent, but I'm not sold on the broad usefulness of striving for 100% coverage. I've worked a little bit with hypothesis, a property-based testing system (if you've ever touched Haskell -- QuickCheck is property-based testing), and I'd love if a lot of our tests could move to something like that which works more off stating assumptions and checking that they hold, with something like what you're talking about doing integration-type testing (which is what a lot of our current stuff is doing, albeit by way of a lot of mocks that aren't always that current and a lot that aren't even functional -- so something running actual firmware code would definitely be better).

As far as implicitly testing the firmware code, I think that's also good and we might even be able to build this such that it can optionally use real devices (somewhere far away from defaults/the public mesh), which could uncover device-specific bugs, potentially. I don't imagine we could systematize that without a lot of resources (especially given how many devices we support these days), but it'd be a cool option to have.

Basically: yes, I like this!

[geeksville]

ooh Meshtasticator looks great (I didn't even know this hotness existed - awesome work @GUVWAF !) and is probably an indirect descendent of that old linux build I was mentioning. awesome!

[geeksville]

ooh the docker 'auto build and run the native build' container stuff is slick.

[thebentern]

I'm game! I think some of the nominal testing that was originally there shook loose from the 1.3/2.0 days when we moved all of the config protos around.

Long term, I'd actually like to have real IoT hardware in play for e2e testing. I think ESP32 based targets could work the easiest, since we have esptool access already in python. I wonder if a USB hub with programmatically switchable ports exists, because inevitably when something goes wrong, power cycling could save us some flakey test heartache.

[GUVWAF]

While I'm not proficient with Python testing, I'm happy to help with some Python glue code to create a basic test scenario of 3 Portduino nodes that talk to each other with a hop in between. I believe with that you can cover a lot of test cases for mesh communication (broadcasts, DMs with ACK, traceroute, etc.) already.