Portenta X8 RPC tutorial is not working at all

[michelgokan]

I followed this tutorial to establish a "Data Exchange Between Python on Linux & Arduino Sketch": https://docs.arduino.cc/tutorials/portenta-x8/python-arduino-data-exchange/

I've tried going through the tutorial with both the latest firmware image (861 (Release arduino-91.2)) and also 746 version, on my Portenta X8, tried both with and without the mid-carrier board.

I have py-serialrpc running as follows:

bash-5.1$ sudo docker compose up
[+] Building 0.0s (0/0)
[+] Running 2/2
 ✔ Network py-serialrpc_default           Created0.2s
 ✔ Container py-serialrpc-py-serialrpc-1  Created0.1s
Attaching to py-serialrpc-py-serialrpc-1
py-serialrpc-py-serialrpc-1  | server: m4-proxy, port: 5000

this is the output of my sudo journalctl -u m4-proxy -f:

bash-5.1$ sudo journalctl -u m4-proxy -f
Password:
Sorry, try again.
Password:
Jul 30 15:30:23 portenta-x8-200fc209dab6fad9 systemd[1]: Started M4 builtin led forwarder.
Jul 30 15:30:29 portenta-x8-200fc209dab6fad9 m4_proxy[750]: Registering service on port  5005  with functions  map[count:5005 len:5005 tty:5005]
Jul 30 15:31:05 portenta-x8-200fc209dab6fad9 m4_proxy[750]: Registering service on port  5005  with functions  map[count:5005 len:5005 tty:5005]

Here is the Arduino Sketch code according to the python-sensor-rpc mentioned in the tutorial without any sensors:

#include <RPC.h>
#include <SerialRPC.h>
#include <Wire.h>

void setup()
{
    Serial.begin(115200);
    Serial.println("BME680 test on M4");
    Wire.begin();

    Serial.println("Trying to find sensor...");
    
    //RPC.bind("status", []{ return bme.sensorID() == 0x60; });
    RPC.bind("temperature", []{ return 100; });
    RPC.bind("humidity", []{ return 200; });
    RPC.bind("pressure", []{ return 300; });
    RPC.bind("gas", []{ return 400; });
    RPC.bind("altitude", []{ return 500; });

    Serial.println("Starting");
}

void loop()
{
  Serial.print("Temperature = ");
  Serial.print(100);
  Serial.println(" *C");

  Serial.print("Pressure = ");
  Serial.print(200);
  Serial.println(" hPa");

  Serial.print("Humidity = ");
  Serial.print(300);
  Serial.println(" %");

  Serial.print("Gas = ");
  Serial.print(400);
  Serial.println(" KOhms");

  Serial.print("Approx. Altitude = ");
  Serial.print(500);
  Serial.println(" m");

  Serial.println();

  delay(1000);
}

...and here is the python code:

import time
from msgpackrpc import Address as RpcAddress, Client as RpcClient, error as RpcError


# Fixed configuration parameters
port = 8884
publish_interval = 5

# The M4 Proxy address needs to be mapped via Docker's extra hosts
m4_proxy_address = 'm4-proxy'
m4_proxy_port = 5001

def get_data_from_m4():
    """Get data from the M4 via RPC (MessagePack-RPC)

    The Arduino sketch on the M4 must implement the following methods
    returning the suitable values from the attached sensor:

    * `temperature`
    * `humidity`
    * `pressure`
    * `gas`
    * `altitude`

    """

    rpc_address = RpcAddress(m4_proxy_address, m4_proxy_port)

    data = ()

    try:
        rpc_client = RpcClient(rpc_address)
        temperature = rpc_client.call('temperature')

        rpc_client = RpcClient(rpc_address)
        humidity = rpc_client.call('humidity')

        rpc_client = RpcClient(rpc_address)
        pressure = rpc_client.call('pressure')

        rpc_client = RpcClient(rpc_address)
        gas = rpc_client.call('gas')

        rpc_client = RpcClient(rpc_address)
        altitude = rpc_client.call('altitude')

        data = temperature, humidity, pressure, gas, altitude

    except RpcError.TimeoutError:
        print("Unable to retrive data from the M4.")

    return data


if __name__ == '__main__':

    print()
    print("============================================")
    print("==       Portenta X8 Sensor reading       ==")
    print("============================================")
    print()

    try:
        while True:
            data = get_data_from_m4()
            if len(data) > 0:
                print("Temperature: ", data[0])
                print("Humidity: ", data[1])
                print("Pressure: ", data[2])
                print("Gas: ", data[3])
                print("Altitude: ", data[4])
            time.sleep(publish_interval)
    except KeyboardInterrupt:
        print('Stopped.')

Here is the Dockerfile:

FROM python:3.9-alpine AS Build

RUN set -ex \
    && apk add --no-cache --virtual .build-dep \
        git build-base python3-dev

RUN set -ex \
    && git clone https://github.com/kpn-iot/senml-python-library \
    && cd senml-python-library \
    && python3 setup.py bdist_wheel \
    && cd ..

RUN set -ex \
    && apk del .build-dep

FROM python:3.9-alpine

COPY --from=Build /senml-python-library/dist/kpn_senml-1.1.1-py3-none-any.whl ./
COPY requirements.txt ./

RUN set -ex \
    && pip3 --disable-pip-version-check --no-cache-dir install \
        -r requirements.txt \
    && rm ./kpn_senml-1.1.1-py3-none-any.whl \
    && rm ./requirements.txt

WORKDIR /app

COPY src/m4_to_python.py .

ENTRYPOINT [ "python3", "m4_to_python.py"]

...and here is the docker-compose.yml:

services:
  python-sensor-rpc:
    #image: hub.foundries.io/arduino/python-sensor-rpc:latest
    image: python-sensor-rpc:latest
    restart: unless-stopped
    environment:
      - PYTHONUNBUFFERED=1
    volumes:
      - /var/run/secrets:/app/config/
    extra_hosts:
      - "m4-proxy:host-gateway"

I basically haven't changed anything in the original python-sensor-rpc.zip and py-serialrpc.zip files, except for in py-serialrpc I also pip installed backports.ssl_match_hostname and backports.weakref packages. So no change whatsoever in the code.

Here is the error after bringing up the container:

sudo docker compose up
[+] Running 2/2
 ⠿ Network python-sensor-rpc_default                Created 0.2s
 ⠿ Container python-sensor-rpc-python-sensor-rpc-1  Created 0.1ss
Attaching to python-sensor-rpc-python-sensor-rpc-1
python-sensor-rpc-python-sensor-rpc-1  |
python-sensor-rpc-python-sensor-rpc-1  | ============================================
python-sensor-rpc-python-sensor-rpc-1  | ==       Portenta X8 Sensor reading       ==
python-sensor-rpc-python-sensor-rpc-1  | ============================================
python-sensor-rpc-python-sensor-rpc-1  |
python-sensor-rpc-python-sensor-rpc-1  | Unable to retrive data from the M4.
python-sensor-rpc-python-sensor-rpc-1  | Unable to retrive data from the M4.
python-sensor-rpc-python-sensor-rpc-1  | Unable to retrive data from the M4.
python-sensor-rpc-python-sensor-rpc-1  | Unable to retrive data from the M4.
python-sensor-rpc-python-sensor-rpc-1  | Unable to retrive data from the M4.
python-sensor-rpc-python-sensor-rpc-1  | Unable to retrive data from the M4.
python-sensor-rpc-python-sensor-rpc-1  | Unable to retrive data from the M4.
python-sensor-rpc-python-sensor-rpc-1  | Unable to retrive data from the M4.
python-sensor-rpc-python-sensor-rpc-1  | Unable to retrive data from the M4.
python-sensor-rpc-python-sensor-rpc-1  | Unable to retrive data from the M4.
python-sensor-rpc-python-sensor-rpc-1  | Unable to retrive data from the M4.

...I'm now getting this in m4-proxy output:

Jul 30 21:00:13 portenta-x8-200fc209dab6fad9 m4_proxy[827]: resolving  register
Jul 30 21:00:13 portenta-x8-200fc209dab6fad9 m4_proxy[827]: Registering service on port  5005  with functions  map[tty:5005]

Jul 30 21:53:04 portenta-x8-200fc209dab6fad9 m4_proxy[20457]: Serving 172.23.0.2:37126
Jul 30 21:53:04 portenta-x8-200fc209dab6fad9 m4_proxy[20457]: [0 0 [116 101 109 112 101 114 97 116 117 114 101] []]
Jul 30 21:53:04 portenta-x8-200fc209dab6fad9 m4_proxy[20457]: {[148 0 0 171 116 101 109 112 101 114 97 116 117 114 101 144] 0 0}
Jul 30 21:53:04 portenta-x8-200fc209dab6fad9 m4_proxy[20457]: wait for response
Jul 30 21:53:05 portenta-x8-200fc209dab6fad9 m4_proxy[20457]: done
Jul 30 21:53:20 portenta-x8-200fc209dab6fad9 m4_proxy[20457]: Serving 172.23.0.2:37128
Jul 30 21:53:20 portenta-x8-200fc209dab6fad9 m4_proxy[20457]: [0 0 [116 101 109 112 101 114 97 116 117 114 101] []]
Jul 30 21:53:20 portenta-x8-200fc209dab6fad9 m4_proxy[20457]: {[148 0 0 171 116 101 109 112 101 114 97 116 117 114 101 144] 0 0}
Jul 30 21:53:20 portenta-x8-200fc209dab6fad9 m4_proxy[20457]: wait for response
Jul 30 21:53:21 portenta-x8-200fc209dab6fad9 m4_proxy[20457]: done
Jul 30 21:53:36 portenta-x8-200fc209dab6fad9 m4_proxy[20457]: Serving 172.23.0.2:37130
...

I don't see any output in py-serialrpc output (I had to pip install backports.ssl_match_hostname and backports.weakref to make dockerfile compile though):

sudo docker compose up
[+] Running 2/2
 ⠿ Network py-serialrpc_default           Created 0.1s
 ⠿ Container py-serialrpc-py-serialrpc-1  Created 0.1ss
Attaching to py-serialrpc-py-serialrpc-1
py-serialrpc-py-serialrpc-1  | server: m4-proxy, port: 5000

Any ideas what's wrong and why the code is unable to call the RPC functions? I can't even see any of the Serial outputs neither on py-serialrpc output nor Arduino Serial Monitor. Without serial support, this board will be useless for many industrial use cases, including ours.

It's very likely that the m4-proxy is not capturing the RPC.bind from the Arduino Sketch side, because I tried a simple client/server on Python side and it was working okay! I'm not sure what's the exact problem :(

[Timespansoul]

I am having the same issue on version 881. I will keep tinkering however the lack of support is ... depressing. Someone please help us fix this. I can only imagine the hours collectively wasted on this issue.

[jcarolinares]

Hello @Timespansoul and @michelgokan

We just released a new version of the tutorial last week that hopefully will improve this user experience:
https://docs.arduino.cc/tutorials/portenta-x8/python-arduino-data-exchange/

Please take into account that this github repository is not the official arduino support and therefore resolutions can take more time than in our professional support.

Please contact my colleagues from support at: https://www.arduino.cc/en/contact-us/ who will check and try to assist you with all you need.

We will also double check the information provided here and make changes if needed, thank you for pointing out these problems!

Regards

[jcarolinares]

Hello, just an update, Arduino is working on addressing this issue during the following weeks, thanks!

[jcarolinares]

Hello @michelgokan I can confirm this is part of the development future plan, therefore content will be updated accordingly when improvements regarding RPC are addressed. Please contact our support team in case you need additional help!

https://www.arduino.cc/en/contact-us/

thanks!

[michelgokan]

Hi @jcarolinares,

Thanks. Well, I was in contact with the support for a long time but they weren't very helpful in finding and resolving this issue.

I know many people have this exact problem; you can check this thread here: https://forum.arduino.cc/t/portenta-x8-rpc-tutorial-is-not-working-on-the-latest-firmware/1286747/2

I had to switch our entire tech stack away from X8, just because of this problem. Please let me know if there is a remedy and I may be able to switch back.