Register Map convenience class for operating on device registers

[jameshilliard]

So I'm not sure if this is something that has been done with pymodbus before but I think it might be nice to have some sort of device register memory mapped convenience class/classes that gets initialized with all known device registers(ie based on device documentation) that would let you operate on registers as fields of some sort and potentially provide proxy methods that do device register read/writes based on mapped fields without having to re-specify field details like address and data types which would be provided in the initializer.

I'm thinking this would provide a cleaner register decoding capability along the likes of BinaryPayloadDecoder but would be initialized with a map of all device registers and allow mirroring of groups of registers along with out of order register payload decoding via field accessor methods of some sort.

Building register mapping classes was something I did in the past with a protocol that shares some similarities to modbus called upb that uses 8 bit register reads/writes instead of the 16 bit register reads/writes that modbus uses. To construct these memory map classes I essentially used a ctypes.BigEndianStructure to essentially provide register maps mirrored from the devices.

The main reason I think something like this would be helpful for modbus is that the existing decoder classes like BinaryPayloadDecoder have some limitations that seem to inherently result in unnecessary code duplication when it comes to register address definitions/mappings, for example BinaryPayloadDecoder doesn't allow for specifying ahead of time a full device register map that can be easily reused across different read/write operations. BinaryPayloadDecoder also requires retrieving register values by doing call order dependent reads on register data fields and thus isn't really suitable for storing register data for accessing later. In cases where we may want to provide two methods for say updating a large group of registers along with a smaller group of registers that is a subset of the larger group there doesn't seem to be a convenient way to decode the registers without essentially duplicating the code calling the BinaryPayloadDecoder decoder methods which is somewhat error prone as we then have to manually build partial register maps in our code calling the BinaryPayloadDecoder decoder/read methods.

Is there a reason BinaryPayloadDecoder uses pointer based decode operations for register decoding rather than say have those methods decode based on _payload address offsets passed to the decode methods or something along those lines?

[janiversen]

I think I understand what you suggest, for the server we have done that in form of the simulator, for the client we have been thinking about a higher level API that do e.g. ReadInt32, but never came around to implement it.

I cannot tell you why the payload functions work as they do, they are very old, and to a large on the way out...today we offer convert_from_registers, convert_to_registers with a datatype parameter as a much more efficient way of decoding/encoding.

I think you idea is very interesting, it is however still a bit vague to me how to convert it into actual code.

[jameshilliard]

In Homeassistant we have a type that do exactly that "datatype=custom". Maybe moving that code back into pymodbus is what you are looking for.

I think this is a bit different different what I'm trying to do. AFAICT that's more for defining a custom datatype on a single register while I'm more thinking of how to abstract out the entire device register map in a friendly way.

For example I have a PDU I'm currently dumping some config info from like this(it has a bunch more registers in addition to these):

@dataclass
class ANENPDUInfo:
    manufacturer: str
    model: str
    hardware_version: str
    software_version: int
    protocol_version: str
    compilation_time: str
    rated_voltage: int
    rated_current: int
    rated_frequency: int
    outputs: int
    output_switch_function: bool
    output_current_function: bool
    output_degree_function: bool
    voltage: int
    current: int
    active_power: int
    reactive_power: int
    power_factor: Decimal
    active_electrical_degree: int
    frequency: int
    l1_voltage: int
    l2_voltage: int
    l3_voltage: int
    l1_current: int
    l2_current: int
    l3_current: int
    l1_active_power: int
    l2_active_power: int
    l3_active_power: int
    l1_reactive_power: int
    l2_reactive_power: int
    l3_reactive_power: int
    l1_power_factor: Decimal
    l2_power_factor: Decimal
    l3_power_factor: Decimal

data = await c.read_holding_registers(0, 58, slave=self.address)
decoder = BinaryPayloadDecoder.fromRegisters(
    data.registers,
    byteorder=Endian.BIG,
    wordorder=Endian.BIG,
)
info = ANENPDUInfo(
    manufacturer=decoder.decode_string(8)
    .rstrip(b"\x00")
    .decode("utf-8"),
    model=decoder.decode_string(32).rstrip(b"\x00").decode("utf-8"),
    hardware_version=decoder.decode_string(2)
    .lstrip(b"\x00")
    .decode("utf-8"),
    software_version=decoder.decode_16bit_uint(),
    protocol_version=f"{decoder.decode_16bit_uint()}.{decoder.decode_16bit_uint()}",
    compilation_time=decoder.decode_string(8),
    rated_voltage=decoder.decode_16bit_uint(),
    rated_current=decoder.decode_16bit_uint(),
    rated_frequency=decoder.decode_16bit_uint(),
    outputs=decoder.decode_16bit_uint(),
    output_switch_function=bool(decoder.decode_16bit_uint()),
    output_current_function=bool(decoder.decode_16bit_uint()),
    output_degree_function=bool(decoder.decode_16bit_uint()),
    voltage=decoder.decode_16bit_uint(),
    current=decoder.decode_16bit_uint(),
    active_power=decoder.decode_16bit_uint(),
    reactive_power=decoder.decode_16bit_uint(),
    power_factor=decoder.decode_16bit_uint(),
    active_electrical_degree=decoder.decode_32bit_uint(),
    frequency=decoder.decode_16bit_uint(),
    l1_voltage=decoder.decode_16bit_uint(),
    l2_voltage=decoder.decode_16bit_uint(),
    l3_voltage=decoder.decode_16bit_uint(),
    l1_current=decoder.decode_16bit_uint(),
    l2_current=decoder.decode_16bit_uint(),
    l3_current=decoder.decode_16bit_uint(),
    l1_active_power=decoder.decode_16bit_uint(),
    l2_active_power=decoder.decode_16bit_uint(),
    l3_active_power=decoder.decode_16bit_uint(),
    l1_reactive_power=decoder.decode_16bit_uint(),
    l2_reactive_power=decoder.decode_16bit_uint(),
    l3_reactive_power=decoder.decode_16bit_uint(),
    l1_power_factor=decoder.decode_16bit_uint(),
    l2_power_factor=decoder.decode_16bit_uint(),
    l3_power_factor=decoder.decode_16bit_uint(),
)

The issue here is that this dataclass pattern only really works well when all registers being read are directly adjacent to each other and in cases where I want to actually read them all at once. It's also limited to the max modbus read length and would want to be able to partially update the register map class and have it cache the device registers(basically have it act as a proxy class for device specific register read/write operations).

What I want to be able to do is have a class representing a specific hardware device sorta like ANENPDUInfo but where I can pre-define all known register names with their address offsets and datatypes to read from and store register data into a using their register names without having to compute specific register offsets other than when defining the register map.

I'm thinking the register map proxy class would be sorta like a home assistant device with each individual register being sorta like a home assistant entity or something like that.

[janiversen]

May I suggest to keep it simple, a first level would logically be to support 1 read registers call, and interpret the result as INT32, CUSTOM, ... that is something useful for most pymodbus users.

Once that is working, then its easy to expand to cover more.

Btw. the information structure is a good example of custom usage, we have seen several configurations in HA doing that.

[jameshilliard]

May I suggest to keep it simple, a first level would logically be to support 1 read registers call, and interpret the result as INT32, CUSTOM, ... that is something useful for most pymodbus users.

Doesn't pymodbus handle that sort of thing with convert_to_registers and convert_from_registers fairly easily already? I mean that could probably be made a little nicer but I'm more looking for a good abstraction for operating on the entire device register map representation of some sort(ie similar to ctypes fields or something where you create a structure mapping class). Or are you referring to building a predefined register map structure like I was suggesting that can act as a read/write proxy but for individual registers?

Btw. the information structure is a good example of custom usage, we have seen several configurations in HA doing that.

Isn't that just for a custom datatype decoder and not really for handling encoding/decoding multiple register items(not sure the correct terminology) via a predefined register map? Home assistant's pymodbus integration itself has some datastructures for mapping multiple registers from the looks of it but it seems like it just operates on each individually and doesn't really handle say mapping data from a multi-register read into multiple variables AFAICT.

[janiversen]

The HA modbus integration provides support for native datatypes like int32 etc, no mapping as such, it also supports a custom datatypes for users who want to read multiple consecutive registers and convert the result according the the struct supplied.

Mapping a whole device, and have it update automatically is an interesting idea (very similar to a SCADA realtime db), surely sitting on top of the current API, but it is a bigger implementation.

[jameshilliard]

The HA modbus integration provides support for native datatypes like int32 etc, no mapping as such, it also supports a custom datatypes for users who want to read multiple consecutive registers and convert the result according the the struct supplied.

Yeah, I just mean this isn't really designed for operating on an entire device register map, but rather is an abstraction more over individual register operations(or adjacent groups of registers that decode to a single python variable).

Mapping a whole device, and have it update automatically is an interesting idea (very similar to a SCADA realtime db), surely sitting on top of the current API, but it is a bigger implementation.

Yeah, I think this is more what I was thinking of as individual register reads/writes are already relatively straightforward to handle in downstream projects. I'm thinking more like a mapping class with passthrough methods that proxy requests to the existing register read/write methods but with request/response caching and register to variable/property mapping capability essentially.