Traceback error when running naive-mpc-optim

[Excentyl]

Describe the bug
On version 0.12.8, naive-mpc-optim sometimes fails. Over the last few days, I have found it tends to fail if ran in the early morning, but works at other times.

To Reproduce
Use a rest command to call naive-mpc-optim:

rest_command:
  run_emhass_optimisation:
    url: http://localhost:5000/action/naive-mpc-optim
    method: POST
    content_type: "application/json"
    payload: >-
      {
        "load_cost_forecast": {{
          ((state_attr('event.smart_meter_electricity_import_current_day_rates', 'rates') + state_attr('event.smart_meter_electricity_import_next_day_rates', 'rates')) | selectattr('end','gt',now()) | map(attribute='value_inc_vat') | list) | tojson 
        }},
        "prediction_horizon": {{
          min(48, (state_attr('event.smart_meter_electricity_import_current_day_rates', 'rates') + state_attr('event.smart_meter_electricity_import_next_day_rates', 'rates')) | selectattr('end','gt',now()) | map(attribute='value_inc_vat') | list | length)
        }},
        "pv_power_forecast": {{
          ((state_attr('sensor.solcast_pv_forecast_forecast_today', 'detailedForecast') + state_attr('sensor.solcast_pv_forecast_forecast_tomorrow', 'detailedForecast')) | selectattr('period_start','gt',now()) | map(attribute='pv_estimate') | map('multiply',1000) | list) | tojson
        }},
        "soc_init": {{ states('sensor.solar_battery_capacity') | float / 100 }}
      }

The following is printed in the logs:

[2025-03-05 21:30:00 +0000] [23] [INFO]  >> Obtaining params: 
[2025-03-05 21:30:00 +0000] [23] [INFO] Passed runtime parameters: {'load_cost_forecast': [0.254432, 0.124774, 0.124774, 0.124774, 0.124774, 0.254432, 0.254432, 0.254432, 0.254432, 0.254432, 0.254432, 0.254432, 0.254432, 0.124774, 0.124774, 0.124774, 0.124774, 0.124774, 0.124774, 0.254432, 0.254432, 0.254432, 0.254432, 0.254432, 0.254432, 0.254432, 0.254432, 0.254432, 0.254432, 0.254432, 0.254432, 0.124774, 0.124774, 0.124774, 0.124774, 0.124774, 0.124774, 0.381647, 0.381647, 0.381647, 0.381647, 0.381647, 0.381647, 0.254432, 0.254432, 0.254432, 0.254432, 0.254432, 0.254432, 0.124774, 0.124774, 0.124774, 0.124774], 'prediction_horizon': 48, 'pv_power_forecast': [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 5.8999999999999995, 160.3, 577.2, 1261.0, 1771.9, 2607.4, 3092.2999999999997, 3474.5, 3798.6, 3988.3, 4119.3, 4153.900000000001, 4093.6000000000004, 3939.9, 3531.1, 3236.9, 2535.1, 2067.0, 1449.2, 914.5, 332.6, 128.3, 11.2, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], 'soc_init': 0.57}
[2025-03-05 21:30:00 +0000] [23] [INFO]  >> Setting input data dict
[2025-03-05 21:30:00 +0000] [23] [INFO] Setting up needed data
[2025-03-05 21:30:00 +0000] [23] [INFO] Retrieve hass get data method initiated...
[2025-03-05 21:30:00 +0000] [23] [INFO] Retrieving weather forecast data using method = list
[2025-03-05 21:30:00 +0000] [23] [INFO] Retrieving data from hass for load forecast using method = mlforecaster
[2025-03-05 21:30:00 +0000] [23] [INFO] Retrieve hass get data method initiated...
[2025-03-05 21:30:01 +0000] [23] [INFO]  >> Performing naive MPC optimization...
[2025-03-05 21:30:01 +0000] [23] [INFO] Performing naive MPC optimization
[2025-03-05 21:30:01 +0000] [23] [INFO] Perform an iteration of a naive MPC controller
[2025-03-05 21:30:01 +0000] [23] [ERROR] Exception on /action/naive-mpc-optim [POST]
Traceback (most recent call last):
  File "/app/.venv/lib/python3.12/site-packages/flask/app.py", line 1511, in wsgi_app
    response = self.full_dispatch_request()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/flask/app.py", line 919, in full_dispatch_request
    rv = self.handle_user_exception(e)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/flask/app.py", line 917, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/flask/app.py", line 902, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/src/emhass/web_server.py", line 472, in action_call
    opt_res = naive_mpc_optim(input_data_dict, app.logger)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/src/emhass/command_line.py", line 686, in naive_mpc_optim
    opt_res_naive_mpc = input_data_dict["opt"].perform_naive_mpc_optim(
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/src/emhass/optimization.py", line 1456, in perform_naive_mpc_optim
    self.opt_res = self.perform_optimization(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/src/emhass/optimization.py", line 497, in perform_optimization
    e=P_PV[i]
      ^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/pulp/pulp.py", line 235, in __rsub__
    return other - LpAffineExpression(self)
           ~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~
  File "/app/.venv/lib/python3.12/site-packages/pulp/pulp.py", line 951, in __rsub__
    return (-self).addInPlace(other)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/pulp/pulp.py", line 919, in addInPlace
    raise const.PulpError("Cannot add/subtract NaN/inf values")
pulp.constants.PulpError: Cannot add/subtract NaN/inf values

Expected behavior
Able to run naive-mpc-optim without failure

Screenshots
Response when attempting to call the rest command in Home Assistant:

Home Assistant installation type
Home Assistant OS

Your hardware

• OS: HA OS
• Architecture: amd64

EMHASS installation type

• Add-on

[RudolfRendier]

Do you have EMHASS configured to replace NaN with 0s?
That setting might impact this behaviour.

Here's a snippet from my config json, in which it is enabled.

  "sensor_linear_interp": [
    "sensor.power_load_no_var_loads",
    "sensor.solar_panels_total_output_power"
  ],
  "sensor_power_load_no_var_loads": "sensor.power_load_no_var_loads",
  "sensor_power_photovoltaics": "sensor.solar_panels_total_output_power",
  "sensor_replace_zero": [
    "sensor.power_load_no_var_loads",
    "sensor.solar_panels_total_output_power"
  ],

[Excentyl]

Below is the corresponding snippet from my config:

  "sensor_linear_interp": [
    "sensor.solar_house_load"
  ],
  "sensor_power_load_no_var_loads": "sensor.solar_house_load",
  "sensor_power_photovoltaics": "sensor.solar_pv_power_total",
  "sensor_replace_zero": [
    "sensor.solar_pv_power_total"
  ],

If you think I need to try a different setting, please let me know and I will happily try it.

[tbrasser]

To me these 2 config options seem mutually exclusive: you either replace NaN with 0, or you interpolate NaN values. Cannot do both.
(or if one is applied after the other, the latter one has no effect because all NaN values have been replaced already)

[Excentyl]

I have an automation which attempts to run naive-mpc-optim every half hour and I have been monitoring my issue for the last few days and have noticed that every day it gives this traceback error between 19:00 until 06:30 (inclusive).

Every day, from 07:00 until 18:30 there are no errors.

I haven't been able to find any NaN's in my data as of yet.

[davidusb-geek]

To me these 2 config options seem mutually exclusive: you either replace NaN with 0, or you interpolate NaN values. Cannot do both.
(or if one is applied after the other, the latter one has no effect because all NaN values have been replaced already)

Not completely true, but not intuitive I'll give you that.
For example if you have NANs at several observations at the beginning and at the end of your sensor (this can happen with PV sensors) and you apply interpolation, then those NANs at the beginning and the end are not interpolated because probably an extrapolation is needed. In that case a seond step replacing by 0 or any other number might be needed.

@Excentyl Add the two sensors (solar and load) to both options (interp and replace) just as posted by @RudolfRendier

[Excentyl]

Thanks a lot @davidusb-geek, that seems to have worked and I am no longer getting the traceback errors.

What I find strange is that I am providing the PV forecast using the runtime variable pv_power_forecast (which seems to be the problem from the logs, assuming that is the P_PV[i] variable), so I do not understand how setting the solar and load sensors to interp and replace fixes the issue. I only ask because I wonder if there is an underlying issue here? I could be wrong, but I thought it would be worth asking the question first before closing the issue?

[davidusb-geek]

When using MPC, even if you provide your own forecast at runtime, the system needs to fetch some data from Home Assistant to implement the current/now values, see here: https://emhass.readthedocs.io/en/latest/forecasts.html#now-current-values-in-forecasts