make files in /app/data persistent

[scruysberghs]

Describe the bug
A clear and concise description of what the bug is.

When I try to run the mpc optimizer with 'load_forecast_method': 'typical' I get the following error message in the log:

  File "/app/src/emhass/forecast.py", line 1071, in get_load_forecast
    with open(data_path, "rb") as fid:
         ^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/app/data/data_train_load_clustering.pkl'

To Reproduce
Steps to reproduce the behavior
pass 'load_forecast_method': 'typical' runtime parameter to mpc or day ahead api calls or set the option in the config.

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Home Assistant installation type

• Home Assistant OS

Your hardware

• OS: HA OS
• Architecture: amd64

EMHASS installation type

• Add-on

Additional context
Add any other context about the problem here.
Running v0.12.6. Was hoping the "Solved missing file problem: data_load_cost_forecast.csv" fix in the latest version would also solve this but it didn't.

Also noticed I had to rerun the ml fit after upgrading to the latest version to regenerate the load_forecast_mlf.pkl .
I guess a good solution would be to map the whole /app/data folder to the host to make it persistent? Haven't looked at addon docker structure in detail but from what I can see the /data folder is mapped to the host. Maybe a symlink from /app/data to /data could solve all these ?

[scruysberghs]

Just tested that even restarting your emhass add-on wipes the de /app/data folder.

This a shell inside the running emhass app-on docker container in HAOS using "docker exec -ti emhasscontainerid /bin/bash"

root@5b918bf2-emhass:/app/data# ls
actionLogs.txt	data_load_cost_forecast.csv  entities  injection_dict.pkl  load_forecast_mlf.pkl  opt_res_latest.csv  params.pkl
root@5b918bf2-emhass:/app/data# #        

Restart the add-on, so new containerid:

  ➜  ~ docker ps -a | grep emhass            
7af41e183809   ghcr.io/davidusb-geek/emhass:v0.12.6                           "uv run gunicorn emh…"   5 seconds ago   Up 4 seconds              0.0.0.0:5000->5000/tcp, :::5000->5000/tcp                                                                                                                                                                                                                                                                                                                    addon_5b918bf2_emhass
➜  ~ docker exec -ti 7af41e183809 /bin/bash
root@5b918bf2-emhass:/app# ls
README.md  data  gunicorn.conf.py  pyproject.toml  src	uv.lock
root@5b918bf2-emhass:/app# cd data/
root@5b918bf2-emhass:/app/data# ls
actionLogs.txt	data_load_cost_forecast.csv  opt_res_latest.csv  params.pkl

Luckily I had made a backup before restarting the container using "root@5b918bf2-emhass:/app/data# cp -r . /data/backup/"
and was able to get the files back using "root@5b918bf2-emhass:/app/data# cp -r /data/backup/* ." ofter the restart.

[scruysberghs]

@davidusb-geek made a small PR to modify the the Dockerfile to fix this.

[davidusb-geek]

Thanks for your contribution and for fixing this issue

[scruysberghs]

Thanks for your contribution and for fixing this issue

must admit I haven't tested the fix yet. The devcontainer builds ok and seems to work but I didn't have the time to try it out with a fork of the add-on pointing to my repo in my setup. Is there an easy way to do this?

[davidusb-geek]

In that case maybe this is the best option: https://emhass.readthedocs.io/en/latest/develop.html#docker-run-add-on-method

Or any of the other methods presented there concerning the add-on

[davidusb-geek]

But for the add-on I'm not 100% sure, but I think that your fix applies mostly to the docker standalone installation method.
What do you think @GeoDerp ?

[GeoDerp]

Yeah I would like to have a look at this issue myself. Looks like there may be a deeper issue at play here. The symlink fix may be a good method of fixing it, I'll try it out. 🤔

[GeoDerp]

@davidusb-geek the docker file is the same for both add-on and standalone 👍

[davidusb-geek]

Yes I know but the HA supervisor seems to have a utility that allow for data persistance after reboot or upgrading. That's what I meant, that this fix will be useful mostly for the standalone docker

[scruysberghs]

Don't know what you mean exactly by "HA supervisor seems to have a utility that allow for data persistance after reboot or upgrading", I've gone trough the add-on documentation and can only find a reference to the /data folder being used to store persitant data on the HA host.
On this page (https://developers.home-assistant.io/docs/add-ons/configuration/) it reads "/data is a volume for persistent storage."

An alternative to symlinking /app/data to a subfolder in /data could be to change line 569 of emhass-master/src/emhass/utils.py from:
DATA_PATH = os.getenv("DATA_PATH", default="/app/data/")

to

DATA_PATH = os.getenv("DATA_PATH", default="/data/")

[GeoDerp]

Sorry @scruysberghs @davidusb-geek I havent really had mutch time to properly read issues recently.

For running EMHASS via docker you can use a volume mount to the /data path :

docker run --rm -it --restart always  -p 5000:5000 --name emhass-container -v ./data:/app/data -v ./config.json:/share/config.json -v ./secrets_emhass.yaml:/app/secrets_emhass.yaml ghcr.io/davidusb-geek/emhass:latest

For running EMHASS via Add-on, change the data path to share:

On this thought however, I do believe there is something weird going on with the data files. Will look into this

[GeoDerp]

The /data path should be persistent unless upgrade or uninstall. The fact that it may not be is a possible sign of issues.
PS. @scruysberghs the emhass[data_path] should be defaulting to /data I believe in a diff rent section of the code. However its good pointing out and I may change that line.

[GeoDerp]

Update: changing the data path to /data/.... I have no idea why i set it to /app/data/. @scruysberghs Ill place you as co author thumbsup

[GeoDerp]

3f7909f

[davidusb-geek]

Merged and released on v0.12.7