[Sleep Log] Gadgetbridge sync support

[joserebelo]

I would like to implement sync support for the sleep logs to Gadgetbridge.

I see that we already send real-time events to Gadgetbridge, but they are currently not being handled. Aditionally, any events that happen while disconnected will be lost.

I was envisioning a mechanism similar to #2889, where we request all the sleep logs since a timestamp and update existing records in Gadgetbridge, along with a button in the preferences to trigger a full fetch of all data present on the Bangle, as that might take some time to execute.

@storm64 do you have any thoughts on this?

[gfwilliams]

I could be wrong here, but as far as I can see from https://github.com/espruino/BangleApps/blob/master/apps/sleeplog/boot.js#L27 sleep log mainly uses the 10 minute 'movement' value from the built-in health event.

This is great because it's not really using any extra power, but also the health app is already storing this information too, all the time. And it's synced already via the {t:'act', mov:...} - we may just not be storing it in Gadgetbridge?

... so in a way, we may just be able to run this algorithm on data that's already in gadgetbridge? Looks like sleeplog does check wear state and whether it's charging though, which we don't store (but charge state at the very least could be sent/stored trivially, and I guess if it's not worn we won't have a BPM reading for that 10 min health event).

That's one thought, but also sleeplog saves everything to a storage file. Perhaps we should consider having some kind of generic file sync built into Gadgetbridge rather than setting out to do it just for sleeplog?

• metadata.json already has a data field listing data files the apps make. Maybe if sync:true is set there, we could make sure that's pushed through to appname.info in the bangle
• when syncing Gadgetbridge could scan those files and figure out a list of files to sync, and could then automatically sync them

Espruino has two different storage types (one fixed length and binary, one appendable and text). There's already the code to handle syncing the appendable one for tracks thanks to @thyttan, but I guess we might need to add something to deal with the binary one.

[thyttan]

Off topic:

@gfwilliams

This is great because it's not really using any extra power

I also like this aspect of sleeplog. But every time I use it for a while I end up suspecting the app of being a big drain on battery somehow. Do you have this experince as well? Could it be writes to storage? I haven't looked at this deeply yet (and don't know if I ever will).

[gfwilliams]

I haven't used it I'm afraid - but it depends how big the sleeplog JSON file is - if it is big it could end up being a huge drain.

Using StorageFile for stuff that gets appended to would be the way to go... Or actually we could extend the 'health' app's file format with extra fields.

• battery and wear state would be a good one
• we could also have a field for 'activity type' which apps could update, so sleeplog could just update that and then it'd be saved automatically. Similarly the 'run' app could automatically set activity type when it's running, the cycle app, and so on.

I feel like having some kind of global activity state would probably fit better with what Gadgetbridge expects?

[thyttan]

If we extended the 'health' file format that way, would that mean we could refactor sleeplog to just display data from the 'health' file? Sounds pretty neat.

[gfwilliams]

would that mean we could refactor sleeplog to just display data from the 'health' file?

Yes - it'd still have to run over the data from the day and count up all the moments of sleep and how long and so on, but it'd just be a case of looking at whether activityType==DEEP_SLEEP and so on.

but it depends how big the sleeplog JSON file is - if it is big it could end up being a huge drain.

I just checked again and I was totally wrong here. It uses StorageFile already so that should really have no big effect on power usage no matter how big the file gets.

[joserebelo]

... so in a way, we may just be able to run this algorithm on data that's already in gadgetbridge?

One one hand, this would work for the current implementation. On the other, I'm left wondering if this would prevent the algorithm from being improved (eg. taking small bursts of real-time data like Sleep as Android does).

I feel like having some kind of global activity state would probably fit better with what Gadgetbridge expects?

I do like this idea, but I'm not sure we should just think about what Gadgetbridge expects :) its current implementation is driven by how the first gadgets that needed saved data, which is not always the most efficient or easier to work with format. It might also be a bit heavier on the Bangle side storage-wise, as we'd need to keep the sleep state for every 10-minute block, instead of a sleep range "from X time to Y" (I did not check how the current sleep log app saves it).

We have been having to merge the data from multiple sources for some gadgets (steps from a table, heart rate from another, sleep from another), so that's not really a problem.

[gfwilliams]

It might also be a bit heavier on the Bangle side storage-wise, as we'd need to keep the sleep state for every 10-minute block

Yes, however right now we store 4 bytes for every 10 minutes, so that's 18kB per month - which isn't a big deal as we have 8MB available. I was thinking maybe we add a bit to that.

• 1 byte could store charge state, wear state, and current activity easily.
• I also fancy storing battery level and altitude (packed into 2 bytes)

Not sure if there's anything else?

[thyttan]

Pinging @storm64 again in case you missed the first one. What's your thoughts on the discussion here?

[gfwilliams]

@storm64 ?

[thyttan]

Semi-monthly ping @storm64

[storm64]

Sorry for me not responding, the working life was a bit too demanding.
It's been a while since I've done the coding, but here are some thoughts on some points mentioned:

1. Recalculating sleep data on the smartphone side:
   Might be possible if we know the "wearing"-state and a movement value with timestamp.

2. Reciving live changes from sleeplog:
   To get changes while connected to Gadgetbridge you could simply add a trigger as described in #add-functions-triggered-by-status-changes-or-inside-a-specified-time-period.

3. Reciving sleep data from sleeplog:
   I'd implementet some utils to get the sleep data for a defined time period quiet fast (depending on the range).
   You could simply run require("sleeplog").getStats(0, {UNIX timestamp of last connection}); on reconnect.
   More informaton on this: #access-statistics

4. Battery usage and storage types:
   To be as gentle to the battery as possible (for my skils), I implemented the following features:

• Using the 10min-movement-values and only checking for "wearing" if no movement is detected.
• Append only status changes with some necessary additional data to a StorageFile.
• Moving the cleaned sleep data over to a normal file and clear the StorageFile every 14days, so it doesn't get to big. See 5. in #timestamps-and-files.

Thanks for working on this topic. I try to be faster on responding, now. ;)

[thyttan]

Nice to hear back from you @storm64 :)

ping @joserebelo @gfwilliams

[32bitx64bit]

I searched up for some reason this didn't show up. Glad to see work getting done on this. Hopefully development moves quickly and efficiently 😄

[32bitx64bit]

@storm64 just giving a ping to see if any progress has been made.

[32bitx64bit]

Just doing the monthly checkup.