Skip to content
This repository has been archived by the owner on Jan 24, 2023. It is now read-only.

Is it possible to set a TTL expiration on keys added to redis? #107

Open
HoukasaurusRex opened this issue Aug 11, 2021 · 3 comments
Open
Labels
question Further information is requested

Comments

@HoukasaurusRex
Copy link

HoukasaurusRex commented Aug 11, 2021

I use this module to stream IoT data to a real-time monitoring application. The storage size of the data grows exponentially with the amount of connected devices, so it's important to use a strategy of evicting keys once storage is full. I know for redis cli and some redis modules, I can set a TTL on keys using an expire attribute or something similar depending on the module, but I haven't found anything like that in the redistimeseries documentation.

Something like this is what I'm wondering is possible with this module:

from redistimeseries.client import Client
rts = Client()
rts.create('expire-soon-test', labels={'Time':'Series'})
rts.add('expire-soon-test', 1, 1.12, expire=10)

What's the best way to attach a TTL to the timeseries data streamed to redis using redistimeseries?

@gkorland gkorland added the question Further information is requested label Aug 11, 2021
@gkorland
Copy link
Contributor

Notice you have three ways to set TTL/EXPIRE/RETENTION.

  1. Set LRU eviction policy on the server level see: https://redis.io/topics/lru-cache
  2. Set TTL on the key level using Redis EXPIRE method see: https://redis.io/commands/expire
  3. Set RETENTION policy on the Series level which will expire old events in two ways.
    1. see: https://oss.redislabs.com/redistimeseries/commands/#tscreate
    2. see: https://oss.redislabs.com/redistimeseries/configuration/#retention_policy

@HoukasaurusRex
Copy link
Author

Thanks for the response, using an LRU is almost what I need, however it's not always appropriate. I'm not sure if there are any protections for recently created keys, and since keys are evicted based on the LRU of random sample sets, isn't it possible to have random holes in relatively recent data if it hasn't been accessed yet?

Using the Redis EXPIRE command is another option, though wouldn't that require creating another redis client connection in the same script? Something like:

from redistimeseries.client import Client
import redis
rts = Client()
rts.create('expire-soon-test', labels={'Time':'Series'})
rts.add('expire-soon-test', 1, 1.12)
redis.expire(name='expire-soon-test', time=10)

It seems to me that I would need to maintain two separate redis connections per device running two operations per data point, which might negatively impact IO performance on the server.

Looking at the RETENTION policy of the series though looks perfect for my use case as I should be able to set a TTL on all keys even from server side. This should be fine for me, thanks for the answer!

@gkorland
Copy link
Contributor

Looking at the RETENTION policy of the series though looks perfect for my use case as I should be able to set a TTL on all keys even from server side. This should be fine for me, thanks for the answer!

You should notice that RETENTION will expire old events not the all series.

As for EXPIRE you can do the following:

from redistimeseries.client import Client
rts = Client()
rts.create('expire-soon-test', labels={'Time':'Series'})
rts.add('expire-soon-test', 1, 1.12)
rts.redis.expire(name='expire-soon-test', time=10)

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
question Further information is requested
Projects
None yet
Development

No branches or pull requests

2 participants