fix(vault): secret is temporarily empty after changed vault config #14209
Conversation
When vault config is changed, in the worker event we first flush the LRU cache, then start to update the secrets from vault provider. There’s a period of time in between that the cache is empty. The `kong.vault.update()` function only lookups cache and will update the secret to an empty string when cache is empty. This can cause plugins to throw nil errors. This commit changed `kong.vault.update()` function to not touch it if not found in the cache.
github-actions
bot
added
core/pdk
cherry-pick kong-ee
When vault config is changed, in the worker event we first flush the LRU cache, then start to update the secrets from vault provider. There’s a period of time in between that the cache is empty. The `kong.vault.update()` function only lookups cache and will update the secret to an empty string when cache is empty. This can cause plugins to throw nil errors. This commit changed the worker event callback to not flush the LRU cache. The cache will be updated when the secrets are fetched from vault prodiver.
|
Pushed a new commit to fix the issue in another way: instead of not touching input table in the pdk update function, in this new commit I completely removed the LRU:flush_all() line in the worker event. The LRU capacity is fixed and the keys has ttls, so there likely won’t be memory leaks. |
What about the configuration stickyness issue (just double checking that stickyness is not back with this change)? |
kong/pdk/vault.lua
Outdated
| @@ -1429,8 +1429,6 @@ local function new(self) | |||
| end | |||
| end | |||
|
|
|||
| LRU:flush_all() | |||
|
|
|||
There was a problem hiding this comment.
The LuaDoc has:
---
-- Flushes LRU caches and forcibly rotates the secrets.
--
-- This is only ever executed on traditional nodes.
I guess this may need to be changed. Also should it be executed with incremental too?
|
Hi @bungle sorry for the late reply, I just took another look at this. I think current approach may not be ideal for these reasons:
Do you think the first commit is okay? The only downside is the behavior of kong.vault.flush() changed slightly. But the function is marked as experimental in the docs, and seems that the usage of the function in kong will not be affected. If it's acceptable, I will revert the second commit and change the failed test "update function sets values to empty string on failure". |
… config" This backs out commit 4f7cf09.
|
@cshuaimin The first commit is ok with me! Thanks! |
The test "resurrects plugin config references when secret is deleted" failed. It is asserting that when a secret resurrect ttl is passed, it should be deleted. However with previous update() function logic the value is only set if it's not nil. This commit changes the logic to set the value based on if there's an error or not.
cshuaimin
force-pushed
the
fix/vault-cache-flush-issue
branch
from
00e069a to
9d82074
Compare
Summary
When vault config is changed, in the worker event we first flush the LRU cache, then start to update the secrets from vault provider. There’s a period of time in between that the cache is empty.
The
kong.vault.update()function only lookups cache and will update the secret to an empty string when cache is empty. This can cause plugins to throw nil errors.This commit changed
kong.vault.update()function to not touch it if not found in the cache.Checklist
changelog/unreleased/kongorskip-changeloglabel added on PR if changelog is unnecessary. README.mdIssue reference
FTI-5936