Fix slow pantry caching by using memory as much as possible

Fix slow pantry caching by using memory as much as possible #313