Problem this feature would solve
Currently, vCache uses InMemoryEmbeddingMetadataStorage, which means that cached data is lost whenever the application restarts. In production environments, this can lead to repeated LLM requests, increased latency, and higher API costs.
Proposed feature
Add support for a persistent storage backend (such as Redis or another database) while keeping the current in-memory storage as the default option.
How this feature would improve vCache
A persistent storage layer would allow cached responses and metadata to survive application restarts, making vCache more reliable for production deployments. It would reduce unnecessary LLM calls, improve response latency, and help lower operational costs.
Possible implementation approach
Introduce a new storage implementation that follows the existing storage interface. For example, adding RedisEmbeddingMetadataStorage and allowing users to configure the storage backend through VCacheConfig without changing the core vCache architecture.
Use cases
This would be useful for applications running vCache as a long-lived service, where deployments, scaling, or restarts should not remove previously cached information.
Contribution
I would be happy to implement this feature if it aligns with the project's goals. I would appreciate feedback from the maintainers before starting the implementation to ensure the design matches the project's direction.
Problem this feature would solve
Currently, vCache uses
InMemoryEmbeddingMetadataStorage, which means that cached data is lost whenever the application restarts. In production environments, this can lead to repeated LLM requests, increased latency, and higher API costs.Proposed feature
Add support for a persistent storage backend (such as Redis or another database) while keeping the current in-memory storage as the default option.
How this feature would improve vCache
A persistent storage layer would allow cached responses and metadata to survive application restarts, making vCache more reliable for production deployments. It would reduce unnecessary LLM calls, improve response latency, and help lower operational costs.
Possible implementation approach
Introduce a new storage implementation that follows the existing storage interface. For example, adding
RedisEmbeddingMetadataStorageand allowing users to configure the storage backend throughVCacheConfigwithout changing the core vCache architecture.Use cases
This would be useful for applications running vCache as a long-lived service, where deployments, scaling, or restarts should not remove previously cached information.
Contribution
I would be happy to implement this feature if it aligns with the project's goals. I would appreciate feedback from the maintainers before starting the implementation to ensure the design matches the project's direction.