Summary
As OpenBalancer grows, adding support for new inference providers should require minimal code changes.
Currently, each new provider integration may require modifications across multiple parts of the codebase. As the number of supported providers increases, this can make maintenance and contribution more difficult.
Goal
Enable new providers to be added through a well-defined extension mechanism with minimal changes to existing code.
Ideally, adding a provider such as:
- Azure OpenAI
- Anthropic
- AWS Bedrock
- Google Vertex AI
- Together AI
- Fireworks AI
- DeepInfra
should primarily involve implementing a provider-specific adapter and registering it with the platform.
Benefits
- Easier community contributions
- Faster integration of emerging providers
- Reduced maintenance burden
- Cleaner separation between routing logic and provider implementations
- Improved long-term scalability of the project
Possible Directions
Potential approaches could include:
- Provider plugin architecture
- Provider registry pattern
- Capability-based provider discovery
- Shared base classes for OpenAI-compatible providers
The exact implementation is open for discussion.
Help Wanted
Contributors with experience in API gateway design, plugin systems, or multi-provider AI infrastructure are welcome to share ideas and submit proposals.
Summary
As OpenBalancer grows, adding support for new inference providers should require minimal code changes.
Currently, each new provider integration may require modifications across multiple parts of the codebase. As the number of supported providers increases, this can make maintenance and contribution more difficult.
Goal
Enable new providers to be added through a well-defined extension mechanism with minimal changes to existing code.
Ideally, adding a provider such as:
should primarily involve implementing a provider-specific adapter and registering it with the platform.
Benefits
Possible Directions
Potential approaches could include:
The exact implementation is open for discussion.
Help Wanted
Contributors with experience in API gateway design, plugin systems, or multi-provider AI infrastructure are welcome to share ideas and submit proposals.