Lab/main dev#2
Merged
Merged
Conversation
…ations This commit adds complete integration of OpenAI's Realtime API (WebRTC-based voice) with CopilotKit, enabling developers to build voice-enabled AI applications with ultra-low latency. Key Changes: - Add useRealtimeChat hook for WebRTC voice communication - Implement bidirectional message bridging between Realtime and CopilotKit - Create complete voice-enabled form filling example application - Add comprehensive documentation and technical guides Technical Highlights: - Fixed critical event name issue (conversation.item.input_audio_transcription.completed) - Enabled whisper-1 transcription configuration for user voice messages - Corrected parameter naming to snake_case (prefix_padding_ms, silence_duration_ms) - Implemented proper tool registration with type: "function" requirement - Added message deduplication to prevent UI duplicates - Created automated clean rebuild script for development The integration enables any CopilotKit application to add voice capabilities with minimal code changes, opening up new possibilities for accessible and natural AI interactions.
…oice actions - Replaced the use of useCopilotContext and useCopilotChatInternal with useRealtimeActionHandler. - Simplified the handling of the "confirmIncidentReport" action by leveraging the new voice action execution method.
- Updated the example to focus on medical intake, including changes to the form fields and voice commands. - Renamed components and adjusted prompts to reflect the medical context. - Added new fields for patient information, symptoms, and medical history. - Enhanced the README files to provide clearer instructions for the medical intake process. - Introduced a new pnpm lock file for dependency management.
- Introduced a new action, `scheduleAppointment`, allowing users to schedule medical appointments with editable fields for date, time, type, and provider. - Enhanced the `IncidentReportForm` component to include an interactive appointment editor for user input. - Updated documentation to reflect the new scheduling capabilities and parameters for the appointment action.
Lab/realtime integration
- Removed the `realtimeMode` property from `CopilotKitProps`, `CopilotContextParams`, and related hooks, simplifying the API. - Updated message handling to include voice metadata for better context in user and assistant messages. - Introduced a new `executeAction` method in `useRealtimeActionHandler` to handle actions triggered by voice, with source context. - Enhanced the `RenderMessage` component to display voice indicators and handle voice metadata appropriately. - Deleted obsolete `RealtimeActionExecutionMessage` and `VoiceTranscriptMessage` types to streamline message types.
- Eliminated the `realtimeMode` property from the `CopilotKit` component, streamlining the layout and simplifying the API for users.
- Introduced detailed logging for message updates and context changes in the CopilotMessages component to improve debugging and traceability. - Updated the useRealtimeActionHandler to manage message order and delays for voice actions, ensuring proper handling of user transcripts. - Refactored message rendering to include a unified source indicator for both voice and text messages, enhancing user experience and clarity. - Improved the useRealtimeChat hook to track message creation order and manage pending user transcripts effectively.
…nscript management - Enhanced the useRealtimeActionHandler to include callId metadata for better tracking of actions. - Updated message handling for voice interactions to ensure proper ordering and processing of user transcripts. - Introduced a mechanism to track pending user transcripts in useRealtimeChat, improving message reliability. - Refactored argument handling in action execution to clean up unnecessary data before processing.
… transcript management - Refined the useRealtimeChat hook to buffer assistant responses until user transcripts arrive, ensuring proper message ordering. - Implemented a mechanism to track pending assistant messages, enhancing reliability in voice interactions. - Updated message rendering to streamline the display of voice and text messages, removing unnecessary source indicators for clarity. - Enhanced logging for message flow and processing, aiding in debugging and traceability of voice interactions.
- Added a loading state for the "connecting" status to improve user feedback during connection attempts. - Introduced an error state with a "Reconnect" button for better handling of connection issues. - Updated the "End Call" button styling for improved visibility and user experience.
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the ✨ Finishing Touches🧪 Generate unit tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. CodeRabbit Commands (Invoked using PR/Issue comments)Type Other keywords and placeholders
Status, Documentation and Community
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR with complete voice/realtime provider and example app.