Home
Create voice commands for VR experiences with Watson services
In this developer journey we will create a Virtual Reality game based on Watson's Speech-to-Text and Watson's Conversation services. The player will issue voice commands, Watson services will convert to text and then pull out the intents and entities, and the application will respond by changing virtual reality.
Cognitive
The powerful immersive experiences of virtual reality can take a user to another world. The means of affecting that world, however, can be a distraction from that experience. We don't normally interact with reality using hand held controllers, so why should we work within virtual reality that way? In fact, much of getting what we want involves using our voice. We ask for things, talk, issue requests and commands, and people and things respond. With IBM Watson services and the Watson unity SDK, we can create these natural feeling interactions in a virtual reality application. The intuitive interactions of voice commands can be easily implemented for a seamless experience, and we will show you how to do that in this journey.
By Author Scott D'Angelo Acknowledgements to Kyle Craig and team
- N/A
In Virtual Reality, where you truly “inhabit” the space, speech can feel like a more natural interface than other methods. Providing speech controls allows developers to create more immersive experiences. The Oculus Rift is the 4th most popular head-mounted VR devices (not including Google Cardboard) and an ideal candidate for Speech interaction, selling roughly 355 thousand units in 2016.
When the reader has completed this journey, they will understand how to:
Add IBM Watson Speech-to-Text and Conversation to a Virtual Reality environment build in Unity.
- User interacts in virtual reality and gives voice commands such as "Create a large black box".
- The Oculus Rift Headset microphone picks up the voice command and the running application sends it to Watson Speech-to-Text.
- Watson Speech-to-Text converts the audio to text and returns it to the running Application that powers the Oculus Rift.
- The application sends the text to Watson Conversation. Watson conversation returns the recognized intent "Create" and the entities "large", "black", and "box". The virtual reality application then displays the large black box (which falls from the sky).
- IBM Watson Conversation: Create a chatbot with a program that conducts a conversation via auditory or textual methods.
- IBM Watson Speech-to-Text: Converts audio voice into written text.
- Unity: A cross-platform game engine used to develop video games for PC, consoles, mobile devices and websites.
blog