Voice-movemenent with an LLM

Why

TLDR: A demo of freeing LLM from the chat box.

At NERD Summit 2024, hosted at the University of Massachusetts, I challenged the audience to imagine and use LLMs outside the comfortable confines of the chat-box modality.

I decided to create a quick example showing how we could quickly create a voice-only experience for moving objects around the screen just by describing them, without needing to learn specific commands (the way most current voice assistants require(d)). Potential uses for environments requiring hands-free work, and certainly for various a11y use cases!

A short demo video recorded after the conference

If you want to update to OpenAI's state-of-the-art model, you can update the model name in pages/api/generate.js

Setup:

1. If you don’t have Node.js installed, install it from here (Node.js version >= 14.6.0 required)

2. In your project dir, install the requirements

   $ npm install

3. Make a copy of the example environment variables file

   On Linux systems:

   $ cp .env.example .env

   On Windows:

   $ copy .env.example .env

4. Add your own OpenAI API key to the newly created .env file

5. Run the app

   $ npm run dev

6. Launch via http://localhost:3000

Things I'd like to get to:

• Respond to:
  • Relative movement
  • Speed
  • Concept of viewport and diagonal movement ("move to the top right corner")
• Give selected objects a clear visual signifier

Built using:

• React
• Next.js
• Framer Motion
• SpeechRecognition (A native browser API!)

This demo was built on top of OpenAI's API quickstart tutorial for Node.js