A semester-long exploration done for Georgia Tech’s Art and AI Vertically Integrated Project (VIP) group. This work utilizes an Amazon Polly + MetaHumans Sample Project, and Open AI’s GPT-3 API. I am responsible for the initial idea, and worked on the project in Unreal Engine alongside my co-lead, Jude McLaren. Together, we lead a group of five others to contribute to the brainstorming, research, and art phases.
Thank you to Steve Cox and Bulldog Game on Youtube for their helpful documentation.
Introduction
‘Artificial Friend Bot’ is the informal name given to this demo/art project. When my group and I first embarked on our semester-long inquiry into conversational AI agents, our goal was to create a series of art to inspire our audience to imagine how a anthropomorphic AI might look and act. But thanks to work done by the AWS SDK’s team, we were able to adapt their implementation of real-time text to speech animation to create a usable, interactive prototype of this idea. Amazon’s project uses visemes (visual phonemes) provided by their Polly API to animate Unreal Engine metahumans only using text. Our project connects Amazon’s animation pipeline with Open Ai’s GPT-3 model, allowing our metahuman, Hanna, to answer all kinds of questions from users in real time!
Demo:
Process:
One of my favorite parts of working on this project was teaching my team how to think about and create immersive 3D systems- which is especially exciting when emerging technology like AI is involved. To help inspire our brainstorming process- I used Open AI's Dall-E to generate images of virtual AI assistants, editing the images with Photoshop for further art direction.
Using Unreal Engine was a great way to introduce people to 3D for the first time, as it cut out a lot of the tedious work of more traditional pipelines. For example, a team member who hadn’t worked with 3D before was able to successfully texture Hanna in Substance Painter- which was only possible thanks to Unreal Engine’s UVd Metahuman and snappy PBR Material system. She did a great job- way to go Rhea!
Rhea's new texture applied to Hanna in Unreal Engine.
While I’m excited about the progress we made in just a few months- this project is far from complete. Some short term changes my team would like to make include speech-to-speech and upgrading to UE5 and GPT-3.5/4. Longer term, it would be cool to integrate custom Metahumans, adaptive tone, and more realistic movements. I’m not sure if I will stay involved with this project in the future, but I hope it continues to develop, even if just to spark some inquiry into the role of virtual humans in a physical world.
---
Amazon Polly + MetaHumans Sample Project, Open AI’s GPT-3 API, Unreal Engine 4 + Blueprints