Me and a ChatGPT NPC in VR / MR (Mixed Reality) on Quest 3 going for a walk, pt.1
Vložit
- čas přidán 31. 12. 2023
- My first Unity and VR development project which is a work in progress, showcasing the capabilities of Quest 3, like mixed reality, and OpenAI features like image processing and conversation through ChatGPT. This project features an NPC with a memory and autonomous movement, notice there are no Quest 3 boundary warning messages, the play area space has no limitations. I used an old phone for internet connectivity which is slow and kept dropping out. Enjoyed working on this project and will improve it.
Edit: this app is now available for $2 at : / tykeai
This app, called TykeAI, is now available from here : www.patreon.com/TykeAI
this is the most awkward date i have ever witnessed
(jokes aside this is really cool)
lol, well, what can I say, I've never been that good with women, even when they are virtual! Thx for the compliment 👍
The Future is getting more and more shaped these days. Oh boy what a exciting trip this will be. And we are just on the beginnings.
Absolutely agree with you. We're so lucky to be alive now, I also imagine what things will be like in 100 years time (if we haven't destroyed our planet by then).
10 years from now virtual reality will be unrecognisable.
Very cool. This can help a lot of people with anxiety
Do u mean help as in practise communication for those with social anxiety or general anxiety? Or as a companion for those who find being outside causes them anxiety? As someone who has experienced anxiety off and on over the years I'd be very happy if this could help these things in some way, the NPC could be tailored to be suitable for these situations.
Thx for the compliment btw.
The staring is kind of creepy... same with the dead voice... but oce they can make that more natural and soften the vocal characteristics (and speed up the process of 'Thinking') it will be cool
Hi, the latency i.e. the "thinking" time is already greatly reduced, u can see this in my later videos. I appreciate the voice isn't to everyone's taste.
This is awesome! I really believe that the combination of Mixed Reality and AI capabilities offers so much potential for crazy things like this. I mean honestly, in a few years it's just a normal thing but now is the creative time! Imagine using ChatGPT (or other AI) not only for talking, but also for image or scene creation or even coding on the fly for Unity (not sure if this would make sense though ;-) Anyway, please keep on experimenting! You have a new follower and fan!
Thanks! Agreed that this type of creation will be common 1 day, I envisage it to be like something like out of Blade Runner 2049. Btw ChatGPT does do image creation today from a textual description, it can also automatically do some basic Unity coding. It can't right now do what this project is but I bet it will 1 day. It has helped me with some specific coding blocks I needed help on, it's the future no doubt. Thanks for your kind words.
She sound like she don't give a sh!t...it's so funny!!!
I like her attitude tbh lol!
Great potential! But the latency is an immersion breaker somehow.
Thanks! I can and will reduce the latency :)
Hardly latency, normal for me when talking to the Mrs 😂😂😂
@@mazzachiefyXD
Super impressed! 💯
Thanks so much!
Interested in seeing how this project progresses
Thanks, I can tell you I've made changes so that the time it takes for the NPC to give a reply is much reduced. Also, the AI aspect to the NPC is now free up to a specific usage threshold (whereas before I was paying for every AI interaction) and the threshold is quite generous. The Text To Speech service for the voice is unfortunately still not free. I can give the NPC a different voice that is free but it is not as realistic as the paid for version.
So cool, reminds me of a Robocop MR video experiment very similar to yours, please share a link for us to experience ourselves! Great work!
Thankyou, comments such as yours encourage me to make this available to others - and I am working on this right now. As soon as I have some news on the development of this I will make a comment/video about it. Thanks again!
Wow, that's the future. I wonder when enthusiasts can develop something like this).
Thx! It'll only get better from here, imagine how good it could be if a big company with a big budget decided to do something like this (they will, 1 day).
Here before this video blows up and becomes a meme
I just purchased the quest 3 and now I’m more than excited of using it for everyday use!
Great choice of vr headset, love or hate Meta they're making a great product in the Quest 3 right now.
@@Tyke18 Btw how is the process of creating this avatar? Is she using multimodal AI to see everything in her environment through your quest 3? Also, in the future, that would be amazing to throw a virtual ball 🏈 with her or for both of us to eat lunch well talking to eachother at the same time.
@@timrod94 She's only able to see through my eyes unfortunately, what happens is she analyzes the image I see through my Quest 3. In the latest iteration of my app, she can't do this right now because I changed from using OpenAI's ChatGPT to using a different AI product which brought several good things with it but unfortunately lacks the image processing ability. It could additionally be added in though, maybe that will come.
@@Tyke18 Got it! Regardless I still can’t wait to start using her. Around what timeframe do you think Tyke will be released so we could start using her?
@@timrod94 I'm hoping for by the end of this month if not before, I don't see why it can't happen, regarding the app almost everything is in place.
Incredible! How performance heavy is this implementation? Will it be viable to be used for multiple interactive characters in a world?
It performs pretty good on the Quest 3, using Quest Games Optimizer (u can see this at the start of the vid) I'm able to supersample the resolution to above the Quest 3's native res and it still runs fine. Multiple/different interactive characters is definitely a possibility, I realise not everyone will be a big Jen Lawrence fan like me XD
Huge Jen Lawrence fan ❤
How did you get the avatar to start walking with you on cue? Does it interpret the ChatGPT responses and perform an action based on the response? Basically what I’m curious about is how you interface the avatar with the chatGPT conversation
You're on the right track, I created Unity Events so that when certain/specific things are spoken by me they trigger the NPC to add a special keyword to its response, this keyword then triggers the event, like the walking animation (or it could be any animation I wish to use according to what I say - so many possibilities), and the keyword is removed from the NPC's response just before it gets sent to text to speech service.
@@Tyke18it’s a good concept. I guess making the NPC listen for keywords is basically like writing a bunch of predetermined if/else scenarios with predetermined reactions to each keyword. Maybe an idea for later down the road is to implement AI into the unity code to help the NPC seamlessly and intelligently interpret what the user wants it to do and then react using AI. But that’s probably easier said than done.
@@keyboardmouse7699 It is like an IF statement but the logic is fuzzy. The AI is involved in interpreting what I say, for example to trigger walking I don't have to say a specific/particular sentence/phrase/word. As long as I make a request in relation to what i want, or imply it, it will get triggered. I could say I feel like going for a walk today do you want to come? Or do you fancy going for a stroll? So it's quite natural in that way. The NPC's response is very specific, its response will contain a START_WALKING keyword, and it's this is what my app detects to trigger the event.
@@Tyke18 very cool! I feel like there’s a lot of potential in this. I’d love to see where you take this.
The AI is serving with that voice lmao.
Are you going to.make this available to the public anytime soon?
How can you bring this NPC into VR games or social platforms with you?
Hi, it's available for $3 on my Patreon page : www.patreon.com/TykeAI it's not possible at the moment to bring the NPC into games or social platforms. Please note, the NPC now has a different voice. It also has no sight ability like in this video but I plan on adding this back in at a later date. Thanks for your interest.
I apologize, I tried with optimizer to record without the boundaries with the passthrough as in your video but the problem is that the video goes faster than normal, please tell me what video options you have set with optimizer so I can try them too? A thousand thanks!
Hi, the video recording options I have in QGO are all at default values afaik as the only thing I have done is turn the boundary off using QGO, I never altered any video settings.
How did you do this? its brilliant, I'm currently using CONVAI but it's nothing like this! love to chat with you about this project.
I've no experience of Convai so can't comment on it, upto now I'm very happy with the Inworld integration tbh. I'm up for chatting no problem, I have a Discord (iirc an invite link is in this CZcams channels description somewhere) so you could or contact me there or I have an email address huntc938 @ hotmail . com, I check the Discord more often though.
After she said she’s the real Jennifer Lawrence he started breathing rlly heavily. I guess she made him nervous
Of course yes who wouldn't be 😅
@@Tyke18 completely understandable brother
Woah…
Epic
Thanks!
It's possible in quest 2 in french?
No sorry, Quest 3 only and English but more languages are coming.
Cool, I want to learn English with the app, it can improve my English spoken.
Good point, indeed it could yes.
Great work? Is it possible for you to share the implementation by any chance?
Thanks! If you mean share the code then I'm not going to share that I'm afraid, I am willing to answer any questions about the app etc, about the concepts and programming tools I've used, just fire away!
wow!!! But is it a bluff or is the artificial intelligence even able to look through the cameras of the quest 3 and analyze the surrounding environment?😱😱 did it recognize that it is a field??? but seriously or did you train her by telling her before giving that answer?? because if it is really like this and it can recognize elements of the real environment it is truly incredible news! But has your app been released on patreon? will it come out on sidequest? or applab? the greatest use I would like to make of it is outdoors!
Hi, thanks! Unfortunately the later versions of my app of don't have this "sight" ability due to using different AI behind the scenes. Changing it came with several important benefits, much improved latency being one, but it also lost the sight ability. I might be able to add it back in later at some point. I didn't tell it what to say. It can see through my eyes, so it can analyze what I'm seeing. It will be released on Patreon, I'll upload a vid when ready, won't be long. Thanks for your interest.
This looks awesome how do i get this on my quest 3 🙏
Tbh i never thought what i was creating would be wanted by anyone else, it was a project initiated from an interest in AI and VR/MR and wanting to do something a bit different, but a few people have expressed an interest in having it so i'm definitely looking into making this feasible. I want to do more to it but the main 1 is to reduce the latency. Will keep you updated and thx for your interest!
Would I need a pc
@@fcukwork Nope it's Meta Quest 3 standalone, runs well on it too.
the neeeeeeeeeeeeeck
🤣 lol note this was 5 months ago and very early days in the development of my app, if you get chance watch my latest videos to see how the avatar has improved.
that ain't awkward one bit 😆
XD
Bro started getting nervous near the end what is that breathing
I'm was very unfit!
How does she show up behind objects, all other MR things ive tried they never are behind an object.
Well spotted, I've used Meta's Depth API to give the effect you are describing, which is known as dynamic occlusion in AR/MR terms.
@@Tyke18 can we plug this into games that dont have it?
@@Tyke18 I cant try your app right now I dont have the means to pay for Patreon and pay for chatgpt. But that API if it can be plugged into other unity games that would be cool.
@@ripsaw2322 Sorry I'm not totally sure on what you mean? The NPC can't be added to other games if that's what you mean.
@@ripsaw2322 Ah ok no problem! By the way, once the Patreon fee is paid ($3) there are no ChatGPT fee's to pay. I gave April for free to March subscribers, and may reduce the $3 in the future if my costs are met (I must pay for AI API calls). The Depth API can be used in other apps, it all depends on if the dev has decided to use it or not.
what are you using for TTS?
Hi, I'm using Eleven Labs, however when the app is released it will use a different voice. Eleven Labs voices are the best imo but they are expensive, so when the app is released I'm going to have use a different voice which won't be as good unfortunately. Down the line at some point I hope to add the option whereby someone can add a voice of their choosing i.e. if they want an Eleven Labs voice and are willing to pay for it.
@@Tyke18 Thanks, I really like this idea as a therapeutic experience. I have debilitating social anxiety, not disabled but it interferes with my ability to maintain meaningful relationships. Do you think you could allow us the ability to adjust the base prompt too? So our companion could have the identity we choose? and if we are supplying our own openAI key will you add support for GPT-4 requests?
@HypeLobbiesHD Hey I feel for ya on the social anxiety, I'm not great in social situations, it can really limit things. About changing the companion identity, it is possible. I use an Inworld.AI product for the NPC, Inworld use ReadyPlayerMe avatars which are highly configurable (how they look etc). In addition to this, Inworld provide what is called the Inworld Studio which is an online easy to use portal to design your own NPC for adding into a game, the Studio allows all aspects of the NPC to be configured, such as personality etc. It's possible I could add such an NPC, designed and created by someone else, into my app. This would require some extra work from me, I would be willing to do this if the reward was there (I'm not talking alot, just enough to cover my time). About support for GPT4, the short version is : it's quite probable in the future. Long version : the early iterations of my app directly used GPT4 however I switched over to Inworld.AI because this meant much lower companion reply response time and was cheaper, plus ease of integration. However I still have the code I was using for these GPT4 versions so it wouldn't be that difficult to create a version for this and facilitate someone to use their own OpenAI key.
@@Tyke18 Interesting to hear that you've switched to Inworld. I've been thinking about starting an LLM-NPC project lately and was considering the options as to whether I should use an LLM API directly or use a secondary provider like Inworld. I kind of know about the advantages of Inworld, such as low latency, context-memory abilities etc. However I would prefer support for my native language (Turkish) too which Inworld does not provide right now. And I guess since Inworld is not pure ChatGPT, but instead uses a special combination of multiple models, you wouldn't get satisfying responses if you start talking a non-supported language to an Inworld character. With a direct LLM API (ChatGPT) call however, I assume I could talk to the NPC in my native language. But again, then I would experience the latency issues etc.
@@istemihanoktay Inworld is going to start offering multi-language support soon, iirc Mandarin/Korean support is due to start in March or Q1 2024, then other languages will follow.
Where can I get this? :D
It looks like Patreon will be the chosen distribution method, or maybe itch.io, but it'll probably be Patreon when I've finished it which should be by the 2nd week of March. I'll upload a vid when it's ready, thanks for your interest!
what are the odds of you running into JLaw on your walk?! lucky you! XD
lol I guess we just move in the same social circles (I wish!)
@@Tyke18 😂
incredible! how do i get this?
Only I have it right now, I'm sharing these video's to show the progress I've been making with it as I'm quite pleased with it. Quite alot of people now have expressed interest so I'm trying to make it so others can try it. It's not finished, there's no user interface for example. Thanks for your interest.
I recognise that field, I half expected to see me and the dog walking across there at some point 😂
@@chicofrostie lol small world!
amazing M8 ¶ ; - )
Thanks 😁
the future of dating is here boys!
She’s struts like Conor McGregor lol
She fights like him too XD
@@Tyke18 😂😂😂
Pleaase let us test it too, at least for money in patreon for example
Sure, it is progressing, I've just added a basic menu to it, it's getting there. Yesterday I bought the website/domain name www.TykeAI.com so they'll be a website for it with instructions etc (there's nothing there at the moment). Keep an eye on future vids from me as I'll be updating those interested on progress and when it's released. Thanks for your interest, it's appreciated.
@Tyke18 man, you're underrating yourself. This is very interesting and helpful, thanks
@@ontarioyoutube2055 Thanks so much! I'm not sure how is best to go about funding this at the moment. Right now, with me being the only user, it doesn't really cost me anything. But once it is released it may incur costs as the NPC's AI free usage threshold may be surpassed (it is free up to a certain point and being as I'm the only 1 using it I won''t pass that threshold). So I'm thinking about the best ways to cater for this i.e. Patreon, or a 1 off purchase price. Patreon makes sense in that the ongoing AI API calls cost would hopefully be covered, so yeah maybe Patreon. Of course if I have some reward left after the costs have been paid then good stuff, it'll spur me on to create more!
Great job, what a fun project! I think you should dump her, she doesn't sound very interested... :/
XD
pretty cool but she sounds a bit... matter of fact and unimpressed based on her voice inflections. Maybe not the best choice of accent to use. Still impressive AI.
Fair points, I appreciate all of our tastes will be varied in such things
👍
She sounds like scarlet johansen when she's isn't interested in the conversation
lol I like Scarlets voice (and her as an actress) so that isn't so bad. When the app is ready to be tried by others it won't have this voice as it is quite expensive to generate.
It’s interesting how you referenced BR2049 and Her in your about page that I resonate with 120%. Serious inquiry: Are you solo deving this and what are your future plans to go about it if I may ask?
Hi, yeh love those movies, I think Pheonix, Gosling, de Armas, Johansson are are great actors/actresses in most anything I've seen them in. Yes I'm doing this on my own, my plans right now are to make it available to others which is nearly complete, then if some people like it to improve upon it, listen to suggestions etc.
I'm not such a big fan of Jennifer Lawrence. Does it come with a Jessica Chastain? I'd buy one if it does. I'd spend all my time playing Scrabble with Jessica, having awkward conversation, just staring at her rack, trying to find words.
The latest version of the app, the 1 I'm releasing soon, it's not J Law, it's an AI companion called Tanya. It wouldn't be legal to release an app that replicated a real person without their permission afaik. Shame because it was pretty cool imo, and it was obviously not the real person.
Well Alice it seems we both have autism, haven’t we?
Why does she sound like Miley Cyrus?
Because Miley Cyrus provided the voice over for my app! XD It's a clone of Jennifer Lawrences voice, it doesn't have this voice now.
Gotcha! @@Tyke18 Please don't think I'm bashing on you. This is a great proof of concept!
@@jamesdewhalley2665 Thx! Hey no problem, people have said it sounds like J Law, Scarlett Johansson, and now Miley Cyrus 😁
That voice is so annoyingly monotone. I wonder if replika can go on a walk too.
It uses a different voice now, I don't think replika can go on a walk.
Thinking.................................................................... Hi.