Local Voice Assistant: Using your Cameras & Speakers in HA
Vložit
- čas přidán 15. 06. 2024
- In this tutorial, we’ll explore how to set up a local voice assistant powered by Assist using your smart cameras and speakers within Home Assistant (HA).
Even if you don't have any cameras on HA, we will show you how to test it with any Android Phone.
Update: Doesn't work with the Echo Players since they don't support local audio streaming
FixtSE Web: fixtse.com/blog/stream-assist
Stream Assist Github Page: github.com/AlexxIT/StreamAssist
00:00 Prerequisites
00:47 Install it on the Home Assistant Server
01:11 Install it on a Different Computer
02:21 Wyoming Integration
03:14 Configure Assistant
04:22 Stream Assist
05:07 Configure Stream Assist
05:35 Wake Word Detection Beep
07:28 Android IP Camera App
09:32 Demo
If you like my work, please consider supporting me on Ko-fi! ☕🎉: ko-fi.com/fixtse
Patreon: / fixtse
or Join this channel to get access to perks:
/ @fixtse.
You can find me on:
Web: fixtse.com/
Instagram: / fixtse
Hope this was useful and if you have any questions, write me a comment below
Thank you for watching (～￣▽￣)～ - Věda a technologie
Keep these Assist videos coming! You just solved a major problem I was having getting wake word working on Android. This is absolutely fantastic! Thank you!
Glad it helped! 🙌
Brilliant❤ Many thanks
It would be interesting to see if a feature can be created that plays the latest news from specific sources such as CNN and others. That's one of the main benefits of using the Google Assistant.
Awesome job!
This is exactly my setup + Fully Kiosk as the media_player. Bonus: run rtsp stream through frigate and automate FKB (screen/screensaver) + StreamAssist based on motion/person/speech detected (Y)
Great, you're a crack. You should get last year's Emi award and this year's award.
Just a little thing, those of us who don't speak or understand English fluently, would appreciate it if you wouldn't be so fast. Thank you, thank you very much
Jajaja thank you, believe me I'm working on that 😅, I hope to keep improving over time 🤞🙌
Thanks for the great video. I have it up and running now. Is there anyway to get extended conversation working? How can I auto trigger wake, so that it listens for my response?
Great video. Keep up the good work.
I wonder if any of the Xiaomi Smart speakers or similar devices can be used as microphones?
Hi, thank you, I don't think so. Unless you can get an RTSP, HTTP or RTMP stream with audio into Home Assistant, it's not going to be possible. For example, to get the RTSP Stream for the Nest Hub Max into Home Assistant, you need the nest integration (it requires a $5 fee if i'm correct), and even after that, you can only get Video, not audio. So it wouldn't work with this integration.
I'm doing some research about this, so expect a video in the near future 😁
How do we configure the actions each command performs when we utter ì to assist. Thanks for the great tutorial.
I'll do a follow-up video with more usage examples, to trigger automations and scripts. It is also possible to configure your own trigger sentences.
hello, how to add the beep path that is inside the www folder?
Great. Can I use Amazon Echo Dots as microphone and speaker and media player?
No, one viewer just confirmed that it didn't work with Alexa. It's because of the way that the Alexa integration had to be implemented (it's a cloud integration, not a local one). Echo devices don't support local playback
Amazing effort! Does it support Google home mini?
Yes it does
I love youre Videos. That was what I waiting for. Great. Thank you.
Do you have a solution for only Integrate microphones instead of a mic from a camera? Because mics a easier to Place. Thank you
Feedback:
- it didnt worked with alexa speaker. didnt get sound from.
- unfortunately i cant run it on my walldashboard with the android up because then the camera is blogged and fully cant use it.
Hey thank you for the feedback 🙌🙌🙌 I was wondering if it worked on Alexa 😕, I was hoping it did since it's just playing a audio file, I'll update the description.
Let me see if I can find a way to use just the mic in the future, you'll never know what can be done with some clever code 🙌🙌🙌
@@fixtse. I integrated now an s3 Box lite and combined it with my echo speaker. Unfortunately the Internal speaker speaks too, Do you know how I can Mute or deactivate this?
Next step esp32 with mic and combined with the echo.
Thanks for sharing, what mic do you use in this video? TIA
The mic on the YiDome Camera, I use the yi-hack firmware to get an RSTP stream from the camera with audio support.
Great, thanks for sharing.😊
Really cool! Can you do a vid on examples of music/media streamers that are good to use with HA. Thanks!
is posible with Homepod? Or Sonos One?
Great. Can I use Google Home Mini as microphone and speaker and media player?
no
Great video 😊 how to do that with an ESP32-S3-BOX-3 ? Thks.
It should be easier, but not with this method. You need to go the esphome route to get the ESP32-S3-BOX-3 working with Home Assistant. I don't have the device, but I've seen that it even supports on-device hot word detection.
Hi, does home assitant green has enough juice to run this pipeline efficiently and fast engough?
No, unless you offload at least whisper and piper to a more powerful machine.
For this could I just use a microphone and a separate speaker? I have a spare pi 3b I could put HA satellite on and use a pi hat then I have a Sonos one. Could they be used together like this instead of a camera?
Not sure, I would have to check how Wyoming Satellites work, but, let's say it is possible, It will require adding that as an option to the integration, so it can redirect the output to the Sonos speaker.
I think you should add this as a question on the GitHub page of the project (on the Issues tab, since the repo doesn't have Discussions Activated), so AlexxIT can give it a look
@@fixtse. thanks I’ll have a look
What cameras have you tried and found work? My one camera in my living room has a mic but I could not get the SST to become active.
Any RTSP, OVNIF or RTMP should work, the integration handles the transcoding of the audio source into something suitable for STT automatically. I use YiDome Cameras with the roleoroleo firmware to support the RTSP protocol, but any camera should work.
@@fixtse. Any camera with a microphone, correct?
@@pjuhl2313 yes, as long as the firmware supports audio over RTSP, OVNIF or RTMP. That is up to the manufacturer
@@fixtse. What integration are you using for your cameras in HA. I'm using Frigate and wondering if I need to allow audio in the config for it to transfer over to HA
@@pjuhl2313 I use frigate too, but I'm using the YiHack Integration Camera, instead of going through frigate.
If you want to use it with frigate, I think, as you said, that you need add audio support on your frigate config files, there are examples on frigates the documentation, I put a link on my frigate article on my website if I recall correctly.
cool. can i use different voice assist not english?
Yes, it is available in different languages, just keep in mind that the accuracy of the detection will vary.
Hi,
tell me how did add LLMs in Conversation Agent, and how you have so much voices?
Check out my other videos for the LLMs part, for the voices I actually show it here, using a docker piper installation
@@fixtse. Sorry my friend,
I saw that after my comment. :/
Merci mon ami ;)
I JUST was wondering why my cameras have so many sensors and I can’t use them to do more. This is perfect! If only I could use my camera speakers as a media player for something like this (not music obvs…).
It is possible, if your camera support two way audio you can use WebRTC ( github.com/AlexxIT/WebRTC#stream-to-camera ) Camera custom integration to add it as Media player.
@@fixtse.Wow thanks! Your videos have stuff I don't see anywhere else. I appreciate you posting the steps so precisely.
@@fixtse. I have read through the WebRTC information and googled a lot of discussions. I can't find much info on the new Stream to Camera option. May make a good video. I can't get it working and see a lot of people struggling also.
Running a Local LLM is the best way to replicate the capabilities of a Google Home Mini or Amazon Echo without having to have an internet connection. It's like having J.A.R.V.I.S. from Iron Man without all the effort that Tony Stark put into creating and training him.
Yes, I think I should upload a short showing some wild answers from the AI I got, some of them are so bad that they are good 😂😂
Do you need a camera? Or can I use a Google mini as mic input?
I have the same question
Yes, you need the camera. Google doesn't expose access to the mic on its devices, so there is no way for home assistant to access that stream.
I understand that a Google Nest Mini 2 generation can be used, what happens to me is that when I put several, it only responds and listens to one. does it happen to anyone?
Interesting, I haven't test this scenario yet, could take a while, but i'll add it to my list and get back to you when I have an answer.
@@fixtse. Thanks for your time and videos!
Can I implement this pipeline:
Amazon Echo Dot > Home Assistant > Custom Wake Word > Fast GPT > Home Assistant> Device Actions > Confirmation via Amazon Echo Dot?
No.
@@fixtse.Thank you very much for your answer. What a shame the pipeline does work with Amazon Echo devices. From my point of view Amazon Echo devices are the best smart speaker.
Google seams to have stopped development of Google/Nest devices and the quality of ESP32 Devices isn't as good as Amazon Echo devices.
I have quite a lot of rooms. What is a reliable hardware reference?
Is it possible to make it without a wake word?
Yes, V1 used to work like that, I never used it, but i guess you just need to call a service to trigger the voice assistant process.
Anyone have any idea as to how to add a Google speaker to HA?
They should work out of the box, as long as they are on the same local network.
How to #Automate Script Execution at logon? Where to copy provided code?
fixtse.com/blog/ollama-home-assistant#automate-script-execution-at-logon
@@fixtse. But where to copy provided code?
will my google home still work with "ok google"
?
Yes.
@fixtSE Would this work for a google nest cam battery 2nd generation (given the limitations of how sdm api works)?
If not, could I use a Wyoming Satellite (czcams.com/video/eTKgc0YDCwE/video.html) as an audio input?
P.S. I have openwakeword running on the satellite device and am using Home Assistant Cloud for STT and TTS.
Hi, I can't answer that question since I don't have the device myself, but you can, If you can get a RTSP, HTTP or RTMP stream with audio from your camera into Home Assistant, yes, you can, if not, it's not possible.
Right now, Wyoming satellite as an audio source is not supported, but you can add a feature request for it on the project GitHub page. I'm sure that if it's possible AlexxIT will consider adding it. (It kind of goes out of the main scope of the project tho, so be respectful if he says that he is not planning on supporting that feature)