3 Seconds of Audio Can Clone Any Voice - Speech Editting with VoiceCraft
Vložit
- čas přidán 6. 09. 2024
- Links referenced in the video:
VoiceCraft Demo - jasonppy.githu...
VoiceCraft Github - github.com/jas...
Hardware for my PC:
Graphics Card - amzn.to/3pcREux
CPU - amzn.to/43O66Ir
Cooler - amzn.to/3p98TwX
RAM - amzn.to/3NBAsIq
SSD Storage - amzn.to/42NgMFR
Power Supply (PSU) - amzn.to/430bIhy
PC Case - amzn.to/447499T
Mother Board - amzn.to/3CziMXI
Alternative prebuilds to my PC:
Corsair Vengeance i7400 - amzn.to/3p64r22
MSI MPG Velox - amzn.to/42MnJHl
Cheapest and PC recommended:
Cyberpower 3060 - amzn.to/3XjtZoP
Come join The Learning Journey!
Discord - / discord
Github - github.com/Jar...
TikTok - / jarodsjourney
If you found anything helpful, please consider supporting me and the content I am trying to produce!
www.buymeacoff...
It is worth to note that both Voicecraft and Audiocraft cannot be used commercially (weights have non-commercial licenses). Most people probably will not care as it is hard to prove, but I guess some people would like to know that. AFAIK you can use Tortoise (and probably RVC) commercially. Thanks for the video though, the results from just 3s of audio are just stunning. I get it that you can copy the voice patterns pretty well if you get like a few minutes of speech, but what you can get here with just 3 seconds is crazy.
Exactly,
Awesome quality for a zero-shoot! I'll give it a try. Also thanks for the update on the upcoming voice clone video! I'm glad it's coming, hopefully during this week!
VoiceCraft seems to be the best one I've heard so far -- at least on CZcams.
Lyrebird was at this level long time ago, but it was commercial, yeah, finally open source caught up!
Any chance you can do your magic and create a one click install that will work on windows if you have an Nvidia GPU? Pretty please?
Just found a CZcamsr who did it on his patreon :)
SWEET finally, we are so BACK!
I can't get docker working, it's breaking on the first cell. Would LOVE a walkthrough.
Cool stuff. Appreciate the update on the next voice clone repository video
I think the voice engine by open ai is just hype now because its from open ai
please do a video on windows setup. I really want something to clone voices from small datasets. I been replacing game audio of Castlevania SOTN. and the PSX voice for Dracula doesn't clone because of only one minute of audio. I really would like to use all the PSX voices but right now been using PSP Dracula. thank you for any help. and thank you for the video.
I made an API server for VoiceCraft as well as added it to my audiobook/dubbing generation app, Pandrator. Both run on Windows and Pandrator has a one-click installer. I'm not sure what I think about it yet, to be honest. I achieve very good results with XTTS, but I cannot experiment with VoiceCraft too much, because generation is very slow on my measly 4GB 3050 (laptop), slower than processing XTTS results with RVC, even. I have only tried the smaller model (though, according to the author, the difference in quality is negligible). Sometimes it drastically changes the pitch, it sounds as though a sentence or a part of one was generated using a different voice altogether. It can be mitigated by playing with the parameters a little, probably. The quality of voice cloning alone is much better than XTTS (I'm not sure about XTTS+RVC), but consistency seems to be worse.
Hi, man thanks for the work..
Does it support apple silicon?
Can you pack one click installer for mac mps?
is there any demo online that Mac users can test, Huggingface?
And does it support silero Indic languages?
❤ Great work
Thanks for Pandrator, much appreciated.
Thanks Man for all you do
Thank you for good content.
I'm just curious about the commercial use .
Since Coqui Public Model License is only for non-commercial use . Is it good to use output of this model in youtube ?
I'm also planning to use tts in my content.
Probably not, I'm not a lawyer lol.
Hey Jarrod. Do you think this would be a viable replacement for tortoise for me to use to make audio books? I really need something soon.
The license is "Attribution-NonCommercial-ShareAlike 4.0 International", whether or not that affects you is a complicated question. Tortoise is free to use for almost *anything*.
If the audiobooks are for personal usage, I don't see why not. But unfortunately, I won't be able to implement this into any of my projects anytime soon
oh, awesome. progress. we need more of it!
How does the cloning do on voices with special effects tied to them? I.e. SHODAN from System Shock? Will a large enough finetuning dataset accomplish good results, or will the inherent underlying training cause issue from not training on voices with special effects/pattern variances?
very cool
I have to say I learned a lot from you especially when it comes to RVC so thank you very much but even with all of that I could never get a voice model that is identical to the original voice
any help with that?
What's the lowest power Nvidia card needed to make this reasonably work
if you don't know the answer to this, installing it is going to be a NIGHTMARE
Hello, why do I need one voice sample in the tortoise auto aggressive fine-tuned model in the voices folder? Why can’t I select just none and will the voice sample affect the quality of the trained voice I mean when you put voice samples in the voices folder after training and selecting the model and wears the prepare a new language tab and how does it work?
thanks for all the insights !!
Love your stuff. I am using this repo and was able to reproduce the output wav files. I need to be able to load the model and then make many inference calls without having to reload the model every time. Do you have insight on how to do this or can you share your visual studio code project on github?
So far, I am loading the model using fastapi and unicorn to host it. I can do inference via a post call the first time, but when trying to run again, I get WARNING:phonemizer:words count mismatch
I think I need to unload or re-execute something to reset it, without removing the entire model.
It works only in english?
Thank you for this useful and wonderful channel. I have been searching for a long time for a program that converts text to speech for Arabic. I have tried many programs and found that they do not reach the desired level regarding the Arabic language. The best one was eleven labs, but it's not free. Also, xtts coqui is good for the Arabic language, but it needs improvement. Currently I am looking for a program that converts text into speech. It can be trained to pronounce the Arabic language.. I do not know how to train xtts coqui in the Arabic language.
Can this do things like laughing, sneezing, coughing or things like that?
have you tried like indian accent or singlish accent ? will it follow?
Can you share your build? This is the quickest way to use VoiceCraft
great stuff!
Thanks :)!
it's getting scarier and scarier
Jarod, How was the inference speed and GPU utilization on your 4090(I assume you're using one)?
Eh, about 5-15 seconds once alignment was finished using MFA. It is pretty fast generally, but I'm sure it's not optimized either
@@Jarods_Journey That's not too bad at all! I'm trying to move away from freaking ElevenLabs for my project. So I'm about to get real heavy into RVC and Tortoise. Gonna hafta binge your videos XD
@@joshuashepherd7189 What is the difference from ElevenLabs and this? I am new to voice cloning and just want to understand the programs to use to do it
Cool. Does it run on CPU or you must have a graphics card.
Need a GPU, will run on CPU, but way too slow.
What about the best open source speech to text? Is there nothing better than Whisper in accuracy?
Whisper is the best afaik
Before I try to istall this: How does it perform with other languages?
What you think about using cloud gpu?
could you make a one click installer for us? that would be greatly appreciated!
Do you have suggestions for real time TTS models that are open source?
Tortoise with the right settings I guess. Select DeepSpeed and low samples (2) / iterations (
Alltalk is the best I have seen better then tortoise and it has a streaming mode it does one word at a time and plays it do you could probably do real time. It's also faster
@@AEONIC_MUSIC It uses XTTSv2, which is very powerful model, but released under terrible Coqui license. Not open source at all. I considered paying Coqui for it, but before I pulled the trigger, they shut down their business and now you have no way to legally use it at all.
Tortoise TTS would be it or xtts if you don't care about licensing. Styletts2 is a good contender as well
Hi, it can talk another languages ?? By the way, tanks for all the work, nice channel to creators.
Just English I think for now
This license allows only non-commercial use of a machine learning model and its outputs. Why even bother...
Did they specify it for outputs? Usually it's been vague but risky, did they finally admit that it's just not usable?
@@FenrirRobu Yup.. for output.
You can use they are not going to re check you
It's cool tech, the abilities for prosody and pitch maintenance from 3 seconds of audio is pretty wild. To put it out in the open means that another project will build upon it, possible with more permissive licenses
@@Jarods_Journey let's see if lucidrains takes up the task.
Does it work well for other languages?
I think it only does English right now
Pleas can use with arabic voice
How does this compare to my boy tort TTs RVC ?
Alltalk is way better and faster at everything then tortoise but im curious to see what this one is like compared to allltalk
Haven't done any thourough comparisons, but, it's pretty darn good lol
How to run in Google colab
Since it was built from Linux, you can probably just follow the readme
Does this come with the webui?
No, it comes with a jupyter note and I have no idea how to use this thing.
No, I built the webui for it to make it easier to use. Someone said they were going to PR their own version of one on the repo, so we'll have to wait and see
Can I run this without GPU any method 😢
I don't recommend it. It seems to run on CPU, but the outputs take much too long to generate.
Does it support CHinese and cantonese , vietnamese ?
Just English
Hello, why do I need one voice sample in the tortoise auto aggressive fine-tuned model in the voices folder? Why can’t I select just none and will the voice sample affect the quality of the trained voice I mean when you put voice samples in the voices folder after training and selecting the model and wears the prepare a new language tab and how does it work?