Run Any Chatbot FREE Locally on Your Computer
Vložit
- čas přidán 2. 01. 2024
- Here's a free open-source tool to run any chatbot locally and offline.
Discover More From Me:
🛠️ Explore hundreds of AI Tools: futuretools.io/
📰 Weekly Newsletter: www.futuretool...
😊 Discord Community: futuretools.io...
🐤 Follow me on Twitter: / mreflow
🧵 Follow me on Threads: www.threads.ne...
🐺 My personal blog: mattwolfe.com/
Resources From Today's Video:
jan.ai/
Outro music generated by Mubert mubert.com/render
Sponsorship/Media Inquiries: tally.so/r/nrBVlp
#AINews #AITools #GenerativeArt - Věda a technologie
The small Phi-2 model is proven to perform better than most 7-13B models out there, including Mistral-7B and LLaMa-13B. It’s good to see it available on the app.
You are complaining that a LANGUAGE model can't do math... You are definitely special 😂
Yea, i asked it to calculate the coefficient of a warp 9 warp drive and it couldnt even do that.@freedomoffgrid
@freedomoffgrid No models can do basic math, they have to use tools. Even the might GPT 4 has serious problems with math.
@freedomoffgrid limitations in the technology it self brother..
Better how?
Great video as always, but you might want to mention the PC requirements when running larger models.
I was wondering the same if mixtral was running well because of his beast machine or not.
A vague rule to go by is make sure you have a few gigabytes of ram more than the file size of the model itself, but the larger the file the better GPU you also need. @@TheMiczu
how did you get it to run on the GPU ?@@qster
@@ProofBenny it will automatically use it, no need to change settings
The current, pioneering and rather primitive LLM models have an average size of a few to tens of gigabytes. In this size they can pack most of the information, works of literature, paintings or films known to mankind. This means that they FANTASTICLY compress data. What is the result of this?
That the data captured from a single user, even if filmed with sound 24/7, will be compressed to such a minuscule size that a precocious 14.4k modem would suffice to transmit it just when the Internet access appears. Besides, the model may be canny enough to find ways to connect to that Internet, attach user data to any file and send it that way to the rulers of our lives. Privacy needs serious work.
We're finally seeing some improvements in UI software for casual people to use. I'm happy how the AI market is developing to be more casual friendly and not just for devs. I wonder how Jan makes money by doing this.
Taxpayers and venture capitalists probably? Who knows though. I am blown away by how tech savvy some people are. Even with only a little bit of knowledge and with AI some pretty powerful workflows are built and shipped. Exciting stuff.
I never thought it was hard. LM studio makes it so easy even my 11-year-old daughter can do it
@@CM-zl2jw Probably not its a gui they not getting tax money to print hello world but if they are maybe its better then sending it to ukraine orgies thanks..
A description of your hardware configuration (CPU, amount of RAM, GPU, amount of video memory, network upload and download speeds) would be very helpful. What might be quick for you might be absurdly slow for someone with a less-capable system or slower network configuration.
I adore YTers who quote what % of CPU or ram is being used without saying. Because that means they think we're family and know every detail of each other. And will be pleased when I show up to crash in their guest room.
This is what I’ve been waiting for. I actually downloaded and started running it before even finishing your video. Great find! Thanks!
I can feel your excitement. I was about to do the same but think I will use discipline and read the comments first 😂🎉. Self control!!
@@CM-zl2jw haha too smart!
So far testing has been pretty good 👍
How did you get it working, every time i enter a message, i get error occured: failed to fetch.
I'd like someone to do a Pinokio, Petals, Oobabooba, Jan framework comparision. Oobabooba and Pinkokio give you a lot of under the hood options I'm not seeing demonstrated here - pre-prompts, token buffer access, etc.
There is also LM studio that does the same thing only with a bit more controls :)
Or oobabooga with infinitely more control and multimodal. It’s the Ui that everyone creating models uses.
Do any of these allow for agents or the equivalent of copilot or autogpt?
From the video the main difference with lmstudio is Jan is open source, but am not sure if it can run on older processors or machines like lmstudio is starting to do..
@@ChrisS-oo6fl Yeah, but it's a pile of code-vomit on Github, which is exactly why normal peeps like me are NOT using it...
LM studio can also run Autogen @@mayagayamsee various tutorials on youtube :)
Thank you ❤
Thanks again, Matt. You are truly a legend. I have searched online for ways to run my own chat bot, and just like magic, you present it. Thanks mate.
Just might be the post we have all been waiting for, way to go Mr. Wolfe.
I'm definitely looking for something like this, but I need to feed it about 100 PDF files that it needs to scan and know intimately, then I would be thrilled.
There are local models trained for way higher token limits upto 20k.
@@USBEN.
What are you talking about? Self hosting an AI needs tolkens? What?
@@americanswan token limit =word context limit it can take in.
Like 4096 slider in the video but upto 20k words
@@americanswan These models are restricted in what amount of content they can "remember". This content is measured in "tokens". GPT-4 has a context window of 128k tokens (which is a lot), which some people say means it could remember and talk about a 300 page book for example. So to find out if there is the right model for your needs, you would have to find out how many tokens the text in your PDFs has. As far as I know, OpenAI has a tool where you can paste text and it tells you how many tokens it has. Then you would have to find a model that has enough tokens in it's "context window" so it could remember the text in your PDFs. If every PDF only has one page with text it would be a lot easier than if it has a 100 pages (since a higher token limit also means higher demands for your computer). The next problem is that in "Jan" there doesn't seem to be an option to input your documents, so you would have to find a similar tool that allows that. At this moment in time I don't think you will find a satisfying solution (if your PDFs have a lot of text). But many people are looking for solutions for exactly your problem (especially since this would be very valuable for a lot of companies). So I'm optimistic something will come up in the next months or so.
@@USBEN.can you point to best working models with higher context in your opinion?
Thanks Matt! Just what I have been waiting for. Now all I need is an image generator like this.👍
you can try stable diffusion to generate image locally on your computer
@@Krisdomain That would be awesome! Thanks! Is it as easy to set up as JanAI or is it difficult?
@@BionicAnimations It is easy as I could do that. However, the images I generated were not nice, maybe I used poor models :(
There are great tutorial to install, I enjoy the Automatic1111 UI, it's prettier to use, but if you want quality images to be generated locally, there is some studying required and follow up installs. Keywords for your search: ControlNet , Negative Prompting, Seed number. This should give you an idea to where you are heading.
@@stribijev Hmm... I've usually seen really good pics with Stable Diffusion. How did you learn how to make them?
I installed Jan... downloaded 4 or 5 different models. 0 have worked. "Error occurred: Failed to fetch"... 🤷♂
I got the same error message
Same @@Dayo61
@@Dayo61 Same, and no help , nothing........no answers to fix it
Hey Matt, I love your content, been with you for almost 2 years now! Have you found a similar tool that allows your to upload files to a local LLM for data analysis? Basically this same thing, but with file upload capabilities?
as someone who isn't very tech savy but who's been wanting a nice jumping on point to offline local chatbots, this is a brillient start.
Videos like this and channels like network chuck have been a huuuuuge help in getting me more involved and helping me understand it better, one step at a time. Incredible work you guys all do, keep up the awesome work. :)
Edit: I also wanna note that AI itself has been getting such a bad rep and it geniunely surprises me. Like i understand the problems of jobs, and bad actors, but any advancement in technology has problems like that which arise. l mean look at how people viewed the internet in it's infancy. And yes, there are bad people who do bad things. But thats not the technologies fault, thats the descretion of the individual who uses it.
When it comes to AI, i prefer a glass half full approach. Sure it can be used for bad, but it can also be a wonderful and amazing thing if we give it the chance to fully evolve and shine and put some measures inj place to help against or deter the bad actors. :p
Amazing video! Thank you.
Matt, do you know perhaps how can we train those local models with the data and files from our hard drive? (word, excel, PDF files, etc)
Train or upload docs for reference
Is there a way you can share documents like pdf with it? If not, are there any alternatives that can do that?
Thanks a lot Matt!
I only know of ChatPDF (at the top if Googled). It's done online, so not ran locally, but it's free!
There are a handful of other good tools for running local LLM like LM studio (mac/PC) and Ollama (mac) that I think are worth mentioning alongside Jan.
LMStudio is closed source though.
I still prefer LM studio due to the ability to modify GPU layers and CPU offsets. Also, LM provides direct access to Huggingface models.
Me too, LM studio will also inform you if your hardware is capable of running the model.
I would love a comparison between the two, I was actually going to ask Matt for that
Do you have any tips for what the recommended settings are for GPU layers and CPU offsets? My laptop is not very robust, sadly, but I did just upgrade to 32gb of ram. That did not fix my high cpu usage when running LM studios, however. It still has moments where it spikes up to 100%
@@jennab176 depends if your laptop even has an independent GPU. Many of the mid to lower tier laptops with just use the CPU’s integrated graphics processing, which at that point modification of GPU layers won’t improve token processing speed. It really depends on your specific hardware configuration. I bet if you post your specs here the community can help you optimize your settings.
More apps for local LLM‘s:
- LM Studio
- Faraday
- GPT4ALL
Commandline (inst. use via UI)
- Ollama ( Ollama Web Ui )
- Text Gen Web Ui
- PrivateGPT
Do you know if any of those allow the user to "upload" documents to them to chat about?
@@missoats8731third try 😂 youtube removes constantly my answers … yes g p t 4 a l l
Great video ! Note that the experimental release has the GPU option.
i get an "error occurred: failed to fetch" message, is it because my video card isn't good enough? i have 128GB of ram
@Matt Wolfe - Thank you for making sense of this AI world for me so that it is easier for me to do so.
what about LM Studio? That is my current fav.
🎯 Key Takeaways for quick navigation:
00:00 🚀 *Introduction to Jan Tool*
- Introduction to a free open-source tool called Jan for running chatbots locally on your computer.
- Jan is secure, offline, and supports various operating systems like Windows, Mac, and Linux.
- Highlighting the tool's simplicity and availability under the AGPLv3 license.
01:11 📥 *Downloading and Installing Jan*
- Demonstrates the process of downloading and installing Jan on a Windows PC.
- The availability of open-source models like mistol instruct 7B, llama 2, and mixol 8X 7B for download.
- Emphasis on the user-friendly interface of Jan, making it easy to explore and install models.
03:13 🌐 *Connecting to Internet for OpenAI Models*
- Explains that while open-source models like llama and mixol can run locally, OpenAI models (e.g., GPT 3.5, GPT 4) require an internet connection and an API key.
- Provides guidance on obtaining an API key for OpenAI models.
- Discusses the distinction between free local models and those requiring an internet connection.
05:24 💬 *Chatting with GPT 4 Using Jan*
- Demonstrates the process of setting up a chat session with GPT 4 using Jan.
- Highlights the cost associated with using GPT 4 due to the reliance on the OpenAI API.
- Showcases the real-time interaction and responses from the GPT 4 model.
06:16 💻 *Local Model Mixol 8X 7B Usage*
- Illustrates the usage of a local model, mixol 8X 7B, without requiring an internet connection.
- Emphasizes the cost-free nature of running local models directly from the hard drive.
- Compares the resource usage and output quality with GPT 4 and GPT 3.5 turbo.
07:50 🔄 *Exploring Diverse Local Models*
- Discusses the variety of open-source models available for download and use in Jan.
- Highlights the flexibility to download and run models locally, even without an internet connection.
- Points out the ease of disconnecting from the internet while using local models for enhanced privacy.
08:59 🌐 *Conclusion and Recommendation*
- Concludes by emphasizing Jan's simplicity, speed, and cost-free usage for local chatbot models.
- Expresses the lack of sponsorship or affiliation with Jan, making it an unbiased recommendation.
- Encourages viewers to explore Jan for running local chatbots easily and efficiently.
Made with HARPA AI
Whoa nice hardwork
Matt - big fan of your videos. This Jan chatbot tool looks great. Thank you for bringing us all the latest and greatest info on AI. 😃
🎯 Key Takeaways for quick navigation:
00:29 🖥️ *Running Chatbots Locally*
- Use Jan app.
- Free and open-source.
- Works offline too.
01:50 🌐 *Variety of Models*
- Llama, Mixol, GPT.
- Open source models.
- Options for customization.
07:50 🚀 *Easy and Efficient*
- User-friendly interface.
- Quick setup.
- No internet needed.
Made with HARPA AI
can you train it with your local documents and chat about them?
How do you know it won’t steal your data or worse? Is it a good idea to be downloading such a large file??
This is great Matt thanks. Now we just need jailbreaked model
i have already found this, its called "Pi GPT" and it will answer the most outrageous and evil questions you can imagine
This looks cool! It wasn’t clear to me from the video, but I believe only open source models can be run locally (versus paid OpenAI models like GPT-4).
That is right, you can see Matt uses his own API key to enable GPT models.
This is because there is no "downloadable" version of gpt. Mixtral, on the opposite, doesn't have an online version (provided by the developers) so if you want to use it you must download it and run it on your computer.
@@stribijev Yeah, that’s when it became clear the title is misleading. I think the video title should be changed to “Run Any Open Source Chatbot FREE Locally on Your Computer”
@@r0bophonic Right. Actually, it is no news, the LLM Studio has been out there for quite a while.
0:09 Answer from Phind 34B Q5: As an AI language model, I don't have specific hardware requirements to run. My existence is based on neural networks and cloud computing, so the minimum resource needed would be a stable internet connection to communicate with the server where I'm hosted.
Something is suspicious about this application. To the question "Why my GPU is running 100%?" I had a very high response time, while the CPU reached 82%, the GPU at 97% (constant) and 52 GB of memory. I didn't have such consumption of resources even when editing videos. Is it possible to make your resources available as happens with the blockchain?
Just a warning to people who have never run local models: Don't expect GPT4 quality nor speed.
🎯 Key Takeaways for quick navigation:
00:29 Open-source *chatbot tool.*
01:50 Diverse *chat models.*
03:13 Requires *internet for some models.*
06:04 Free *local models.*
07:50 Privacy-focused.
08:16 Easiest *way to run chatbots.*
Made with HARPA AI
Thanks! Lobo, why not a video about the computer you need to run locally these models? Minimum RAM CPU-GPU, NVIDIA type and so on? Could be very useful for those thinking to buy a new PC to play with AI at home.
That's EXACTLY what I want. I tried asking the local PC shop but really they just know gaming and were basically offering me the most expensive of everything
The lowest spec that I have been able to run 7B models (4-bit quantised) in LM Studio is GTX 1060 6GB laptop with an old Core i5 and 16GB RAM.
If I would build a dedicated PC, I would go with RTX 4060Ti 16GB GPU, any modern 8 core CPU and 32GB RAM.
This guy isn't technically qualified
He's just running a CZcams channel
@@konstantinlozev2272 I've been having fun lately via LM Studio, running dolphin-2.0-mistral-7b.Q5_K_M.gguf. My machine has 16GB RAM and an RTX 2060
@@40g33k You can certainly know when you are running on the GPU Vs the CPU. What is more, GPU memory use (the critical metric) can be monitored with MSI Afterburner.
Hardly rocket science.
Best AI news. I tried for a couple of weeks to run LLAMA like 6 months ago and found it very challenging.
i wish you mentioned how much ram does one need to run this the fact its using 75% of your ram doesnt say much because you didnt mention how much ram you have
I run 64 gb and i suspect he does too, being a content creator.
the only issue i have with these, like Jan and LLM studio is the chat history or memory, anyway to have the local models save a memory locally so they can remember the chats forever?
How fortunate for me! I was just thinking about having an offline chat, and BOOM! Here ya go! Thanks, Matt, I love the content! Keep it up!
This one is likely one of my favorites yet! Thank you!
Amazing! Now we just need a simple Windows program like this that transcribed Audio/Video to text, using something like Whisper AI. Every other option out there isn’t user friendly like this. Hopefully in 2024
Descript is really good for transcription.
🎯 Key Takeaways for quick navigation:
00:00 🚀 *Introduction to Jan App and its Features*
- Introduction to Jan, a free and open-source tool for running chatbots locally.
- Jan supports various operating systems, including Mac, Windows, and Linux.
- Overview of Jan's compatibility with both open-source and closed-source models.
01:25 🛠️ *Installing and Exploring Jan App*
- Installing Jan on Windows and accessing the chat page.
- Explanation of the absence of pre-installed models and how to explore available models.
- Listing various open-source models like Mistol, Llama, and Mixol 8X 7B.
02:57 ⚙️ *Installing Mixol 8X 7B and Managing Models*
- Installing Mixol 8X 7B model and understanding its size.
- Mentioning pre-installed closed-source models, GPT 3.5 Turbo, and GPT 4.
- Highlighting the flexibility of setting up multiple models for comparison.
05:12 💬 *Interacting with Chatbots using Jan*
- Creating a chat thread, setting custom instructions, and selecting models.
- Demonstrating the usage of Open AI models like GPT 4 (requires API key) and Mixol 8X 7B.
- Emphasizing the cost associated with using Open AI models for demo purposes.
07:36 🌐 *Running Jan Offline and Exploring Model Variety*
- Discussing the ability to run Jan offline with local models like Llama and Mixol.
- Highlighting the freedom to download and use various models, including uncensored or fine-tuned ones.
- Describing Jan as an easy, fast, and free solution for running chatbots locally.
08:59 👏 *Endorsement and Conclusion*
- Clarifying the video's non-sponsored nature and expressing excitement about Jan.
- Encouraging viewers to explore Jan, emphasizing its simplicity and intuitive interface.
- Concluding by positioning Jan as a valuable tool for easy access to local chat models.
Made with HARPA AI
🎯 Key Takeaways for quick navigation:
00:00 *🤖 Introduction to Jan Chatbot Tool*
- Introduction to Jan, a free and open-source tool for running chatbots locally on your computer.
- Jan is accessible on various platforms including Windows, Mac, and Linux.
- Users have the flexibility to download, modify, and build their own versions of the tool.
01:11 *🔍 Exploring Available Models*
- Overview of the models available for download and use in Jan, including both open-source and closed-source options.
- Highlights the availability of popular models like GPT 4, GPT 3.5, Llama, Mixol, and others.
- Users can experiment with different models and compare their outputs to find the best fit for their needs.
04:31 *💬 Setting Up Chat with OpenAI Models*
- Demonstrates the process of setting up a chat with OpenAI models such as GPT 4 and GPT 3.5.
- Users need to obtain API keys from the OpenAI platform to use these models.
- Provides insights into the cost associated with using OpenAI models, which varies based on usage.
06:04 *🛠️ Running Local Models with Zero Cost*
- Shows how to utilize local models like Mixol and Llama without requiring an internet connection.
- Highlights the advantage of running local models with zero cost, aside from the power consumption of the user's GPU.
- Offers a comparison of performance between local models and OpenAI models in terms of speed and resource usage.
07:36 *🔒 Privacy and Flexibility of Jan*
- Emphasizes the privacy benefits of running chatbots locally on one's computer with Jan.
- Discusses the flexibility of Jan in downloading and running various models, including uncensored or fine-tuned models from other sources.
- Highlights Jan as a user-friendly and efficient solution for accessing and utilizing local chat models.
08:16 *🌟 Conclusion and Recommendation*
- Concludes by recommending Jan as an easy, free, and efficient solution for running chatbots locally.
- Suggests that Jan may be the best entry-level tool for accessing and using local chat models.
- Encourages viewers to explore Jan further for their chatbot needs.
Made with HARPA AI
Mistral under Jan freezes up my MacBook. 16 gg may not be enough ram?
FABULOUS! Thank You! Brilliant. "Works right out of the box". IS there a way to run it on the GPU instead of the CPU?
Was waiting for this video for a while. Thank you
Wow! Best practical video I've seen in many months! Thanks, Matt!
Awesome Matt very easy to follow. Happy New Year 😊!😊
GPT4all is easy to and can question local docs
Gpt isnt preinstalled. Because its running from the API it dosent need to be installed
getting error message:```Error occurred: Failed to fetch```
please help
On top of that, Mistral requires 24 gigs of dedicated ram on your GPU to run so that’s also kind of a downside lol.
Also, thanks for sharing this much simpler local AI, I’m excited to try it out
But.. it's local and offline that's a very good upside
@@dark_mode Exactly!👍
I like oobabooga. You can choose and install whatever model you like from hugging space. You can even talk to them not just chat. There are some models that are extremely good at role-playing.
6Gb of GPU ram for the Mistral 7B model is the minimum. Though more is better.
CPU inference can be used but it is quite slow. Unless you want same X rated content it is better not to do it lol.@@tigrisparvus2970
i was just messing around with this last night!!!!!! dude, you are on TOP OF IT MATT!!!!!! lets go to a Padres game!
This is definitely it. You hit the nail on the head. This was much like what DiffusionBee to bring Stable Diffusion to those who weren't prompt proficient but it's pretty crazy! Happy New Year !
Can we upload ppt,docs and then ask these models questions based on the files uploaded?
Thanks for recommending this. Been looking for something like this for a long time. Can recommend which model is good for SEO research and article generating?
Does any of this apps/models support external api calls and upload files?
Matt, when do u think we will get the kind of advanced tools that you don't need to be a coder to maximize the capabilities. I'm a filmmaker, and I don't have the technical proficiency to do a lot of what might seam easy to many of your watchers. But I know how to use the tools once they are simple for the average person. - Love your videos!
2024, probably, maybe.
From my experience playing with AI over the last year.
Didn’t you notice that Matt doesn’t quite _think_? He parrots what people of Twitter say or what ChatGPT writes for him.
@@natalie9185 He obviously does his research and reports it in a clear and visual way. I'm def a fan.
Another great update. Thanks Matt! We appreciate you.
Hi, Could you help me?
What can you advise? The book is created, the outline is created, there is an understanding about the number of chapters and their titles. It is necessary to structure the text, link and divide it into chapters. Who of AI is the best at this? Free and paid options? Thank you in advance
lm studio as well. paired with open-interpreter you can use local llms to help with automating tasks on your pc. includiong opening things and creating files as well as create and execute code
I know there are some Models that also allow us to input images, similar to gpt vision. Is there any program that simplified that process?
Another advantage of these free models is that they are less censored than GPT/Claude
the models are not working. I always get that response: Error occurred: Failed to fetch
'Even the 24 gigabyte only took about 3 or 4 minutes to download ' *Cries in Australian Internet Speed
Great Video, What PC configuration with Windows do you recommend to use a Mistral version?
It installs everything on the C-Drive. App, Models, everything - without an option to store it on an external drive. I prefer to keep my AI apps and models on a separate external drive. Until that is fixed, never mind. I have no interest in filling up my OS Drive with LLM Models.
Hey Matt,
Can I customize the models after downloading so that I can train it on my own data?
Nice. Super valuable video. Thank you! You talked about the small cost for using one's own OpenAI API key, but do you know (or anyone here in the comments know) if those conversations are private, or does OpenAI use them to train the model like they do with using regular GPT3.5 or 4?
When using the API the personal data is not used for training in theory.
Not used in training. I think that there is also an opt-out toggle in paid version of regulat chatgpt.
@@InnocentiusLacrimosayes. True. You can opt out of sharing data on chatGPT but then it doesn’t save the chat history.
Wondering if there are any malware or security issues to be aware of....
Finally. Ollama STILL doesn't have the "coming soon" Windows version and my antivrus software won't let me download LM Studio. Great find, Thanks Jan. Can't wait to see Marsha. 🤣
Love your videos matt always informative
This is EXCELLENT @Matt thanks for sharing this.
I've done all the cybersecurity training, and refresher training, and in-person training, and read the reminder emails, yet here I am about to download and install some more mystery software because of some dude on CZcams.
Go, Matt! Great video as always!
Thank you, Matt, for the engaging podcasts-I'm a regular listener and always come away with new insights. Your discussions have sparked my interest in the burgeoning field of AI, and I find myself in need of a more powerful PC to keep up with the demanding tasks. I'd love to know the specs of your current setup. I'm considering an upgrade to an Intel Core i9 12900K with 64GB RAM, and I'm torn between the RTX 4070 and 4070 Ti, as the top-tier 4090 is beyond my budget. Any recommendations or insights you have would be greatly appreciated!
Podcast? Title?
I've been looking for a way to run a secure local model for a while and Jan definitely looks like one of the easiest.. However I use GPTs mainly for data analysis. Is it possible to attach files for analysis using this application?
Matt. Thank you for your hardwork.
Man I hate ChatGPT 4 limitations that they had placed in. It wasn’t on there before.
How am I supposed to get stuff done with 50 prompts 😣
Anyone know way around this?
I feel the same way. For me it's 40 prompts every 3 hours. Plus, everything seems to violate its terms. It's very annoying now.
@@BionicAnimations yeah 100% and you continually have regenerate responses or start a new chat 💬
As for some random reason it crashes.
Lol. It was so awesome just 4months ago.
Hope it changes some time in the future.
You have just saved my life! This is what I have been looking for!
Until we get a decent open source, dont forget that gpt3.5 turbo is also free...
Definitely going to try that Matt! on a M1 Mac mini 16GB
Great video
Consider making a video on how to customize or train on specific topic
Interesting post, Matt. Thanks for sharing.
I think the takeaway is that very soon, maybe a year, you'll get GPT4 levels locally. GPU sales will skyrocket again and you'll be paying a fortune to get one 🙂
AI on GPU is a bridge technology. GPUs were what we had that worked best for AI when AI was still niche. Now that AI is everywhere we are already seeing the first dedicated AI chips hitting the market. AI is probably what will finally give ARM chip architecture its breakthrough. We are seeing a major technology shift that will probably end with a completely different computer architecture. GPUs will either disappear completely or return to their original purpose. That will of course not change the fact that the demand for chips in general will skyrocket.
@@trnogger I'd forgotten about the new architectures. Cheers
Thank you so much, there are so many software that do similar things but are a pain to install and run. This is super easy.
will it automatically use gpu instead of cpu? or do you need to change settings?
Great content, including your website. Thank you!
Is this a potential replacement for a business intranet search, where guidance and instructions are stored but unsearchable unless you already know the keywords you need?
I want someone to be able to ask "How do I deal with a mentally unstable caller" (or whatever their semantics dictate) and actually get the best page for supporting vulnerable customers insted of "no results" because none of the keywords match, or (even worse) get results that show how to improve the stability of the VOIP software.
Great work thanks Matt!!😊
Thanks, Matt! This is great! Do you know if these models can be trained in this platform to be customized for specific areas?
Thank you I've been following your work for awhile this was great information I mean all your news is but this is crazy it's fast an free an so easy i mean wow that's awesome
Can you see and investigate if there is any useful AI that is not for sex chat or writing a VC? Like is there any AI that can use a webcams to take images of a room, use segmentation and then give suggestion if it time to clean, declutter, put something into a drawer or just suggest rearrangement and updates? This may sound a bit silly and I would think so too 1 and a half rear ago, but then I got a smaller stroke and I then can eat food, place the plate on a side table and then forget it, and even it is there my brain do not do the connection to remove it directly, but if some ask me why I have a plate of old food in the bookshelf I can remove it.
So just now I looking for any use of AI and home automation that can help, the problem with home automation is that it cannot confirm that something been done.
Btw, I like your videos and they keep my hope up for good and useful local AI in the future.
Wow you are very impressive in the creativity of solutions you’re thinking about. What if AI just recognized food, clutter etc and told you? You could run it on a schedule, the report could be written as an imperative (not “I see food “ but “clean up the food on the desk.” Then you don’t need it checking up on you per se, just run 1x a day automatically and it will keep pointing out the food until it’s gone.
Error occurred: Failed to fetch. What's the fix?
What's the difference between this and LM Studio?
Can you explain Custom Instructions? Or provided a link for info? Thanks 🙏
In Chat GPT, the paid version, you can click your user name and then find the custom instructions stuff. In there you can enter who you are, what you do, your interests, how you want CGPT to reply etc. I find it more useful than creating a custom GPT.
Is this an ad? LM Studio has been doing this for much longer and supports more models.
I just tried installing it on windows 10 but bitdefender thought it was suspicious and broke my Firefox install (blocked related applications it said).
I've added this issue to git and reply is that it's fixed in the next release. I'll retry then.
Great video 👌
It works, but here is a very annoying issue!
you go to download more engines/select a different one,
and that area is on the bottom right screen, YOU CANNOT SCROLL DOWN TO CHOOSE ANOTHER ONE!
you have to go to the top left corner then hit the second icon which shows all AI.