Unlimited AI Agents running locally with Ollama & AnythingLLM
Vložit
- čas přidán 6. 06. 2024
- Hey everyone,
Recently in AnythingLLM Desktop, we merged in AI Agents. AI Agents are basically LLMs that do something instead of just replying. We support both tool-call-enabled models like OpenAI but have even now have a no-code way to bring AI agents to every open-source LLMs like with Ollama or LMStudio.
Now, with no code required, you can take any LLM and get automatic web scraping, web-browsing, chart generation, RAG memory, and summarization all autonomously and running locally.
If the future of AI is agents, AnythingLLM is where it is going to happen!.
Download AnythingLLM: useanything.com/download
Star on Github: github.com/Mintplex-Labs/anyt...
Chapters:
0:00 Introduction to adding agents to Ollama
0:45 What is Ollama?
1:08 What is LLM Quantization?
1:28 What is an AI Agent?
2:54 How to pick the right LLM on Ollama
5:11 Pulling Ollama models and running the server
5:45 Downloading AnythingLLM Desktop
6:17 AnythingLLM - Initial setup
7:21 Sending our first chat - no RAG
8:22 Uploading a document privately
8:43 Sending a chat again but with RAG
9:10 How to add agent capabilities to Ollama
10:45 Add live web-searching to Ollama LLMs (Free)
11:41 Using agents in AnythingLLM demonstration
13:24 Agent document summarization and long-term memory
14:35 Why you should use AnythingLLM x Ollama
15:00 Star on Github, please!
15:06 Thank you - Věda a technologie
@TimCarambat I had to pause the video just to leave a comment! I'm deeply impressed by the excellence and simplicity of the content presented here. It's truly remarkable to have access to such tools, created by a team that clearly demonstrates passion and a keen ear for what we all think and wish would be great to have, and at every update, distilling all p of these wishes into a few simple clicks within this amazing piece of technology! I'm immensely grateful for the opportunity to experienceh the brilliance of software engineering and development of Anything LLM, especially within the context of open-source communities. Participating in the advancement of genuine and incredible open tools is a privilege. Thank you Tim! I will be promoting this project to the moon and back, because this deserves to be known.
This is so incredibly kind. Sharing with team!
haha I was just about to leave a comment when I read yours. I feel the same. What a champion Tim is. I do not know if I will ever install AnythingLLM but I think I will donate to Tim regardless.
Aye, I was interested in anythingLLM a while back but chose another project for my inference server. I've found getting half decent agent capabilities to be a huge time sink for someone with my skill set (I'm a physical security guy, not a programmer) and the results just weren't worth the time invested.
Even basic agent capabilities with RAG, memory and so on in a package that I can just plug into ollama sounds awesome.
Prepping the server now. Here's hoping.
Tim, thank you for making the world a better place with this awesome tool! :)
You built an amazing piece of software. Thank god that I stumbled across this video.
Thank you for explaining quantisation in details for niebiews.
Im a chronic video skipper but watched this back to back. Great explanations and can't wait to try this out! Would love to see more videos, tutorials or even lectures from you. You really have a knack for explaining things!😊
PS I've starred on Github!
What I love about your tutorials is that you succinctly explain all the things that come across during the tutorial. Thanks!
Big thanks man. This video helps alot for me as an beginner to understand how good a local llm is and which Usecases we have. Thumbs up for this great video.
I had no idea you had a channel talking about your software. Im a big fan of your work!
This is the easiest all-in-one platform. Thanks. More videos please ❤
Great this will make LLM more understandable for many ppl.
Damm,... finally found the tools that i been looking for..MAN you save my day, i have been crazy stuck finding webui for my ollama remote server..your a gift from heaven keep it up your helping alot of people like us..thank you so much..❤❤❤😂😅😊😊
Thank you! Will test it for sure. I think you guys are on the exact right path 😎👍
Amazing Tim. Keep up the good work.
Yo honestly it feels great when guys like you make your software completely free and i also think you should keep a option of donation. after seeing guys like you i will make something great and i will make it completely free to use and open source. again thanks dude!❤.
Fantastic Tim! Mine doesnt have agent config, guess i need to delete and udate, ill try that, looks great! keep up good work, i love anythingllm i really do!
Awesome vid! Really impressed with how you presented the information. 🙏 thank you
This is so dope. Great no-code solution and it's awesome that it's open source.
Good stuff, will try it out. Subscribed. Looking forwards to seeing how this develops.
Incredible.. gonna make building tools so much easier. Cant wait to see more agent abilities added!
Anythingllm is awesome. Glad to hear custom agents are on the roadmap. It's the big hole in capability. Also need config to change agent promt. I scan a lot of code and the @ is used often to define decorators.
This is awesome work. I looked at the other simple to install Windows front ends and stumbled on this. Pretty cool stuff and I love how you can add documents and external websites to feed it information. An offline LLM is soooooo much more preferred. The only item I don't understand is why you could just ask a regular question once you provided the document, but used @agent when asking to summarize a document.
IMO, i find having a local LLM that even is **only** like 75% as good as on online alternative is just much more rewarding.
Like i can be on an airplane, open my laptop, and start brainstorming with an AI. Pretty neat.
Next evolution would be a local AI on your phone but i dont think we have that tech _yet_
@TimCarambat
I'm excited to see the features you talked about work with the ollama like in the video for the agent, as of now, its same as before I updated, but it's exciting to think of the future.
Awesome! thank you for this. looking forward to more information/details/examples on using agents w/AnythingLLM!
This is by far the easiest and most powerful way to use LLMs locally, full support, like and sub. And many thanks for the amazing work, especially being open source.
🫡
looking forward to utilizing AnythingLLM, it looks really awesome! congrats on creating such an impressive application! thank you!
Can't wait to use this. Thank you!
Wow this is amazing... I'm gonna go star you!
You are amazing. Thank you 🎉
Funny ! I heard yesterday for the first time about AnythingLLM during an AI-info event.... and discarded the idea of giving it more attention because it was presented as "just another local RAG support". And now I stumble across this video by chance - and the additional agent functionality changes everything ! BTW, very well presented , this feature !
My immediate idea & feedback: if there was ANY chance to model custom agents in Flowise and re-import the JSON exports of this Flowise flow as input for an AnythingLLM custom agent, you'd save yourself the trouble of designing your own agent editor AND would start with a comparably large installed base. OK, maybe that's just wishful thinking..... but maybe I'm also not the only one with this wish to facilitate local agent building ;-)
Great job! This is wonderful! I will be responding after using to let you know my thoughts if you care to see them :)
Great work. Thanks a lot 🙏
Hi Tim, thank you very much for the great video showcasing open source llms, and tools like anythingllm to create agents. I followed your video and successfully was able to do everything in your video. Are there other agentic videos for other usecases you made, look forward to see them. Cheers
This explained the rag and agents parts I couldn't set up. Great educational content for those who are not programmers. Appreciate your explanations being without that much of "pre-supposed" know-how, that coders have - which is most tutorials on youtube...
I still didn't get why there's a difference between @agent commands and just regular chat
In a perfect world, they are the same. AnythingLLM originally was only rag. In the near future @agent won't be needed and agent commands will work seamlessly in the chat.
So @agent is temporary for now so you know for sure you want to possibly use some kind of tool for your prompt. Otherwise, it's just simple rag
There were some concepts I didn't quite understand; for example, tunneling from the Windows PC to the Mac (if it's on your local network, why work with VPN protocols rather than client/server - due to needing a stateful connection vs. 200 response code or something?). But the interface itself is brilliant! And I think that when it becomes agent-swarm-capable it's going to be a much better option for me than Crew AI, as it feels more intuitive, I am just going to need multiple agents working together. I have never installed a local LLM, but you have inspired me to give it a try. Thanks!
Great video! I also want to know how well the RAG function of AnythingLLM performs. It's important that text, images, and papers are handled properly and meaningful chunking are achieved
A fantastic beginning! When do you think we willbe able to create our own agents?
Amazing work 👏
this is seriously very neat tool👏👏👏 Pls add some feature to custom develop agents with function calls. It will be helpful for our local automations.
This is shown in the UI that we will be supporting custom agents soon!
Really enjoyed this video. It's a great educational intro to these related AI tools, while featuring your very valuable product! 🫡
Would love to see this run stable diffusion and comfy ui workflows
Great work Tim! If using 'AnythingLLM' in the 'LLM Provider' section, can I load other LLMs that are not listed? Like the '8b-instruct-q8_0' you mention? So I don't have to rum Ollama separately to load a model?
So training/finetuning is coming up as well? Loving the progress and process updates, keep up the great work Tim!
how'd you know!?
We will likely make some kind of external supplemental process for fine-tuning, but at least make the tuning process easy to integrate with AnythingLLM.
RAG + Fine-tune + agents = very powerful without question
@@TimCarambat That's awesome to hear!! I created an agent to get insider info, that's how I know of course!
@@FlynnTheRedhead !!!!! I thought i was hearing clicks during my phone calls!!!
Been struggling to get custom agents to integrate reliably with external tooling, using frameworks like crewui with local LLMs. Would love a video guide explaining best practices for this
Really awesome demonstration. I am excited about agents. Would be nice to be able to build custom tools in python for agents to use.
I've been using it for months. Love it! Will you enable voice to voice soon?
We just did in our most recent update. TTS is live for all, STT is only live for the docker version. There are some restrictions and limitations we need to work around to get STT to fully function cross-platform. It will be solved soon
Wow, this looks *_amazing!_*
I’m just starting to experiment with local LLMs and wanting to play with agents; this looks SO easy! I’m going to download and set it up right away.
I’m also interested in Open Interpreter for having an AI assistant do things on my local machine. Can this interface with that, or is it really meant as a substitute/enhancement to it?
(Also, how can I support your project? I gather your biz model is selling the cloud service, but my usage will be purely local. Anywhere I could send a token few bucks?)
very cool!!! subbed!
Hi Tim, thank you for your dedication and effort in teaching us about local LLMs. I have a medium-spec computer with 4GB VRAM and 16GB RAM. The last time I installed ALLM, the inference speed was a bit slower compared to other alternatives. How does it perform with the new version? Thanks again.
Unfortunately, i doubt much would change on the inference side. When you say alternatives, what were you using? You might get slower responses in AnythingLLM vs just chatting via CLI in ollama, but that is because we are adding that valuable context to the prompt. More tokens = more work on the LLM to respond!
I'm so happy I came across your video. Thank you. I am having trouble on where you to get the base link that you pasted in @6:36 mark to install the ollama3
I was able to follow along from your other video to install it. Thank you I'm now a new sub.
Great Work :)
Great work - thanks for sharing - Question - how can we send data to the agent via webhooks - is this a possibility?
Debug mode would be ideal. Agent to scrape the web just exits without any error even though I do have search engine api defined
Can it also use dalle but unrestricted?
Looks really good and simple. I tried PrivateGPT using conda/Poetry and could never get it to work, so jumped into WSL for Windows connecting to Ubuntu running ollama, via WEBUI. Works great, but this just looks so much easier. Will have to give it a try. What I do like with the WEBUI I have is I can select different model, and even use multiple models at the same time.
Yeah, we didnt want to "rebuild" what is already built and amazing like text-web-gen. No reason why we cant wrap around your existing efforts on those tools and just elevate that experience with additional tools like RAG, agents, etc
Amazing software!! Congratulations and thank you! Very similar to MemGPT server but seems easier to set up and use. I wonder whether you can save a whole company database (i.e. ERP data: products, materials etc.) in it and being g able to ask questions about it? Also can you instigate more than one agent simultaneously?
In theory, this would be better delegated by some purpose-built agent that can traverse the data. Currently, we only have one-agent conversations but the code _does_ support multi-agent. We just find it to be really messy and cumbersome when many agents are once are trying to do something and your Ollama instance is already at max use generating tokens!
Awesome info……thx
Awesome tutorial Tim! Can this extract specific data from PDF files and save it to an Excel file?
Great things shown.
Tx for all the work and commitment.
🎉 Here is a kind of dedicated use case I am interested to get acces:
I am a mind mapping addict. I use Mind Manager, that stores the mm in .mmap format.
I would like to ask ANYTHINGLLM to help me scan all folders for mind maps on different subjects and Rag & summarize on them, without having to export all mmap files in another format. Is this doable at this stage? What else should have or have created?
Hey, thank you for putting out anytingllm. I have two questions: 1. When I ask the llm to remember something, is that long term memory stored on my pc on a server? 2. is the summary part of the long term memory necessary? And I have a feature request for an automatic long term memory. Meaning that I don't have to say specifically what to remember but that the llm will be able to recall the entire chat history - eveything i have ever said in that thread. Is that in the picture?
This is incredible..
Hi Tim.. I am absolutely impressed with the capabilities of AnythingLLM. Just a small query..how can I deploy it on a cloud machine and serve it as a chat agent on my website?
I actually want to add few learning resources as pdf for the rag document of this llm so that my users can chat with the content of those pdfs on my website.
I also want to understand how many such parallel instances of similar scenario but with different set of pdf is possible? For instance, if I am selling ebooks as digital product to my users, can I have unique instances autogenerated for each user based on their purchase?
We offer a standalone docker image that is a multi-user version of the desktop app. It has a public chat embed that is basically a publicly accessible workspace chat window. You can deploy a lot of places depending on what you want to accomplish: github.com/Mintplex-Labs/anything-llm?tab=readme-ov-file#-self-hosting
For this, you could do one AnythingLLM instance, multiple workspace where each has its own set of documents, and then a chat widget for each. This would give you the end result you are looking for
@sergiofigueiredo1987, @TimCarambat, I agree with Sergio. Wow! I have Ollama installed locally on a Windows machine in WSL. (I was leery of the Windows preview, but I may switch because NATing the Docker container is a pain.) I also pondered how to build a vector DB on my machine and integrate agents. You guys have already done it!
thanks a lot for your video, this the first video that i see and it's really simple to understand, i have a question, if for example the model that i use know that the capital of france is paris, how can i change that information and make the answer different from paris? best regards
Brilliant! Could it use my own computer as a data source for the agents? Can I scrape my mac?
Many thanks for nice introduction !
Is there a way to configure this LanceDB ? Is there a doc how it's integrated with the AnythingLLM ?
There is nothing to configure, it is preinstalled and saves to the same location as the application's main storage folder!
WOW! Mindblown!
The RAG is fantastic! I uploaded a 'Davidson's textbook of medicine' and was able to ask questions and what not out of it! Thank you for the AnythingLLM Desktop. Thank you! Thank you! I have no more words!
Did you ask it questions that validated that the agent/bot was not straying from its training data, and into realms of general knowledge - or hallucination? I have a gigabytes-large proprietary data set that I need to train on, without straying. Open to ideas from anyone reading this comment. Thanks!
@@agentred8732 The answers were pretty much on the point. I am trying to upload a much larger 'Harrison's Textbook of Medicine', but seems there are limits to the size of the book that can be uploaded.
So far, no hallucination, but I may not be pushing it to the edge.
I was told that if we train it on too much material, the output gets generalised and loses depth - I am not yet able to test it with more material.
Which model are you running? I tried llama 8B q8, and when I asked question about "Release Notes History.pdf", the results were ... bad.
@@alpha007org OpenAI API.
really nice!
This is perfect it just needs more tools and agent customization like crew ai and it is going to be an absolute killer for the ai industry.
Will be coming soon! Just carving out how agents should work within the context of AnythingLLM and should be good.
Also, it would be nice to be able to just import your current CrewAI and use it in AnythingLLM - save you the work you have done so far
Hi,
How does quantization type affects the system resources needed to properly run that model?
Great video by the way!
It mostly impacts the RAM and overall storage side of the GGUF modelfile. It's tricky to determine the exact requirement decrease because it has to do with the specific model parameters and other factors. Im not aware of a simple equation or expression that is a direct calculation for all models.
In general, lower quant -> Smaller file size and memory footprint when loaded, but much worse output performance
What agents do you have planned for future?
Phenomenal
@Tim do you know anything about home assistant, home automation application. Reason i ask is they already have some intergration with LLM but not with agents and not specialized for home assistant auto automations. When you have time check it out and see if its possible to integrate this with home assistant that would be great. Great job with the video!
@TimCarambat
Hey Tim it wont let me select anything under Workspace Agent LLM Provider even though everything is setup and working, obviously ollama is running and everything else in anything is using ollama fine in the app, but this selection option doesn't show like yours does.
@Tim: I am a professional translator (English to French), and I've just discovered AnythingLLM. Sometimes I have to translate confidential documents that cannot be shared on the cloud. They need to remain locally on my own computer. Once the translation is done, they have to be encrypted to be sent to clients.
Could I use AnythingLLM to help me with the translation process?
Could I use it with my actual Lexicum, glossaries and personal dictionaries? Most are PDF or DOCX files.
How would I do that? What are the first steps?
Many thanks if you can give me some hints on how to proceed.
I'm now a new subscriber! 😊
Literally sitting here wondering this when you dropped the video
Thank you so much the only problem i have is i cant seem to find anything llm github pdf file. Where do i download it from?
Does Internet Search Work for You ?
I configured the agent to use Google Custom Search Engine but search does not return any results.
With some models you _might_ have to word a prompt more directly. Like even explicitly asking it to call `web-browsing` and run this search. Which i know breaks the "fluidity" of conversation, but this is just a facet of the non-determinisic non-steerable nature of LLMs and trying to get them to listen.
Mostly, its the model that needs to be better so it can follow prompts more closely, but its also not always that simple!
Hey, just found you on a random youtube video suggestion. Love this concept.. A few questions, how deep into a website can this scrape? Can it read a sitemap or robots.txt and download all the data, summarize, etc? Can I hook it into different LLMs? For instance, assign agents to different LLMs? Most importantly, if we're using a vector database, can I feed it rows and rows of data to remember forever?
The one in the document uploader is a single site, but we have a deep website scraper as you mentioned.
You can use a different LLM per workspace and also per workspace-agent. So yes.
The vector database we use runs locally and is built in. It works like any other and yes does persist information - so yes to the last point as well
Would it be possible to see a vide of setting up your Ollama models on Anything LLM, I followed these instructions but my ollama models never load.
Amazing ⭐⭐⭐⭐⭐
wOOHOO I GOT IT NOW! ID LOVE A UPDATE BUTTON LOL!
It probably just was not refreshed yet. I think we have it on a 1 hour expiration to check so it may have been in between checks
Your the best
Great video! however, i followed every step you described in every detail, but i could not make the agents communicate with outside world. in any ''search'' or 'webscrape'' request, the model is hallusinating, and presents data that are already to its knowledge insted of real time data (i.e. current gold price ). i used llama3 Q8, i inserted google api and id code, i also tried the other search engine.. nothing. the logs show that it really creates json commands, but nothing comes in from the internet.... any help ?
I think it's a very good application, easy to use, and after testing it for a day or so, I have some wishes.
1. direct commands Bypass Agent LLM in Agent mode. It takes time for the agent to understand the sentence and convert it into internal command, and url parsing sometimes fails depending on the agent. For example, a command that scrapes the specified URL and shows the result, or a command that lists the currently registered documents with numbering. And a command that summarizes the document by this number instead of its full name.
2. I wish there was a way to pre-test the settings in the options window to make sure they are correct, such as specifying LLM or search engine.
I hope this application is widely known and loved by many people.
Hi, Tim. Great work. I'm trying to use Google. But... it is free for 100 queries per day. How I can control it or limit it? Other options are payable :)
Hello. I was just playing with RAG. It seams that the acuracy and results are very poor. I tried with laama 3, wizardlm etc. LLM is unclear of my questions. Is the context windows too short? LLM is giving answeres in a hindsight
Can I edit my question and regenerate the result in AnythingLLM? I use OpenAI GPT-4o api, but I don't find the edit button in AnythingLLM UI.
Will not load models in the linux version when I select local Ollama
Great!!!...
Thanks for the tutorial Tim. For some reason I am not able to get web search working. I am using the same setting you showed in the video. Can you do another video with more detail on setting up the google search engine for this purpose?
I have the same issues, the agent will answer that it doesn't need to use any function and will answer its alucinarions, if I give it the direct order to scrape it will trow a lack of openid key, I think is a work in progress still
Is the model just refusing to call the tool at all or when it does call the tool it says it failed?
@@TimCarambat the model will tell me no need for using any tool I got this and then hallucinate
@TimCarambat Can this run on an old slow server and connect to ollama on a fast server, or does AnythingLLM use a lot of local CPU when invoked?
Actually, this is a perfect combination. AnythingLLM using an external LLM and embedder is no more overhead than just running an HTML page - seriously.
The only demanding process is if you use the built-in embedder, and that is really only when you are embedding documents. Depending on the size of your documents you could crash the server with the built-in embedder. For reference, our hosted starter tier is 2vCPU and 2GB RAM and we squeak by.
If it's more than that, you are golden.
The vector database is so lightweight and fast it is legitimately a non-issue.
can you upload a video showing how to embed a chat widget onto a website using the llm ran locally on ollama?
@TimCarambat can you please let me know wich model is anythingLLM using to generate embedding and if it is possible to choose another one? thanks
We use the huggingface.co/sentence-transformers/all-MiniLM-L6-v2 by default, 384 dimension
So it works only with Ollama or also with LM Studio which is my LLM provider, as for many people Ollama does not work on windows?
I didnt go over every provider in the window, but lmstudio is supported as well and I was going to make a video showcasing that provider because there are many more models to choose from
What is the chunking method you use to create the vectors?
Is there a way that the user can control the method of chunking?
Say : Short, Token Size, Semantic, Long etc...
We currently use a static recursive chunk splitter. So basically just character counts. You can modify those chunking settings in the settings when you go to "embedder preference". So you can define max length and overlap
i tried anything LLM and RAG sort of works but I can never pull anything factually from my uploaded text files which are configuration files.
Is this a model issue? I was using Llama3 Q8 via ollama and llm studio.
I know it's a huge ask, but it would be great if it could listen to a inputs and active windows. it could be really cool if it could capture and describe my workflow, i could analyze what i am doing, and than generate macros for me.
Hello! Thanks for uploading this...very helpful. I'm stuck on something though. When I try to setup agents for Ollama, it says that agents only work with OpenAI currently. When I try to scrape sites like you do in the video using Ollama, the AI tells me that it can't. Am I missing something?
I have the same issue, but i am using lmstrudio as backend
You are able to use Ollama as you agent correct? If that is the case, are you using a small quantized model? Sometimes models have issues calling tools when they were built for that. Our system we implement works well, but we dont "force" the model to call a tool, it still has to generate a valid response to call it.
guessing things have changed but I cannot seem to find where my programmatic access api key would be
Also, will there be a text to speech and speech to text option?
It is a pending issue at this time, yes