If they had an option to load multiple models at the same time (if there's enough ram/vram), it would be cool. The current workaround is to dockerize an ollama instance and run multiple of them on the same gpu.
Wow. I am impressed to find one useful AI related channel. I mean you show things running with your code, you state real problems you find and you discuss your own results. Please continue with that 🙏 and thank you very much!
Thank you for another very informative video. It would indeed be cool to hear more about using Ollama and local LLMs with AutoGen and for a fully local RAG system.
Greater canter through the recent updates, have to say I am a fan of ollama and have switched to using it almost exclusively in projects now. Not least as it's easier for others in my team to pick up. Really short learning curve to get up and running with local LLMs.
Pls. create a video about hosting an LLM server with Ollama on Google Colab (free T4) available via API. That might be a cost efficient way of hosting "local" models.
Essentially this is based on llama.cpp embedded in Go but stranglely cannot handle concurrency. Love ollama and use it a lot but to run it in a production setting you have to basically spin multiple ollama server each of which can take a queue. In other words a load balancer setup with niginx or something.
I just noticed some of these updates yesterday and it let me simplify some bits of my stack and remove litellm. It's actually kind of scary how quickly all of this stuff is becoming commodity parts.
@@samwitteveenaiI'll throw in a suggestion - using DSPy for an LLM agent with tool usage! Imo DSPy seems really powerful for bootstrapping examples for optimal answers. Let's say we have an LLM agent that has the purpose of performing five or six different main purposes with one or two functions for each purpose. Can use DSPY to optimize the pipeline for each of those purposes, it would be amazing.
heh, run ollama run llama-pro:text "what are you" .. about 10 times and confirm that i'm not going crazy, it's the model . . . that thing is outputting it's fine-tuning data verbatim .. AFAIK
Ollama is Awesome however there are some minor issues with oLlama: 1. Single threaded, so can not run on server serving single url to team. It’s big issue, I don’t want everyone in my team install ollama in their machine. 2. With Stream response its not easy to create client app as the response is not same is OpenAI 3. CORS issue, so need a wrapping around the APIs, which means you need to install ollama and install api wrapper on every machine
Maybe if you actually take the time to check for yourself, you’ll notice that there is a web interface available, just need to point to your ollama instance, exactly the same as chatGPT, actually it is even better 🙃
Does Ollama support the same grammar specification that restricts your output, the way llama.cpp does? That’s a great feature which I’ve used in a project recently to force JSON output.
i love this series of introduction ollama!!! a lot!!!
Thanks for pulling this together. Really like the /show /save capabilities. Suggests new ways of creating and updating model files.
If they had an option to load multiple models at the same time (if there's enough ram/vram), it would be cool. The current workaround is to dockerize an ollama instance and run multiple of them on the same gpu.
good tip!
Wow. I am impressed to find one useful AI related channel.
I mean you show things running with your code, you state real problems you find and you discuss your own results.
Please continue with that 🙏 and thank you very much!
Wow so cool how local LLMs are progressing! So many ideas - can't handle hahah
Thank you for another very informative video. It would indeed be cool to hear more about using Ollama and local LLMs with AutoGen and for a fully local RAG system.
would be awesome for some tutorial videos on how you make those automated tools would be awesome to know how to do things like that
the logs feature is a game changer!
ollama is the nest bro. i use it witz web ui its amazing
Greater canter through the recent updates, have to say I am a fan of ollama and have switched to using it almost exclusively in projects now. Not least as it's easier for others in my team to pick up. Really short learning curve to get up and running with local LLMs.
totally how I feel about it. it is simple to the point and the code is open source. I have got my team using it and everyone picks it up quickly
Pls. create a video about hosting an LLM server with Ollama on Google Colab (free T4) available via API. That might be a cost efficient way of hosting "local" models.
Essentially this is based on llama.cpp embedded in Go but stranglely cannot handle concurrency. Love ollama and use it a lot but to run it in a production setting you have to basically spin multiple ollama server each of which can take a queue. In other words a load balancer setup with niginx or something.
👏🏻👍🏻
👏👏
I just noticed some of these updates yesterday and it let me simplify some bits of my stack and remove litellm. It's actually kind of scary how quickly all of this stuff is becoming commodity parts.
totally agree things are moving so quick
great video as always! Would you consider to cover lepton ai? looks like a great way to host llm on local machine
I just saw on Matt Williams channel that Ollama now run on Windows natively. Just thought I'd mention it to you.
yeah I saw they have been in beta. I don't use Windows but glad it is out.
I serve a model with ollama and I hooked it up to a discord bot :D
Great video! Can you please cover stanfordnlp's dspy, amazing library!
Yeah I have been working on a few ideas for this. Anything in particular you wanted me to build etc with it?
@@samwitteveenaiI'll throw in a suggestion - using DSPy for an LLM agent with tool usage!
Imo DSPy seems really powerful for bootstrapping examples for optimal answers.
Let's say we have an LLM agent that has the purpose of performing five or six different main purposes with one or two functions for each purpose. Can use DSPY to optimize the pipeline for each of those purposes, it would be amazing.
@@samwitteveenai I'd love to see some app that uses dspy with langchain and pinecone maybe.
Does ollama can working without internet connection when i'm using a local llm ?
yes it doesn't need an internet connection once you have downloaded it locally
does this mean it can run on windows now? it has been saying windows version coming soon on its website.
pretty sure they are still working on it and getting close.
can you actually stop ollama (linux) somehow ? or it runs forever and ever on the background ?
sudo systemctl stop ollama if you are using systemd.
heh, run
ollama run llama-pro:text "what are you"
.. about 10 times and confirm that i'm not going crazy, it's the model . . .
that thing is outputting it's fine-tuning data verbatim .. AFAIK
Ollama is Awesome however there are some minor issues with oLlama:
1. Single threaded, so can not run on server serving single url to team. It’s big issue, I don’t want everyone in my team install ollama in their machine.
2. With Stream response its not easy to create client app as the response is not same is OpenAI
3. CORS issue, so need a wrapping around the APIs, which means you need to install ollama and install api wrapper on every machine
great points!
I just don't get how it doesn't need an API for the OpenAI models.
its not running the OpenAI models it is using a mirror of their API to run local models
@@samwitteveenai thanks man. I guess I have to test it to find out.
I always thought that it is a pig
command line??? are you kidding?? This is super unconvinient + confusing + NOT for regular people! Not even half way close to LM Studio
Maybe if you actually take the time to check for yourself, you’ll notice that there is a web interface available, just need to point to your ollama instance, exactly the same as chatGPT, actually it is even better 🙃
Learn CLI and stop whining.
what is better llama.cpp or ollama?
I'm pretty sure ollama is built on top of llama.cpp
Does Ollama support the same grammar specification that restricts your output, the way llama.cpp does? That’s a great feature which I’ve used in a project recently to force JSON output.
@@mshonleif you need Constrained grammars I suggest you use localAI Is very easy to implement
local