Run any AI model remotely for free on google colab
Vložit
- čas přidán 7. 07. 2024
- 🦙 Installing ollama.ai on google colab with Ease! 🎉
In this video, I'll show you how to easily run ollama.ai and other large language models (LLMs) remotely using the handy platform 'Google Colab'.
Say goodbye to having not enough compute power to run your dream ai model! 🚀
We use Ngrok, to expose our ollama server from the colab server to the internet. Ngrok is a software which allows you to create secure tunnels and expose local ports to the internet.
🔧 What is Ollama?
• Ollama.ai to install l...
Ollama is an open source tool that simplifies the installation process of Llama2 and other large langue models (LLMs). It provides an easy-to-use interface for installing llms on your system, without requiring you to manually configure
any dependencies or settings. With Ollama, you can quickly and easily get started with Llama2 in just a few minutes! 🚀
📔 Jupyter Notebook
github.com/marcogreiveldinger...
📝 Installation Steps
In this video, I'll walk you through the step-by-step process of installing ollama on google colab with a python jupyter notebook. From downloading the tool to running your first remote AI, I'll cover it all! 💻
🎬 Watch and Learn
Whether you're a seasoned AI user or just getting started, this video is perfect for anyone looking to install and use LLMs remotely without any privacy concern.. So sit back, relax, and let me
guide you through the process of using google colab with Ollama! 😊
00:00 - 00:42 Introduction
00:43 - 03:21 google colab coding
03:22 - 04:14 start your jupyter notebook
04:15 - 06:34 link your local machine to the remote host
06:35 - 07:10 Outro
🔗
VIdeo is based on the example provided by ollama: github.com/jmorganca/ollama/t...
github.com/marcogreiveldinger...
ollama.ai
colab.google/
ngrok.com/
#googlecolab #ollama #ollamaai #privateAI #chatgpt #privategpt - Věda a technologie
Great stuff!!
Ngrok is now asking for auth -- solved this by adding
await asyncio.gather(
run_process(['ngrok', 'config', 'add-authtoken',''])
)
before:
await asyncio.gather(
run_process(['ollama', 'serve']),
run_process(['ngrok', 'http', '--log', 'stderr', '11434']),
)
please i use windows how to use ngrok with him?
@@lamechemohh9113 you mean Ollama? You will need to run WSL2 (if you have any Win version that is not the Home edition).
Thanks a lot for adding this! Just pinned the comment
and by the way, I updated the github repository to reflect your proposals :)
@@lamechemohh9113 use WSL for Windows.
great stuff bro, keep them coming, thanks again.
Thanks! I will :)
Thank you so much. I was killing my Intel mac with the LLM questions xD. This gives a good rest for it.
Perfect!
If you want to learn more about ollama.ai, head over to my initial video about it :)
czcams.com/video/vUxAkCcag5s/video.html
Thank you!
awesome man
Thank you.
I could run this succesfully in the terminal, but how can access the model or the collab through jupyter notebook instance?
fantastic
This is awesome stuff! Would like to know after this up can we connect this to Webui or Anythingllm?
Found a Gem 💎
That’s a fantastic video! Do you know if Ollama has OpenAI API compliant endpoints? So we could use Google Colab as a “Backend-as-a-Service” for some time in our chatbots :) One way I saw people doing is to create a long audio (like 12 hours of audio), loading it in the Google colab, and giving it a play, it’s a silence audio. It seems to work to keep the session opened for more time.
There is currently an issue at the ollama gh project, so feel free to check that out and track the progress :)
github.com/jmorganca/ollama/issues/305
And good tip with the audio sound, never thought of that ... 😄
great tip
I imagine it's costly to run LLMs.. is there a limit on how much Google Colab will do for free?
I'm interested in creating a Python application that uses AI.. from what I've read, I could use ChatGPT4 Assistant API and I as the developer would incur the cost whenever the app is used.
Alternatively, I could host a model like Ollama, on my own computer or on the cloud (beam cloud/ Replicate/Streamlit/replit)?
As a 3rd option, could Google Colab work in my situation? Is OpenAI's Assistant API totally different from the API to programmatically interact with llama2 , mistral , etc?
Hi amazing stuff!
Is there a way to connect the ngrok to jupyter notebook? Thanks!
Thanks for the video. One question though, how can I avoid downloading the language models every time I run Colab notebook? Can I save Ollama and its models in Google drive and retrieve them when running the notebook?
but can't run stable diffusion this way?
i got windows machine, do i need ollama locally tooo?
how do i save the progress, because everytime i run it, it downloads the model all from the start?
use your google drive bro, pay for 100gb. its worth it if you're an ai guy
canw e load 70B model with this same technque ?
can we just make it as api end point and create good frontend on top of it?
Yes, kind of. The url which is getting exposed via ngrok, it is also usable as url in front ends especially built for ollama.ai
Check out my other ollama linked video, there I show how to start up a front end for that. (Last section)
hey, how to the export step on windows?
I have the ollama installed
just use "set OLLAMA_PATH=the_url"
i got a 403 forbidden error, but replacing
run_process(['ngrok', 'http', '--log', 'stderr', '11434']) with
run_process(['ngrok', 'http', '--log', 'stderr', '11434', '--host-header="localhost:11434"']) fixed it for me.
Fixed it for me too, thanks
for me worked too, thank you!
THANK YOU
You are a life saver!!!
great spot! I already updated the script on github :)
How can we keep our downloaded LLMs permanently, eg. on a mounted Google Drive?
It would speed up the start of inference in a new ollama server start.
Yes, that's a brilliant idea! You can save those in google drive with this snippet for example:
import os
# Mount Google Drive
from google.colab import drive
drive.mount('/content/drive')
# Create a folder in the root directory
!mkdir -p "/content/drive/My Drive/My Folder"
# Start Ollama with a path where models are stored
OLLAMA_MODELS=/content/drive/My Drive/My Folder ollama serve
@@techwithmarco that's great, thank you! :)
4:33
The model is downloading on colab or locally?
Also can u plz tell what command changes will be there if we are using windows terminal?
The model is being downloaded on the remote machine (colab)
The commands stay the same, if you use WSL2 on windows with ollama.
This is Fire Can you help me connect open Interpretur Like this with So I can Give access to my computer But it wont load my PC that much
You can
Error: something went wrong, please see the ollama server logs for details
am getting this error after running export OLLAMA_HOST= ...
what should i do????
See the other latest comments or check out the new version on github. Should resolve the issue :)
@@techwithmarco okay I will check
@@techwithmarco thank you for the reply
Thanks for the video. You mentioned disconnecting the runtime. So I am assuming google will itself shut the running notebook in few hours. Do you know for how many hours continuously we can run this?
I just googled because I did not know, but apparently 90 minutes if you do not interact, or absolute 12 hours
@@techwithmarco so if we leave the server running colab tab ideal. I am assuming it will auto shut in 90 minutes.
Honestly I am not sure because I haven't used it for that long in one run.
I would assume it will be up for 12 hours because the tunnel is working in the background and the jupyter notebook is still running :)
It will ask for are you a robot? captcha if you're inactive for a while , you could write a small javascript fn in browser to randomly click some ui elements but yeah 12 hours is the hardlimit after that you can't connect to a GPU instance for another day or so
Very usefull video also the code
btw i can't get it running on windows
what would be the way to set OLLAMA_HOST variable on window
set OLLAMA_HOST= "--" doesn't seem to work it still runs on local machine
I think it depends on the terminal and shell you are using. Are you using the standard windows terminal?
Hii @@techwithmarco , thanks for chiming in. I'm actually experiencing the same issue as @abhishekratan2496 , but I'm running it on the Ubuntu terminal. Setting the OLLAMA_HOST variable doesn't seem to work for me either. Any insights on how to resolve this would be greatly appreciated! Thanks.
@@TirthSheth108 Okay that's weird. I just used it a few days ago and it worked perfectly. I'll investigate and let you know :)
did anyone figure this out? it just tells me that export is not a recognized command
Good do you have idea to benmarch mixtral on colab ?
No sorry
I am not that deep into AI stuff so that I know how to benchmark the performance 🥲
Where did you store the .ipynb file so ollama could access it?
See your other comment :) it is just a local file
Please what about windows user?
You can use ollama with WSL2, it is not available yet in windows
i get error like 403 forbidden
replace:
run_process(['ngrok', 'http', '--log', 'stderr', '11434']) with
run_process(['ngrok', 'http', '--log', 'stderr', '11434', '--host-header="localhost:11434"'])
That fixed it for me.
@@ricardomorim9444 tkx very much guys, solved my issue.
Great tutorial. Runs super fast compared to other methods. How would we ingest a document and query its contents? @techwithmarco
You can either use the a cat command to preview the content for ollama like in minute 5:44 or use langchain for example and make some custom code to read some specific files. There are cool examples on github like this: github.com/ollama/ollama/tree/main/examples/langchain-python-rag-privategpt
@@techwithmarco Thank you for the reply. I am on Windows using the beta version of Ollama and $(cat file) doesn't work from the command prompt and Powershell only uses local Ollama not remote. Do you know if there is an equivalent command for cmd?
Thanks!