Running a Hugging Face LLM on your laptop
Vložit
- čas přidán 3. 08. 2023
- In this video, we'll learn how to run a Large Language Model (LLM) from Hugging Face on our own machine.
Blog post: www.markhneedham.com/blog/202...
Notebook: github.com/mneedham/LearnData...
Other videos showing how to run LLMs on your own machine
• Running Mixtral on you...
• LLMs on your own machi...
• Running Mistral AI on ...
• Hugging Face GGUF Mode... - Věda a technologie
You explained completely and perfectly without wasting the audience's time! well done
Thanks!
except for the part that i have got no clue what the required software to start is. what os to run on nor how to run the code in step 2. the farthest i got was git bash for the transformer and thats about it. i just want to run some local ai, why does it have to be so obtuse??
@@norwegiansmores811 It's running in a Jupyter notebook - jupyter.org/ but you can run the code anywhere that can run Python code e.g. a Python REPL or a Python script.
@norwegianmores811 I think the easiest way to run local AI as of June 2024 is now this - lmstudio.ai/
Awesome! Thanks for this video.
Thanks for this video!!
Super video!
That's Great! Thank you!
Super helpful and easy to understand!
Glad it was helpful :)
Absolutely wonderful video! to the point and well explianed! way to go! thanks a lot!
Thanks - very kind of you :D
thank you very much for this great explaination
This was an extremely informative video. Really appreciate it.
Thanks, glad you enjoyed it!
Thank you very much, you helped me a lot
this video saved my day
Super helpful 👍
Random idea, but a video on how to run an embeddings model on a laptop would be really cool 😀 Could even combine embeddings + text2text for more specific answers. Or even t5_3b + selenium to create something similar to bing chat. Anyway, wish you luck on CZcams 😊
Sorry, I didn't see this reply! I've got a notebook with that idea sketched out, so I'll create a video for that soon. On holiday at the moment, but will do it when I get back home!
thx
Awesome content, love your channel! Video is very informative and concise, thanks. As a friendly suggestion, you might want to give a couple of secs at the end for the video for slow people like me to hit that well deserved like button :)
Thanks for your kind words! Let me see if I can figure out a good way to implement your suggestion 🙂
Thank you! I finally downloaded a big llama model.. lol 😹
Winning!
thanks sir , however I want to know
1- how one can integrate specific set of models (pre-trained) ones in to Rstudio ? so that one can simply run examples on data "proprietary in my case " locally within R
2- is there a way to ask the inference API for tasks different from the typical sentiment classification of text for example "multi-entity tagging" , "modalities" ....etc
your input is highly appreciated
An API key is not needed if the model is downloaded and run locally.
So long as you have a behemoth of a machine. 16GB Ram = 100% memory usage 😭😂
I don't think anyone can afford an expensive laptop lol
Do you know how to run it on live servers!! How to get?
It is needed to download the model!
thanks Mark, very nice video, super clearly put!
could you please suggest, what could be the reason if (when trying to set the wifi off) the output of those lines of code is "ModuleNotFoundError: No module named utils"?
utils should be referring to this file - github.com/mneedham/LearnDataWithMark/blob/main/llm-own-laptop/notebooks/utils.py - so in theory that's independent of WiFi connectivity. If it can't find that module you could copy/paste those functions into the notebook and use them like that.
Hey, great tutorial! I also found your blog on this and followed that, but I always get this error: Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
This is my first time experimenting with LLMs, so any assistance would be greatly appreciated.
Oh I'm not sure about that error - I haven't seen that one before. Since I made this video I've been playing around with another tool called Ollama which I found easier to use. It might be worth giving that a try to see if that works for you?
czcams.com/video/NFgEgqua-fg/video.html
What is the configuration of your local environment
i want to work with a model that is tagged as 'text-generation' how do i run it?
What is the difference w.r.t to using the classical:
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)
Thanks in advance!
I deeply appreciate your video! Although I have a question, does this still works when the model file is a .safetensors or .pth file, not a .bin file? Thank you!
Yeh I think it should work with both of those.
Hi Mark! This video is very helpful, may I ask do you think fastchat can be used in combination with Qdrant for RAG? Thank you in advance
Yeh you can could combine it with any database to do RAG.
Hi Mark, great video. May I know your notebook and the configuration? I’m thinking switching to MacOS to play around with Gen AI.
I'm using the latest version of Jupyter Lab and I have it set to dark mode with pretty much every one of the views hidden so that I can use as much of the screen as I can.
Not sure if that answered your question, so feel free to follow up!
I am getting an error/ info log from transformers (twice) stating "Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained." The model then generates only a bunch of whitespace, no matter the input. I have followed through your steps and made sure the files were downloaded at the expected location. The behavior occurrs both with and without setting legacy=False.
Does it work after you see that message?
@@learndatawithmark getting the same error and hang when running in colab or locally. seems like the pipeline("my query") never resolves
Same thing happens to me! Hanging and giving random replies! I am using mac m3
Thanks Mark for this video. A quick question- Is this safe to pass some PII data to one of the open source hugging face models that require the hugging face API token ? If No, how can this be resolved in deployment so that there is no risk of data leakage ? Please guide through this.
It depends.
If you are passing your HF API token because you're using the HF inference endpoint then your data is getting sent to the HF API.
If you're passing it because you're downloading a model that requires token auth then your data will only be local to where you run the model that you download.
hi Mark - super helpful. can i run all of this in terminal?
You can. You can use a Python REPL or even the iPython CLI. Or you could put it all in a Python script and run that.
If you're getting a wacky error trying to perform `AutoTokenizer.from_pretrained(model_id, legacy=False)`, do pip install protobuf==3.20.1 and restart the jupyter kernel
Good tip! I get that error somewhat randomly but never quite figured out the combination steps that result in it happening!
I'm probably missing something, but where are you using the downloaded files? you are entering model_id in .from_pretrained(), how is it finding/using the downloaded model?
It's reading from the ~/.cache directory. So it constructs a file path based on that directory & the model id
Has anyone built a GUI for this?
i personally found disabling your wifi from a jupyter notebook to be bad ass
Haha, thanks. It took me a little while to figure out how to do it!
Hello there Mark i was wondering if i could use this method to download other ai models for example text to image models?
Yes you should be able to use a similar approach. There's a good guide on image to text over here - huggingface.co/tasks/image-text-to-text
Hey, can you tell me about your system info, i am using mac m3 and its not giving any response and running continuously?
I use a Mac M1 with 64GB RAM. I think it's a 2021 edition. I've found in general that the quantised models work better on my machine - either using Ollama or llama.cpp.
czcams.com/video/YDj_ScvBpKU/video.html
czcams.com/video/NFgEgqua-fg/video.html
What is the editor you are using on localhost ?
I'm using a Jupyter notebook in the video
hey um, i don't know if you'll read this in time, but I have a problem:
pytorch_model.bin: 0%| | 0.00/13.5G [00:00
Hard to know exactly why - maybe connectivity with Hugging Face or maybe your internet or maybe the download tool?! You could try going to Hugging Face directly and click through to files and download them directly to see if it helps.
Hi, wich webinterface you use for python scripts? I want to use it to :)
This is Jupyter Lab - jupyter.org/
sorry i'm just starting with this, the code you're writing in the beginning, what is the website called?
Do you mean this one? huggingface.co/
under the "..." in huggingface there is a "clone this repo" which copies all stuff onto your PC. seems simpler to me.
Probably works for this one but sometimes there will be multiple different versions of the same model and it'll take up all your free space if you do that!
1:03 - Thanks for this clarification. I'd done quite a bit of Google searching and scouring the Hugging Face website for this information. I found nothing of value. I'm a computer enthusiast / gamer and not a professional machine learning engineer. Since embarking on running an LLM locally on my previous daily use desktop, I've noticed its near impossible to find a model's resource needs. GPT4 says a 7b parameter model would consume about 48 GB memory. I asked it what size model would fit in my 12 GB Nvidia 3060, it said about 3.2 billion. My question for you is, why is it that everyone in this space who seems to offer a model (or talk about them) never includes something like a system requirements descriptor? Is it one of those situations where, if you need to ask, you probably don't have enough resources? Thanks for any insight you can give on this phenomenon.
My impression is that most of the models being created are assuming that you have insanely good GPUs to run them on!
Since I created this video, there's been a lot of work done by a guy called TheBloke on Hugging Face to 'quantise' the models, which effectively means that the amount of resources required is reduced, but the quality of the model is slightly reduced too.
I've found those models work a lot better on my laptop.
The Bloke is using a format called GGUF, which is kind of a defact format for LLM models. I made a video showing how to run one of his models on my machine - czcams.com/video/7BH4C6-HP14/video.html. That video uses a tool called Ollama which works on Linux/Mac - czcams.com/video/NFgEgqua-fg/video.html
There is also another library called CTransformers which lets you choose whether to run models on the GPU or CPU. I've found the 7B parameter quantised models work reasonably well even on the CPU. I should probably create a video about that I guess! But in the mean time, this is the link - github.com/marella/ctransformers
@@learndatawithmark thanks!
How can I use this downloaded model next time ?
It will already be there so if you try to use it again there won't be any need to download it
Great vid, however I’m getting a value error, failure to import transformers error even though I used pip to do that, wondering if it’s a python version issue, I’m using 3.10, wonder if anyone has any ideas ? Thx
Can you share a script with all the code you ran and I'll try to reproduce?
use 3.9 its more stable
I keep following along until about 12 seconds in, where you start typing into something and you say let’s open up age something, and carry on typing into whatever it is you’re typing into - I can’t get that far, I don’t know what to type into
I'm using a Jupyter Notebook, but the code would work in any Python environment or script
jupyter.org/
@@learndatawithmark ah thanks, that’s interesting - I’ve never heard of that
Hi, i try to find someone who uses GGUF directly and locally without using a .bin to launch it because I would like to launch it under python, is this possible? Or should I do something else?
You can do this using CTransformers like I did in this video - czcams.com/video/S2thmwdrYrI/video.html
I think you might even be able to do it with HuggingFace transformers, but I haven't tried it myself.
@@learndatawithmark if one day you make a video on this, I would like to see it, in fact what I would have liked was to discuss with the model directly with python without going through any interface and to give it a personality with json like we have could do it with webui (but without webui) I tried various methods and honestly I find so little explanation. I had the idea of making my own bot as I saw in "wifu" mode in the sense that it is totally customizable and we give it a personality with a long term memory. The basic idea was to have a small model just for me. I'm just frustrated to see bots that don't even remember talking to us 2 seconds before. xD
@@The_Little_One_Of_Darkness it sounds like you want to keep the history of the chat messages between you and the LLM? I showed how to do this in memory on this video using Ollama, but it can be adapted to another approach - czcams.com/video/MiJQ_zlnBeo/video.html. I can across a tool called MemGPT which I think attempts to solve this problem, but I haven't tried it yet - memgpt.ai/
sir if wifi is on then they model is working properly or not?
Yes it should work without wifi - but you will need a connection to the internet to download the model.
Huggingface is a single point of failure.
An index of alternative download URLs
for LLMs is needed for when Huggingface
is down, such as now (Feb 29, 0200 GMT)
hi. please help me. how to create custom model from many pdfs in Persian language? tank you.
So many steps missing in this video...
Pinned
Seems unnecessarily complex... isn't there like an online space to use this stuff without having to write a bunch of stuff just to download it?
Yeh I think with a bunch of the models you're able to run them on the Hugging Face website on the right hand side of the page. And then in general there are many services that offer APIs that you can call. The approach describe in this video is only for if you don't want to use those services.
When I run this: "os.environ.get("HUGGING_FACE_API_KEY")" I get"None". Is it normal?
I guess I need to set the HUGGING_FACE_API_KEY variable to my token beforehand.
You would need to set that environment variable before running your Python environment otherwise yeh it'll be none