Build A Full Stack Chatbot in Rust (feat. Leptos & Rustformers)
Vložit
- čas přidán 10. 07. 2023
- A walkthrough of building a ChatGPT clone using Rust on the frontend and the backend. Quick introduction to the Leptos framework, Rustformers/llm, and open source language models.
Code: github.com/Me163/rusty_llama
Leptos TailwindCSS example: github.com/leptos-rs/leptos/t...
Camera: Canon EOS R5 amzn.to/3CCrxzl
Monitor: Dell U4914DW 49in amzn.to/3MJV1jx
SSD for Video Editing: VectoTech Rapid 8TB amzn.to/3hXz9TM
Microphone 1: Rode NT1-A amzn.to/3vWM4gL
Microphone 2: Seinheiser 416 amzn.to/3Fkti60
Microphone Interface: Focusrite Clarett+ 2Pre amzn.to/3J5dy7S
Tripod: JOBY GorillaPod 5K amzn.to/3JaPxMA
Mouse: Razer DeathAdder amzn.to/3J9fYCf
Computer: 2021 Macbook Pro amzn.to/3J7FXtW
Lens 1: Canon RF50mm F 1.2L USM amzn.to/3qeJrX6
Lens 2: Canon RF24mm F1.8 Macro is STM Lens amzn.to/3UUs1bB
Caffeine: High Brew Cold Brew Coffee amzn.to/3hXyx0q
More Caffeine: Monster Energy Juice, Pipeline Punch amzn.to/3Czmfox
Building A Second Brain book: amzn.to/3cIShWf
Keyboards
Redragon K552 - amzn.to/3oNtpD7
Keychron Q1 - amzn.to/3YkJNrB
Keebio Iris - keeb.io/collections/iris-spli...
Purple Gradient Keycaps on Iris - amzn.to/3UZq93f
Corne v3 - shop.beekeeb.com/product/pre-...
Apollo themed keycaps on Corne - amzn.to/3IXKPUc
Chocofi - shop.beekeeb.com/product/pres...
Piantor - shop.beekeeb.com/product/pre-...
Kinesis Advantage360 Professional - amzn.to/3Ce5zUf
Glove80 - www.moergo.com/collections/gl... - Věda a technologie
Join us on Discord discord.gg/2G35Sp6kpQ
Does it have a bridge to Matrix?
It's so cool you use Rust on backend and frontend the same time! Would be great to see more videos on using Rust to build full stack! And the bot itself is super fun!
nice, glad to hear this one worked for many folks - looks like I'll be making more!
@@codetothemoon Please do!
Super interesting video! I love watching someone go trough a project small-ish but complete project. Would love to see more videos like this in future!
I’m just starting to code again. Last time I wrote code was in 1997 and was promptly discouraged as I am quite dyslexic. Not anymore with ChatGPT, holy heck has it been helpful for me! Watching projects from start to finish, and the methods for doing so is what I need to see and learn. =]
Rusty Bros... Get ready for another great ride
🚀
Thanks for this! You always bring out quality content.
glad you got something out of it! 😎
Your videos are some of the most timely, spot-on code tutorials available. It is as if I get an idea to do a project, and out of nowhere, you post a video showing me just exactly how to approach it. Really amazing stuff you produce.
someone's got a crush... but is it maybe likely that folks are about building chatgpt clones now, perhaps? :)
@@adicide9070 bahaaha :D
Just too nice, two things I wanted to have a look at united:
a Leptos example and
how to access G. Gerganov's language models in Rust.
So, thanks a lot! 🦀👍
thanks! glad you got something out of it!
Loved the video, please more Leptos!
this is the most disgustingly insane thing I've seen in a while. I will be trying trying both the llm integration and the all rust NoJS stack immediately
Great let us know how it goes! hoping to be able to build more stuff like this...
this is awesome!! excited to learn from this vid
nice! hope you get some value out of it!
That was the rustiest video I've seen in a while! Thank you! Keep up the good work. Really like the content and presentation. Also a bit of offtopic. I've noticed that you've switched from neovim to doom emacs several months ago. Maybe you can share your experience? I'd be very interested.
thanks for the kind words! my primary reason for switching to doom emacs is really org mode and org-roam, so I can take notes and manage knowledge in the same place I'm writing code. To me that's very valuable. I did made a video primarily about org mode ("Why I Prefer Emacs over vim") but I would like to do a more in depth comparison between doom emacs and a Neovim configuration framework like NvChad (which I've also been very impressed with)
Super useful, thank you!
nice, really happy you got something out of it!
Wow, great job!
thanks, glad you liked it!
Awesome tutorial!
The pattern you mentioned at 15:10 or so is very similar to the way things are done in PHP land via a package for the Laravel framework called Livewire. Essentially, two parts: A Livewire PHP component class, and a Blade (html templating engine) file that references it. So you might have
class MyComponent extends Livewire {
public string $someValue = "Test";
public function save() {
// Do saving stuff here
}
}
and then your HTML can just do stuff like
and
and you never have to write any of the transport logic.
Great work as always! Love your teaching style. Any plans on paid courses?
thank you! I do have something along those lines in the works...
Super awesome! ❤
thanks, glad you liked it!
Great video!
thank you!
This is awesome, thanks
nice, glad you got something out of it!
Can confirm I was able to get this working on my desktop. It's got a ryzen 5 so responses do take a bit longer, but just wanna say awesome guide and ty.
Still trying to get my head around rust, and I don't use js/ts at all, so I just copied the tailwind stuff from the repo and it was fine.
The rust error handler is such a joy. I just went step by step and fixed everything (I had lots of typos from typing by hand to get used to the syntax), and leptos built at the end just fine.
Cool to see Doom Emacs! The Monokai themes are nice but I prefer the default Doom One theme.
I agree the default one is pretty good. it's fun to switch things up now and then, maybe I'll give it another shot
Trpc i think gives you the same functionality in JavaScript land for not having to write fetches manually and just calling a function name.
This is awesome. Can you release some content on how you got to the level of rust you are today? What projects do you think could elevate a novice??
thanks! Everything I've learned about Rust has come from making videos about it and doing little projects like the one in this video. But most of my knowledge of software development comes from my ~13 years in the industry prior to learning Rust 😎
As for projects that can elevate your skills, the best choice is always the project that you're most excited about. I don't think one specific type of project lends itself more to learning than another - I think for any given type of project there are are ways of approaching it that are more suited for a beginner and other approaches that an expert might take.
Examples: make a game, make a webapp, create generative art, create a command line utility, etc. All of these are suitable for beginners and experts alike, though the approach might differ depending on the level of experience.
Great job man! What font are you using in your editor?
server actions are what you're looking for at 14:00
Great Thanks for Sharing.
thanks for watching!
Exceptional example showing a lot of interesting and useful things/techs! Instant subscriber! :D
The `[profile.dev.package.ggml-sys]` specifies to compile the listed dependency package (or all if you write `[profile.dev.package."*"]`) with optimization level 3, the same that's used when compiling with --release: that way the dependencies (that don't change often unlike our code under active development) can __run__ fast while our code will still __compile__ faster and have debug data
thank you, very happy to have you onboard!
and thanks for clarifying the meaning of `[profile.dev.package.ggml-sys]`!
Hey there! This's awesome! I have been wanting to do try full-stack Rust but for some reason I cannot get Leptos to deploy properly online. Locally everything works but when deployed to any host (e.g Github pages or AWS), it always shows an empty page LOL
I do have my backend running in Rust though via Actix Web :D
Thanks!!
thanks for watching!
Thank you for this, what is the plugin you're using to find your imports?
hey, man thanks for this awesome video, I wanted to ask, which hardware specifications does your laptop has
thanks for watching! It's an M1 MacbookPro with an M1 Max and 32GB RAM
Would the wizard-vicuna-13B-GGML model work if I switched it out with the current model you are using? Amazing video by the way taught me all the stuff needed to bootstrap a fullstack Rust project, also thanks a ton for the bonus tailwindcss integration part haha
Yes, actix!
😎
15:00 honestly sounds a lot like some of the benefits u get from graphql but for a little less verbosity (depending on which gql libraries/frameworks u use)
Great stuff!
But `unwrap_or_else(|err| panic!(...))` looks a bit silly to me, considering the behavior of unwrap is to panic with the error message (and expect(...) to panic with a custom error message).
supabase and featherjs offer similar front/back integration for ease of dev…
thanks for making this vid!!
ohh nice thanks for letting me know, I haven't used either of those - maybe I should check them out. Thanks for watching, glad you liked it!
Would be cool, if the LLM itself is streaming, to use a streaming HTTP call, updating the bot's message every time the client receives a chunk of the message. This is also what ChatGPT does-it's not just fancy effects! That way you could even halt the response while it's still running mid sentence as well.
Funny you mention this - after the video I actually implemented what you’re describing using web sockets. It’s in the repository if you’d like to have a look 😎
@@codetothemoon Yeah, that's cool! Though I specifially meant with Server-Sent Events since it's only a one-way streaming connection that is needed, and so there would not be a need for change of protocol nor the overhead of the WebSocket connection on the client. Though, WebSockets works too!
yeah I can't remember why I went with WebSockets instead of Server-sent events. I think maybe to keep the conversation context handy for the next prompt from the user? which you could do without WebSockets but there wouldn't be a guarantee the next request will go to the same host. not sure.@@dealloc
What is your neovim plugin for the file manager? I've looked over to your github but I didn't find your dotfiles. Great video btw!
Ah nvm you are using emacs
yep doom emacs! of which I'm a big fan
F#/websharper has this function call inside of api call. i tought this is coolest thing I ever seen but, in time this abstraction is to much for mind. I prefer standard http api call with http client(fetch,axios etc). Also last few month using htmx with maud(server side html generator :P) make things easier and more hypermedia style.
While there are robust web frameworks like React or Angular available for frontend development, it's worth considering Rust only for backend tasks to ensure high performance.
I would like a Tailwind video! :D
nice thanks for the feedback! will try to make one...
@@codetothemoon nice!! It would be cool to see it integrated with Leptos. Thanks for all your work and for taking the time to answer me :))))
you got a sub
nice! very happy to have you onboard!
I've got most of the leptos components working great
but for some reason I can't seem to get Rustformers LLM tools to work with the LLM; I've snagged the same Vicuna model you linked.
I've git cloned the entire repository, and I've tried targeting a single node from it, and I've tried targeting the directory that all the nodes live in
Should I be zipping up the repo into a single file or what is the best way to connect the Rustformers/LLM tool to my actual LLM?
what do you mean by "targeting" in this context? Make sure to specify the path to the language model in the `.env` file in the root of the repository (by default it's set to the path on my system). More broadly, make sure you've followed all of the steps in the readme (I just updated it a bit yesterday) github.com/Me163/rusty_llama/blob/main/README.md
Qwik has a pattern like Leptos's server APIs!
Wow. Thanks.
glad you got something out of it!
The Leptos Server Functions pattern is effectively TRPC
yeah I took a look quick look at TRPC - at first glance it appeared to be similar but not quite as concise as clean as what Leptos has. could be completely wrong about that - maybe I just need to do a deeper dive.
Nice! what theme you using in Emacs?
thanks! doom-monokai-pro
Great video! What window manager do you use?
thank you! I use Amethyst - I think it's fantastic, but it's also the only window manager I've really used for more than a trivial amount of time 😎
@@codetothemoon what is the code editor? thanks
@@guisoliven8022 doom emacs 😎
Danke!
thank you so much for the support!!! 🙏
Some checkpoints in the video would help.
you seem to be a lot into rust and so am i, however i've never used lisp much and i presume you did since you are a emacs user.
what comparison of the two would you make in term of workflow ?
why did you use the loop inj 37:11?
Great
thank you, glad you got something out of it!
It was really a great learning@@codetothemoon
Quick question, if I wanted to enable CUDA support where would I do that (just kind of a general direction thing). I went to the docs for Rustformers and *attempted* to try their implementation, however that didn't seem to do anything. This may just be a matter of "git-gud" for doc reading.
I actually had the same question, but in my case pertaining to metal support. I found instructions on how to enable it for the Rustformers command line tool, so my next step was going to be looking at the code for that to see what it's doing under the hood. Wasn't able to get there prior to making this video unfortunately :/
Good timing! I've just added CUDA/OpenCL support yesterday. Yeah, you're right, our docs on how to use GPU acceleration could use some work. I'll admit, documentation isn't really my strong suit.
@@lukaskreussel669 nice, thanks Lukas! Really appreciate your efforts!
correct me if im wrong, but i think you can call the backend functions in a similar way in the T3 stack can you not? Also really great video.
Trpc which is what t3 uses for that l, is about as close to serverfn as your gonna get as far as I know when doing client and server in typescript
never tried T3 myself, been hearing great things about it though. I may have to give it a spin!
I also had to install "npm install -g sass" to make "cargo leptos build" work
oh interesting thanks for pointing this out!
can you do a video that, which areas rust is helpfull in using in react app like that.
you're asking what the advantages of using Rust on the frontend are? Type safety guarantees (because TypeScript is a superset of JavaScript, it can't really do such a thing), no garbage collection which yields better performance for certain use cases. Language isomorphism for projects where the backend is also in Rust (such as this one). Basically any reason you'd prefer using Rust over some compile-to-js language on the backend would likely apply to the frontend as well.
A link to your dotfiles please :) specifically vim, that's some awesome status line!
it's actually doom emacs! theme is doom-monokai-pro. other than that, everything else is pretty much what you get out of the box.
@@codetothemoon I haven't noticed the window title :facepalm: Thanks :) Great content (regardless of the inferior editor preference).
i think the closest thing i can think of to this in the JS world is NextJS server actions which is still beta
nice, I may have to check those out!
Very easy to understand! I still didn't understand the whole node_ref thing. Why is that necessary?
Noderefs allow you to pass a reference to a specific dom node around to say a function or closure, that way the closure can make changes and do things to the dom node and you don’t have to try to inline the statements as part of the dom
Interesting tutorial! I will definitely give it a try. I have a question. Is it also possible to define your own applications, similar to OpenAi? Example: Can I link a specifications document (.pdf/.text file) and then make requests for it? And then the bot gives me an answer in the context of the document.
thanks! what you're describing is absolutely possible, and there are a couple approaches you can take
1. If you're using a language model with a large context window, you might be able to simply prefix your prompt with the entire contents of the relevant documents
2. Implement RAG (Retrieval augmented generaton) yourself - where semantic search is done on the documents in question and the pieces deemed most relevant to the prompt are extracted and included in the prompt to the language model. There are databases that provide semantic search capability like Pinecone and Nuclia that can help with this
trpc implment a similar pattern for api requests
15:13 you might be looking for trpc? It kind of accomplishes the same thing.
tRPC does the typesafe server side api thing you want.
thanks - I took a look at tRPC and it looked close but not quite as clean as what Leptos gives you. I didn't spend too much time investigating though - maybe I just need to dig a little deeper
I cannot compile the code on Windows 11 - both repository codes from the "video" branch and my hand-written variant gives the following error: error: linking with `link.exe` failed: exit code: 1120
I think the project is a bit high level for me for now, will need to learn th rust syntax and come back.
N00b question: If I wanted to integrate such a chatbot into a website, what would be the best approach?😮
It really depends on the nature of the website that you're looking to integrate it into. You could use this project as a starting point and built the rest of the application around it, or you could use this project as an example for building something similar into whatever tech stack your website currently uses.
How to get a Tailwind CSS, for some reason it doesn't work on my end. And how to swap over Axum from Actix? Thank you
check out this example if you haven't already: github.com/leptos-rs/leptos/tree/main/examples/tailwind
re: axum, I'm actually not sure as I've never actually done this. maybe check the Leptos docs and if it's not there, ask on their Discord server - Greg is very active on there answering questions!
Idk if im doing something massively wrong but inference doesn't work for me. I think cpu bottleneck as the request doesn't complete at all. i have an older ryzen 5 but can i not use gpu for inference? i have a 3060ti.
I think there is a way to enable CUDA for inference (or Metal) but I hadn't looked into that yet prior to creating this video. Maybe hop on the Rustformers/llm Discord, I'm pretty sure CUDA is supported, and I'd think the 3060ti should be plenty to run Vicuna 7B
Are you using leptos with stable or nightly rust ?
just took a quick look and it actually looks like I used nightly when I was making this video - that was a mistake, I had originally intended to use stable. Up until very recently, I believe Leptos required you to use nightly.
However, the code in the GitHub repository does use stable, so take a look at that if you want to avoid nightly.
@@codetothemoon If you read the leptos readme, it is explained that nightly permit to use a simpler syntax. But nightly should not be used in a long term project.
Thank you for your very instructive video. By curiosity, I have tried to generate it on Windows, but Visual studio 17 compiler fail with the C code of Rust former llm. I will try on my Linux computer or with WSL.
what code editor do you use. not like neovim, thanks
doom emacs!
@@codetothemoon thanks
It's possible to use this lib to load personal documents and query them?
this is achievable but not without a bit of effort. the biggest obstacle there is the limited input size of the models, so in most cases you can't just prefix your prompts with the entirety of the documents. Many work around this by doing a semantic search on the document base to prune out the parts most relevant to the prompt, then prepend those excerpts to the prompt that is given to the language model. The problem with this approach is that there isn't a guarantee that the semantic search will surface all of the context necessary to accurately respond to the prompt.
Now create_action needs to take a function which returns a future
Why not liveview? And yes rust and elixir have liveview that can do SPA without JS(only for websocket) render everything at the the backend
I'd always been curious about this approach, and maybe it would be a better fit. it seems like it's well supported in the Elixir world - what do you think the best approach in Rust land would be? This crate? github.com/lunatic-solutions/submillisecond-live-view
I just do a insta like when I see a video about rust with more than 10min.
Is it possible to train this LLM to use some own database/documents?
Nice content! thks for share
this is kind of something I'm actively researching. Fine tuning using your own documents is an option, but I don't believe Rustformers supports this so you'd have to use a different library and I'm not sure how effective it is. The other option is the "few-shot" approach, where you provide your documents inline with your prompts - this is an easier and more effective approach as long as your documents fit in the context length of the model, but that can very quickly become limiting. To overcome the context length limitation, another option is to put your documents in a vector database and use semantic search to find the parts most relevant for a particular prompt, then feed just those pieces into the prompt that goes to the LLM. I think this is currently the most common pattern for this use case - check out something like NucliaDB (open source), Pinecone (not open source), or Weaviate (open source)
Guys, why am I getting error: linking with cc failed exit status 1 when I run cargo leptos watch...
I prefer warp and Elm (elm-ui)
oo which font do you use!?
Monaco
@@780Chris thanks :D
How about dioxus?
it's extremely high on my list of stuff to check out!
смотрю через перевод нейросетью, классно
What font is that?
I think it was likely Fira Code, if not it would be Monaco. I made the switch from the latter to the former around the time of this video
i'm a big fan of svelte, so poor to hear it slower than vue and leptos???
I'm a huge fan of Svelte as well! Incredible framework. I took a cursory glance at krausest.github.io/js-framework-benchmark/2023/table_chrome_114.0.5735.90.html and I'd personally describe the performance difference between it, Leptos and Vue as negligible at the moment. I definitely wouldn't switch from Svelte to Leptos for the performance - it'd be for the ability to use Rust on the frontend.
Doom Emacs
much
use theme in video ?
doom-monokai-pro!
I hate my NixOS right now. I can't figure out how to get cargo-leptos to install. I'd probably already have it done with ubuntu or similar
I posted this. Right when I figured out the build. Hallelujah!
nice glad you got it sorted out!
@@codetothemoon gotta love a 45 minutes video turns into a 4 hour setup 😅. It's all good though. At least I learned how to get past it for next time.
@@codetothemoon Update. I never got it to work. I'm having a can't find cc error and it's a bit of a pain to figure out on NixOS to the point I needed to stop.
I was getting the model loaded and everything on the build I did. Unfortunately I couldn't get my version or your version. Working.
Yours less than mine.
Thanks for the video anyway. I'll have to come back later and try again sometime.
@@Script_Alchemist can you paste the exact error you're getting? or hop on the Discord and I can try to help there! discord.gg/2G35Sp6kpQ
i think trpc is similar but not completely
yeah that was my impression as well - it seems like it aims to automatically manage the contract between the client and server, but just a quick glance at the examples led me to believe that it wasn't quite as clean as what Leptos offers. I may need to do a deeper dive to see whether that is actually true...
Is rust good to learn vs go
Really depends heavily on both what you enjoy working on and your personal taste in programming languages
error occurred: Failed to find tool. Is `gcc.exe` installed?
yikes - I don't have any experience building this on Windows, is that what you're on?
@@codetothemoon Yes
holy, spent like 3 hours trying to figure out why mine didnt compile. turns out future_utils is not futures_util. *eye roll* there should be a law about naming your libraries too close to another. lol
7:24 I think naming things is nobody’s forte 😅
hah thanks, glad I'm not the only one! 😎
I don't think you're going to have any FE perf issues for what your building haha "
An input field, button and some message box's
The bottleneck is the language model inference, which uses a non trivial amount of compute power. By “relatively fast” on the MBP, I’m talking about 10 full seconds or so
@@codetothemoon The LLM for sure! I just meant when you where talking about different FE framework pros / cons 🚀
Yeah this is a great video, but I also don't see the relevance of showing that FE performance chart here, That said, the video does go on to explain the other advantages to using a rust-based FE framework, including sharing type definitions across FE/BE, and that cool leptos feature where the FE can automatically (via macros, I think?) invoke a BE function to trigger an HTTP request.
@@mikejamesbelanger the main reason is that until recently, Rust frontend frameworks had performance profiles that were inferior to most of the popular JavaScript frameworks, so it was often cited as a good reason not to use Rust on the frontend. With these numbers it seems like that can't really be used as a reason anymore.
@@codetothemoon Oh I see. Fair enough.
nice video, but for me I'd like to keep the BE and FE apart, feels unnatural to write FE using rust but amazing video nonetheless and thank you!
thanks! hopefully it offered a new perspective even if it didn't turn out to be your thing!
Another opinion people (like myself) have is that it'll never be too early to see all of the JavaScript mess, language and tooling both, burn in hell 😂 so doing frontend using other languages (and Leptos especially thanks to its transparent function calling with client-server type guarantees) looks super-interesting 😀
Who designs a light mode app in 2023... Weird man... just weird.
lol, definitely one of my biggest regrets from this video. Feel free to submit a PR for dark mode - I'll approve 😎
@@codetothemoon hehe thx for the vid
Ain't nobody got 43 minutes to watch you type everything
Then don't watch.
You have the GitHub repo.
The most stereotypical CZcams comment. Pathetic.
@@elpolodiablo5486 i didn't
@@tylerweirtube i am sorry that i dont have 45 minutes to watch someone type every single command into the terminal. This video could be shortened into a 10 minutes.
Just a macbook pro 🥱
I realize Macbook Pro is pretty high end for a personal computer, but in this video I am speaking about it in stark contrast to powerful rack mounted servers with multiple GPUs that are typically used for LLM inference
@@codetothemoon I know, but it sounds funny. Btw, nice video!
Anybody else get this error?
error[E0063]: missing field `n_gqa` in initializer of `ModelParameters`
--> src/main.rs:62:36
|
62 | let model_parameters = llm::ModelParameters {
| ^^^^^^^^^^^^^^^^^^^^ missing `n_gqa`
cargo leptos watch fails with this issue:
Error: at `C:\Users\JDyla\.cargo
egistry\src\index.crates.io-6f17d22bba15001f\cargo-leptos-0.2.5\src\lib.rs:43:66`
Caused by:
0: at `C:\Users\JDyla\.cargo
egistry\src\index.crates.io-6f17d22bba15001f\cargo-leptos-0.2.5\src\config\mod.rs:52:74`
1: Error parsing line: '"C:\Users\JDyla\Downloads\Wizard-Vicuna-7B-Uncensored.ggmlv3.q4_K_S.bin"', error at line index: 4
I have tested multiple version of the ggml to remedy this but to no avail. Did anyone have any similar problems ?
Thanks!
thank you thank you so much for the support!