DeepSeek Coder AI 🐍 The Best Coding Model I've Tested? (Open-Source)
Vložit
- čas přidán 16. 11. 2023
- In this video, we use a new coding rubric to test a coding-specific LLM called DeepSeek Coder. It's an incredible model that is fine-tuned for coding tasks, and today, we're going to see if it's as good as they say.
Enjoy :)
Join My Newsletter for Regular AI Updates 👇🏼
www.matthewberman.com
Need AI Consulting? ✅
forwardfuture.ai/
Rent a GPU (MassedCompute) 🚀
bit.ly/matthew-berman-youtube
USE CODE "MatthewBerman" for 50% discount
My Links 🔗
👉🏻 Subscribe: / @matthew_berman
👉🏻 Twitter: / matthewberman
👉🏻 Discord: / discord
👉🏻 Patreon: / matthewberman
Media/Sponsorship Inquiries 📈
bit.ly/44TC45V
Links:
Coding LLM Leaderboard - bit.ly/49E7jFH
DeepSeek Coder - deepseekcoder.github.io/
Use RunPod - bit.ly/3OtbnQx
How To Install LLMs on Runpod - • Run ANY LLM Using Clou...
RunPod TextGen UI Template - bit.ly/3EqiQdl - Věda a technologie
What tests should I add to the new coding-specific rubric?
Can you add some regex generation, explanation and regex fixing? Make sure that the questions are unique!
Build an e-commerce store with one item, many, items with features (size, color, material), subscription business; secure back-end.
CLI based todo app.
C# code tests, C++ code tests, see if it can create the snake game in Unity Engine or Unreal Engine. I tried this with GPT 4 and it failed, it was able to create the code and tell me step by step how to set things up Unity, but it failed in being able to fix an error in the movement logic. It understood the problem that I described but was unable to come up with a solution.
React app, Regex, some coding challenge from a page, etc.
VS Code tutorial would be great!
There's a million of those
Let’s make it a million and one. I for one don’t find it obvious how to run a local (or in my case cloud) open source coding LLM and plug that into VS code copilot. This would be much better that the existing copilot since you can train on your own code base (and it’s already amazing)
@@ryanfranz6715 use pieces
VSCode tutorial please
Yes please!
Yes, please. A tutorial for LLM in VS Code would be great. How do we deploy them, local Vs cloud and how would a basic workflow look like? Thank you 😇
Totally would love a tutorial on setting it up with VSC.. the least amount of money I have to give to microsoft the better.
100th agree :)
Please make the tutorial, this looks like a ton of fun!
And if possible not only for VSCode but also for neovim. At least if there are neovim plugins for this open source models, please mention it. Cause I use a proprietary plugin for coding and would love to switch to the open source one.
I love it!! Snake works perfectly! The 7b model is also impressive! Reaching also pretty close to GPT4 for a 7b and its also capable of creating Snake Locally!
Crazy, right?
I appreciate these overviews, and this is great to see! I teach undergrad CS and before teaching full-time my work was in creative text generation, creative agents, and using AI to support creativity (glad I defended before GPT-2 dropped). I've been hoping to find coding models we could run on our local servers for (at least a subset of) students to eventually use, but don't have the time to keep up this the wild progress without videos like these. +1 for sure on VS Code usage, anything on deployment and practical workflow integration is fantastic. Great vid as always!
Oh, and in terms of some niche use cases for different kinds of code benchmarks, beyond generation, code reviews, checking if requirements are met or violated, and checking test case coverage have all been almost fully hallucinations in the limited tests I've done, so these are always appreciated (assuming the model is meant to support instruct)
Insane. What parameter model did you use for the test, you didn't say I think?
EDIT: If it was the 33B then I think it would be worth the hassle to test the other models or at least the 3B or 7B because that's I think what pretty consumer level hardware can run pretty easily
This was one of the better code testing processes I've seen you do on model's. I don't expect most of them to get things on the first try but after some encouragement.
It would be great to see creative and effective ways to use open source llm models to build useful applications.
For formatting the !=, i think the term is ligature and different fonts can include them.
Ligatures are special characters in a font that combine two (or more) troublesome characters into one. For instance, in serifed text faces, the lowercase f often collides with the lowercase i and l. To fix this, the fi and fl are often combined into a single shape (what pros would call a glyph).
Awesome, thanks for letting me know.
Ah -- a ligature. Thanks for the info.
"troublesome characters" lool.. is that answer from AI haha....
funny af
Thanks for this deepseek video.
I have tested it and it really blows my mind.
A copilot video will be highly appreciated.
a local copilot would be wild if you can get that to work!
Really appreciate everything you do. Thank you!! Tip delivered!
You are amazing, one of the few channels that I have with all the notifications enabled. Of course, I would love that you make a video with the installation of local VSCode programming assistant. Thank you very much.
Really happy that you changed the way you test the coding models, it really hurt me to see you only ask one very basic coding question that every model was getting right and then a super complex question that no model was getting right. With this new format the improvements are more visible. Also would love to see a way to use this as an AI code completion like copilot. (Maybe find a way to use it in Visual Studio as well as Visual Studio code, I've found some projects for VSC code but none for VS :( )
Finally... this is awesome. Thank you for reviewing this model.
Go for it Love t see the Visual Studio off line integration, thank you Matthew
Thank you for doing that multi-turn error testing, that's what all coding LLMs testing should go into!
I love the direction this is going 🙂
Thanks for sharing 🙂
The score-results vs GPT-3.5 and GTP4: I'm guessing all the scores are for un-quantized models? It would be interesting to see how the scores were for all different quantized versions of the open models too.
My favorite moment by the end of the week when I discover new impressive project with your video 😊.
Lovit
Awesome video, please make a video about using an open source llm as a copilot on visual studio code. That sounds very interesting.
Thank you for another interesting video. Please do make a video that implements a local copilot. I use Olama, Litellm, Conda, and VScode on my M1 Mac Mini. I've also installed Autogen and Langchain with this setup. So far, my only problem has been using Docker to run code from Autogen, but it runs fine on the CPU. IMO, you are working on the most important projects for helping the community march towards AGI.
Amazing. Super duper keen to see how to integrate this as a copilot substitute in VS code!
Hey Matthew, Really enjoy your videos. Just a question on the code challenge section. If you change the engineer prompt to "Please find the issues with this code and explain in detail: ". You will find the outcome to be more inline with expectations.
Awesome, thanks for sharing this.
As a programmer with 20+ years of experience I can say that now I'm starting to become impressed. The code (in this case quick sort) identification element is extremely valuable on its own. Finally an local AI model has as much smarts as a high schooler.
Running this now on an RTX4090. It works but damn it's slow. So "local" may not be the right word yet.
@@ArianeQube Do you know what the bottleneck is? Are you sure it is your gpu?
As a programmer with 16+ years of experience I can say that you sound like you have been sleeping if you think it wasn't interesting before this.
@@foobars3816 As an anime watcher with 0 years of coding experience, I believe he was referring to the code, not artificial intelligence.
It's true that Machine Learning is impressive and it's been achieving amazing feats, but the code it was generating by itself meant nothing, at least not until now that it's making some code that can actually be used.
As a programmer with less than 1 year of experience i can say that this is not yet interesting for me.
Let's get that tutorial, save us some time haha. Love your work Matthew, thanks for your contributions to the AI space.
You mentioned Phind and I wanna make sure you know that while Phind did release an open model Phind/Phind-CodeLlama-34B-v2 but later iterations have all been proprietary and behind a commercial cloud offering. So when people say "phind is the best" they generally aren't talking about Phind/Phind-CodeLlama-34B-v2 rather they are taking about the proprietary commercial offering.
That's really good to know, thanks for sharing. Is there a significant difference between v2 and the subsequent closed-source models?
I asked Phind who it is... and I was told that there is GPT 3.5 under the hood, modified / optimized by the company for coding
@@matthew_berman it's known to be significantly better, but we don't know how they built it
YES!!!!! VS Tutorial for the win!!!!!
When running the aider benchmark tests it also scored very well
This is the ONLY model aside from OAI 3.5 and 4 that will actually write good code, python, js, bash, or whatever I have tried. Great Video!
Very cool :) Can you make a few videos about AI code assistants for C++ ?
Definitely want to see how you set this up. I wonder if you can ask it to increase the size of the tiles in Snake game without breaking? Aider couldn't do that.
The ≠ formatting is a rendering option. VS code supports it. Under the hood it's still the same text.
I love your code model testing!!
Super cool! +1 for a tutorial on the vscode integration, is possible to use a local LLM for coding without the cut and paste in the files?
I'm sure it's possible, especially with the VS code integration.
@@matthew_berman If you could show us how to have a coding model understand an entire relatively large code base at the same time it would be the biggest leap in AI since chatGPT
I'm going to test this model with the continue vscode plugin. It should work and prevent some copy and paste. I'd love to see other options though. So +1 for a video
I have a suspicion that the snake game was in their training dataset, that's why it always nails it.
very likely.... most of the code gen llms so far have been disappointing... feels like code pairing with a junior fresh out of the uni...
id like to see it actually build a proper lambda script that pulls something ... transforms something and serves it... in an optimal manner...
or even better... a snake game but over telnet.... 14.4k style
It might very well be in the training dataset... but it passed almost every test thrown at it... even the "what does this code do" test... which I found to be very impressive. As these models get better and better we need to expand the testing to see if it can handle variations. But... this is so impressive. I don't recall any of the other models that Matthew has tested as doing this well before (even if we take snake out of the equation).
Good stuff as per usual. This one hits all my needs... So let's get this.into autogen and see what kind of damage we can do.
Thanks for this powerful information
Very good, Sir. Thanks for the video.
While it is impressive, it might not mean much. LLM CZcamsrs have been using the same tests to evaluate models for the past six months. If I were a model creator, to get some hype, I'd make sure that the model I'm training passes those "tests."
I would be really interested in seeing a video on how to set up this model in VSCode!
Vscode tutorial of course - but would like to see the code suggestion/completion feature as well in it
Wild! Loved it!
A tutorial for LLMs in VS Code would be super awesome!!
Awesome!
I want this as an API.
Yes please to the VS code open source code pilot. Folks working behind a corporate firewall would be rescued by that!!! 🙏
Love this!
9:38 - it's just a 'font ligature' feature. working with a special font and modern editors
Yes, please. A tutorial for LLM in VS Code would be great.
So what's next on the game test front? Breakout? Space Invaders?
🤔
Bro how you said that the background is ugly lol😅😂😂 which is true.
And that’s the thing I like about you is that you are true to the core, brutally honest. Who can agree
Interesting that it used the unittest framework which is pretty much secondary to pytest these days. I would ask it specifically to use pytest. I had this problem all the time when generating python code with tests. It’s also very interesting to generate test fixtures as well with parameters to make really good unit tests. Pytest is awesome.
Do that vs studio guide. It sounds great, I will definitely watch it.
Yes please! Can we please have a tutorial for DS integration in VS or any other IDE. Thanks!
this is AWESOME!
Isn't it great to be astonished almost on a daily basis by the progression of AI ?
I will have to get this going sometime this weekend to see if it can help me finish up my Inventory management app.
One question: Say I wanted to use an LLM to make a local AI for use by my company and website.
Can you give an overview of what is required as far as training etc..?
I would love to see this... ❤
I test this model to beat leetcode. It works like a beast, almost every coding challange is solved even for the hard problems. I guess we have a practical coding model already. Even better, it's open source.
Yeah, a tutorial for how to install an LLM as a copilot alternative would be fantastic.
It would be great if you could briefly discuss the licenses these adopt and the type of hardware you’d need depending on the parameter sizes.
I run the 6.7B on 3090 using ollama and it's super fast.
It's performance is almost indistinguishable from gpt3.5.
Please I would LOVE to see this tutorial in how to do It to run it on local. Thanks a lot for your great work and aim to share knowledge with everybody.
Would love copilot tutorial
I'd really like to see a video on how to combine an embedding model with something like this. For some reason, (using llama_index) whatever model I use to generate the index/embeddings, it never works together with the model that I want to use to retrieve from the vectorindex (sometimes I get shape/dimension errors, sometime other errors), it's really frustrating!
First open source LLM into awesome level
+1 for visual code intergration, would also like to know how it performs on C#, and if I can add the libraries and documentation and preferences what I normally use to this model like embed it. Similar like what you can do in a custom GPT
Dude, i can't believe you have to ask if we want to know how to hook this up to VS code. This would be next level
Could you test the models on strongly types languages instead of dynamically typed languages like Python and JavaScript ?
Very impressive !!!
Yes please do the local coder tutorial!
I think it'd be super interesting to see if we can get this to work with aider
it’s a slept on model. Competes very well against the GPT4 coders for sure
Hi Matt: I find your videos very informative. Is it possible that DeepSeek was pre-trained to ace the Snake game?
I am scientist not an experienced coder. I used DeepSeek locally and was able to get the Snake game to work. However, It failed miserably when I prompted it to create another application (e.g app to analyze stock performance). GPT4 gave me excellent code for the same and other tasks. You may need to refresh your rubric; what do you think?
Hi Matthew, fantastic stuff, please do make a tutorial for this DeepSeek Coder AI in VS Code.
That'd be great! A tutorial on using it with VS Code
Vs Code Tutorial would be really great
The gap between opensource LLMs and close ones is closing super fast. I'm wondering if there will be a day that we will get like an AGI which is based on an open source model. But with the pace and the "secret sauce" OpenAi is running, they are still keeping a large pace between them and the rest of the world.
Definately the best coder!
This year: Snake
Next year: Optimizing Crysis to run on a raspberry pi
When are you gonna release the llm finetune vid?
Yes I need to know how to run this model in vscode on local
We need a 70b coding model!!!! It would crush GPT-4
Tutorial +Pointer App, thank you
What's +pointer?
@@matthew_berman your highlighted mouse pointer
Your own free local copilot? Hell yeah, that would be interesting!
This is really handy.
Thanks!
Thank you!!
You’re doing the copilot thing right? :) just got a new laptop with 48gb of ram and I’d love a free , local copilot (especially if it is half decent at rust)
Edit: actually I did that myself with continue extension and ollama as host :)
hey thanks for the awesome video. where can i learn more about how such tools are developed. i'd like to discover how the sausage is made
hi, pls let us know how to setup the LLM in visual studio as a replacement to copilot. thanks.
Using it as a VsCode editor would be awesome!!!
Tried 7b locally, impressive!
Yes we need tutorial on vs code
When are we going to get LLMs for other roles beside coding? I want a product owner, business analyst, software architect, and QA.
I want to see an integration with visual studio :D
A tutorial will be highly appreciated, thanks
VS Code Copilot Tutorial would be helpful!
Did you test the base model or the "instruct" version?
Maybe it's good on basic, widely known code like snake. But it fails on something more complex which wasn't included in dataset I guess.
Hey Matthew... is there a model working for 3js yet?
VS STUIDO TUTORIAL WOULD BE GREAT.
CAN YOU DO A BREAK DOWN ON HOW TO BUILD SOMETHING COMPLEX LIKE THIS FOR SOMEONE WHOS KINDA OF GOT THE HANG OF IT BUT DOESNT KNOW EVERYTHING.
INSTEAD OF JUST A PY CODE WHEN THINGS GET COMPLEX AND LOG CAN WE MAKE IT START TO SUGGEST JS FILES AND CSS FILES AND GUI WINDOWS OR WHATEVER, ITS JUST AS IT DEVELOPS AND GETS TO FAR IT BECOMES ERRORS AFTER ERRRORS ALL IN ONE .PY FILE AND IM NOT SURE HOW TO TACKLE BIG COMPLEX TASKS LIKE THIS
I want to build apps and websites to make money but dont want to learn coding or pay someone else to do it, do you think ai will be able to do this in the near future?
The first model to create the Snake Game !