I Tried Every AI Coding Assistant
Vložit
- čas přidán 7. 05. 2024
- Which of the AI coding assistants will turn you into a 10x developer? ChatGPT? GitHub Copilot? Tabnine? AWS Code Whisperer? Bard? Sourcegraph Cody? CodiumAI?
Prepping for your frontend interviews? Use code "conner" for a discount on my product FrontendExpert:
www.frontendexpert.io/conner
🎬 TikTok: / connerardman
💼 LinkedIn: / connerardman
💻 Video/Coding Gear: www.amazon.com/shop/connerardman
Timestamps:
0:00 Tabnine
4:31 ChatGPT
5:37 GitHub Copilot
9:43 Bard
10:49 AWS Code Whisperer
13:51 Sourcegraph Cody
20:40 CodiumAI
Business/brands 👉 youtube@connerardman.com - Věda a technologie
Correction on Sourcegraph Cody:
I wasn't fully using this one correctly. You can download an additional app and link it to your GitHub repository, and it does a _much_ better job of understanding the repo. I'm not sure why the UX doesn't prompt you to do this from the start, but it does fix some of the problems in the video such as over-indexing on node modules. However with all that said, I honestly still wasn't able to get it to perform many tasks that were particularly useful, and the larger the repo I gave it, the less reliable the responses seemed to become. It could definitely be worth a shot in your projects to see if it is useful to you though! 😃
Also for the generating code comments, Cody has a built in commend to generate docs which should do a much better job :)
you didn't try Codeium though. it's different from CodiumAI
What is the "additional app"?
@@ivanolmo4109 There's a little icon in the bottom of the Cody chat that links to a download of a desktop app that asks you to link the repo, and it does a better job of indexing the code if you do.
@@ConnerArdman It's already different now. I did see a prompt about this after I installed Cody but was confused as to why it was telling me about a desktop app. I moved on and now I don't see any reference or links to the desktop app. There own website doesn't have any mention or link to it either. I also asked Cody to assist me with downloading it and it gave me a dead link.
FFS
UPDATE: ooohhhh... the desktop app isn't available for windows...... SILLY ME
I like that the video starts by jumping right into reviewing the extensions instead of a long intro that ends with a sponsored segment. You get points for that from me
I really despise that kind of CC's behavior.
Nothing wrong with that. Content creators need to make a living too. In the end, you're getting that content for free.
@@Homiloko2 That argument goes out the window when you can't block the ads.
@@FaastTex Well, that's a fair argument. My ads are still blocking fine so I even forget they exist
I genuinely thought something was wrong at first because I'm so used to the 1 minute intro for EVERY SINGLE VIDEO regardless of subject matter
The examples he uses (factorial, mergesort) are trivial with simple, well known solutions implemented in almost all languages. No one writes their own mergesort or factorial function.
I'd like to see a comparison of these various tools using something more realistic. For example how well would they implement envelope encryption using various cloud providers? I hope they would produce code that is at least a good starting point for envelope encryption. If I get some time maybe I'll do my own trial using these various tools.
What would be some other realistic examples to test these tools with?
bump
These are all terrible at real-world coding. Your best bet is probably GPT-4 since it is much better than the others at having a back and forth conversation. You can kind of reason it into giving you correct code or potential errors sometimes. Still not good though.
imho coding assistants may be more useful working on trivial yet time consuming coding tasks like generating html from a visual description or refactoring existing code. if your prompts end up longer than the assistants code output you're probably not using your assistant well
Personally, I use copilot for work and I never rely on it for new implementations. I only use it as a refactoring/convenience tool. Works great for that. Can speed up my workflow by several orders of magnitude. I might use chat-GPT as a starting point for something I have not done before -- as a smarter google search -- but I wouldn't trust it either. AI is there to augment, not replace, IMO.
@@renynzea - That sounds good, but speeding up your work by two orders of magnitude means you would complete an entire day of coding in 5 minutes.
Don't get carried away.
I literally haven’t seen your channel before but I gotta comment about how much I appreciate that your video just starts. There isn’t 3-5 minutes of just bs when the topic is literally explained in the title. THANK YOU!
like a bad anime lol
We need a comparison with more non-standard functions/customized requests.
Factorial is too likely to be an example in the training data of the language models.
To determine these tools' capabilities and usefulness for actual work we'd need some tests to get code that solves a problem the tool most likely hasn't seen before, or atleast not with the same constraints or not with the same custom properties (like sort this array of strings alphabetically and add an argument to select whether to treat uppercase and lowercase the same or have all lowercase letters come after all uppercase letters like ASCII code based sorting).
I agree. One place I've noticed Co-Pilot really shines is in Makefiles. If your targets are well named, and use wildcards, it's able to deduce the idea reasonably well. Overall, I've noticed these tools suffer the same issues as humans. It will only provide bulk prompts in atomic segments, but struggles to isolate sub-segments of these chunks to interleave a modification to existing code. Almost like it was trained to copy and paste from stack overflow...
exactly, that problem was in the domain of llama with 7b of parameters.
why are u trying to use these tools for tasks that aren't common in the training data?
The whole point is to get a good aswer that is not in the training data. The zero shot capabilities.@@derschutz4737
@@derschutz4737 Because you are always confronted with non-standard tasks in programming. Most of the time there is already an existing system that you need to work on, which dictates constraints that you need to account for, and that's where the main time sink in implementing a big project is.
Excellent video. What I love about your presentation is the speed. You speak at a great speed that gets to the point, and gives us exactly the amount that we need without any fluff. So many times when I watch other people's videos they go slowly and I just have to speed up the video to 1.5 speed at least to get through it. This is my first video of your I have watched and I am really glad I did. Thanks.
Thanks, glad you liked it!
This comparison would have been more useful (perhaps) if the selection of tasks posed to each agent were non-trivial, novel, non-encapsulated tasks. Ie: something you couldn't just google "how do I code " type questions to find a definitive answer.
I love how this is so straightforward, thank you
I really like the head-on approach you took on this. I found it easy to follow your comparisons as you went and I appreciate your analysis. I'm doing my own research and found your video quite helpful!
Im only about 8 months into my psuedo coding journey (im in school for InfoSec, and programming takes a bit of a back seat as its not necessary to build a program, only to understand how said code functions so we can break it) but co pilot as been an absolute godsend. Not to code for me, but with lack of any kind of instructor to assist me, its sorta like a mini teacher. I dont let it do it for me, rather help me understand what i did wrong.
I wish I had ChatGPT in school! Sometimes grasping concepts or just needing that extra little bit of context goes a long way in biology. Now I use it for coding but it would've been just as useful back then.
Super interresting comparison. Thanks Conner for this actual topic 🙏
The list is missing Codeium, which I use a lot and is pretty spot-on more often than not in my opinion. Wanted to see how it would rank against Copilot and similarly named CodiumAI
Second this. Absolutely love Codeium. It's free and good compared to Copilot that was behind queue wall, then pay wall, not sure about its current state but idc
Same
Codium is great
Isn't Codium in the video? 20:40
@@BrianSamThomas Codium is. Codeium isn't
Also, Cursor IDE and their coding assistant could be mentioned. It works pretty well for me :)
Cursor IDE should be added, I agree.
Totaly agree, using ChatGPT and bard out of the box isnt fair compared to the others, should absolutely included Cursor
This was a nice comparison. Although the test cases were too simplistic, TBH. E.g., you could have asked to complete code to parse a snippet from a text file given within a preceeding comment. Stuff which isn't literally found in text books ...
you did a fantastic job in this comparison , thanks a lot for the video
22:36 - What’s interesting is that I think most compilers that perform tail recursion optimization do the same thing (i.e. simply convert them into a loop).
thank you for the video!
literally was searching for something like this
My company is probably going to go with tabnine (starting to test it now), mainly because their models can be deployed on-prem. Also because they're modular, one model per language, and they say they'll update them whenever new versions of languages / big frameworks come out.
It's also supposed to have some degree of repo awareness (they told us they stuff the context with the content of files that seem related to the current one), and full repo awareness is on their roadmap (if it does work, it should be a big plus compared to generalist LLMs).
That said, its reasoning capabilities are definitely not great right now.
How much does it cost? I'm playing with Github Copilot. It is pretty good at fixing bugs and explaining what the code does, etc.
We're also using tabnine and chatgpt. I like tabnine's ability to code ahead of me. It doesn't, usually, get in my way and is a helpful assistant. I like its speed or should I say rhythm.
@@dak2009 nice
so, do your company using tabnine now?
there's also codeium to , really great they were even featured on a changelog podcast
These AI tools need better names lol, Codium and Codeium cannot both exist 😂
@@ConnerArdmanclassic naming dilemmas, the only way I was initially able to tell both of them apart initially is that one only did tests and the other one was trying to be copilot (it's actually good)
Codeium really doesn't get talked about enough... It's recent updates makes it beyond fire🔥🔥🔥
i uninstalled tabnine once codeium came out. Love it until it’s not free anymore 😂
IntelliJ and JetBrains among many others were built on top of Codeium ;)
Cody works really well in projects since its context comes from your entire project and not the single file you're in. It's also doing this locally if I remember correctly by creating it's models in PostgreSQL.
I'm not sure if the others do this but 'How can I get XYZ from ABC model" has workers really well, or Create Test for this, will often bring in correct model relationships and factories in Laravel.
lol, Just read your pinned comment 😬
As many other comments have pointed out already, you used very standard functions every computer science student knows about (factorial, mergesort) for testing the AI assistants. The problem is, these functions are most likely occurring many times in all of the models' training data - and obviously computers can memorize and parrot smart-sounding answers / code. If you do a round 2, maybe you could consider tasks that are not really that hard at all, but a little off the beaten path, you can't just google the solution which also means it probably is not in the training data
This was awesome! Thank you for doing this!
from wikipedia : 170 is the largest integer for which its factorial can be stored in IEEE 754 double-precision floating-point format. This is probably why it is also the largest factorial that Google's built-in calculator will calculate, returning the answer as 170! = 7.25741562 × 10306.
170! = 7.25741562 * 10^306
in a way this comparison reminds me of benchmarks
you feed it some example tasks but what i never see in videos like this is the MEAT
Like for example, write me some code that aggregates data from x, y, z web API sources and combine them in an application for accounting, and by accounting i mean the client has some god damn custom requirements, also add the deployment pan, the human readable guide docs, have meetings with the managers, etc. The actual part of working.
Yes I understand developers only want to sit at their PC and code but at least 30% of your time is not spent working
Incredibly helpful video! Didn't even know about Codium AI, just started using it and it's performing really well in test generation 💪
Nice Video man.
But you missed these AI Coding tools 🛠️.
1.Cursors IDE - Use whole codebase knowledge.
2.Codeium free- Free version of Codeium like Copilot.
3.Phind AI - Extension from Phind AI in VsCode.
Good Additions!
Phind is, in my experience more precise, while Codeium often tends to trial and error and keeps looping its suggestions.
Where Phind totally shines of all the plugins is fining configuration and settings for eg temux etc. It can save you hours of searching, and you don't have to read meaningless comments on stack* sites etc.
what abt blackbox
Or IBM’s watsonx
in my experience (and I code primarily in R) chatGPT 3.5 gives code that doesn't really work, straight up makes up packages/functions (phantom memories) to solve the issue, and in general is only able to help me 1 out of 10 times
5:55 It suggested "factorialize" because it's a verb-ized form of the word "factor," since you're writing a function. If you were writing some property, the noun form would be preferred.
But factorialize is a really good word
Thank you so much for starting the video right away u beautiful human
this man explains so well, great video sir, great audio quality as well
Thanks!
great video, very informative, and I especially love how you get straight to the point, and kept it engaging throughout the whole video :)
however, there's this boxy sound from your microphone that makes me dizzy, and I used chatgpt to help me explain how to fix it lol:
1. Cut 250Hz Range: Start by applying a parametric EQ and reduce the 250Hz frequency range by about 4dB. This should help remove the boxiness.
2. Boost 3.5kHz Range: Next, boost the 3.5kHz frequency range by about 3dB. This should add more clarity.
3. Adjust Resonance: Play with the resonance (Q, bandwidth) settings to achieve a smoother curve between the 250Hz and 3.5kHz points on your EQ visualizer. Aim for a gradual transition, creating an almost straight line through the 1kHz area.
4. Fine-Tune the 1kHz Area: Slowly cut a notch in the 1kHz area to refine the sound and make it sound more natural. This can help reduce any unwanted "wispy-ness" (very technical term).
Optional: Boost your volume by about 1-2dB if needed.
Thanks! And yeah, the issue isn’t the mic or the EQ. The audio was peaking when I recorded, and I did my best to save it in post because I _really_ didn’t want to have to rerecord this one 😅
For more complicated functions you need more detailed prompts. At the end of the day, it just saves you some typing time given enough context.
great video. Would love to see your evaluation of Replit Ghostwriter
Amazing video!
As this is a fast-paced technology, curious to see how that comparison would be in 1/3/6 months time.
Commenting on how the video immediately starts is amazing
your vid is awesome , so what are the Ai editors free to use that are like github co-pilot
My tests playing with Sourcegraph Cody, there is a step you need to do that creates embeddings for a given repo before asking things about said repo.
Yeah checked the pinned comment 👍
Awesome!, we need a sequel
Now you need to test JetBrains AI coding assistant and Google Duet AI coding assistant (both released since this video).
Jetbrains ai is using chatgpt behind the scenes and they hide that they use chatgpt, so there is no jetbrains ai
@@Sp1tfire100The model they use is less interesting than how well it integrates with the IDE. The model can often be changed.
We need more of theses
I use the same ones you are using except the documenting one (probably will at some point).
On rare occassions, I go with Bard.
I had an interesting case where ChatGPT and Bard disagreed both with Copilot about efficiency of a JS loop. Copilot was going with a more readable code and claimed it is also more efficient, but after running tests, I found out it is about %5 less efficient and personally did not find it readable.
I hear you on the chatgpt thing. You can tell it to cite its sources in its response and it will do it. Sometimes itll reply saying it did not get it from a source and it cant find a source. If it’s something super important then you may not want to use it if there is no source.
Specify .gov and .edu it will find files it shouldnt i discovered that one while writing my history paper on ww2
I absolutely love Copilot since it came out, and in my opinion the only AI that's worth paying for.
With that said, CodiumAI looks absolutely amazing and useful. Testing is tedius and optimizing is tedius.
If you are student, you can have it for free.
Hello. Really good coverage on the AI coding tools. Can you try the AI assistant of JetBrains please?
I am using Assistant AI that is in preview for JetBrains editors, and it looks to be similar to ChatGPT but it is built-in to the editor.
I've been playing with the JetBrains AI (PyCharm) and found it to be quite useful for boilerplate, or those times you're too lazy/busy to manually generate a small utility. Liked the way you can ask successive questions/prompts and get the code enhanced.
Is it just me or is the JetBrains Assistant kinda sassy when it corrects you?
i can build something like copilot inserts, in fact i did similar. and for my own stuff i have hundreds of ideas on how to make it better. problem: if you want it smarter, it will cost more. because all the mechanisms to make it smarter cost tokens and time. but models rapidly get cheaper, better and faster. so in a year from now, you have a mind blowingly smart assistant
This really helped me get started on a new project, and make everything just so much easier as a learning developer
right
nice channel and content. i like how you direct ur videos!!
tabnine deserved better. i find it works well for code completion. i use a simple vscode-openai with custom ai key to have a chat, and i actually like the disconnect between code and chat.
so u chat with a bot?
In the first factorial example, another thing you and AI missed is that the 'n
for cody in the first question it suggest the
copilot falls by a cathegory when you actually use it on exhaustive in house projects. Sure it know well how to do a factorialize function but with custom code you cannot do much. It works well if it can find something similar on the internet.
Best coding AI review I have seen. Really wish you had included the one I use, Codeium (different than CodiumAI).
No BS, straight to the point
You deserve my sub ❤
Extremely useful test! Thanks! 🙏
At 16:33 I would argue that your prompt "JSDoc comment" is why it took so long to get a response and why it wasn't what you were expecting. The prompt itself is too ambiguous. Would be interesting if you tried it again but provided more context in the prompt, like "JSDoc the factorial function"
Edit: it even says in the reaponse that there's too little context in the prompt 😅
Every intro to every CZcams video needs to be like this 😅😅😅 I wasn’t even ready 😂
11:11 I might be incorrect, but instead of setting the cursor on a blank line by itself, maybe you need to set it directly in front of the closing curly bracket without an empty line in betweet to avoid the empty line when working with AWS?
Dude, factorial is a traditional thing. Are those good for a totally new stuff like Svelete, Qwik or stuff?
codium ai made a mistake at the end of the video
-it throws an error if the number is less or equal to 0
- after that it checks if the number equals 0 to return 1
the first condition must be changed to strictly under 0
Great video overall thanks for your effort
Yeah nice catch, I missed that while filming 😅 Although the test suite handles this properly, so it would at least catch its own error in this case.
The day when I will be able to open Firefox repository or LibreOffice repository in my editor and having an AI assistant explaining and helping me learn about the repo, which saves me months of reading and looking. I'll say that AI assistant a thing. Right now, they just copy and paste code. I'm hoping for the big thing in the near future.🙏
great video... agree about chat gpt not being ideal for specific less than idea coding contexts...
I use Phind, I code in python and asked it to transform to js (wich I'm learning) or to help me correct my code, I really like the explanation it gaves me and "talking" with my code on vscode directly is a plus //it has a free plan wich is the one I'm using
phind is good but for me it too often confuses languages. just a few days ago, I asked it something about calculations with dates and random data in ABAP and it quoted sources about python… and if I can believe what it says on the website, it was using gpt4
yeah this is my AI coding stack along with the Codium PR agent and Cursor with an openai key
instantly clicked the like button after hearing you jump into things immediately without an intro. I don't even know if the video is good yet but you deserve a like for that.
*edit: yup the video's great lol. Subscribed
Awesome video thanks. I only wish you had also included Codium (different than codiumAI)
Great comparison. Thank you.
Hey Conner,
Do you mind sharing what software you used to record and add the camera effect (gradient border)?
Thanks
I just use QuickTime to record and Final Cut Pro / Motion for all the editing and effects
Very interesting video. Which is the best AI for Excel VBA?
Really helpful, thanks!!
Test cases... fine... they work. What happens when you throw a full size project at it and ask it to do a task more complex. Like getting data from UI and creating a model object and send it to the save / upload api? Or find a logic error?
My main problem with my own spaghetti code is that it's been harder to follow the logic and all the async events. I think for that we need an AI that could follow an execution thread. Or an AI to optimize memory usage. Every software out there seams to be bloated to the max. I made it my mission never to open a 'node_modules' folder again.
You absolutely need to try Cursor.
I was sent here by Matt Wolfe. Thank you for this great overview! Your precise comments per assistant help me come to a decision which assistant to try out in the future :-)
Awesome, glad you found it helpful!
Bard has the advantage of being able to access the internet. You can give him an API documentation and he'll make you a wrapper
thank you for your efforts
Straight into the content was a great choice!
You didn't even touch Phind, Deep Seek, Tabby which is as per with copilot, even surpasses it for me in some cases during some bash scripting which surprises me too. And for cody it doesn't have any idea if you ask it randomly, you have to register the repo either from local disk or from github with the cody app to give it access to your code files then it works like magic. Out of a blue opening a folder and a file in vscode and asking the extension it will fail every time because it doesn't perform the scanning. Only file discovery scanning that it does for any open folder will fallback to node modules like yours.
Pretty surprised they didn't know about Phind tbh
Nice review-thank you. Liked and subscribed.
The terminology "non-negative" if you want greater or equal to zero. (Zero is neither negative nor positive. Zero is even and not odd.)
For what it's worth: Zero is neither negative or positive for the case of integers, but can be negative or positive if it is floating point. From Bing Copilot: Caution: Representations allowing negative zero can lead to errors if developers overlook the differences between +0 and −0 in certain operations.
I use co-pilot and chat or talk with chat gpt about what I am coding and ask questions if I have one.
Works like a charm.
co-pilot can also suggest code based on other classes and it mimics your comment and code style previously used throughput the project.
Also a hilarious bit is learning OOP.
Writing the dummy classes with methods, that print animal behaviour.
It knows Penguins cannot fly, but dive and all kind of context specific behaviour.
it is damn impressive.
Amazing reviews!
At the pace this field is progressing you could update this video every month 😊
Straight to the point thanks
I wonder if one issue with the jsdoc comments there is your editor automatically inserting " * " for new lines; might confuse them (not that it should, but clearing that line I guess their suggestions would be more reasonable).
Calude also does a good job of writing code
Actually O(n^2) factorial is kinda correct, since the intermediate results get bigger and bigger, and require more and more time to calculate.
I would like to see you test out Llama3 70b model using Groq inference engine to get your thoughts for coding compared to Copilot especially.
Any videos on double in the future? Seems promising over these utilizing Claude 3
Copilot chat has a @workspace macro that does what Cody tried to do
The basic functions like pulling in a mergesort, or a factorial function doesn't really save the programmer any time. It's the difference between 10 seconds and 30 seconds to run over to a stackoverflow post or personal repository to grab one. The extra constraints are what will break it and cause real time to develop - it's that time reduction that makes these tools worth it.
Not to mention, then you have to still proofread the ai's output just in case, so you really don't save time there more than likely.
Tabnine test was a little unfair. You could posture that getting it to initially write some simple test then using the generate test function it would generate tests like the others you already have. Should have used that function with no other test examples to get a more fair comparison to Copilot
What a heavenly delight to start right out with relevant stuff. I didnt notice this annoyed me before. I hope this will become a thing
You should also try Gemini (the new bard with better models) and google's specific gemini-based code helper (their rival to copilot) called Code Assist
You went straight to the point so I went straight to the subscribe button
good comparation, useful, tanks
The merge function probably isn't written automatically as it could be expected by you that it should be written above the mergeSort function. So it just waits for you to place the cursor before it suggests
There is also phind and deepseek!
Codepilot not throwing an error is actually superior. Exceptions are expensive so properly handling error scenarios or error response is preferable (OPINION).
Awesome thank you !