CoPilot Review: My Thoughts After 6 Months
Vložit
- čas přidán 1. 06. 2024
- I've been using CoPilot for 6 months now, and I wanted to share my thoughts on it. It's a great coding tool, but there are some shortcomings.
Twitch
Everything is built live on twitch
Twitch : bit.ly/3xhFO3E
Discord: discord.gg/ThePrimeagen
Spotify DevHour: open.spotify.com/show/4MLaDrQ...
Editor
All my videos are edited by Flip. Give him a follow! / flipmediaprod He is also open to do more editing, so slide deeeeeeeeep into his dms.
Join this channel to get access to perks:
/ @theprimeagen
Links
Linode: linode.com/prime
/ discord
Twitch: / theprimeagen
Insta: / theprimeagen
Twitter: / theprimeagen
VimRC & i3: github.com/ThePrimeagen/.dotf...
Keyboard 15% off bit.ly/Prime360 USE CODE PRIME360
#coding #neovim #typescript #programming #vim #softwareengineering #codinglife #webdesign #webdevelopment #webdev #javascript #rustlang #rust #twitch #twitchstreamer #programmerhumor #codinghumor #software #softwareengineer #softwaredeveloper #softwaredevelopment - Věda a technologie
6 months.... IT HAS SCREWED ME MANY TIMES. I hope you like it :)
I started using co-pilot less than a month ago and I've already spent hours trying to work out what on earth was going on ... copilot wrote code.
So 100% with you, boilerplate code is freaking awesome, just gets the fluff out the way quickly so you can get down and dirty.
But don't let co-pilot code :D
thank you for that coverage. more videos like this prime🙌
and thanks for all ur videos/rants/advice so far....has been super helpful 🙂
As junior I'm tring to use it only to fill variables and write comments faster. When it suggests me whole bunch of code sometimes I read it or just ignore it. Like chat gpt it gives some ideas but I would not trust it enough to just copy paste.
Didn't you write tests for generated code? :)
I agree with you, copilot is great for boilerplate but in the end you need to know your game and be in controll of what is going on.
So basically copilot is a great excel corner-drag-fill-thing
Yup
Perfect analogy!
Perfect
Mr Paperclip’s grandson.
Realistically, we often aren't giving it enough information to make good judgments - it has to guess from code we're writing, what we really intend.
Comments do help in creating context.
And, of course... more powerful models are coming (though they'll likely cost more).
Working with these LLMs is like managing a team of brilliant high school interns. They're brilliant, but they're also teenagers. Manage them wisely and they'll multiply your abilities (at least within some narrow domain). Let them walk all over you and they'll write garbage faster than you can pick it up.
Can confirm. Those kids wrote my username
@@barbietripping 😂😂😂😂
Have the LLM write you a unit test first, then run the snippet it produces from the unit test. ;)
noooo, you can't manage teenagers
You are overestimating these tools. It is mostly like training a parrot and then asking him to write your code. Both Parrot and AI don't know what they are doing but they are pretty good at repeating things.
You know how the worst part of your job is combing through a junior dev’s PR to make sure they aren’t going to break everything if their code gets merged? Now with CoPilot you can have that experience writing your own code too!
I love that part of code reviews. Once I'm on a roll to find potential issues in a PR I don't hold back.
I once had a PR with 400 loc changes and I must have put a comment on every other line, sometimes multiple ones.
The best part is seeing them do something with that feedback.
More senior engineers tend to think they know better and cannot be steered to a better solution without a lot more effort.
Exactly. Just seems like you’re signing up to work with an idiot that is constantly trying to merge garbage code. No thanks.
Having spent a few months letting copilot write bugs for me, I'm no longer worried that it's going to take my job. Also, adding copilot to cmp seems to be the better option for me since it's easier to ignore bad suggestions. Love the video, been waiting for the review!
i tried! i was reallllly thinking about it
Wait for them to launch the captain. It's copilot what did you expect
@@ronniebasak96 Yeah, the name implicitly means you're the captain, so you can already guess what Co-pilot had to offer
I am using it with cmp too, but I do not understand how/when it decides to spit out multiline suggestions.
What is cmp?
copilot is the best if you're using it to make development fun. If you let it take care of filling in the boring things like lists of constants, filling out predictable arguments, filling out syntax that's easy to forget, you end up having way more fun writing out the logic that requires actual thinking. If you use it to fill in logic, even if it did it correctly 100% of the time, you're going to get disillusioned with the whole dev experience and you won't have any fun feeling confused and like you're not leveling up your skills at all.
You must still be new
@@greglocker2124 I am. I'm only a month past the trial period. Anything I should know?
@@JFrameMan I think hes burned out developer and he meant that because you still have fun writing code, you must be new into programming... dont listen to him, programming is fun, just dont get stuck on one position for too long...
3:43 There is actually even a better way to prevent this bug than adding the condition:
Instead of using a condition you can match against the slice of self.frames (match &self.frame[..]) and then use [first, .., last] as pattern to guarantee that first and last are not the same element, while being both more readable and less error prone.
i love the energy, the talent, the editing, keep up the good work !
Co pilot has only been good for me as a small intellisense snippet engine. Its been dead wrong when I let it write more than 50 characters honestly.
agreed
Same experience here. Its wonderful as a boilerplate elimination tool.
@@josephp.3341 nay
@@josephp.3341 Use a language with less boilerplate then? Yeah, I know, radical idea 😜
@@adriandmochowski9391 Redux has entered the chat
Coming from the opposite end of the spectrum (i.e. noob coder), I've been using copilot to write simple 1 or 2 page python scripts, eg for api pulls and sorts.
For basic 'get a task done', where you're not concerned about quality coding, just getting a simple job done, it's been a dream come true.
Yes I hit many errors, but not the type of issues I used to have that would make me quit and figure out a non-code manual way around my task.
The gamechanger for me is I don't have to read several books, watch hours of tutorials, and become an expert at troubleshooting to get simple things done.
AND yes I do learn a lot of python commands and syntax tips from CoPilot in a much more effective and faster way than I would through standard learning methods.
In fact when copilot gets something wrong it's sort of a game to figure out what it's suggesting vs what you're trying to do.
Copilot has really improved my scope in terms of backend programming.
All helper functions I need are auto generated, even the ones I didn't know I needed.
Also in terms of translation of a requirement I copied somewhere or a json I need to manually write
Super helpful video, I've been using Copilot myself recently for the past month and have had the feeling I spend more time reading and rejecting bad code than I save from accepting good code. Love the idea of just rejecting the logic from the get go.
Using it for a month, I really enjoy it writing boilerplate for me, but I enjoy it more when it writes unit tests for me!!! I've got some whole suit of tests entirely written by Copilot with few fixes from my part!
Yeah, I've found this, too; copilot is really good for spitting out unit tests for your code. It even comes up with test ideas I should have thought of. I'm just like tab tab tab tab tab tab tab thanks for the 6 new tests
how to ask copilot for unit tests? just comment that you want unit tests?
Can you write the test cases or what you want instead first, then have copilot fill in the code?
Exactly my experience as well. I love copilot for taking over the boring work for me, so I can spend my time actually tackling the difficult stuff. Writing boilerplate, filling translation files, mapping data from one structure into another and interestingly documenting my code are now all things I spend less time on, since copilot usually gets these things done for me.
I find that it's good for rewriting existing code. E.g. a couple days ago during a Scala code review, I saw someone had a long chain of matching on Optional values, and I wrote `// instead we use a for-comprehension`, and Copilot perfectly rewrote the code as a for-comprehension.
Using it for a week and I love the autocompletion so far, no amount of vim skills could save you that many key strokes, like writing arguments + types a have never been that fast, when I'm doing something obvious/easily understandable and it spots it or the pattern it's also cool, it takes some corrections but it's faster than from scratch.
I still have a key mapping to toggle it tho since it can be annoying if you're focusing on doing something non trivial and it proposes you nonsense.
The fps function should use (n-1) for the calculation rather than n. Consider having 3 frames, one per second. So the duration difference between the first & third is 2 seconds, then you're dividing by 3.
Thank you for taking the pain for us all! I've been thinking about jumping into using Copilot but honestly I don't write all that much boilerplate in my current job.
It definitely shines the most on those line-by-line suggestions when you name variables and functions intelligently, it can pretty quickly work out what you're trying to do let you just tap away at tab whilst confirming each line is what you were going to type anyway.
You are legendary already. Thank you for your service--every video I watch levels me up.
Great summary. It kind of reflects my thoughts about Copilot as well! 🎉
The summary is my exact experience of copilot. For languages I consider myself competent in, I love copilot. It helps smooth out the boilerplate and I generally can provide it with enough context to guess what I'm up to.
For languages I'm learning, I generally turn it off and debug interactively with gpt
Another great use for Copilot is inline documentation, like in C# you can document functions with XML and I find that Copilot is a treat in those situations, and if the comment is invalid you just CTRL Backspace a bit and correct it.
@ThePrimeagen not related to the video at all, BUT I just discovered "The Last Algorithms & Data Structures Course You'll Ever Need", and I'm so fucking excited. I've learned this stuff several times, but I feel like everything kinda leaves my mind if I'm not practicing... and it has been some time. Getting ready for job interviews, this feels like a godsend. I fucking love you man. Thank you so much! Keep doing what you're doing brotha!!
Love this so much, been using Co-Pilot and had a few moments where I'm like "No wait, that's a bug!"
The biggest difference between me and you though is that when I used Co-Pilot, it had actually wrote code for an edge-case and I created a bug by trying to correct it 😂
Great video. Hit the hammer on the nail at the end when saying it basically works extremely well when you are already at the top of your game and just needing to smash out boilerplate. Personally I would add one suggestion/clarification to the last section.
If you're a junior, I would also highly advise AGAINST using copilot, but I would extend that to any level of engineer that is starting at a new (existing) company, and coming into a codebase they have no knowledge of yet. Knowing the ins and out of the codebase you are in plays a big factor in how quickly you spot copilot's mistakes/inaccuracies.
Other than that Awesome Vid!
If you are at the top of your game, why are you writing boilerplate? All that means is that you haven't been able to create an efficient environment for yourself and your team.
Copilot is such a complete tool, not only writes code for you, also writes bugs for you! Freaking awesome!
Thanks for the overview, really helpful! 👍
I started writing mongoose schemas, and its suggestions were great. When creating a few specific schemas, it even suggested fields that slipped my mind. I loved it for that. However, that's all I used it for, so far. I've been making sure my data models can handle the feature set I want before coding anything else. We shall see what happens when I actually get into things.
Really enjoyed your vid, keep it up!
My experience exactly after 4 months.
My take is that you code faster in the beginning but the gain becomes less and less.
Coding is much more tiring because you don't have to write the stuff that require less thought. You're always working in the hard bits.
It won't replace good programmers.
It will reduce the number of good programmers because beginners will rely on it.
so your advice to new learner is to avoid using CoPilot?
@@thomasdinh2k No, learn to use it like this guy. It's an intellisense monster for sure. Don't let it write your code. Use it to fill in the blanks.
Wrong. The "hard bits" are precisely the precious, _creative_ bits. The part that the most productive (and precious) programmers are more _willing_ to do. (like doing your pushups vs watching TV😄)
after 6 months, you've delivered .. you the best ..thank you ;)
Totally agree with this take. I'm a full stack web dev and have also been using it for about 6 ish months and have come to very similar conclusions. I will say that for me it's worth the subscription fee just for the efficiency gain I get from it auto filling boilerplate. And every once in a great while it does spit out some logic that is actually sound...
it's also great at translating my logic and teaching me the proper syntax when given a large codebase. i'm surprised at how quickly it can learn quirks like that
What I find really usefull is wiring unit tests with copilot. Have you tried it? This works really great if you have existing tests and want to add more cases. Just name the function properly and copilot does most of the work :)
It's also good for coming up with recent (up until 2019, or later with web plugin) libraries and functions to use. If you aren't sure where to start, it will try to do something with some functions and libraries that maybe didn't exist while I was learning to code. Then I will delete all the logic, research the APIs, and write my own using those newer tools it suggested. Having it create boilerplate in places where it can't screw up much. Because yes, it often gets creative with the bug creation.
I've never used co-pilot but it looks quite handy. Nice video and explanation!
One other thing that Copilot is really good is generating code for an API that you're unfamiliar with. I had to do some stripe scripts to check that all invoices were generated correctly. It would take like 30 minutes of me looking at the documentation, working through all that text, instead Copilot generated the code for me in 3 minutes. But it was still important that I knew some details on how the API worked beforehand, so I had the opportunity to fix some of the bugs before they happened.
If I hadn't knew the Stripe API details beforehand, maybe it would take like 10 minutes. Still better than 30.
I think this is where it's most dangerous as well though, especially if one starts to do this regularly. Imagine it writes some code for an API that you don't understand, you haphazardly review it and then there is some subtle bug that isn't caught. I wouldn't want to let copilot write any code for me for which I don't already know what it should more or less look like beforehand. That being said, I think it can be really useful in quickly getting to know the API, and writing some rough prototype code.
@@aure6898 I definitely think the same. It's good for early API exploration or obvious/unsurprising code generation.
Actually, since I made that comment, my perception of Copilot's usefulness has been reduced significantly. It seems to be good only at doing the obvious (pattern repetition, copy pasting, etc) and requires a lot of guidance in most places. In my compiler, it just doesn't understand much and resorts to dissing my code by saying "this is a hack but should work for now".... frankly, Copilot is not wrong there lol but it's just not that useful.
Agreed. Copilot is awesome at doing boilerplate. It's also pretty good at doing CLI parsers given help outputs. (In Haskell at least.)
I mainly use it to do all the tedious stuff like autocompleting several nested brackets or parenthesese and generating boilerplate. It's really good at picking up patterns which comes in handy many times.
I started playing with wgpu in rust recently. And there is so much boiler plate. After a few hours of working on that project copilot just started smashing out those texture descriptors, buffer descriptors and so on. It saves you a lot of time, but only on the stuff that's annoying about coding. If you are not exactly sure what you need, it most likely won't do a good job
div by 0 is defined in IEEE754, you do get a inf in rust too
edit: for floating point types
Hey man. Love your content! Good job. Also wanna ask what headset do you use?
Not sure if you got a chance to test it out with CMP completion. It's very similar to how you used to use tabnine for completion but gives you copilot instead. Imo it's a way nicer experience since it won't just blurt out a full function (since you're not using it anyway). I'm curious to see your thoughts on it
I find that for C# Unity development, it's really good at filling in the boilerplatery stuff, like the caching of get components. It's also good at knowing what kinda stuff you'll likely do with certain variables, probably because gamedev code is so similar a lot of the time. But yeah, I only use it to fill in a line of code, but still, I use it quite often and would probably feel the pain without it.
Unity dev here too, I use it the same exact way. One line at a time. I also find it incredibly good at writing SQL queries/strings, as it knows what I want to fetch and can just know what my database schema looks like somehow. Try using vscode on a plane one day, you’ll find yourself typing then stopping to wait for the auto complete and then realize a few moments later it’s not coming 😅
@@NicolasSouthern Okay, Mr. 1000 lines of code each day. Maybe you should disable auto completion or even better remove all your so-called productivity tools.
@@rentefald nah, I’ll keep using them. These productivity tools assisted me in getting a product from theory to market in probably half the time and it’s currently generating a solid income for me. I don’t need to write every for loop to feel good about myself.
Thanks for the suggestion though!
@@rentefald this is a tool that is like a power tool in the hand of an experienced woodworker versus a power tool in the hands of an inexperienced woodworker: the experienced woodworker will get their project done faster, the inexperienced woodworker will likely botch their raw lumber that much faster than using hand tools, and may take off their hand in the first place.
I find it comical someone in the field of automating things as the job description is screaming at others to avoid automating the task of automating things as much as feasible: it's the epitome of extreme hypocrisy personified.
I just tried copilot today writing a varint encoder/decoder in Typescript. I agree with your assessment, and I would also like to add that its really good at writing JSDocs if you ever make something public facing in Javascript.
I'm so glad they've made a Copilot vi plugin for you.
Great video! I program in several languages with different IDEs (VS Code or Jetbrains, actually) for production and teaching. The little I have used Copilot has matched your experience of boiler plating. But it makes me wonder: the example with filling in the Media type with Typescript is something that Jetbrains does with syntactic analysis. For this, AI seems to be the long way to the goal. But I find that VS Code is lagging behind substantially on syntactic analysis and try to compensate with AI. Does that match your experience and is it the better path?
You could be right. There are some things that it does so well. Like at the end with auto completing audio FPS. It can get pretty dang fancy with logical boilerplate
For that first examples, what happens if you specify to do it without memory allocation? Does being more explicit give you better results?
This is a great concise review of where it falls short. It has screwed me majorly twice by generating incorrect code which looked right at first glance. Like honestly I got lazy and let copilot fill in the blanks for code I could have written myself faster than I could have verified. The two outages eviscerated any marginal time improvement spent typing.
Honestly if you have boilerplate in your codebase instead of fixing it with copilot, you might want to ask why you have boilerplate everywhere. Every line of boilerplate you write you gotta read too at some point.
-Out of context
You really made my life a bit better, I'm not in my best moment, but you give me the motivation enough to keep going cause you're a "real" whatever that means, just want to say thank you dude.
I know you really liked the DTO functionality of co-pilot, but almost all existing basic intellisense can already do it, just thought I'd mention that, love the vibe of your video, quite informative, thanks.
Would be interesting to hear your take on the GPT4 based one after months usage.
i like the autocompletion, used the copilot chat for hints to fix my code and use mostly the /doc command, when a method/function is finished it creates a really nice ///summary in c# 😊
Excellent take on Copilot! Thank you!
I've found it fantastic for fleshing out comments. They always need more work but sometimes getting the bulk down is great
Love the keyboard sound. Could you please give more details about it? which keys?
I really like how you let CoPilot complete your subscribtion suggestion, that just feels so natural 😂
That’s what I thought as well and here I was am I using it wrong ???! 😂😂
Great video and most importantly a no BS AI will replace kind of video as well ❤
Thank you for sharing your experiences.
As always great content 🙂
Ahhhhhh WOW the quicksort example is so perfect and perfectly points out the problem with these a.i. tools.
Yes, it does. People don't realize exactly how sinister these problems are. Because quick sort has a step-by-step solution, most problems I solve don't
Honestly the best thing copilot has done for me is speeding up writing unit tests
Not only will it take out any of the boiler plate around, for example, mounting components, but also, if you give a good enough name to your test case and it is a simple case, it easily create it for you
You do realize that what is basically does is stealing licensed open source code? Using it is unethical.
@@bdkamil95 do you realize that no one fucking cares? Blame Microsoft for it, not the consumers
After spending all day today writing unit tests, this is the straw that is making me try copilot.
@@javierflores09 consumers are just as guilty as the creators are.
@@bdkamil95 no they are not, you are using a phone that was most likely made from child exploitation in China yet I am not blaming you here for it, but Apple or Samsung, or whatever phone manufacturer you are going with. You can't blame the consumer because they can do little about it other than protesting, besides rather than being unethical it is just of dubious moral, it isn't as simple of a problem as people make it seem
Hey @ThePrimeagen question probably worth a video. How much time is ok for a junior/senior developer to pickup a a small code base and understand and be able to do modifications/features ?
The problem with your question is "small code base" is such a nebulous term.
Small-but-complex code and small-but-simple code is one aspect, but what do you define as "small" versus simple and complex?
It helped me improve my c++, particularly dealing with stl
thanks for your sharing experience. i test not copilot, but chatgpt, to create unit test. it was 1:10+... 1 time ask gpt for the function, 10 times more asking for unit test of that function due to runtime error.
Thank you, Mr. ThePrimeagen!
Had approximately the same experience after 24h of using it but was thinking I was using it wrong. Thanks for your video. A good boilerplate assistant but not sure it's worth the price.
Nailed it. 😂 I use it mainly for line completion, boilerplate, and on occasion, it will surprise me with a logical suggestion. $10/mo likely makes me 10% more productive.
The more you know about what you are doing and what you want, the best the collaboration with such auto-completion AI tools.
I think it's just the beginning though, and soon enough we will see much better code generation and UIs to use them more efficiently.
I am 1 day into having CoPilot in my JetBrains IDEs... and it is, for sure more than a snippet generator. I have so much to learn, but giving it project context has it returning code that looks like I wrote it, albeit sometimes/often quite wrong. I felt I was missing a place to prompt... but, I guess doing what I was doing yesterday, by deleting the wrong parts and letting it continue to suggest may be the way. Day 2 now, I guess I have a side pane with other "guesses".. I'll see how that works. Thanks for the hints on how to get it to do the right thing...
1 / 0 is Infinity in JS :)
Great Video Doc!
Thanks! This is actually how I feel about all AI. Thanks for taking one for the team for 6 months. lol
Thank you for the video, I had the same issue with copilot and dumped it. I maybe give it another shot!
It works really well when it has context of your well maintained, large repository. Otherwise it is boilerplate filler or a way compare how others would write the same thing.
Great video bro!
The Go LSP takes care of a lot of boilerplate without having to use any fancy AI. I just start initializing a struct object, double activate the suggestion keymap, and all the struct initialization lines are populated.
And the best part about not using an AI to write my bugs is that I get to write them myself.
Amazing review!
I extremely rarely let copilot write multiple lines of code, but as single-line automcomplete iv found it incredible
This is exactly the video I needed in my life.
Def agree with this. I feel like co-pilot has been great at typing out things for me that had pretty clear in-context "templating" provided by me, but I would never trust the code it actually provides on its own. It's also great if you're just mentally blocked and you just need some kind of starting place to work with.
I like this short no thinky videos, easier on the brain
can you elaborate?
@@ThePrimeagen i watch your videos to chill and lighten my mood, good to turn the brain off and watch an entertaining but enlightening shortish video now and again. Compliment, and probably good for the add revenue, that's all :)
I hope to see a new video soon about the new copilot X with gpt4!
Something that copilot was amazing at, generating translations. Something incredibly time consuming and tedious when you're a solo dev.
Thank you so much for your advocacy on Rust. I have a question somewhat unrelated to this particular video. There exists currently something called Dart/Flutter which is allows one to deploy on desktop, mobile, and web with one language and framework. Currently one can use Rust with for instance rocket, a web back-end framework, SurrealDB, a rust based database, Tauri, a desktop environment, and Yew, a web front-end framework. What I would like in the future is something like Flutter where you have Rust and one system/framework/transcoding that can deploy to all three, desktop/web/moble. Essentially Flutter but for Rust. Are people working on that? Such a thing would be beyond my abilities currently. But no one has accused me of thinking small.
i agree this is how i use copilot too, sometimes wish it was a setting to turn off those long suggestions.
i also usually don't do the "comment driven development" thing BUT i do find that it is useful for writing unit tests honestly
You can ctrl+right to accept suggestions word by word instead of as a block. I use it quite frequently bc it's annoying when you just need some boilerplate and it thinks you need a whole 15 lines implementation of the thing
Massive thank you for this.
One thing to mention when working for a company is data privacy because regular subscriptions have telemetry and data collection.
Not only data privacy but also licensing. It’s known for basically stealing large pieces of licenses open source software without respecting its licenses. There’re ongoing legal issues against msft and other related companies due to that fact.
I tend to just have copilot for things that i cant be bothered to open google for. Like hey, I want to know how to sort an array in reverse. And those simpler things it tends to do very well. But longer code blocks tend to be slightly wacky, but can be used if you have good tests for the function already imo
What's the colorscheme in Vim? It looks like the repository set up uses rose-pine, but rose-pine has different colors.
what was very important for me was implementing a logic to only get the first line of copilots suggestion
Copilot has been an **insane** boost to my productivity in all ways. Just a matter of learning what it's good at and where it falls short. Easily the best dev tool I've ever used.
It'll be interesting just how much better it copilot-x is with the GPT-4 model and added features. I've gotten AMAZING use out of having it write scripts for me that deal with APIs I'm unfamiliar with/other things like that.
Finding all the little things add up and take FOREVER where as validation of endpoints used in a script is much faster.
As well, having it write tests/boilerplate for code is sublime.
On your second point, I find that my copilot only RARELY recommends more than to the end of the current line. Which I do like, since often times the rest of the line is very obvious. If I want more lines, it will recommend a next line after I go to it, and so on. I use it like an actually smart auto-complete. Like context aware fill or something.
I don't know what this guy is saying but he's saying it in such a compelling way I subscribed
Same experience. For autocomplete, boilerpate type stuff it is good, but for logic it usually gets it wrong if beyond something simple. It definitely helps but I l am not worried about anyone losing a job anytime soon.
Great Video! Have you tried Copilot Labs in VS Code?
Nice, v
ery informative video, thanks for sharing.
As a noob coder, this guy is talking Swahili! Good job.. we need people like you
Thanks for this video. I can't code yet so you helped me dogde a bullet.