Autonomous Flight with a Camera: Horizon Detection Algorithm
Vložit
- čas přidán 25. 07. 2024
- I built a computer vision autopilot system for an RC plane. The plane is able to maintain level flight and execute controlled turns using only input from the camera and no other standard aviation sensors such as a gyroscope, GPS, altimeter, air speed indicator, etc.
This video explains the horizon detection algorithm that allows the system to work.
To see the project overview, check out this video: • Computer Vision Autopi...
GitHub Repo:
github.com/timmarkhuff/horizo...
Chapters
0:00 Project Recap
0:48 Image Thresholding
3:08 Finding Contours
5:46 Filtering Contours
7:09 Diagnostic Mode
7:40 Fitting a Line to the Contour
8:48 Limitations
9:46 Conclusion
#python #opencv #drone #rcplane #computervision #robotics #aviation #fpv - Věda a technologie
Super cool! Thank you for sharing!!
Great work!
really cool - thank you for walking through the code here. awesome work
Glad it was helpful!
Very informative.❤
great work
This is awesome
Cool stuff!
What happens in a steep dive, when the sky is out of view?
A steep dive would be a problem. It needs to always keep the horizon in sight. The autopilot program I wrote only allows level flight, slight turns, and slight ascents/descents so it isn’t a problem. This could be improved with a wider angle lens, but I haven’t really explored that too much.
How did you perform testing of the code and the airplane in general? Did you use simulation software to model the plane and test the code?
I started by flying the plane manually and recording footage. I used that to develop and test the horizon detection algorithm.
It became tricky when I started developing the flight controller. I had no simulation software, so most testing was in the air. I did create a simulated horizon out of poster board, which I held in front of the plane to test that it made sensible responses to changes in the horizon. Once in the air, I found that the PID controller was not well tuned (lots of oscillation). I had to do trial and error over many flights to get the PID parameters tuned reasonably.
One thing that you could easily play with is how you do the rgb-to-greyscale conversion. There's a reason why we say "the sky is blue", so taking the greyscale value as the usual 33%red+33%green+33%blue seems to not be the smartest way to go about this.
What makes this complicated is that 0%red+0%green+100%blue might not be optimal either, especially when there's blue lakes in the picture. Since lakes will also have some blue tone, but a less overexposed kind of blue.
Maybe you can compare the color of your sky with the color of the most problematic (brightest) patches of ground. (Which I define as any bright areas on the "ground" side of your horizon line.)
Step A: Take the average sky color minus the average problem patch color (And since the sky is the sky, you can probably get away with sampling only every 10x10 pixels of your downscaled image).
Step B: Normalize the resulting RGB space vector - by which I mean to a sum of 1, not an euclidian length of 1. (Problems can occur when the problem patches are brighter than the sky itself, see the first fix idea below.)
Step C: Use those numbers as the mixing ratio for the greyscale conversion.
You'll probably need to come up with a way to obtain a representative sample of problem patches, despite them appearing (by design) only rarely.
My first idea would be to not use the same threshold as for your actual horizon detection, but 90% or so of it (to pick up on darker patches that COULD become problematic with a slightly changed sky color.) And my second idea would be to keep a record of past patch colors across several seconds of footage, and average over those.
Great ideas!
Interesting approach to this problem. There are some downsides to using visuals for such a vital system. Is this just meant as a proof of concept, because for production a simple IMU would probably be faster and less complex on the software side. But nevertheless, great video 👍
Yeah, there are definitely more established and straightforward ways of doing this. My project was about exploring what could be accomplished with computer vision. Who knows, maybe it could function as a backup system when IMU data is not reliable? Or maybe it was just a fun project :)
Can you please show us your electronics set up?
My next video will be on the hardware set up. I have some improvements in mind, so I will make those first, and then shoot a video. Stay tuned!
The hardware review has been published! czcams.com/video/MBLD8sjiYiU/video.html
Great code review, very clear and concise.
I think segmentation would definitely give you better perception outputs, but would depend on how quickly you could perform inference. What HZ do you get out of your control loop right now?
Depending on that, you might be able to get away with using a USB compute stick attached to the raspberry pi to augment processing power.
My program runs at 30 FPS now. I chose this FPS because the Raspberry Pi camera only seems to be able to get frames at that rate. My horizon detection algorithm runs at something like 250 FPS. I think you are right that adding a graphics accelerator (like the Coral Edge TPU) would help run a semantic segmentation model and possibly get better results.
@@timmarkhuff I think you might be able to get away with running an even simpler algo - whereas segmentation takes an image input of m x n pixels and returns an output image of m x n with many categories of object, you only need to extract one feature (horizon angle). So you might be able to build a neural net that runs leaner and outputs 30Hz
Was flying the clubs' C172 one day and the turn coordinator ball was way the hell out the side, very odd.. I did everything in my power to center it, but my head wouldn't let me, so I went instrument only for ten seconds and the ball magically centered. Eyes outside again, and as if by some black magic the damn ball headed off to the side again! What was it? A cloud on the horizon was on an angle.... You will need some checks on your algorithm - I did on mine.
Ok, what about night? I assume that the algorithm not will be working fine, because of camera with night mode return back images in different colours.
That's a problem I haven't tackled yet. Presumably the program would have trouble with that, but perhaps with the right kind of camera, it could work.
Is there another video in this series coming?
This series has been on pause for a while, but I might resume soon. What kind of videos were you looking for?
@@timmarkhuff something that covers the basics of the code and how you starting building the software would be interesting. Even if it was high level and not super detailed.
And where exactly is he flying? What is the ultimate goal and who assigns it?
At this point, the program just flies level, like a simple autopilot. Also, the user can command the program to turn, and autopilot will manage the turn. In the future, I would like to add user-defined waypoint missions, but that will take a bit more work.
@@timmarkhuff Is there a desire to turn an airplane into a living being? With irritation and reaction to irritation
@@Koroleva_O_A haha, that might be a bit beyond the scope of this project, but it sounds interesting!
@@timmarkhuff I will not deceive, it's not about super technologies, everything is much simpler. A living being is just an airplane that receives commands from the surrounding reality, and its behavior is set by the surrounding reality. So it's not something sky-high, everything is much simpler. Create the same thing as any living being, only in a primitive form.
Commands are set by the environment, in psychology this is called an irritant. If you translate this into the language of technology, then the command is called a definition. This is the result of measurement and/or identification. The definition has a magnitude. This value is translated into the amount of irritation.
Some examples. The plane sees a person a few meters away from it. A person is identified and identified as a dangerous being. The danger must also have a magnitude. The higher the definition of danger and the smaller the distance between a person and an airplane, the higher the amount of irritation. On the way out, you will get what you can see in the behavior of any bird that is afraid of you, and that flies away from you. The closer you are to it, the greater the magnitude of fear (in living nature, all hazard identifications are set by constants and are regulated by the distance between, which a living being determines by sense organs, in our case a video camera). That is, the plane will constantly fly away from us because it is afraid of us.
If this is quite difficult, then first you can teach the plane to just take off when it has a need to move somewhere. This is also a reflex. A living being moves not only when we scare it, but also when it is hungry, for example.
The magnitude of the reaction corresponds to (and is proportional to) the magnitude of the irritation. The higher the irritation, the more pronounced the reaction. For example, to spin the engine to a greater number of revolutions. For a living being, this is the amount of muscle tension. If there is no danger, then turn off the engine (this is also an example of how real living beings are programmed to relax their muscles).
In principle, it seems to be nothing complicated. Or is it too complicated? Will you be able to cope with such a task?
@@timmarkhuff Aren't you interested anymore? Or is it difficult for you?
wow , we did it with ardupilot 10 years ago :) but cool work anyway