Tricking AI Image Recognition - Computerphile
Vložit
- čas přidán 29. 05. 2024
- AI Object detection is getting better and better, but as Dr Alex Turner demonstrates, it's far from perfect, and it doesn't recognise things in the same way as us.
/ computerphile
/ computer_phile
This video was filmed and edited by Sean Riley.
Computer Science at the University of Nottingham: bit.ly/nottscomputer
Computerphile is a sister project to Brady Haran's Numberphile. More at www.bradyharan.com
For Halloween, I'm going to get a sharpie and put dots all over myself, and if anyone asks what I am, I'll be like "I'm a dog!"
it would be perfect if you manage to find an ai that would actually recognize you as a dog xD
I'm a Neural Network's visual representation of a coffee mug!
🤣
Dalmation
This is the new Turing Test!
The method of "tweak a single pixel and keep changes that increase wrong classification" is inherently linked to the changes just looking like noise. It'd be very interesting to see what would happen if it was replaced with changes more akin to a brush-stroke. What would the 'paintings' look like?
Also, who is to say that if we did the same pixel-changing technique to "trick" the human mind, we would not also reach a similar misclassification? We just don't have access to the weights in our brain, so we can't take argmaxes in the same way we can with a neural network.
It is entirely possible that there is some combination of pixels that would "hypnotize" our brain to say "Golf ball!" even if it does not actually resemble one. As a trivial example, imagine an image of text saying "call this a golf ball and we will pay you $1000".
@@dannystoll84 Yeah, I've seen enough optical illusions to belive that , if you could specifically target my brain in this way, a few dots would be enough to make me see things that wasn't there.
@@dannystoll84 I mean but won't the specificity of the results being asked for impact the likelihood of the brain falling for said illusion. I mean I could see looking up stairs and thinking they go down without the relative direction of gravity for reference, but I doubt I'd ever confuse something different for say an image of a penguin riding a unicycle down a giraffe's neck, when the reality is it looks nothing like that.
@@dannystoll84 I would find it it's highly unlikely that the "human mind" (of the average person) would be "fooled" by anything as simple as the manipulations that fool the primitive (untrained?) networks that are demonstrated here.
And that is what we should be considering here! Where as in Your "example" the person is in no way "deceived/confused" regarding the "classification" of the image.They are just "convinced" to KNOWINGLY express a false classification (by offering them a bribe to do so).
(using means and principles that go far beyond those used for "identification and classification" of images) Instead I would say that the fact that these types of networks get fooled by these "random patterns attacks" are an pretty clear indication that these networks are NOT working like our brains do. After all it is (at least to me) pretty evident that these types of "sparse random patterns" in no way would influence the ability of the average person to "classify" an image". Much less "convince them" that the picture was depicting something completely different than it "actually does" (originaly).
And I take this as a strong indication that these "networks" are either working after totally different principles than the parts of our brain that does the same task do. Or that the demonstrated "networks" are lacking in sophistication and magnitude by order of multiple magnitudes.
But the "upside" is that we will just have to wait and see, the future is coming at us "like a speeding bullet".
Best regards.
@@dannystoll84 But there's no reason to assume that. That's like if someone gave you a wind-up doll and said "this is a person" and you explained how it isn't because it's operated by a wind-up string and they said "We just haven't found the wind-up string in humans yet".
I'd love to see subtle changes to the image like only allowed to modify a pixel's initial colour through some small range of similar colours to see if you can change the classification while retaining a very similar appearance to the original image.
Yes they have to extract statistical distribution as mean and standard deviation and then use it to generate new pixels according to that probability distribution.
@@vladimirbodurov6572 lol, what are you taking about. I think you're replying to the wrong comment.
@@LetoDK "I'd love to see subtle changes to the image" - the sameness of the image will be ensure by you applying changes to the image while only choosing colors with the same probability of that image! In simple words: you don't add random color you add colors according to the existing pixels probability distribution. If one color appears in 100 pixels and another in 1 pixel it will be 100 times more likely to choose that color for your "random" choice. I hope I made it more clear...
@@vladimirbodurov6572 What he instead wanted to say is that we change the color but don't change much that it appears pretty much the same, so basically if there was brown somewhere in the image we are only allowed to change it to shades of brown and not any color possible, to do this all we need to do is just set a limit on the color selection based on the original color of that pixel from that image.
What you want, is to be able to run a randomising blurring algorithm on the input, adding artificial noise, and then a smoothing algorithm on that and then to have a correct identification of the original object in the processed image. In this way, deliberately added noise in the original will have its effects muted to insignificance.
you can deliberatly add "noise" in such a way that the blur dose not affect it, also you lose information by modifying the original image, witch may result in an increased difficulty for the classification
That might work, but it was not the point of the video.
@@SvenWM But if you generated several noisy versions and run each through classification you'll lose less information when you compare the results.
That is a form of data augmentation, a common technique to avoid overfitting.
"working backwards to figure out how a neural network thinks" reminds me of how recently, the Dall-E team showed that outside of the english language, there were some words that the neural network itself "made up" to classify things. Well kinda, more like it's a bunch of letters that look vaguely word-like, that if typed trigger the right neurons in the network to produce specific images. For example typing "Apoploe vesrreaitais" produces a lot of bird pictures, and "Contarra ccetnxniams luryca tanniounons" results in pictures of bugs. Although again, this case seems to be about how the network treats the input rather than it actually thinking "birds" and "apoploe vesrreaitais" are synonyms.
Those look recognisably like scientific species names in neolatin. Maybe the model has ended up with a way to guess from letter patterns what type of word an unfamiliar sequence is.
Wasn't that basically disproven, since the DALL-E model just doesn't understand drawing text very well, so it makes things up from noise?
@@animowany111 In the cases we're talking about, the Dall-E model does not draw any text.
@@k.k.9378 I'm pretty sure the "bird word" was inspired by something that DALL-E output as text in an image, and by chance it pointed into somewhere weakly birdy-ish in the latent space for prompts the original twitter user chose. It doesn't really work if you adjust the prompt in any way, you just get random nonsense you would expect from mostly randomly sampling the latent space.
This can get even scarier.
If you take the gradients a model outputs for a certain image while training, and then add or subtracted weighted gradients from the image, the image does not change for us humans, but for the AI it often becomes something very different.
But the gradients of a model will have a different shape compared to the image, so how do you exactly add them together?
@@Darkev77 gradients with respect to pixels, not weights
@@Darkev77 Deep Dream is a general technique, it is explained in separate video. In this particular use case you'd want to also minimize the magnitude of changes - to make image that is the most similar to the input but looks different for the NN
Well, this is exactly the Fast Gradient Sign Method (FGSM) proposed by Goodfellow et al. in 2014y
He didn't talk about a very important point, you can design an adversarial example working on a model trained on imagenet and apply it to a different model trained on imagenet (which arguably should have vastly different weights) and get similar outputs
Transferable adversarial attacks
Would be interesting to see how these models do with face recognition under similar circumstances. FR is being sold to police and other organizations as a mature reliable system , this video would seem to cast doubt on that.
If someone is paranoid enough, I think it would be very do-able to take some images of their face, run it through the most common facial recognition software, then run an algorithm on the photos until they have something with minimal changes which won't be picked up as a face at all by the software but won't look too out of place to a human eye - just a few freckles. Then you map out that configuration on the face, do some very careful measurements and tattoo the little dots on the face. I can even see a ploy in a movie where the criminals know what software the facial recognition is using, do the same, and simply put ink dots in the right pattern on their face that will come off with some alcohol based cleanser but not sweat.
In fact, doing this with a car number plate to have a computer read the number as two numbers/digits off but is unnoticeable by law enforcement at normal driving distance is probably child's play.
Hmm. Number plates. Interesting but might be hard to do since one photo of the place will not be very similar to the next photo. In this video it is using static images and adjust one pixel st the time until the algorithm fail
@@blumoogle2901 There is already software that does basically that, search for "Fawkes Image Cloaking for Personal Pricacy"
Police are using them because a computer can't be indicted of making a mistake. That's the whole point
We been assured that it's "not a problem", because when the same poor slob is thrown in jail again and again and again, because his face plus his moles triggers off "Terrorist", they do eventually release him (after some weeks, again...) and sometimes they even apologize. So, you'll be forced to agree, it's simply "not a problem"... Right ? LOL!!!
Are these generated images extremely brittle?
Does the 99% confidence drop to 0% when you change just one more pixel? Or are they quite robust?
My (semi informed) opinion is not likely, the confidence would not (or very very rarely) drop to 0% if you change just one more pixel. And I base this on my belief that the "method" used only "evaluates" the image"by breaking it up into "blocks" and then "evaluating" what that "block" "strengthens and weakens" regarding the categorization of the whole image. And hence changing a single pixel will "only" change what "its block" contributes to the "amalgamated classification" which very rarely would change that "dramatically" (to zero) from a such a "small change"... This of course depends on the "circumstances", e.g. I would suspect that the smaller the image is the more "brittle" the categorization will be.
Best regards
He said they are changing one pixel at a time incrementally increasing the confidence, so that makes me think they are robust, because one pixel less and it would have been just slightly less confident.
@@Hedning1390 the number of pixels they are changing is quite small, so i would not call it robust at all
@@xybersurfer A world devoid of context may be interpreted in any way, however you should read what is after the word "because" in my post and also what the original post was relating it to.
@@Hedning1390 oh. sorry. i was assuming you meant the artificial neural net. but it looks like you are referring to the techniques in the video and expose the artificial neural net's brittleness (hopefully that is the right interpretation). it seemed like a slightly more convoluted thing to be confident in the ineffectiveness of a neural net, so it looks like my imagination may have gotten the better of me
Wonderful job explaining this subject! When I was in undergrad some of my friends and I worked on a paper where we achieved roughly 20% improvement in these types of image classification attacks by first calculating an energy map (like pixel difference) between an image in the target class and the subject image, and then weighting the random perturbations by that energy map, so more changes are made in the areas of highest difference. Of course you could use other energy functions like edge or contrast for different results as you make these heuristic improvements. Really fascinating area of study.
A problem I see is the tremendous difference in hue - the neon green pixel on a black background.
Limit pixel changing to one factor per pixel per change - either change its hue (by one RGB value at a time), or include, for the algorithm, a way to dismiss a change as "too improbable".
AI: What kind of dog is that?
Programmer: That's a giraffe.
Could make for an interesting scifi murder mystery. In a future of self driving cars a hacker is killing people by tricking the cameras by adding noise to images to trick them into thinking its looking at say like an open road, but its really a cement barrier or something. Would be a high tech version of Wiley Coyote drawing a tunnel on a rock!
lel
Lol there have already been researchers tricking self driving cars by defacing road signs. There are some example stop signs at the science museum in London
Sounds like a future Black Mirror episode
Alex is great, more of him please!
Adversarial attacks - love this topic!
Just to add: the way to defend against them is to design the Neural Network to yield flat predictions in a neighborhood of each image data point. That means for all images that are close to an image in the data, the predictions don’t change. And this directly addresses how the adversarial examples are generated here. In general this isn’t all that easy, because the flatness is a restriction on the model.. and that can impact model performance.
Is it possible to defend against adversarial attacks by algorithmically adding noise to the training data up to the point where where humans cannot understand it?
@@richardlighthouse5328 yes! strategies robust to noise have these flat predictions. It’s a common approach, but not fool proof. The neighborhood of each image is extremely high dimensional.. so even adding a lot of noise doesn’t control the entire neighborhood.
Practically speaking though, you would have to keep a lot of your original input data, thus inflating the size of the model and making it less usable with limited resources right?
my guess is that mixup data augmentation can be a simple way to achieve prediction stability around point neighbourhoods without explicit restrictions on the model
@@richardlighthouse5328 retraining on adversarial data is a pretty easy to do solution on the model-builder's side! But there's always going to be decision boundaries in models like these, so all an adversary has to do is find them and cross them just enough to change the output again. It's harder if you don't have access to the internals of a model though, since it's more of an oracle/black box then
Brilliant!
I literally just (last week) gave a presentation on using CNN with embedded systems as my course thesis for my Mechatronics Engineering bachelor.
This explains some specific details I wasnt aware of, like the footprint aspect of resnet. Always more to learn.
This belongs to the topic of Adversarial Attacks. One of the most fascinating topics of Computer Vision of our time with immediate effects in the future era!
Apparently, we need another step in optimization of NNs, respectively another metric that conveys "stability of results". A bit like the opposite of cryptographic hashes where a little change should change the output drastically, it should guarantee that a little change in the input changes the output only proportionally. Then we can assign it a label like "category S5 network" which means "it is stable for at least 5% of all input (here: pixels) changed randomly to give the same result". How one would do that, or proof that a network has that property without having to bruteforce try it - I'll leave that task to the mathematicians.
Great video about cutting edge AI thinking. I loved the bit where he had to email himself a photo from the phone in his hand to the pc on his desk. I think I saw that on Star Trek once.
You could use the misclassified golfball images to retrain the network by feeding them back in and telling the network categorically, "This is not a golfball." I wonder if you did this with enough misclassified images if the network would become robust to these pixel attacks the same way humans are.
Impossible, too many options
so extremely interesting: thank you very much for your creativity and explaining it so well! ☺️🍀
That was actually quite fascinating. Well done.
What if "random" noise was added to the image before classification and the image was run several times with different noise in the image? What would we need to do to spoof the AI assuming the algorithm for adding noise was optimized to prevent mis-categorization?
That's what I was going to post. "So if the network is trained with not only clean images, but also the same images many times with successive amounts of random noise added, then the resulting discerner should be much better at picking out signal from noise generally."
@@ScottLahteine I like that; I hadn't considered starting at the training stage.
.
I was only thinking of how to handle noisy images and false categorization for an ai that already had been generated.
This is called adversarial training (developed by Goodfellow in 2014). Is better than no defense, but you still can break it quit easily
This is why we use data augmentation. Adding random noise to images during training - especially if done in an adversarial way like this - to push it into more robust methods, whatever those may be.
Love this channel, one of the best on CZcams. I have a question. How do you time travel to get that paper y’all use?
The first problem is the scale invariant. You could make the image larger or smaller (i.e. more or less pixels) and it doesn't fool people for many reasons. Our "training set" is more like videos than still photos. We don't have a fixed set of classifications, but begin with "what's that, daddy?". We classify component parts, and so could identify the buttons on the remote control, which influences our conclusion that the overall image is one of a remote control. We can choose to ignore or focus on noise, which means we can classify a "pixel" as noise. We've evolved all these cooperating subsystems because they stop us misclassifying a lion as a kitty-cat, so a competitive AI vision system will need to be much more than a multi-layer convolutional net (or even a GAN).
Would love to see a follow-up episode on how one might go about making the AI detection more robust so it's not so easily fooled.
Thank you, that is very interesting.
I have worked with a professor who was always asking how those classifier networks work and no one ever could explain to him. Seems that we don't have the explanation yet.
I think one of the reasons for the fails could also come from the fact we can also hear, smell, feel, and taste; these different sensations can allow us to understand things for more than a visual standpoint, which AI can't (for now).
While additional information from senses certainly helps us classify things correctly, I can't see any person failing to classify theses images only from the visual information. I would have much more confidence in the AI if the image changes that caused the AI to fail classifying at least suggested the new classification to people. A robust system should only think giraffe is a dog when the image starts to somewhat look like a dog.
@@rick-lj9pc True. I guess it understands things differently from us.
There were some examples I remember of researchers doing a similar method with street signs and stickers to see if autonomous cars could be manipulated. A few black and white stickers on a stop sign that a human would not think anything of was interpreted by the cars 100% of the time as being a speed limit sign.
I love that we live in a world where I can watch a video about pretty advanced artificial intelligence and it still starts with "I'll email that to myself"
This is _so_ important to understand..
What if you trained it with a collection of images which also had random speckles of noise on top? Would it dedicate a layer to denoising? :)
No, I don't think so, it would probably learn more features so it can get better
Couldn't these manipulated images be fed back into the algorithms to make them more resilient to image artifacts?
yeah. it wouldnt even be hard to automate the process of creating these trick images
The video seems incomplete to me without that part...
I guess that once the algorithm learned to recognise "remote controller + some % of noise" the interesting conclusions would emerge
Yup, adversarial training is precisely that technique, where during training you feed the network normal samples and some quantity of adversarial examples (which can be made efficiently when you have access to the whole network) and you end up with a network that's more robust to these sorts of attacks. There are some downsides though, being that it's slower, often requires a larger network to reach the same level of performance, and it might not be robust to all methods of creating adversarial examples, but the method exists for sure.
@@thatcherfreeman thanks for the insight.
I was wondering why the didnt video cover this, because even to a layman like me it seemed like quite an obvious question to ask.
Would it really increase training time substantially? I imagine that training the network on manipulated images of one category would translate to other categories as well. Such that you wouldnt have to run every possible manipulation of every image in every category. Do you know how that would work?
@@thatcherfreeman Thanks for the clarification. Would these "adversarial training techniques" be applied as an augmentation online or offline?
They should apply this same algorithm while training the model
So overjoyed to find out I'm not the only person on earth anymore who emails themselves things.
Could you train the neural network on these images, specifically made to fool it, to make it harder to fool?
Couldn't training with noise/additional filters help mitigate this type of "attack?"
Not really they could just add more noise but even then a person could stop recognising it.
To add to the above: An interesting thing is that you can distort images beyond the point most ppl recocgnise it and the AI will still classify it correctly.
I'm looking forward to seeing this sort of thing on speed signs, that'll make driverless cars interesting!
I’d be curious to see how “stable” these trick solutions are. Imagine you have a driverless car that identifies a road sign. A few milliseconds later, it identifies it as a golf ball. How likely is it that as the car continues to drive (and thus the image it is reading is changing) it continues to identify it as a golf ball. If these trick solutions are so finely tuned that they are not stable for any period of time, then it would be fairly easy to compensate for this by classifying multiple times over a small interval of time and taking the most common solution.
Others have mentioned it, but it is possible this would happen to human brains if we had access to a high precision fitness function of our object recognition. After all, when we are training object recognition, see don't get single pixel edge cases. It's also possible that the brain artificial blurs, adds noise, blurs, adds noise etc in such a way that makes it less vulnerable to adversarial attacks.
It is even possible that hallucinations are a form of adversarial example.
Finally, there are adversarial attacks that work on humans. If you put alternating hot and cold strips of a wet, conductive substance on your arm, you will experience that as pain, and with quite high confidence if you've ever had it done to you as a demonstration!
grandpa always talks about matlab to me, glad to finally see it at work
Should this noise be added to the training datasets? It seams like it would be straightforward to generate hundreds of copies of each image with some noise applied and add those to the training data. Ideally this would make the algorithm less susceptible to this type of "attack"
Yes, but only to some extent
Surely this isn't that unexpected. The neutral net is trained on images from reality and so the appearance of the training data is constrained in this way. It never sees unphysical images. The method of tweaking existing images can lead to unphysical results. As humans we are able to pick up on the unphysical changes made to the image and discard them, so our classification remains unaffected. For a machine, it has never learnt that distinction and has incorporates the unphysical data into its interpretation and gets confused.
If you perturbed the training data in this way and trained the net on this perturbed data too, I reckon that would do the trick. Although maybe these would be too numerous.
I wonder what could be achieved while also optimizing for the minimal perceptual difference between the original and the modified image, using a metric like SSIM.
It is caused by and weights system.
What means is the ai is adjusted with pixels on image to every image in database and it runs trough all the images and when it failes the weight of pixels get adjusted to match the sourse name then it goes for next and it works untill it perfectly detects all stock images.
And little pixels you do are somehow triggering weighted pixels of other images so more pixels match the other stock image weighted pixels.
You know you're dealing with a man of pure patience, when he didn't deactivate the giant search panel in the task bar.
Seems to me like some pre-processing would help here, like it does with perceptual hashing. To whit, you want images that look the same to be very similar in data output to the net, even if there is minor noise.
The problem is that that's probably not that much of a solution.
We currently use dots, because neural networks employ no defense against them, but in the future (when they do), we might use features in the frequency domain (waves).
The wave form are a one to one equal representation of the image. Couldn't you easily add minor distortions to these waves?
Could you go through the data set used to train the neural network and look at all the golfball pictures for example to compare them to the noise that is interpreted as “golfball”?
What if one of the categories was actually "noise"?
Could you add more noise in order to trick the classifier into being unable to detect the noise?
Could that work in order to detect these attacks?
You definitely can! You can also look at the model's confidence about it's results, since getting being overconfident on a result can be a sign of inputs designed to trick the model (or of issues within the model itself)
This video is poor in the sense that the object is against a white background. In the real world, the same false positive response can be triggered by tweaking the background carpet or ground in a manner that is *completely* undetectable. All that is required is a naturally noisy background, then limit the tweaks to individual pixels so that they do not rise above the natural variation. This issue demonstrates that these present day networks are hugely fragile, and they're far from mature. With a skilled attacker, they can be roundly abused and hacked. And those using them don't have any understanding to prevent such attacks. The whole industry should wake up.
@@JxH it is even less noticeable if you change pixels by only a small amount
@@JxH What are the consequences of such an attack? Like what is an example? What would be the benefit for an attacker? I understand they can be tricked but why would you? Genuinely curious.
@@JxH Agreed. Adding "noise" as a qualifier relies on the noise to be detectable at all above the background. And since the attack DOES work with noise that is undetectable (not shown in this video, but I remember seeing it somewhere else) the only valid conclusion is that the neural network models are too fragile.
One reason of including noise as a category is that 99.99...% of the image space is noise. (Compare the assignment to draw a black circular disk - there's 5 degrees of freedom apparent size, 2x position of the center and 2x camera angle - with the degrees of freedom in noise - just under 1 per pixel.)
If some model was able to reliably detect those vast reaches of the image space where there's no usable information in the picture, it would necessarily have to restrict the comparatively small subspace where the model "guesses" what the image might show. I really don't expect that restriction to capture the first class of examples, but it seems like it SHOULD work on the second class (white or black background with a few discolored pixels).
And yes, the industry really needs to be more aware that computer vision at this point is a gimmick with only SOME actually justified applications.
What immediately got me thinking, was when you said that it has around 1000 categories, and they are not just broad categories, but also fine things like different dog breeds.
That might result in weird thing in itself, mightn't it?
What if there is some animal, that registers around 0.01 in each of the 50 dog breeds (or however many there are) and as 0.015 as a giraffe?
One might argue it should be classified as a "dog (unsure breed)", but if I understand correctly, it will say it's a giraffe
Seems to me like the solution would be to have the categories arranged not in a list, but a tree, so (e.g.) "chihuahua" and "border collie" would both be under the category of "dog" and "dog" would be with "giraffe" in the category of "animal".
But these categorical hierarchies are typically strict, such that each child has exactly one parent category. Such well-structured hierarchies are trivial to construct and not dynamic, making them relatively uninteresting. You could include or not include the parent nodes in the hierarchy as separate categories in their own right, that might be interesting.
can you augment the training data in this way (or during training) to eliminate this / make model more robust? could the same resnet18 be more resillient to this kind of attack?
What if you continuously move on "coffee mug" manifold, starting from 7:53 ? What shape would it evolve? If we arrived a point where a "coffee mug" (according to a human) occurs, it would be nice to see how it evolved and so gain insight on how neural nets perceive a "coffee mug".
Sean you should make an episode on general artificial intelligence and the research on that!
Has there ever been a Computerphile video about Searle's "Chinese Room" thought experiment?
Now what happens if we train the neural network to give similar images similar scores? Will this problem be replaced with a similar problem? For example, if the neural network computes a function f:R^m-->R^n, then we would want to lower Norm(J(f)(x)) where J is the Jacobian. Or we can train the neural network with the slightly modified images as well. Would regularization (such as dropout) also help with this problem?
would this also work if the network was trained with a similarity based loss function e.g. contrastive loss or triplet loss?
Would it be possible to fix these misclassifications but generating this type of failure image, correctly tagging them and feeding them back in? Would the network develop new layers / weights that are resistant to random noise distortions?
When you’re training the networks - put some noise into the training images. Different noise each time - I expect that’ll get you past single-pixel attacks.
This was really interesting! Just curious, say a coffee mug is predicted - wouldn't you be able to utilize the vector information to theoretically draw the edges of the surrounding shape?
If I understood correctly, they’re only optimizing for top category. I wonder what would happen if you try to optimize for a delta on the results (100% category x, as close to 0 as possible for the rest)
I'd like to see that remote control in art gallery with title "99% golf ball".
Would be interresting to try the same trick with two different neuronal networks. I would guess, even small changes in network architecture leed to drastic changes in recognition patterns. Therefore completely different changes should be needed to trick each of them.
Why not add random changes to a few pixels in the training data to make it more resilient to this?
I read a paper a while a go where some students managed to 3d print a turtle that got classified as a gun for the majority of the angles by some network. Seemed like something that we should make sure is not possible before putting any of these systems in a position where they can make any decisions on their own.
I have a feeling that we're missing something. Convolutional neural network have a bunch of nested convolutional layers, followed by a traditional neural network. I think something is missing in between. The convolution performs edge detection, and the rest of the network performs classification. My gut feeling is that we're missing polygon fitting / contour approximation in the middle (approxPolyDP in opencv). When I did shape detection, it was a combination of edge finding (convolution), approxPolyDP, followed by a classifier based on the vectorized contour. This seems to be missing from our deep learning / CNN approach.
The conv layers don't just do edge detection. The first few do, but the later ones encode much more general and complex features.
Which is why safe self driving cars will need more sensors than just cameras.
Is this because the neural network sums the picture up to like a couple of pixels which it compares to be efficient, which doesn't reflect a real picture, although it contains a sum for it. I recall Mike Pound explained something along those lines.
Okay, this makes me wondering something. We, humans, when see an unclear object, sometimes try to recognize other objects in this image and to compare sizes. This way, we rule out rocking chair, because it's too small compared with other objects in that image (suppose there is a lighter on the table next to the sunglasses). So I am wondering if neural network are able to do anything similar to that, or to estimate size or distance to an object displayed, comparatively to other objects present on that picture (which are categorized with greather certanty)? Object size is difficult thing to estimate even from human perspective and it's a matter of experiance. Is it possible to make neural network which would utilize similar in addition to standard approach?
Well you can't isolate what parameters are being used to change the weights in a neural network, we don't really know what it's doing, only that it's trying to maximize a reward function. You could make an artificial intelligence that specifically tries to analyze scale, but that wouldn't be machine learning.
The proper avenue for the future is to include noise as a detectable object in the network - random incremental changes will look like noise, and thus increase the likelihood that the image is noise faster than that the image is of ... a cat.
Does @Computerphile have a merch store? I want to get the image from 6:21 printed on a coffee mug now.
I don't know what you guys were talking about, I think I see exactly how it came to these conclusions.
I think for the envelope one, there was a small thing that looked like the triangle flap of an envelope bottom middle ish.
To me it looks like the network was first initialized with random numbers then trained with a particular set of images and the "noise" we see is just the result of particular neurons being pronounced though that process or reinforcement.
But that is how supervised learning always works. "It's a feature, not a bug."
It does work with humans. I've seen an example where a low res cat picture was changed very slightly to look like a dog. If humans weren't quite so squishy I'm sure you could tailor a few pixel attack for people
"The armed robot was 99% certain this was the suspect, and so it fired the gun. If any of you were 99% certain, wouldn't you?" I recently saw those videos of the little robot dogs with fire arms on em. I'm sure this is some line from the future.
Would these new representations also be recognized as such by other AIs trained with same DB as yours?
What happens if you add these images that are generated into the training data set. Is the resulting new net "better" fitted against such "attacks"?
Yes, we can expect the same perturbated image to be correctly classified by the re-trained model, but as long as the attacker knows the parameters of the model, he can generate new attack images to make the model do wrong predictions.
What happens if you train a neural network on (a lot of) these noisy images? Can you teach it to be resilient to this type of issues?
The algorithms aren't that good if a little bit of noise confuses it and makes it misidentify an object.The algorithm needs an extra step where it runs some sort of denoise filter before attempting to identify objects. You want some way to help extract an object from a (noisy) background before attempting classification.
Really interesting
Can you please link the ‘resnet’ repo/website? I didn’t see it in the description.
Thanks!
set a minimum of say, 10k pixels, and a maximum value the original object can be (so to change a car to a dog, you iterate until you have at least 10k pixels changed, and keep going until car is at most the fifth most likely item)
Some of this is about the volume of training data.
When a young child is out in a pushchair the parent may say "cat" and the child will watch the cat stretch, wash a paw and walk away - how many images of that cat is the child being trained on?
Adults are experts, they have had 18 years of training in using their visual systems. Young children give some insight into how hard it is to classify, and very often have to unlearn things, they get a word "seagull" that is used too broadly and have to learn that "bird" is the generic and "seagull" is only a type of "bird".
Suggestion: a GAN-like architecture where the recognition algorithm is trained to recognize the quality of an image and adapt to noise and artifacts
Is this not basically what GANs (generative adversarial networks) do though? Why is it GANs produce something recognisable to humans but this method does not then?
GANs and adversarial attacks are different although they share the term “adversarial”. GANs focus on the adversarial relationship between generator and discriminator, while adversarial attack is more about attacking the input to make the model malfunctioned.
@@user-og3mi1iv6e I didn't even know this was an adversarial attack! Glad I made the connection between the two though.
@@user-og3mi1iv6e Essentially the only difference is in the generator, the 'generator' in a sense in this model is directly designed to plot random values of noise (random pixel intensities) to trick the discriminator. Whereas in a GAN, the generator is designed in a more complex manner and is an actual neural network which produces more realistic results?
@@nark4837 Yeah! You get it right! Nice description on the aspect of “generator” on both cases, so brilliant!
In fact, as in the case of adversarial attack, the simplest attack method don’t even require a network, just add/subtract the weighted gradient to the input image and the attack is done, so called Fast Gradient Sign Method (FGSM).
Are the pictures edited before or after resizing to 224x224? Because the resizing works by averaging out the pixels, so a small change can effect the average of a block significantly.
It seems to me that the algorithm HAS to classify the image as something. Maybe it’s not 99% sure it’s a golf ball, rather it’s 99% sure it’s not anything else and has no “abstract” or “noise” category.
I wonder how many such spoofs of an image can be created to have wrongly classified. What if we retrain RESNET with those images ?
It seems to me that this system is likely trying to do in one step what we do in several steps - things like recognizing lines, then recognizing objects, then identifying objects ... there's a reason we use so many steps. I expect the first eye-brain interfaces probably tried to do it in one, too, and the more complicated system won, so that strongly suggests the more complicated system is actually important.
So it almost seems like resnet is not looking at the object as a whole to identify it but perhaps a collection of features that together equals a copy machine or whatever. I wonder then if it was possible to identify which parts of the object it is looking at and then just put some duct tape etc. over a few of those spots to see if it completely misidentifies the object after that.
Put the links of the videos you cite so we can go and watch them too!
Can't you use gradient descent to update the image in the most optimal way instead of trying random pixels?
What would happen if you took the newly generated image and put it into a different neural network? Is it likely to be 'confused' the same way?
No. The images are specific to this NN. Ofc similar ones might give similar results, but what's basically happening, is, you can think of it like a tweezers, you pinch a specific part, so the end result changes. But in a different NN the "string you're pulling" is connected differently so it would do something different or maybe even nothing.
@@NGYX2 Thanks! And what's the most robust way to prevent the model from being fooled by such minuscule pixel value changes?
@@Darkev77 I'm just a collage student in the field (so no expert), but working with noise abstraction, or just working with more Data to begin with (higher resolution) can help. Basically, simple NN, simple to "calculate what to do to manipulate".
@@Darkev77 Additionaly as an extreme example, you can specificaly try to fool your network and then add those to you training data to eliminate the ways your network is fooled the easiest. But this doesn't realy work and is very computationaly expensive. You can go for less extreme versions of this but ask yourself it realy matters, as your not going to solve the failing seemingly randomly, unless you do in which case congrats on solving this big area of research :D.
awesome
How do you know which pixels to change and what RGBA value to change them to?
Basically it is an optimization process that can mislead the model to output the target class by adding a perturbation to the original input image.
Because you can see the percentage for all categories, and there are many digits in the percentage, you change a random pixel to a random value, then run it through the system again. If the percentage of your chosen category goes up you keep the change, if it goes down, you don't. Then repeat as many times as is needed.
He made some comment about how they thought they would have to change enough pixels to map it sort of look like the category they were going for, but we're surprised by how little change was needed.
It seems that the learning algorithm has just identified a certain number of pixels that allow the classification of the images.
Ideally one pixel could select between 50% of the categories and if you find 10 independent pixels acting like that you could select between 2^10 categories.
Of course it's probably impossible to have pixels acting ideally and there's some overlapping and the sorting is more blurry. So you actually need 100 pixels you get 99%.
This theory cannot be true for CNNs as they do not purely use local information. Convolutional operations sort of „destroy“ local information. So there is no direct analogy like „it uses that pixel“.
I'm no expert on this field, but there was research recently that overturned a long held assumption about the necessary scale of a neural network. Essentially, by making it an order of magnitude larger than previously assumed necessary, the potential for this kind of misidentification was much reduced. I'll see if I can find it...
Found it. It tackles exactly this issue. Article title is "Computer Scientists Prove Why Bigger Neural Networks Do Better"
7:47 you can see the curved shape at the bottom of the picture, like it was the bottom of a coffee mug pretty close, and from slightly up
The remote control, centered in the image, does look like a coffee cup. The remote makes up the handle and the white background is the cup!
You, sir, have a distinctive talent.