Open-Source Vision AI - SURPRISING Results! (Phi3 Vision vs LLaMA 3 Vision vs GPT4o)
Vložit
- čas přidán 1. 06. 2024
- Phi3 Vision, LLaMA 3 Vision, and GPT4o Vision are all put to the test!
Be sure to check out Pinecone for all your Vector DB needs: www.pinecone.io/
Join My Newsletter for Regular AI Updates 👇🏼
www.matthewberman.com
Need AI Consulting? 📈
forwardfuture.ai/
My Links 🔗
👉🏻 Subscribe: / @matthew_berman
👉🏻 Twitter: / matthewberman
👉🏻 Discord: / discord
👉🏻 Patreon: / matthewberman
👉🏻 Instagram: / matthewberman_ai
👉🏻 Threads: www.threads.net/@matthewberma...
👉🏻 LinkedIn: / forward-future-ai
Media/Sponsorship Inquiries ✅
bit.ly/44TC45V
Disclosures:
I'm an investor in LMStudio - Věda a technologie
For future test :
1 - Ask unrelated question of a image - [Image of a car] Tell me whats wrong about my bicycle
2 - gradually zoom out of a big chunk of text in a image to see how many word the model can read
3 - A Dense detection task : Describe each element of the object in a json format with a predefine structure
4 - If possible multiple frame from a video to see a glimpse of action understanding
Including object position with x/y or u/v space (0 - 1 per axis), MAybe even bounding boxes.
For future vision tests consider things like:
1) Finding objects - Where is waldo in this picture?
2) Counting Objects - How many bicycles are there in this picture?
3) Identifying Abnormal Objects - How many eggs in this box are broken?
4) Identifying Partially Obscured Objects - Imagine a hand holding cards - What cards are in this poker hand?
5) Identify Misplaced Objects - Which of these dishes is upside down?
What about adding gifs? GPT-4o is also capable of analyzing moving gifs... Not sure on the others, but that adds another dynamic to the tests.
There are quite a nr of these mentioned projects readily available via GitHub projects
It's impressive how open-source image-to-text templates are eventually doing so much better than proprietary and paid ones. 😲
For the captcha gpt4o is clearly the winner. It understands what you mean given the context and doesn’t just repeat all the letters it sees in the image.
That's exactly what I came to say
The question was "what letters are found in this image?", the question wasn't "what letters are found in CAPTCHA field?" Therefore, Phi-3 vision model answered the actual question. GPT4o simply assumed that the task is to break the captcha by reading it. Sometimes less means more, in this case assuming less about the user's intentions would yield better results.
It doesnt matter
@@mrdevolver7999Your cognitive assumption is that it’s right. A random number generator could answer “1 + 1” as being 2, by our pure chance. Therefore, we don’t know if a right answer was a fluke.
Imagine how annoyed you would be, if you ask someone what the letters are and they say 'CAPTCHA'.
8:40 up for interpretation.
"Photos" isnt really a standalone "app" per say, and its not the app itself that is taking up the space, it's the individual jpeg photos, which would take up the same amount of space even if you somehow didnt have the "Photos" app installed anymore.
If a person asked ME that same question, i'd also answer Whatsapp. Since that's something you can tangibly uninstall.
If they asked "what is taking up most space?" the correct answer is "Your photos". But if the question is "what APP is taking up most space", its Whatsapp.
agree
Also with the context of "my space", it's undeniably not photos
It's wrong for a LLM not to at least mention a different possibility if the question seems ambigious.
On the question of the size of the Photos app, GPT noted that 133 GB is larger than the max size of your phone’s storage and thus indicates that it’s possibly using cloud storage and isn’t the actual amount used by Photos on your phone. That was a really perceptive answer, so bonus points to GPT for that 😊 and perhaps that discrepancy is why the other AI seemed to be ignoring the Photos app.
I think if you just consider the output quality GPT-4o is the best.
But if you also take the speed, that fact that phi3-vision is local and open-source into account phi3-vision is the most impressive one.
Pro-tip: Try uploading a photograph you've taken or a work of art into GPT-4o and ask it to behave like an art critic (works great vanilla, but even better with custom instructions).
GPT-4o's ability to dissect the minutia of photography is absolutely wild... even to the point of giving suggestions for improving.
I wonder how long it is until photographers realize what kind of a tool they have available here. I just get a kick out of posting photographs and art and asking for critiques and ratings. It's so, so good.
Oh man, that's actually wild and has a ton of use cases! "Hey GPT, what do you think of this tshirt design I just made for my POD business?" --> *proceed to incorporate the suggestions it has to make a better product* 😮
Really good tests. Thanks a lot.
I love your work. I never miss an episode. I love how you test the LLMs.
The Llama V was probably finetuned for providing verbose descriptions of images. There are other finetuned models that focus on ORC or image labelling
your channel is great, actually useful information, cheers
Nice video these kind of videos I immediately add to my playlist forever
This is really awesome quality content mate, really love your work 😊🚀👍
Very very useful Matt, thanks!
Great test, excellent pace , A+ 🤟
Photos and Apps seem to be distinguished in the storage section of the IPhone. So if you question the largest apps the LLM ignores the photos...
Phi3-Vision is awesome
Please test your meta glasses
I have tried dropping a simple software architecture diagram on them and asking them to extract the entities, hierarchy and connections into a json file, which usually works quite well.
great channel. love your work. FYI - to read a QR code, the QR code must have white space around the outside. This allows the model to pick up on the 3 big position marker squares.
Object counting and the comparison of sizes are always good tests...
Awesome video! I was wondering how Phi-3-Vision fares compared to other vision-capable LLMs. I watched your video while I was working on my own Phi-3-Vision tests using Web UI screenshots (my hope is that it could be used for automated Web UI testing). However, Phi-3 turned out to be horrible at Web UI testing (you can see the video from my tests in my CZcams channel, if you are interested). It's nice to see that it fares much better with normal photos! Thanks for making this video - it saved me some time on testing it myself :)
Thank you!!
Good stuff. Thanks. Results are very prompt dependent
LM Studio is infamously bad for vision.
In order to get it to work you have to follow the following rules:
1. Start a new chat for each photo question.
2. Reboot LM Studio for every photo question.
It’s tedious, but it can start hallucinating after the initial question.
What is LM studio? Interface and UI on top of Ollama, safe to assume this?
@@ayushmishra5861 LM Studio is a stand alone app - Its easy to use and powerful
@@ayushmishra5861 it’s a one click install GUI for working with any LLM model locally with full customization.,I would say it is the best tool available right now.
I have to wonder in the case of the AI messing up the photo app taking the most space. If it recognises your photos are separate from the actual app
Cool. Nice hoodie Matthew.
Nice video. Try testing a video clip and try testing smaller 7B’s vision LLM’s
Thank you for your tests.
Is there a particular HF model you're using. Tested a couple with LM Studio they didn't work 🤔
GPT4o doing the analyzing on the cav prompt wasn't to call up python to look at the image but actually using Python to generate a csv output of the image since you asked for it to make the image data into csv.
Thanks for this video i was planning on downloading both models. You saved me some time. You should try some abliterated models
Thanks and today I had to use vision to troubleshoot router lights. Maybe use photos of real life objects to see how well it can help.
Matt, instability with temp -> 0 is one sign we have an MoE.
Hello Matthew. Do the following test: there are many photos in one directory. Will these LLMs be able to sort photos into folders, depending on their subject matter? For example, photos in nature, photos of a house, photos of animals.
do you have any plans ou tutorial link for phi3 vision local?
Hey Mathew great video. I would test the models to identify elements in a web page to see if they are useful for building browsing agents. Thank you
Matthew, please test handwriting recognition as well when testing image models! This would be a massive usecase useful to anyone who likes to use pen and paper for personal journalling.
hello Methew,
I would be interested to see how well the models transcribe handwritten pages. I think this is an interesting use case for digitizing your own notes. Please add such a case to your Vision test. Thanks!
Maybe try out a (slightly) tilted scanned version of a printed excel document.
One could try adding noise or other means of disturbing the image and testing those disturbed images against the models and see which model handles the best.
I think you need to adjust your System Prompt for Llama to ask for concise answers and for it to ensure it accurately responds to the intended action of the prompt.
I'm wondering if it can read blueprints for both mechanical and electrical schematics and then find the part. I can give you examples if needed.
Hi Matthew
I think a great test which none of the vision models are yet great at is to convert a bit map graph to data.
Eg a stacked bar graph, 3 or 4 series, 2 or more categories
It would be a life changing productivity hack!!
Great channel.
We definitely need some other way to test prompts for the visual models. I would use a long explanation system message to point local LLM to use, scan and convert the picture to a text and use it in order to "read" user prompt with the graphical embedding
Keep in mind that had you run phi-3 vision locally (it is the smallest and easiest to run at only ~4B parameters, and it's open weight) it might have performed better on the identify a person questions, as it seems azure blurs the faces of every image that you upload similar to copilot.
For future tests. Give it a flowchart and have it explain the flow and convert it into code
Nothing to do with this episode in particular but one important question that no one appears to be asking Sam Altman and all the other AI ceos is when will our AI become proactive rather than simply reactive. That will be the next big game changer
Probably never. I'd say that is the first step humans could give to surrender their control over the world and the last step humanity would give as the dominant species too.
@@ronilevarez901 I think input -> output will remain the base function for some time, and that more sophisticated models will be able to accept long-term commands, like constantly checking your calendar and reminding you of dates without having asked for a reminder.
I think it is because of the risk that may pose. but it is worth a try... I would like to say initiative rather the proactivity. However, I get your point
@@marcusmadumo7361 Technically, initiative in AI is called "Agency".
@@marcusmadumo7361 technically, initiative in AIs is called 'Agency'.
Where are the links on models?
I've actually had pretty good luck with llama 3 dolphin. I tried using the lava variant and I came up with kind of the same results.
Think you need to update that AI test soon! :D
GPT-4o did very well on the iPhone questions. I wonder if it had some specific training related to OpenAI's partnership with Apple.
Can the models produce an specific json shape, for instance the iphone screenshot, can we generate a json with the app name, and GB of storage, percentage taken, percentage left etc? Thanks for the content very cool, I alway watch your videos
Ask them to interpret maps. For example, is there a park nearby? What is the name of the park? How about a school etc. This is useful for real estate.
It would be interesting to see if they can explain some software architectural diagrams. Some entity relationship diagrams and such.
What laptop are you using where you are able to load 14gb of model weights without quantizing it into vram?
That is a pretty odd (incorrect) title. GPT4o is NOT Open Source. Maybe you meant Open Source models compared to GPT4o?
Are any of them open source in the truest sense of being able to modify both the code and the weights and biases of the model to alter it's behaviour in a very specific and directed way?
You can never judge the vision capability of a model merely based on the description or detection, it should also be able to localise the objects with good precision, which is where most models fail .
The image descriptions of Llava are perfect input for an image generation LLM. LLava's output sounds very much like the results from "/desc" command of MidJourney. I am pretty sure that if you put the output of Llava as input into MidJourney, you get nearly exactly the same image.
The ui recognition stuff is interesting from a designing agents point of view.
How about estimate the coords of a certain point? Useful for mouse clicking
Some ideas for tests:
1) When providing an image of a small maze, can it describe the shortest route ?
2) What if you provide a map of a maze and a 2 d image of a vehicle as seen from above. The maze has some paths where the vehicle can't pass as these.paths are too narrow
3)Can it calculate the area of a geometric figure when you provide an image of the sizes?
4) what if you provide an irregular figure and ask the area.
I’m envious of Matthew’s AI hair stylist. Only joking Matthew. Love your videos 😊
It would be interesting to have each of the models assess both their own answers and competing models answers to see if they accurately answered the question.
@Matthew Berman maybe you could have included Gemini Advanced (which has Gemini 1.5 pro)
That is very interesting that for a lot of use cases the free open source version does better.
Which is good because generally with a lot of things to do with image recognition you want to do it locally instead of pinging an api.
I guess one way to look at it could be that 4o is better at understanding details but the open source models are better at description?
When working with data on chat-gpt 4o, try to instruct it not to use code interpreter.
I use it to the openai api to OCR purchase orders, I get better results when I instruct it not to use code interpreter.
Would be good if you presented a scoreboard at the end based on your testing tactics
Ask if it can measure distances between points in a photographic image, not in pixels but in the actual length of the object like for example the side of a building from different angles see if it can give a reasonable estimate of it's size if not an actually accurate figure. I'm also interested if they could convert an image of some object into a CAD file as well but I suspect they weren't trained like that.
Can you try uploading a graph picture (such as CPI) YoY line graph and ask the models How they understand It? Which average pace it's moving? Which date intervals it's growing or reducing?
Phi3 did the best. Another note... I used GPT4O to write a lesson plan for my engineering courses. Instead of working hard, I uploaded several pieces of data in different formats. In one portion of the prompt, I uploaded a pdf, a word DOC an XLS. I asked it to provide response and then output in XLS, PDF and CSV formats. The first time through, GPT4O worked perfectly. The second time through, I had to start over. Kinda weird, but all in all the models are getting really good. It also helps that I'm an engineering teacher that loves to watch Matthew Berman.
@Matthew Berman - Maybe you could have it analyze a photo of a spreadsheet with color formatting and some obvious calculations (averages or sums at the bottom) and see if it can create an excel or google sheets file with the data, color formatting, and calculations. Another one (more of a stretch) - give it a photo and return whether it is AI generated or not. Or alternately, whether it was modified in some way (say by photoshop)
GPT40 is getting hammered this days, I wonder if that has any effect on the results it spits out sometimes. Really great new set of tests!
Can you make avideo those models challenged with udestanding charts, graphs, diagrams etc. And how well those models extract meaning data from this kind of images? Are rthose models capable to i.e. understand an UML style activity diagram?
Nice! It would also be good to see if it can read and solve a formula--something with a summation or integral and divide that is hard to express with pure ascii.
Edit: Models are also bad at analyzing music notation. You could take 2-4 bars from a Bach Chorale arranged for keyboard/choir and asked about pick-up notes, time signature, key and chord progression like a quiz from a first year music theory class.
I tried searching for the model in lm studio and couldn’t find it! Where can I download it from?
With that converted_table.csv, it's only using Code Interpreter to do the final conversion from a table to csv. If you just ask for the table I think it will just output the data and not use Code Interpreter.
Maybe try to use them for reading a webpage and scraping the right data from it
Gpt4o seems to have the best understanding of 3D physical space, including direction, coordinates, mass, speed, collision, risk avoidance, obstacles, etc.
Llama3/llava seems to be trained to describe images for low vision/blind people. Maybe it needs a bit of prompt tweaking to be less verbose.
Quite impressed with phi-3 vision, gonna get that installed.
Also, there's PaliGemma from google which is really amazing but it's going under the radar because it's not accesible in LM studio and Ollama (yet?). There's a HF space with a demo, it can describe images, OCR, segmentation masks...
Tried to install it locally by cloning the repo but i failed.
I think the best vision model is CogVLM based non llama3-8B
I would like to see for the vision tests a form that is filled out with both text and handwriting. Tell it to create a organized and well formed JSON object using the data found on the filled in form.
For the GPT-4o testing, I think there may be some personalization settings that may be affecting the results. The responses seem too succinct to me.
What is the name of the bright circle around your mouse?
I think for whatever reason, LM Studio isn't sending the text with the image to the model. If you send an image, it's going to always ignore any text prompt you send along with it. Not sure if that's for every model, but you should probably consider sending the image, and then sending the text separately.
here is what i found. if you copy the vision model file into a folder with an uncensored model and rename the vision model file matching the model. it will load and work much much better. i tested it with the dolphin 2.9 model.
thnx for sharing matthew.
actually chatgpt took the crown here
System requirements for these models?
thx, curious how it handles written text
I test vision models by showing them an obscure character from Super Paper Mario. They don’t usually get it correct and it’s probably not the best way to test them.
Geoguessr (or pics from Google Street View from various areas) would be a good test I think. I had very good luck with it using 4o.
Hi, could you test a chessboard description ? Why not asking what would be the best next move. Thanks
For bing/copilot faces are automatically blurred before the image is shown to the model. Might be the same with phi3 on azure. Just something to keep in mind.
Is it possible to do full OCR usimg these model??? 🤔
Thanks for the great video! It would be great to show the vision model a screenshot of the game pong or the game snake and ask to write the python code to make the game as shown on the image
Would have been cool to compare with pali Gemma
I've used Claude to get help with Unity and I sent him a lot of screenshots, and he was amazingly good!
Gpt and CoPilot failed miserably instead 😅
Its much fair if you test all these three on azure ai platform to get fair results (yes, both llama3 and gpt40 are available)
RE: Visio test suggestions
1) Use an image of a CAPTCH requiring the user to use a slider to move a puzzle piece and ask for instruction on how to solve it
2) Use an optical illusion image and ask a relevant question
3) Use an image to ask it to create 3 question about the image you can use to test someone's vision logic
I'm not sure that QR code if fully standard compliant. I had to do some clean-up on Gimp to get my phone to be able to read it. Mainly it seems the two biggest issues are, first the way the 3 big squares are colored (changing the 3 layers around, from originally dark gray, black, dark gray; to black, white, and black made it look more normal; but there''s something that is still kinda off, QR codes are supposed to work both normal and inverted, but I could only make my phone read it if I de-inverted it (so that be 3 big solid squares are black instead of white, and the 3 surrounding layers the opposite of what I previously described), the more common visual), I'm not sure what's wrong with it that is making it not be recognized in the inverted form. I'm not quite sure how much it can be attributed to failure of the QR code generator, as failures of the more common QR code reader libraries, or even ambiguities of the standard itself though; I've only got a very superficial familiarity with the standard.
Here's one for you to use - can any of them say what the X,Y pixel position of something is within the image? like an icon
Please test if it can decode Steganography images.
You can ask the AI not to editorialize and give a clear descriptions. It works.
And one answer you got you thought it should say the Photos app. It might have not considered Photos an app. Just the memory used for your photos.
What was temperature set at for Llama?
What is temperature in AI context, and what difference does it make?
Please test their ability to analyze screenshots of web pages and determine mouse coordinates to click for potential web automation applications. For example, show a website and ask, "What coordinate should I click to log in?"