Shocked at the results Meta Llama 3 vs. Microsoft Phi-3 vs. OpenAI ChatGPT 3.5
Vložit
- čas přidán 23. 04. 2024
- I gave the below question to all 2 local large language models (Meta Llama 3 & Microsoft Phi-3) and 1 hosted model (OpenAI ChatGPT) and was shocked at the results
Question: There is a cake on a table in the dininig room, I walk over to the cake and place a plate on top of the cake, I then pick up the plate and take it into the kitchen.
Which room is the cake currently in?
The results surprised me but I think we need to do more testing, drop a comment with your ideas!
*update* I did get around to testing llama 3 over 100 times and can confirm it IS smart 98% of the time. Full video with python code available here • Meta llama 3 unexpecte...
I got the first test answer wrong myself. I thought there would be no cake in any room because when I moved the plate to the kitchen, my dog remained alone with the cake in the dining room.
Can you please do a video on how we can leverage Llama3 to do data analytics, e.g. what’s the top 5 sales in California.
Good
Try the rabbits in the garden exercise: There are two rabbits in the garden. In the garden is a whole. the first rabbit falls in the hole. How many rabbits are left in the garden? response should be still two because the hole is also in the garden. You can give the llm an second chance by explain the hole is also in the garden and that therfore both rabbits are in the garden. Now you tell it that the second bunny also falls in to that hole that is in the garden. It should responds , that both rabbits are in the garden.
Great one! I love it. I was also going to try the “adding doubt” approach to any answer with a followup “are you sure?” to see if it flip flops.
how can train Phi-3 with my data
You have a couple of options you can go with supplying additional context with a RAG and use Python to automate and/or Langchain or you can go the fine tune approach. Checkout hugging face for more details huggingface.co/docs/autotrain/en/llm_finetuning
Can you make a video about that?
I'm a beginner and I don't understand much
@@kingfunny4821 can do! In the meantime If you’re interested in how to interact with Python checkout my latest video czcams.com/video/lPGBLYn2OiE/video.htmlfeature=shared
Why are they being compared to GPT 3.5 rather than 4?
I knew 4 would knock it out of the water, I thought I would give it a chance against 3.5
Because meta is a open source software and open source alwasy behind the closed source bwcause investor like the possibility to earn money from it..
So i love meta for it but we need to realise they will fell behind of other closed source llm
I’ve seem historical errors in Llama3…
It's interesting hey! In my latest video I tested it 100 times. It got it correct 98 times
I interpreted ollamas version as correct. It states a few things there. It says think about it. You placed the plate on top of the cake which means the cake is now on the plate, this is true if cake transferred to the plate when you placed the plate on top of the cake which would happen in real life. We're also assuming the cake has icing icing is sticky bits of cake stick to the icing that stick to the plate. It also very specifically stated in it's next paragraph, (on the plate) meaning the cake that's stuck to the plate is currently located.