Swift MLX Chat with Local LLM macOS Application Short Demo | Mistral | LLaMA | SwiftUI
Vložit
- čas přidán 22. 02. 2024
- This is a demonstration video of a macOS Application that utilizes Apple MLX Framework to chat local LLM Models such as Mistral & Code LLaMA #mlx #llm
Check the GitHub Project repository
github.com/alfianlosari/mlx-s... - Věda a technologie
Source code on GitHub Project Repository:
github.com/alfianlosari/mlx-swift-examples
Thank you so much for your hard work, you are as always one of the first to show new technology.
This is awesome! Thanks for sharing the swift implementation. Just had Gemma run on Mac using MLX. Thanks to you all :)
nice :D
(I have not written Mac OS apps before, would love to learn how to create LLM apps and looking forward to the video)
@@AndrejKarpathy your humble nature of not only teaching the best and looking forward to learn from all is very inspirational. 🙏
Waiting for tutorial!
great demo can you show how to run llama3 on iOS ?
Do you have any plans to create a tutorial for using the Claude API to make an app? Claude's features are amazing.
Hi nice suggestions. Will definitely explore Claude API
Great video! Curious how this is different from running the model in CoreML, like say from llama.cpp?
MLX can be used by AI researchers for developing and deploying ML Models from scratch. It is also optimized for M-Series Apple Silicon Chip. It is inspired by PyTorch, Jax, and other popular frameworks.
I think CoreML is more suited for converting and optimizing existing ML models so it can be used on Apple devices.
I think maybe MLX is more performant and efficient compared to CoreML as it is optimized for M series, the initial benchmark shows that it outperforms PyTorch.
Awesome! Terima kasih! Can’t wait for the tutorial. Also are you able to also hook up a local whisperer as well? So everything is local?
Yes, everything is local running on Apple Silicon. I only tried Mistral7B & CodeLLaMA, the MLX documentation says is supports: LLaMA, Mistral, Gemma, and Phi.
Thanks
You’re welcome!
What is the spec of you mac system, do you need a powerful machine to run it?
I'm using MacBook Pro 14 2021 with M1 Pro & 16GB of RAM. If you use LLM with 2b of params, 8GB RAM should be enough. For 7b LLM, you will need at least 16GB RAM
Crazy, would it be runnable on iOS?
Currently no, also the RAM on iOS is too small for model with 7B params. On my mac, Mistral7B uses around 10GB of RAM.
Can you share the code of App?
Here you go! github.com/alfianlosari/mlx-swift-examples
@@XCA Thank you
Thank you so much for your effort. Btw I am getting an error
"Internal: Could not parse ModelProto from `/Users/xxx/Mistral-7B-v0.1-hf-4bit-mlx/tokenizer.model`"
Do u know how to fix it?