Sign Language Recognition with esp32-cam using tflite
Vložit
- čas přidán 20. 08. 2024
- I implement a tensorflow lite (tflite) model on the ESP32-cam for American Sign language. The model is a stand-alone model and requires no network connection to run. The projects uses a pcb from pcbway to connect the esp32-can to a tft lcd screen, thus eliminating much of the messy wiring. #esp32cam #tflite
Notes:
I'm using board version 2.02 of the esp32. The higher versions had a bug with the TFT_eSPI library for outputting jpeg files. Although, I didn't end up using the jpeg files.
The pin connections for the esp32 and tft are below. Note: if you are using the TFT_eSPI library, you need to define the pin connections in the User_Setup.h file,
TFT_MISO 13
TFT_MOSI 12
TFT_SCLK 14
TFT_CS 15 // Chip select control pin
TFT_DC 2 // Data Command control pin
TFT_RST 16 // Reset pin (could connect to RST pin)
//LED pin connected to 5v.
github:
github.com/jon...
Kaggle dataset:
www.kaggle.com...
Kaggle notebook:
www.kaggle.com...
pcbway:
pcbway.com/g/Q...
person detection training instructions:
github.com/ten...
tflite instructions for microcontrollers:
www.tensorflow...
Converting tensorflow to tflite interesting video:
• Convert Tensorflow/Ker...
Thank you mastaruu
Impressive 👏
Thank you, for the kind comment.
I am now getting this error when I try to upload the model onto my esp32s3 error: 'static void tflite::MicroErrorReporter::operator delete(void*)' is private within this context
58 | error_reporter_ = new (micro_error_reporter_buffer) MicroErrorReporter();
this is also the same case for the layers and all the lines that have to do with a micro error reporter. I am confused please help
Can you upload other sketches or is it just this one
Can you tell which tensorflow version using for notebook and Arduino
Hi. Sorry, I've updated it a few times since the project. It would've been something like 2.15 (?)
@@jonathanr4242 hey sir, i try use code in time 29.05 calloc function but this error"exit status 1 invalid conversion from 'void*' to 'uint8_t*' {aka 'unsigned char*'} [-fpermissive] Invalid". Can you help me to fix it, thanks !!!
Great work. Keep it up!
Thanks!
Impressive stuff!
Thank you! Chris!
a software guy once tried to design a PCB and this is the result
😂😂😂
Hi, great job ! I saw your repo and I observed that the American Sign Recognition's model is the same than Person Detection's model, and in my Arduino project (using input->dims->size and output->dims->size), I get 4 and 2. I'm not sure what I'm saying but in case you made a mistake, could it be possible to get the right model ? Thank you !
Hi. thanks for your comment. Its been a while since I worked on that. I remember having to play with those numbers a bit, but it's difficult for me to remember exactly what I did. I'm getting old.
hi, a question, do you think is possible to do object recognition on esp32? with tflite?
Yes absolutely. You can also try edge impulse, which should generate some code that you can load straight onto the esp32.
I'm trying to take an image via esp32 cam. I will do object recognition. But I have a problem. I encounter such an error: ValueError: Input 0 of layer "sequential_2" is incompatible with the layer: expected shape=(None, 240, 240, 3), found shape=(None, 240, 3). I don't know what I should do. can you help me ?
Hi. The best way to debug an error like this is to print out the shapes of all your arrays at each step. For an image should be (None, 240, 240, 3).
Impressive
Thank you.
I find this camera to be really slow. I always thought it was a tcp limitation until watching this video. I not sure why a USB web camera(serial) is so much better than these parallel cameras. Any ideas?
I think it depends on the microprocessor. With the esp32-cam I would assume most of the latency is copying the image into memory and doing jpeg compression. Atomic14 has some videos where he speeds this up (I think?)
hello sir! may i know what are the specifications of each component you've used? thanks a lot!
I'm using an esp32 AI thinker cam and a TFT ILI9341 screen. I think it's 2.8 inches. I made some pcbs to connect the esp32 AI thinker to the TFT. If you email me your address, I can send you one I have left over, or you can get it made by pcb way.
@@jonathanr4242 i appreciate it sir! thank you! what is your email address so we can be in contact? this is for our research study and we would like to have a sign language translator in our own language with the idea of your work. is it possible that we could use a 7 inch tft display? would it work the same on the esp32 that's somehow compatible? thank you a lot again
btw, is it possible to connect it to a 7" tft lcd? or even use a ch340 esp32? thank you
@@t22marg90I don't see any reason why not, as long as you have a camera.
Very informative Video, Thank you for this excellent video small doubt can I know the TensorFlowLite_ESP32 version in Arduino.
I was using 1.0.0
i use esp32-cam AI thinker to upload your code, but problem is Camera init failed with error 0x105ets Jun 8 2016
ok. these errors are very vague. It looks like the camera isn't working.
impressive 2:45
Thanks!
Cops need this immediately please contact someone I have seen them beat the crap out of a deaf person
I think it would be good for a lot of people to get access to technology.
hey sir, i try use code in time 29.05 calloc function but this error"exit status 1 invalid conversion from 'void*' to 'uint8_t*' {aka 'unsigned char*'} [-fpermissive] Invalid". Can you help me to fix it, thanks !!!
Hi. Thanks for you question. The error messages can be a bit vaque. It's a bit hard to know what's going on. Are you using my trained network or did you train your own?
@@jonathanr4242 thanks for answer my question, sir. But i solve it, but when i run your code example camera capture error, can i have send you image about error for you through Instagram 🤔🤔
@@jonathanr4242 i have only esp32-cam but i want test in your code and test img, how can i do it to display result of prediction in monitor of arduino ide, sir
11:39:00.626 -> E (506) camera: Camera probe failed with error 0x105(ESP_ERR_NOT_FOUND)
11:39:00.626 -> Camera init failed with error 0x105ets Jun 8 2016 00:22:57
@@jonathanr4242
@@nguyenquoctoan4024Hi, I think that means that your camera is not working. The first step is to get the camera to display to the output tft.
Is there any way to use laptop screen instead of TFT display?
Yes. Absolutely. You can stream the image from the esp32. Or you could flick the esp32 & just use the laptop camera and processor which would be faster and more accurate. But less difficult and fun to implement.
Can you show me the way to do this? I don't have a TFT display but want to show the output on my laptop screen. What should i modify in your code?
@@s.mtaron4908 you mean with the esp32 cam or do you want to use a Web cam?
@@jonathanr4242 just as u did using the esp32cam but instead of the TFT display I want to show the output on my laptop.
@@s.mtaron4908 Hi. In this project here, I use websocket for the person detection part to output to a laptop. github.com/jonathanrandall/xiao_vs_aithinker
Thank you mastaruu