RI Seminar: Dieter Fox: Where's RobotGPT?

www.ri.cmu.edu/event/ri-semin...
Dieter Fox
Professor, University of Washington
Senior Director of Robotics Research, NVIDIA
Where is RobotGPT?
Abstract:
The last years have seen astonishing progress in the capabilities of generative AI techniques, particularly in the areas of language and visual understanding and generation. Key to the success of these models are the use of image and text data sets of unprecedented scale along with models that are able to digest such large datasets. We are now seeing the first examples of leveraging such models to equip robots with open-world visual understanding and reasoning capabilities. Unfortunately, however, we have not achieved the RobotGPT moment; these models still struggle with reasoning about geometry and physical interactions in the real world, resulting in brittle performance on seemingly simple tasks such as manipulating objects in the open world. A crucial reason for this problem is the lack of data suitable to train powerful, general models for robot decision making and control.
In this talk, I will discuss approaches to generating large datasets for training robot manipulation capabilities, with a focus on the role simulation can play in this context. I will show some of our prior work, where we demonstrated robust sim-to-real transfer of manipulation skills trained in simulation, and then present a path toward generating large scale demonstration sets that could help train robust, open-world robot manipulation models.
Bio:
Dieter Fox is Senior Director of Robotics Research at NVIDIA and Professor in the Allen School of Computer Science & Engineering at the University of Washington, where he heads the UW Robotics and State Estimation Lab. Dieter’s research is in robotics and artificial intelligence, with a focus on state estimation and perception applied to problems such as robot manipulation, mapping, and object detection and tracking. He has published more than 200 technical papers and is the co-author of the textbook “Probabilistic Robotics”. He is a Fellow of the IEEE, AAAI, and ACM, and recipient of the 2020 IEEE Pioneer in Robotics and Automation Award and the 2023 John McCarthy Award. He was an editor of the IEEE Transactions on Robotics, program co-chair of the 2008 AAAI Conference on Artificial Intelligence, and program chair of the 2013 Robotics: Science and Systems conference.

[@yuehchuanjohnson4262]

🎉🎉🎉

[@andrzejreinke]

are the sldies avaialble?

[@cmurobotics]

No, but this video is available. ;)

[@andrzejreinke]

@@cmurobotics it's hard to copy all references and google them :D

[@williamjmccartan8879]

Just a little update, Sanctuary AI, has just announced collaboration between Microsoft, Accenture, Nvidia, Magna International, and the release of iteration number 7 of the Phoenix model, I'm going to guess that this Canadian company is going to be the one to keep your eyes on, thank you for sharing your time and work Dieter, this was really interesting and helpful understanding some of the challenges that the industry is facing, thank you for sharing this material online CMU, this information is important for everyone to be aware of and the speed at which its happening, peace

[@aryangod2003]

Beware of Sanctuary! Sanctuary is a is a FRAUDULENT Company, venture capital scam. I have a Masters in Artificial Intelligence (University of washington), I know of the founder Geordi Rose, he has been implicated in DWave "Quantum computer"- which is widely thought by experts not to work. He is a talented serial Fraudster There is not a Single AI specialist in their team no academic pedigree from Stanford, MIT, Carnegie Mellon, Berkeley. Beware of this Company