Ashesh Jain

Research Engineer

Zoox Inc

Menlo Park, CA

I am currently a research engineer at Zoox, where I am working with an excellent team to build autonomous vehicles. My expertise are in real-world robotics systems with focus on machine (deep) learning, 3D perception, computer vision, and sensory-fusion.

I obtained PhD in Computer Science from Cornell University in May 2016. I was also a visiting research scholar at the Stanford AI Lab (from 2014-2016) where I started the Brain4Cars and RoboBrain projects. I graduated from IIT Delhi in 2012 with a Bachelors degree in Electrical Engineering.


  • Best student paper award at CVPR 2016 (Deep learning on spatio-temporal graphs)

  • I recently graduated. [PhD thesis, May 2016] [CV].

  • Structural-RNN accepted as an ORAL to CVPR 2016.

  • Our paper on sensory-fusion RNN-LSTM for driver activity anticipation is accepted to ICRA 2016

  • I recently gave talks at Oculus, University of Washington Seattle, Keynote at the ICCV workshop on Autonomous driving, BayLearn Symposium, Qualcomm, and Zoox Labs on: Deep Learning for Spatio-Temporal Problems: On Cars, Humans, and Robots (ppt with videos, 300MB) (pdf, 30MB)

  • Neuralmodels: A deep learning package for quick prototyping of structures of Recurrent Neural Networks and for deep learning over spatio-temporal graphs.

  • Brain4Cars driving data set and sensory-fusion RNN code.

My recent talk at BayLearn 2015 on Brain4Cars


My research interest lies at the intersection of machine learning, robotics, and computer vision. Broadly, I build machine learning systems & algorithms for agents – such as robots, cars etc. – to learn from informative human signals at a large-scale. Most of my work has been in multi-modal sensor-rich robotic settings, for which I have developed sensory fusion deep learning architectures. I have developed and deployed algorithms on multiple robotic platforms (PR2, Baxter etc.), on cars, and crowd-sourcing systems.

Learning From Natural Human Interactions For Assistive Robots

PhD Thesis, Ashesh Jain, May 2016 [PDF]

Journal Publications

Brain4Cars: Car That Knows Before You Do via Sensory-Fusion Deep Learning Architecture

Ashesh Jain, Hema S Koppula, Shane Soh, Bharad Raghavan, Avi Singh, Ashutosh Saxena

Tech Report (under review), January 2016 [arXiv] [Code and Data set]

Learning Preferences for Manipulation Tasks from Online Coactive Feedback.

Ashesh Jain, Shikhar Sharma, Thorsten Joachims, Ashutosh Saxena

IJRR 2015 [PDF]

Conference Publications

Structural-RNN: Deep Learning on Spatio-Temporal Graphs

Ashesh Jain, Amir R. Zamir, Silvio Savarese, Ashutosh Saxena

CVPR 2016 (Full ORAL) (Best Student Paper) [PDF] [arXiv] [supplementary] [Code] [Video]

Recurrent Neural Networks for Driver Activity Anticipation via Sensory-Fusion Architecture

Ashesh Jain, Avi Singh, Hema S Koppula, Shane Soh, Ashutosh Saxena

ICRA 2016 [PDF] [arXiv] [Code]

Car That Knows Before You Do: Anticipating Maneuvers via Learning Temporal Driving Models

Ashesh Jain, Hema S Koppula, Bharad Raghavan, Shane Soh, Ashutosh Saxena

ICCV 2015 [PDF] [Code and Data set] [arXiv]

Brain4Cars: Sensory-Fusion Recurrent Neural Models for Driver Activity Anticipation

Ashesh Jain, Shane Soh, Bharad Raghavan, Avi Singh, Hema S Koppula, Ashutosh Saxena

BayLearn Symposium 2015 [Extended abstract] (Full ORAL)

PlanIt: A Crowdsourcing Approach for Learning to Plan Paths from Large Scale Preference Feedback.

Ashesh Jain, Debarghya Das, Jayesh Gupta, Ashutosh Saxena

ICRA 2015 [PDF]

RoboBrain: Large-Scale Knowledge Engine for Robots

Ashutosh Saxena, Ashesh Jain, Ozan Sener, Aditya Jami, Dipendra K Misra, Hema S Koppula

ISRR 2015 [arXiv]

Anticipatory Planning for Human-Robot Teams.

Hema S Koppula, Ashesh Jain, Ashutosh Saxena

ISER 2014 [PDF]

Beyond Geometric Path Planning: Learning Context-Driven Trajectory Preferences via Sub-optimal Feedback.

Ashesh Jain, Shikhar Sharma, Ashutosh Saxena

ISRR 2013 [PDF]

Learning Trajectory Preferences for Manipulators via Iterative Improvement.

Ashesh Jain, Brain Wojcik, Thorsten Joachims, Ashutosh Saxena

NIPS 2013 [PDF]

SPG-GMKL: Generalized multiple kernel learning with a million kernels.

Ashesh Jain, S. V. N. Vishwanathan, Manik Varma

SIGKDD 2012 [PDF | Code]

Video Demonstration

Brain4Cars: Sensory-fusion deep learning for smart-cars

Interactive human-robot learning from coactive feedback

Anticipating driver maneuvers few seconds in advance

Structural-RNN: Deep Learning on Spatio-Temporal Graphs


  • Invited talk at Oculus (Facebook), April 2016

  • Keynote at the ICCV workshop on Autonomous driving. Title: Deep Learning for Spatio-Temporal Problems: On Cars, Humans, and Robots. Dec 2015

  • Invited talk at Zoox Labs (autonomous driving startup), Dec 2015

  • Talk at University of Washington Seattle Department of Computer Science, Nov 2015

  • Invited talk at Qualcomm Deep Learning Research Center, Nov 2015

  • Oral at BayLearn 2015 on Brain4Cars: Sensory-fusion Recurrent Neural Networks (Video)

  • Invited Talk at RSS Workshop on Model Learning for Human-Robot Communication, July 2015

  • Invited Talk at ICML Workshop on Machine Learning for Interactive Systems, July 2015

  • Invited Talk at ICRA Tutorial on Planning, Control, and Sensing for Safe Human-Robot Interaction, May 2015

  • Invited Talk at IIT Kanpur Department of Computer Science. RoboBrain and Learning from Weak Signals, Feb 2015

  • Stanford Semantics and Geometry Seminar. RoboBrain and Learning from Weak Signals, Feb 2015

  • Stanford Robotics Seminar. Learning from Weak Signals, Nov 2014

  • Introductory talk at LPCHS workshop RSS 2014. Learning from Humans. (Slides)

  • Cornell AI Seminar and ISRR 2013. Beyond Geometric Path Planning. (Slides)

  • ICML Robot Learning workshop 2013. (Slides)

  • Oral at SIGKDD 2012. (Video) (Slides)

  • Invited spotlight at Mysore Park Workshop on Machine Learning 2012. (Video) (Slides)

  • Lecture at Indo-German Winter Acadmey 2010. (Slides)