Mobileye's quest to put Deep Learning inside every new car

In Amnon Shashua's vision of the future, every car can see.  He's convinced that the key technology behind the imminent driving revolution is going to be computer vision, and to experience this technology, we won't have to wait for fully autonomous cars to become mainstream.  I had the chance to hear Shashua's vision of the future this past Monday, and from what I'm about to tell you, it looks like there's going to be a whole lot of Deep Learning inside tomorrow's car. Cars equipped with Deep Learning-based pedestrian avoidance systems (See Figure 1) can sense people and dangerous situations while you're behind the wheel. From winning large-scale object recognition competitions like ImageNet, to heavy internal use by Google, Deep Learning is now at the foundation of many hi-tech startups and giants. And when it comes to cars, Deep Learning promises to give us both safer roads and the highly-anticipated hands-free driving experience. 

Mobileye's Deep Learning-based Pedestrian Detector

Mobileye Co-founder Amnon Shashua shares his vision during an invited lecture at MIT

Amnon Shashua is the Co-founder & CTO of Mobileye and this past Monday (March 23, 2015) he gave a compelling talk at MIT’s Brains, Minds & Machines Seminar Series titled “Computer Vision that is Changing Our Lives”. Shashua discussed Mobileye’s Deep Learning chips, robots, autonomous driving, as well as introduced his most recent project, a wearable computer vision unit called OrCam. 

Fig 2. Prof Amnon Shashua, CTO of Mobileye

Let's take a deeper look at the man behind Mobileye and his vision. Below is my summary of Shashua's talk as well as some personal insights regarding Mobileye's embedded computer vision technology and how it relates to cloud-based computer vision.

Mobileye's academic roots
You might have heard stories of bold entrepreneurs dropping out of college to form million dollar startups, but this isn't one of them.  This is the story of a professor who turned his ideas into a publicly traded company, Mobileye (NYSE:MBLY). Amnon Shashua is a Professor at Hebrew University, and his lifetime achievements suggest that for high-tech entrepreneurship, it is pretty cool to stay in school. And while Shashua and I never overlapped academically (he is 23 years older than me), both of us spent some time at MIT as postdoctoral researchers.

Deep Learning's impact on Mobileye

During his presentation at MIT, Amnon Shashua showcased a wide array of of computer vision problems that are currently being solved by Mobileye real-time computer vision systems. These systems are image-based and do not require expensive 3D sensors such as the ones commonly found on top of self-driving cars.  He showed videos of real-time lane detection, pedestrian detection, animal detection, and road surface detection. I have seen many similar visualizations during my academic career; however, Shashua emphasized that deep learning is now used to power most of Mobileye's computer vision systems. 

Question: I genuinely wonder how much the shift to Deep methods improved Mobileye's algorithms, or if the move is a strategic technology upgrade to stay relevant in the era where Google and and competition is feverishly pouncing on the landscape of deep learning. There's a lot of competition on the hardware front, and it seems like the chase for ASIC-like Deep Learning Miners/Trainers is on.

The AlexNet CNN diagram from the popular Krizhevsky/Sutskever/Hinton paper. Shashua explicitly mentioned the AlexNet model during his MIT talk, and it appears that Mobileye has done their Deep Learning homework.

The early Mobileye: Mobileye didn’t wait for the deep learning revolution to happen. They started shipping computer vision technology for vehicles using traditional techniques more than a decade ago. In fact, I attended a Mobileye presentation at CMU almost a full decade ago -- it was given by Andras Ferencz at the 2005 CMU VASC Seminar.  This week's talk by Shashua suggests that Mobileye was able to successfully modernize their algorithms to use deep learning.

Further reading: To learn about object recognition methods in computer vision which were popular before Deep Learning, see my January blog post, titled From feature descriptors to deep learning: 20 years of computer vision.

Fig 3. "Deep Learning at Mobileye" presentation at the 2015 Deutsche Bank Global 

Auto Industry Conference.

Mobileye's custom Computer Vision hardware

Mobileye is not a software computer vision company -- they bake their algorithms into custom computer vision chips. Shashua reported some impressive computation speeds on what appears to be tiny vision chips. Their custom hardware is more specific than GPUs (which are quite common for deep learning, scientific computations, computer graphics, and actually affordable). But Mobileye chips do not need to perform the computationally expensive big-data training stage onboard, so their devices can be much leaner than GPUs. Mobileye has lots of hardware experience, and regarding machine learning, Shashua mentioned that Mobileye has more vehicle-related training data than they know what to do with.  

Fig 4. The Mobileye Q2 lane detection chip.

Embedded vs. Cloud-based computer vision
While Mobileye makes a strong case for embedded computer vision, there are many scenarios today where the alternative cloud-based computer vision approach triumphs.  Cloud-based computer vision is about delivering powerful algorithms as a service, over the web.  In a cloud-based architecture, the algorithms live in a data center and applications talk to the vision backend via an API layer.  And while certain mission-critical applications cannot have a cloud-component (e.g., a drones flying over the desert), cloud-based vision system promise to turn laptops and smartphones into smart devices, without the need to bake algorithms into chips. In-home surveillance apps, home-automation apps, exploratory robotics projects, and even scientific research can benefit from cloud-based computer vision.  Most importantly, cloud-based deployment means that startups can innovate faster, and entire products can evolve much faster.

Unlike Mobileye's decade-long journey, I suspect cloud-based computer vision platforms are going to make computer vision development much faster, giving developers a Heroku-like button for visual AI.  Choosing diverse compilation targets such as a custom chip or Javascript will be handled by the computer vision platform, allowing computer vision developers to work smarter and deploy to more devices.

Conclusion and Predictions

Even if you don't believe that today's computer vision-based safety features make cars smart enough to call them robots, driving tomorrow's car is sure going to feel different.  I will leave you with one final note: Mobileye's CTO hinted that if you are going to design a car in 2015 on top of computer vision tech, you might reconsider traditional safety features such as airbags, and create a leaner, less-expensive AI-enabled vehicle.

Fig 5. Mobileye technology illustration [safety.trw.com].

Watch the Mobileye presentation on YouTube: If you are interested in embedded deep learning, autonomous vehicles, or want to get a taste of how the industry veterans compile their deep networks into chips, you can watch the full 38-minute presentation from Amnon's January 2015 Mobileye presentation. 

I hope you learned a little bit about vehicle computer vision systems, embedded Deep Learning, and got a glimpse of the visual intelligence revolution that is happening today. Feel free to comment below, follow me on Twitter (@quantombone), or sign-up to the vision.ai mailing list if you are a developer interested in taking vision.ai's cloud-based computer vision platform for a spin.

Follow @quantombone

You asked, we listened. VMX will be available to run locally.

The following post is a result of my team launching a Kickstarter campaign two weeks ago and upgrading one of our rewards based on all the feedback we received from backers and potential backers.  We initially intended to launch the VMX project as a service meaning that it would only run over an internet connection to our serves.  But there were scenarios where this was not appropriate. Some people didn't have a fast enough internet connection at home, some people were worried that it would be too expensive to use our product, and some people couldn't use software which required an internet connection at work.  The VMX Project, our flagship computer vision in-the-browser software, will not run using a local object detection server.

VMX Project: Computer Vision for Everyone

(Cross-posted from VMX Project Kickstarter January 5, 2013 update and post on blog.vision.ai )

Over the last few weeks, we've listened to many backers (and potential backers) talk about our technology and would like to thank everyone who gave us valuable feedback. Many of you didn’t like VMX being offered only as a service (requiring an internet connection), so we decided to offer a local VMX installation in addition to making VMX available as a service. We didn’t anticipate such great demand for VMX running locally on people’s own computers and networks, but we are dedicated to letting developers have an exceptional computer vision experience and are eager to give our users what they want. 

Once the early-access period (March 2014 - June 2014) is over, VMX developers will have the option to receive a single-machine VMX license and install VMX on their own computer. With VMX running on your computer, you won’t have to worry about running out of VMX Compute Hours, accidentally making your data public, and most importantly: it won’t require an internet connection. You will also have the option of communicating between VMX running on your computer and our servers. You will be able to download object detectors, download the models you create during the early-access period, as well as back-up your object models and import them into the VMX as-a-service servers. 

During our official launch in Summer 2014, a single-machine VMX license will be available to VMX Developers for $100. Kickstarter backers will be able to simply trade-in 100 of their VMX Compute Hours to obtain one single-machine license and download the software for their own use. 

The local VMX software will be installable directly on a computer running Linux. For VMX developers running MS Windows or Apple OS X, we will provide a Linux Virtual Image for download which will contain a pre-installed, and fully configured instance of VMX.

We hope this will make all VMX users more excited about our technology.

--the guys from VISION.AI

The "Blank Check" of Entrepreneurship

Two seconds left.  You can feel your quads burning.  Your form on the ice is rock solid.  As you race past the last defender, the only thing standing between you and the win is the goalie who you’ve beaten one-on-one countless times.  You’ve been practicing for this moment your entire life.

You accelerate towards victory, when suddenly… BOOM. An unanticipated body check, from who else, but Steve Blank. You are slammed against the wall and your entire game-plan is thrown off-course.  S.G. Blank from the opposing team has just issued his signature move, the "Blank Check" -- you know it’s going to hurt tomorrow.  Even minutes after the impact, you can still hear Blank taunting you with his infamous saying, “Get out of the building!”

   

Getting burned on your way to victory is precisely what happens when you try to execute a business plan devised inside your company walls.  As Steve Blank would say, “No business plan ever survives first contact with the customer.” When Steve Blank, author of “The Startup Owner's Manual,” the genius behind the Customer Development process, and entrepreneurship professor at Stanford, gives you a message, he means business.  You better write this one down.

Today’s post is dedicated to his iconic message, “Get out of the building.” Your initial goal as an startup founder is not to execute a business plan.  Thinking you can build it, and then they will come, is thinking inside the building.  You have to go into the world, meet potential customers, talk to them, and learn from these interactions.  This is precisely what getting out of the building means.  You have to search for a business model.  If you plan on scoring without pivots, you are likely planning to fail.

According to Steve Blank, a startup is a temporary organization whose goal is to search for a scalable business plan.  And the magic can only happen when you go outside your comfort zone, when you talk to people.  This is exceptionally difficult for technical founders to grasp.  Dear technical masterminds with an itch for entrepreneurship: “getting out of the building” is the single most important piece of advice taken from all of Steve Blank’s writings.  Years of training, building things, and being on the forefront of technology have likely given you a skewed perspective on what the world wants.  To become a successful entrepreneur, you must first undo the damage of over-education.  Once you learn how the world thinks, your technical talent won’t go away, and you’ll be in a great position to lead a great company.

I wrote this blog post for a few reasons. Foremost, as a first-time startup founder, I've been reading endless books on the subject and writing helps me remember what I learned. I guess this blog is now about computer vision and entrepreneurship. Secondly, this post serves as a note-to-self because I've been guilty of engineering products to death and skipping customer development altogether (as you'd expect from a Robotics PhD).

To learn more about Steve Blank and his ideas on entrepreneurship, the Customer Development process, and Lean startups, take at look at the following resources:

Steve Blank

Steve Blank’s Entrepreneurship Blog (has lots of great video links)
http://steveblank.com/

Steve Blank’s Free Udacity Course “How to Build a Startup” (this was my first ever MOOC!)
https://www.udacity.com/course/ep245

Steve Blank’s book “Four Steps to the Epiphany,” which is the most influential book I read in the past 10 years.  As groundbreaking as Kuhn's “The Structure of Scientific Revolutions” was to science, Blank's "Epiphany" will likely go down in history as the one that changed the course of entrepreneurship.
http://www.amazon.com/Four-Steps-Epiphany-Steve-Blank/dp/0989200507

P.S. Another great book I just finished reading, Made to Stick, contains a similar point referred to as “The Curse of Knowledge.”  The curse of knowledge happens when your message fails to get across because you assumed everybody else is a knowledgeable as you. Or as Steve Blank would say -- you built a feature-rich product flaunting the benefits of advanced features without understanding that the world isn't filled with technical experts. Nobody cares about your features. Not yet. Get out of the building.

VMX: Teach your computer to see without leaving the browser, my Kickstarter project

I’ve spent the last 12 years of my life learning how machines think, and now is time to give a little something back.  I’m not just talking about using computers, nor writing ordinary computer programs.  I’m talking about Robotics, Artificial Intelligence, Machine Learning, and Computer Vision.  Throughout these 12 years, I’ve witnessed how engineers and scientists pursue these problems, at three great universities: RPI, CMU, and MIT.  I’ve been to 11 research conferences, given many talks, wrote and co-wrote many papers, helped teach a few computer vision courses, helped run a few innovation workshops centered around computer vision, and released some open-source computer vision code.

But now, in 2014, most people still struggle with understanding what computer vision is all about and how to get computer vision tools up and running.  I’ve decided that a traditional career in Academia would allow me to motivate no more than a few classrooms of students per year.  A rough estimate of 100 students per year across a 30 year career is a mere 30,000 students.  What about everybody else?  One could argue that some of these students would become educators themselves and the wonderful art of computer vision would reach beyond 30,000.  But I can’t wait.  I don’t want to wait.  Computer vision is too awesome.  I’m too excited.  It's time for everybody to feel this excitement.

So I decided to do something crazy.  Something I wanted to do for a long time, but only recently realized that it would not be possible to do inside the confines of a University.  I recruited the craziest and most bad-ass developer I’ve ever encountered and decided to do the following: convert advanced computer vision technology into a product form that would be so easy to use, a kid without any programming knowledge could train his own object detectors.

I’ve been working non-stop with my colleague and cofounder at our new company, vision.ai, to bring you the following Kickstarter campaign:

VMX Project: Computer Vision for Everyone

What if your computer was just a little bit smarter? What if it could understand what is going on in its surroundings merely by looking at the world through a camera? Such technology could be used to make games more engaging, our interactions with computers more seamless, and allow computers to automate many of our daily chores and responsibilities. We believe that new technology shouldn’t be about advanced knobs, long manuals, or require domain expertise. 

The VMX project was designed to bring cutting-edge computer vision technology to a very broad audience: hobbyists, researchers, artists, students, roboticists, engineers, and entrepreneurs. Not only will we educate you about potential uses of computer vision with our very own open-source vision apps, but the VMX project will give you all the tools you need to bring your own creative computer vision projects to life.

VMX gives individuals all they need to effortlessly build their very own computer vision applications. Our technology is built on top of 10+ years of research experience acquired from CMU, MIT, and Google. By leaving the hard stuff to us, you will be able to focus on creative uses of computer vision without the headaches of mastering machine learning algorithms or managing expensive computations. You won’t need to be a C++ guru or know anything about statistical machine learning algorithms to start using laboratory-grade computer vision tools for your own creative uses.

In order to make the barrier-of-entry to computer vision as low as possible, we built VMX directly in the browser and made sure that it requires no extra hardware. All you need is a laptop with a webcam and a internet connection. Because browsers such as Chrome and Firefox can read video directly from a webcam, you most likely have all of the required software and hardware. The only thing missing is VMX.

We're truly excited about what is going happen next, but we need your help!  Please spread the word, and if you're even mildly excited about computer vision, consider supporting this project.

Thanks Everyone!
Tomasz, @quantombone, author of tombone's computer vision blog

P.S. I'm not telling you what VMX stands for...

What my mother taught me about computer vision

“Wake up, Tomek.  Pack your bags.  We’re moving to America.” 

These were the words my mother whispered into my ear as she roused me from a deep sleep.  There was no alarm clock and no preparation (at least not on my behalf). I was eight years old, and it was a typical January morning in Poland.  It was 1992, and beside a brief venture into Czechoslovakia a few years earlier, I had never left Poland before.

I can still remember those words like they were uttered yesterday.  I remember both the comfort of a child being woken up by the reassuring words of one’s mother as well as the excitement of what those words meant.  It was a matter of hours until I would experience my first international flight, my first multi-lane highway, my first supermarket, and get my first dose of American television.

What I learned from my mother is that sometimes, you just have to pack your bags and go.  That is the lesson my mother taught me, and it wasn’t delivered in the form of a university lecture.  It was an action.  An action that would be the single most influential event in my life.  Moving to the Land of Opportunity from Poland wasn’t something you could not be excited about.

There is a certain kind of excitement that occurs when you make such a bold move in your life.  It requires a certain kind of courage, a certain kind of entrepreneurial spirit.  A certain vision for the future and a certain willingness to take a calculated risk.  A vision that might be filled with uncertainty, but when the uncertainty is drowned by hope, any residual fear just melts away.

My mother never taught me anything about quantum mechanics.  She never provided me with extra tutors hat would one day help me get into a good college, no guidance on how to get into a great PhD program, no etiquette lessons on how to become a respected scientist, etc.  But she gave me the courage and confidence to know that if you want something in life and you have the willingness to pursue it, you can get it. The courage that my mother's actions instilled in me have been more influential in my personal development than any single formal source of knowledge so far.  Thanks mom.

Computer vision is all about the future.  It is all about risks.  It requires a certain entrepreneurial spirit that cannot be attained within the comfy confines of the ivory tower.  I see a world where the way we interact with machines is drastically different than today.  I see a future where we are no longer slaves to our smartphones, where automation will allow us to embrace our human side.  A future where technology will allow us to be free from the worries and stresses which saturate contemporary life.  Computer vision is the interface of the future.  It will allow for both machines to make sense of the world around them, and for us to interact with these machines in a much more intuitive way.

But this sort of change cannot happen without a change in attitude.  As of December 2013, computer vision is simply too academic.  Too much mathematics simply for the sake of mathematics.  Too much emphasis on advancing the state-of-the-art by writing esoteric papers and competing on silly benchmarks.  As a community we have made tremendous advancements, but we have to take more risks.  We have to let go of our egos, and stop worrying about our individual resumes.

I no longer believe that the sort of change I want to see in the world is going to happen by itself.  I want computer vision to revolutionize the way we interact with computers.  I believe in Computer Vision the same way I believed (and still do) about America. Computer vision is the technology of the future, it is the technology of opportunity.  But this cannot happen as long as I continue to portray myself as solely an academic figure.  I know that the way I’m approaching life now is much riskier than getting a traditional job/career in the sciences.  It’s strange to admit that my last day at MIT has been much more exciting for me than my first day at MIT.  I am excited.  My fledgling team is excited.  After our product launch, we’re hoping you will participate in our excitement.  I think the fun times are only beginning. The only limits we have are the ones we impose upon ourselves. 

“Wake up computer vision.  Pack your bags.  You’re moving into everyone’s home.” 

Teaching Computer Vision Innovation

Tinker. Reason. Experiment. Innovate.

This past August I was involved with the MIT Skoltech Innovation Workshop, the second edition of the successful 2012 workshop.  The motto of the workshop was "Tinker. Reason. Experiment. Innovate." and the goal was to get students accustomed to thinking about innovation in an entirely new way.

“Innovating requires behavioral change. This year students have taught us that we can reliably reproduce the experience we created last year, to effect the change in behavior needed to adopt innovation as a way of thinking,” said Dr. Perez-Breva, PhD, who conceived the workshop and directed both editions.

Skoltech.ru

This was the second time I helped out with the workshop, and just like last year I played the role of a technical advisor for several computer vision-focused teams.  As an aspiring computer vision entrepreneur, the experience of instilling each team with expert computer vision and machine learning knowledge was invaluable.  This gave me an opportunity to better understand how a younger generation of students thinks about computer vision as well as gauge their ability to get technology up and running.  Overall I was impressed with student progress over the short final project period and I learned several important lessons which I want to share with you today.

1.) There is a high barrier to entry when it comes to using state of the art object detections.  Many great tools have been produced by the research community and work quite well on standard datasets straight out of the box.  Unfortunately, true innovation requires utilizing computer vision techniques in novel scenarios -- scenarios for which new datasets must be created.  The entire process of creating an object dataset, image labeling, and preprocessing is not as straightforward as you might think.  This means that students lost valuable time just on creating the right input for their object detection systems.  Training object detectors should be easier.  Because iteration is inevitable in most successful projects, we need a faster way of building vision-enabled apps.  I wish there was an interactive and real-time way of training object detectors.

2.) Innovation is all about seeing opportunities where others either see problems or nothing at all.  As a technology advisor I had to remind the students that their goal was to think about product which could change the world in a positive way.  Once a team devised a product for a specific market, I would help with the technology by acting as a hired expert.  Only after the teams were able to generate ideas I would discuss the feasibility of their proposed solution and help them brainstorm ways of improving the underlying technology. This sort of open-ended thinking is generally not taught in a classroom -- where the goal is to work on predefined problems.  As an innovator, you have to come up with the problem, the solution, and convince a broader audience that your solution novel and likely to succeed.  Homework, with its predefined trajectory for success, is the antithesis of innovation. Too much homework and students get accustomed to being assigned defined tasks.  What this means is that to engender a new generation of entrepreneurs we have to give them fewer predefined problem sets, and more open-ended team-based projects.

3.) The project teams responded very well to my enthusiasm and positive reinforcement.  I enjoy working with teams, but in all of my school projects I have always taken the role of lead engineer.  The innovation workshop was my opportunity to work as a coach. I found that I love working with ambitious and talented teams, as they have not yet been perverted by the overly-refined tastes of academics, but are filled with their own dreams of changing the world.  During a one week intense development period, I rationed my time to 30 minutes per team per day.  I did not answer questions over email, and used a timer to make sure each time only got 30 minutes.  I told each team that I could help them with any technical issue they had -- ranging from C++ linking errors to brainstorming about a new machine learning algorithm on the board -- but that it was up to them to decide how to best make use of my time.  This helped each team better utilize a technical advisor's time, as such advising sessions are quite expensive for in the real-world (i.e., in startups).  The teams treated me with a high degree of professionalism, and they were quick to realize on their own that it was not worth using me for low-level coding questions.  It doesn't take much to get me excited about computer vision, and I feel that after most of the advising sessions I was able to significantly raise team spirit.  When teaching or advising, it is important to have the students feel positive about their work after each meeting.  For every discouraging word, it is wise to sprinkle in encouragement and an overall positive attitude towards student progress.  If you're going to teach, provide enough positive encouragement so your students leave your office so energized that they are dying to get back to their research projects.  I have found this teaching innovation experience to be more fulfilling than any of my other teaching experiences to date.

Overall, teaching innovation has helped me realize what is missing in the world of PhD Academic research.  There is big difference between pure research and true innovation, and while those skills are both instrumental in startup success, there is a big difference between being a die-hard researcher and a die-hard entrepreneur.  I now know what my education has been missing. Thank you MIT SkTech Innovation Workshop for the wonderful experience, and helping me refine certain skills which will likely make more more valuable during my own entrepreneurial ventures.

Related Content:

Innovation Workshop 2012: http://mitsloanexperts.mit.edu/luis-perez-breva-prototyping-technology-innovations-tinkering-reasoning-and-experimenting-innovation-is-a-process/

Innovation Workshop 2013: http://web.mit.edu/sktech/news-events/innovation-workshop-2013-retrospective.html