Three Fundamental Dimensions for Thinking About Machine Learning Systems

Today, let's set cutting-edge machine learning and computer vision techniques aside. You probably already know that computer vision (or "machine vision") is the branch of computer science / artificial intelligence concerned with recognizing objects like cars, faces, and hand gestures in images. And you also probably know that Machine Learning algorithms are used to drive state-of-the-art computer vision systems. But what's missing is a birds-eye view of how to think about designing new learning-based systems. So instead of focusing on today's trendiest machine learning techniques, let's go all the way back to day 1 and build ourselves a strong foundation for thinking about machine learning and computer vision systems.

Allow me to introduce three fundamental dimensions which you can follow to obtain computer vision masterdom. The first dimension is mathematical, the second is verbal, and the third is intuitive.

On a personal level, most of my daily computer vision activities directly map onto these dimensions. When I'm at a coffee shop, I prefer the mathematical - pen and paper are my weapons of choice. When it's time to get ideas out of my head, there's nothing like a solid founder-founder face-to-face meeting, an occasional MIT visit to brainstorm with my scientist colleagues, or simply rubberducking (rubber duck debugging) with developers. And when it comes to engineering, interacting with a live learning system can help develop the intuition necessary to make a system more powerful, more efficient, and ultimately much more robust.

Mathematical: Learn to love the linear classifier

At the core of machine learning is mathematics, so you shouldn't be surprised that I include mathematical as one of the three fundamental dimensions of thinking about computer vision.

The single most important concept in all of machine learning which you should master is the idea of the classifier. For some of you, classification is a well-understood problem; however, too many students prematurely jump into more complex algorithms line randomized decision forests and multi-layer neural networks, without first grokking the power of the linear classifier. Plenty of data scientists will agree that the linear classifier is the most fundamental machine learning algorithm. In fact, when Peter Norvig, Director of Research at Google, was asked "Which AI field has surpassed your expectations and surprised you the most?" in his 2010 interview, he answered with "machine learning by linear separators." 

The illustration below depicts a linear classifier. In two dimensions, a linear classifier is a line which separates the positive examples from the negative examples.  You should first master the 2D linear classifier, even though in most applications you'll need to explore a higher-dimensional feature space. My personal favorite learning algorithm is the linear support vector machine, or linear SVM. In a SVM, overly-confident data points do not influence the decision boundary. Or put in another way, learning with these confident points is like they aren't even there! This is a very useful property for large-scale learning problems where you can't fit all data into memory. You're going to want to master the linear SVM (and how it relates to Linear Discriminant Analysis, Linear Regression, and Logistic Regression) if you're going to pass one of my whiteboard data-science interviews.

Linear Support Vector Machine from the SVM Wikipedia page

An intimate understanding of the linear classifier is necessary to understand how deep learning systems work.  The neurons inside a multi-layer neural network are little linear classifiers, and while the final decision boundary is non-linear, you should understand the underlying primitives very well. Loosely speaking, you can think of the linear classifier as a simple spring system and a more complex classifiers as a higher-order assembly of springs.

Also, there are going to be scenarios in your life as a data-scientist where a linear classifier should be the first machine learning algorithm you try. So don't be afraid to use some pen and paper, get into that hinge loss, and master the fundamentals.

Further reading: Google's Research Director talks about Machine Learning. Peter Norvig's Reddit AMA on YouTube from 2010.
Further reading: A demo for playing with linear classifiers in the browser. Linear classifier Javascript demo from Stanford's CS231n: Convolutional Neural Networks for Visual Recognition.
Further reading: My blog post: Deep Learning vs Machine Learning vs Pattern Recognition

Verbal: Talk about you vision (and join a community)

As you start acquiring knowledge of machine learning concepts, the best way forward is to speak up. Learn something, then teach a friend. As counterintuitive as it sounds, when it comes down to machine learning mastery, human-human interaction is key. This is why getting a ML-heavy Masters or PhD degree is ultimately the best bet for those adamant about becoming pioneers in the field. Daily conversations are necessary to strengthen your ideas.  See Raphael's "The School of Athens" for a depiction of what I think of as the ideal learning environment.  I'm sure half of those guys were thinking about computer vision.

The School of Athens by Raphael

An ideal ecosystem for collaboration and learning about computer vision

If you're not ready for a full-time graduate-level commitment to the field, consider a.) taking an advanced undergraduate course in vision/learning from your university, b.) a machine learning MOOC, or c.) taking part in a practical and application-focused online community/course focusing on computer vision.

During my 12-year academic stint, I made the observation that talking to your peers about computer vision and machine learning is more important that listening to teachers/supervisors/mentors.  Of course, there's much value in having a great teacher, but don't be surprised if you get 100x more face-to-face time with your friends compared to student-teacher interactions.  So if you take an online course like Coursera's Machine Learning MOOC, make sure to take it with friends.  Pause the video and discuss. Go to dinner and discuss. Write some code and discuss. Rinse, lather, repeat.

Coursera's Machine Learning MOOC taught by Andrew Ng

Another great opportunity is to follow Adrian Rosebrock's pyimagesearch.com blog, where he focuses on python and computer vision applications.  

Further reading: Old blog post: Why your vision lab needs a reading group

Homework assignment: First somebody on the street and teach them about machine learning.

Intuitive: Play with a real-time machine learning system

The third and final dimension is centered around intuition. Intuition is the ability to understand something immediately, without the need for conscious reasoning. The following guidelines are directed towards real-time object detection systems, but can also transfer over to other applications like learning-based attribution models for advertisements, high-frequency trading, as well as numerous tasks in robotics.

To gain some true insights about object detection, you should experience a real-time object detection system.  There's something unique about seeing a machine learning system run in real-time, right in front of you.  And when you get to control the input to the system, such as when using a webcam, you can learn a lot about how the algorithms work.  For example, seeing the classification score go down as you occlude the object of interest, and seeing the detection box go away when the object goes out of view is fundamental to building intuition about what works and what elements of a system need to improve.

I see countless students tweaking an algorithm, applying it to a static large-scale dataset, and then waiting for the precision-recall curve to be generated. I understand that this is the hard and scientific way of doing things, but unless you've already spent a few years making friends with every pixel, you're unlikely to make a lasting contribution this way. And it's not very exciting -- you'll probably fall asleep at your desk.

Using a real-time feedback loop (see illustration below), you can learn about the patterns which are intrinsically difficult to classify, as well what environmental variations (lights, clutter, motion) affect your system the most.  This is something which really cannot be done with a static dataset.  So go ahead, mine some intuition and play.

Visual Debugging: Designing the vision.ai real-time gesture-based controller in Fall 2013

Visual feedback is where our work at vision.ai truly stands out. Take a look at the following video, where we show a live example of training and playing with a detector based on vision.ai's VMX object recognition system.

NOTE: There a handful of other image recognition systems out there which you can turn into real-time vision systems, but be warned that optimization for real-time applications requires some non-trivial software engineering experience.  We've put a lot of care into our system so that the detection scores are analogous to a linear SVM scoring strategy. Making the output of a non-trivial learning algorithm backwards-compatible with a linear SVM isn't always easy, but in my opinion, well-worth the effort.

Extra Credit: See comments below for some free VMX by vision.ai beta software licenses so you can train some detectors using our visual feedback interface and gain your own machine vision intuition.

Conclusion

The three dimensions, namely mathematical, verbal, and intuitive provide different ways for advancing your knowledge of machine learning and computer vision systems.  So remember to love the linear classifier, talk to your friends, and use a real-time feedback loop when designing your machine learning system.

Barcodes: Realtime Training and Detection with VMX

In this VMX screencast, witness the creation of a visual barcode detection program in under 9 minutes. You can see the entire training procedure -- creating an initial data set of labeled barcodes, improving the detector via a 5 minute interactive learning step, and finishing off with a qualitative evaluation of the trained barcode detector.

The inspiration came after reading Dr. Rosebrock's blog post on detecting barcodes using OpenCV and Python (http://www.pyimagesearch.com/2014/11/24/detecting-barcodes-images-python-opencv/).  While the code presented in Rosebrock's blog post is quite simple, it is most definitely domain-specific.  Different domain-specific programs must be constructed for different objects.  In other words, different kinds of morphological operations, features, and thresholds must be used for detecting different objects and it is not even clear how you would construct the rules to detect a complex object such as a "monkey."  If you are just getting started with programming and want to learn how to construct some of these domain-specific programs, you're just going to have to subscribe to http://www.pyimagesearch.com/.

Writing these kinds of vision programs is hard.  Unless... you address the problem with some advanced machine learning techniques.  Applying machine learning to visual problems is "the backbone" of what we do at vision.ai and computer vision research has been a personal passion of mine for over a decade.  So I decided to take our most recent piece of vision tech for a spin.  We try not to code while on vacation (a good team needs good rest), and I don't consider using our GUI-based VMX software as hardcore as "coding."  Unlike traditional vision systems whose operation might leave you with an engineering-hangover, using VMX is more akin to playing Minecraft.  I figured that playing a video game or two on vacation is permissible.

Eliminating the residual sunscreen from my hands, I rebooted my soul with an iced gulp of Spice Isle Coffee and fired up my trusty Macbook Pro.  I then grabbed the first few vacation-themed objects from the kitchen. (And yes, I'm on vacation for Thanksgiving -- the objects include canned fruit, sunscreen, and a bottle of booze.)  Then it was time to throw the barcode detection problem at VMX.

Step 1: Barcode Initial Selections

30 seconds worth of initial clicks followed by several minutes worth of waving objects in front of the webcam is not hard work.  5 minutes later we have a sexy barcode detector.  Not too bad for computer vision in a non-laboratory setting.  While on vacation, I don't have access to a lab and neither should you.  A sun-filled patio will have to suffice.  In fact, it was so bright outside that I had to wear sunglasses the entire time. (Towards the end of the video, a "sunglasses" detector makes a cameo.)

Please note that he barcode is not actually "read" (so this program can't tell whether the region corresponds to canned pineapples or sunscreen), the region of interest is simply detected and tracked in real-time.

Final Step: Tweaking Learned Positives and Negatives

This video is an example of a pure machine-learning based approach to barcode detection.  The underlying algorithm can be used to learn just about any visual concept you're interested in detecting.  A bar code is just like a face or a car -- it is a 2D pattern which can be recognized by machines.  Throughout my career I've trained thousands of detectors (mostly in an academic setting).  VMX is the most fun with object recognition I've ever had and it lets me train detectors without having to worry about the mathematical details.  Once you get your own copy of VMX, what will you train?

To learn how to get your hands on VMX, sign up on the mailing list at http://vision.ai or if you're daring enough, you can purchase an early beta license key from https://beta.vision.ai.

So what's next?  Should I build a boat detector? Maybe I should train a detector to let me know when I run low on Spice Isle Coffee? Or how about going on a field trip and counting bikinis on the beach?

VMX is ready

I haven't posted anything here in the last few months, so let me give you guys a brief update. VMX has matured since the Prototype stage last year and the vision.ai team has already started circulating some beta versions of our software.

For those of you who don't remember, last year I decided to leave my super-academic life at MIT and go the startup-route focusing on vision, learning, and automation.  Our goal is to make building and deploying vision applications as easy as pie. We want to be the Heroku of computer vision.  Personally, I've always wanted to expose the magic of vision to a broader audience.  I don't know if the robots of the future are going to have two legs, four arms, or they will forever be airborne -- but I can tell you that these creatures are going to have to perceive the world around them. 2014 is not a bad place to be for a vision company.

VMX, the suite of vision and automation tools which we showcased last year in our Kickstarter campaign, is going live very soon.  VMX will be vision.ai's first product.  While VMX doesn't do everything vision-related (there's OpenCV for that), it makes training visual object detectors really easy.  Whether you're just starting out with vision or AI, have a killer vision-app idea, want to automate more things in your home, you're gonna want to experience VMX yourself.

We will be providing a native installer for Mac OS X as well as single command installer for Linux machines based on Docker. VMX will run on your machine without an internet connection (the download plus all dependencies plus all necessary pre-trained files is approximately 2GB and an activation license will cost between $100 and $1000).  The VMX App Builder runs in your browser, is built in AngularJS, and our REST API will allow you to write your own scripts/apps in any language you like.  We even have lots of command line examples if you're a curl-y kind of guy/gal. If there's sufficient demand, we'll work on a native Windows installer. 

We have been letting some of our close friends and colleagues beta-test our software and we're confident you're going to love it.  If you would like to beta-test our software, please sign up on the vision.ai mailing list and send us a beta-key request.  We have a limited number of beta-testing keys, so I'm sorry if we don't get back to you.  If you want a hands-on demo by one of the VMX creators, we are more than happy to take a hacking break and show off some VMX magic.  We can be found in Boston, MA and/or Burlington, VT.  If you're thinking of competing in a Hackathon near one of our offices, drop us a line, we'll try to send a vision.ai jedi your way.

Geoff has been championing Docker for the last year and he's done amazing things Dockerizing our build pipeline while I refactored the vision engine API using some ideas I picked up from Haskell, and made considerable performance tweaks to the underlying learning algorithm.  I spent a few months toying with different deep network representations, and modernized the internal representation so I can find another deep learning guru to help us out with R&D in 2015.

4 VMXserver processes running on Macbok Pro

We're going to release plenty of pre-trained models plus all the tools and video tutorials you'll need to create your own models from scratch.  

We will be offering a $100 personal license and a $1000 professional license of VMX.  Beta testers get a personal license in return for helping find installation bugs. Internally, we are at version 0.1.3 of VMX and once we attain 90%+ code coverage we will have VMX 1.0 sometime in early 2015.  We typically release stable versions every 1 months and bleeding edge development builds every week. 

The future of vision.ai 

In the upcoming months, we'll be perfecting our cloud-based deployment platform, so if you're interested in building on top of our vision.ai infrastructure or want to have fun running some massively parallel vision computations with us, just shoot us an email.

Sponsor Your Favorite Object Detector + VMX Smile Detector

Many of you asked if the VMX Project will come with an initial set of object detectors. Yes! VMX will come equipped with a library of pre-trained object detectors. We are committed to providing you with an amazing VMX computer vision experience and want to give you as much as possible when you start using VMX.

Today, we’d like to introduce a special “sponsor your favorite object detector” reward. We’re introducing a new $300 pledge level to our Kickstarer page, one which lets you sponsor a object detector that will be come inside the VMX pre-trained object library. In addition to sponsorship, you will obtain all the other perks of being a $300 level backer: 650 Compute Hours, a local VMX install, early-access, the VMX cookbook, and VMX developer status. By sponsoring a detector, your name will appear inside the model library when a VMX user mouses over your favorite detector. This is your chance to make a pledge which will have an ever-lasting effect on our project. Consider the number of people that at some point use a generic car detector! Each time they visit the VMX model library, you will have your own claim to fame. “Look mom, I sponsored the car detector!” 

We have 100 slots for the $300 “sponsor an object detector reward,” and the name of the backer sponsoring the an object detector will appear as you mouseover the object model in the model library. This way, your name will be inside the VMX webapp model library, in addition to the wall of backers on our company page. You will be able to choose your name, your best friend’s name, your twitter handle (such as @quantombone), or your nickname. Sorry, no profanity allowed.

We will release the list of 100 object detectors which will come with VMX at the end of January. Sponsors will get the chance to choose their object detectors on a first-come-first-serve basis. If you are the first one to become a sponsor, you will get to choose “face,” “car,” “guitar” or whatever other object you might be excited about! As always, you can change your pledge level and reward.  So act now and don’t forget that by sponsoring an object detector you are supporting our dream project come to life! 

And for those of you interested in seeing more VMX action shots, here's a new video showing off VMX detecting smiles.  This one was taken with Tom's iPhone because the screencapture software on his computer slows everything down.  No post-processing, this is as fast as the prototype runs. Enjoy!

(Cross-posted from the VMX Project Kickstarter Update #10)

My kickstarter explained using only the 1000 most used words

A wordle using all words from the VMX Project Kickstarter original text

Describing new technology without sounding intimidating is a daunting task, especially when you have a kickstarter about emerging technology which intersects the fields of artificial intelligence and computer vision!  Above you can see a Wordle (a visual collection of most frequently used words, size proportional to frequency) I made from the original text of my Kickstarter named VMX Project: Computer Vision for Everyone. Below, I summarize the Kickstarter using only the 1000 most used words. Below, I wrote a summary using this reduced subset of English words. To quickly explain: the Up-Goer Fixe Text Editor (inspired by this XKCD) lets you type some text and makes sure you don't use uncommon words. Here are just a few of the words are banned from the summary: vision, code, product, item, detect, robot, automate, program, machine, automatic, object, and teach. I took a half hour, decided to use the word "VMX" because it is the product name, and this is the summary I came up with.

VMX: "Computer seeing" for everyone

VMX is all about "computer seeing." To make a computer recognize things like cars, people, faces, hands, dogs, couches, or bottles is not easy. If these things could be recognized in real-time either from a picture or live, we could make new ways of playing games with computers, play different music/shows when different friends' faces are recognized, or control your computer with your hands.

This type of "computer seeing" problem is very hard to try to answer yourself. I have been studying "computer seeing" for 10 years, and made VMX, which is a new way of getting stuff recognized in pictures by computers. You use VMX when you want something recognized in a picture. It is very fast -- you can add new stuff to be recognized in a few minutes. With VMX, you won't have to know how "computer seeing" is used by a computer to recognize things in pictures, and won't have to read hard books or papers. VMX is made for everyone to enjoy "computer seeing" -- It is easy, saves you lots of time, and can be quite fun. You can make VMX recognize and act on the things in your home, your friends' faces, or even body parts like hands. Because VMX is so easy to use, you spend more using "computer seeing" in fun ways and we let you easily share your cool ideas with other people, if and when you want to.

If you give us 100$, we'll let you use VMX early, and you'll get more use time (2x to 3x) than people paying later in the year. For 25$, you'll still get more use time (2x to 3x). The more you help, the better deal on use time you get, and get to play with VMX earlier!

Thanks for for helping us make VMX come to life! Merry Christmas!

[This post was inspired by the Christmas entry from Scott Aaronson's blog, Shtetl-Optimized, called "Merry Christmas! My quantum computing research explained, using only the 1000 most common English words", and it's a great exercise for unlearning the curse of Knowledge!]