Brand Spankin' New Vision Papers from ICCV 2013

The International Conference of Computer Vision, ICCV, gathers the world's best researchers in Computer Vision and Machine Learning to showcase their newest and hottest ideas. (My work on the Exemplar-SVM debuted two years ago at ICCV 2011 in Barcelona.) This year, at ICCV 2013 in Sydney, Australia, the vision community witnessed lots of grand new ideas, excellent presentations, and gained new insights which are likely to influence the direction of vision research in the upcoming decade.

3D data is everywhere.  Detectors are not only getting faster, but getting stylish.  Edges are making a comeback.  HOGgles let you see the world through the eyes of an algorithm. Computers can automatically make your face pictures more memorable. And why ever stop learning, when you can learn all day long?

Here is a breakdown of some of the must-read ICCV 2013 papers which I'd like to share with you:

---

From Large Scale Image Categorization to Entry-Level Categories, Vicente Ordonez, Jia Deng, Yejin Choi, Alexander C. Berg, Tamara L. Berg, ICCV 2013.

This paper is the Marr Prize winning paper from this year's conference.  It is all about entry-level categories - the labels people will use to name an object - which were originally defined and studied by psychologists in the 1980s. In the ICCV paper, the authors study entry-level categories at a large scale and learn the first models for predicting entry-level categories for images. The authors learn mappings between concepts predicted by existing visual recognition systems and entry-level concepts that could be useful for improving human-focused applications such as natural language image description or retrieval. NOTE: If you haven't read Eleanor Rosch's seminal 1978 paper, The Principles of Categorization, do yourself a favor: grab a tall coffee, read it and prepare to be rocked.

---

Structured Forests for Fast Edge Detection, P. Dollar and C. L. Zitnick, ICCV 2013.

This paper from Microsoft Research is all about pushing the boundaries for edge detection. Randomized Decision Trees and Forests have been used in lots of excellent Microsoft research papers, with Jamie Shotton's Kinect work being one of the best examples, and it is now being used for super high-speed edge detection.  However this paper is not just about edges.  Quoting the authors, "We describe a general purpose method for learning structured random decision forest that robustly uses structured labels to select splits in the trees."  Anybody serious about learning for low-level vision should take a look.

There is also some code available, but take a very detailed look at the license before you use it in your project.  It is not an MIT license.

---

HOGgles: Visualizing Object Detection Features, C. Vondrick, A. Khosla, T. Malisiewicz, A. Torralba. ICCV 2013.

"The real voyage of discovery consists not in seeking new landscapes but in having new eyes." — Marcel Proust

This is our MIT paper, which I already blogged about (Can you pass the HOGgles test?), so instead of rehashing what was already mentioned, I'll just leave you with the quote above.  There are lots of great visualizations that Carl Vondrick put together on the HOGgles project webpage, so take a look.

---

Style-aware Mid-level Representation for Discovering Visual Connections in Space and Time, Yong Jae Lee, Alexei A. Efros, and Martial Hebert, ICCV 2013.

“Learn how to see. Realize that everything connects to everything else.” – Leonardo da Vinci

This paper is all about discovering how visual entities change as a function of time and space.  One great example is how the appearance of cars has changed over the past several decades.  Another example is how typical Google Street View images change as a function of going North-to-South in the United States.  Surely the North looks different than the South -- we now have an algorithm that can automatically discover these precise differences.

By the way, congratulations on the move to Berkeley, Monsieur Efros.  I hope your insatiable thirst for cultured life will not only be satisfied in the city which fostered your intellectual growth, but you will continue to inspire, educate, and motivate the next generation of visionaries.

---

NEIL: Extracting Visual Knowledge from Web Data. Xinlei Chen, Abhinav Shrivastava and Abhinav Gupta. In ICCV 2013. www.neil-kb.com

Fucking awesome! I don't normally use profanity in my blog, but I couldn't come up with a better phrase to describe the ideas presented in this paper.  A computer program which runs 24/7 to collected visual data from the internet and continually learn what the world is all about.  This is machine learning, this is AI, this is the future.  None of this train on my favourite dataset, test on my favourite dataset bullshit.  If there's anybody that's going to do it the right way, its the CMU gang.  This paper gets my unofficial "Vision Award." Congratulations, Xinlei!

This sort of never-ending learning has been applied to text by Tom Mitchell's group (also from CMU), but this is the first, and serious, attempt at never-ending visual learning.  The underlying algorithm is a semi-supervised learning algorithm which uses Google Image search to bootstrap the initial detectors, but eventually learns object-object relationships, object-attribute relationships, and scene-attribute relationships.

---

Beyond Hard Negative Mining: Efficient Detector Learning via Block-Circulant Decomposition. J. F. Henriques, J. Carreira, R. Caseiro, J. Batista. ICCV 2013.

Want faster detectors? Tired of hard-negative mining? Love all things Fourier?  Then this paper is for you.  Aren't you now glad you fell in love with linear algebra at a young age? This paper very clearly shows that there is a better way to perform hard-negative mining when the negatives are mined from translations of an underlying image pattern, as is typically done in object detection.  The basic idea is simple, and that's why this paper wins the "thumbs-up from tombone" award. The crux of the derivation in the paper is the observation that the Gram matrix of a set of images and their translated versions, as modeled by cyclic shifts, exhibits a block-circulant structure.  Instead of incrementally mining negatives, in this paper they show that it is possible to learn directly from a training set comprising all image subwindows of a predetermined aspect-ratio and show this is feasible for a rich set of popular models including Ridge Regression, Support Vector Regression (SVR) and Logistic Regression.  Move over hard-negative mining, Joseph Fourier just rocked your world.

P.S. Joao Carreira also created the CPMC image segmentation algorithm at CVPR 2010.  A recent blog post from Piotr Dollár (December 10th, 2013), "A Seismic Shift in Object Detection" discusses how segmentation is coming back into vision in a big way.

---

3DNN: Viewpoint Invariant 3D Geometry Matching for Scene Understanding, Scott Satkin and Martial Hebert. ICCV 2013.

A new way of matching images that come equipped with 3D data.  Whether the data comes from Google Sketchup, or is the output of a Kinect-like scanner, more and more visual data comes with its own 3D interpretation.  Unfortunately, most state-of-the-art image matching methods rely on comparing purely visual cues.  This paper is based on an idea called "fine-grained geometry refinement" and allows the transfer of information across extreme viewpoint changes.  While still computationally expensive, it allows non-parametric (i.e., data-driven) approaches to get away with using significantly smaller amounts of data.

---

Modifying the Memorability of Face Photographs.  Aditya Khosla, Wilma A. Bainbridge, Antonio Torralba and Aude Oliva, ICCV 2013.

Ever wanted to look more memorable in your photos?  Maybe your ad-campaign could benefit from better face pictures which are more likely to stick in people's minds.  Well, now there's an algorithm for that.  Another great MIT paper, which the authors show that the memorability of photographs could not only be measured, but automatically enhanced!

---

SUN3D: A Database of Big Spaces Reconstructed using SfM and Object Labels. J. Xiao, A. Owens and A. Torralba. ICCV 2013. sun3d.cs.princeton.edu

Xiao et al, continue their hard-core data collection efforts.  Now in 3D.  In addition to collecting a vast dataset of 3D reconstructed scenes, they show that there are some kinds of errors that simply cannot be overcome with high-quality solvers.  Some problems are too big and too ambitious (e.g., walking around an entire house with a Kinect) for even the best industrial-grade solvers (Google's Ceres solver) to tackle.  In this paper, they show that a small amount of human annotation is all it takes to snap those reconstructions in place.  And not any sort of crazy, click-here, click-there interfaces.  Simple LabelMe-like annotation interfaces, which require annotating object polygons, can be used to create additional object-object constraints which help the solvers do their magic.  For anybody interested in long-range scene reconstruction, take a look at their paper.

If there's one person I've ever seen that collects data while the rest of the world sleeps, it is definitely Prof. Xiao.  Congratulations on the new faculty position!  Princeton has been starving for a person like you.  If anybody is looking for PhD/Masters/postdoc positions, and wants to work alongside one the most ambitious and driven upcoming researchers in vision (Prof. Xiao), take a look at his disclaimer/call for students/postdocs at Princeton, then apply to the program directly.  Did I mention that you probably have to be a hacker/scientist badass to land a position in his lab?

Other noteworthy papers:

Mining Multiple Queries for Image Retrieval: On-the-fly learning of an Object-specific Mid-level Representation. B. Fernando, T. Tuytelaars,  ICCV 2013.

Training Deformable Part Models with Decorrelated Features. R. Girshick, J. Malik, ICCV 2013.

Sorry if I missed your paper, there were just too many good ones to list.  For those of you still in Sydney, be sure to either take a picture of a Kangaroo, or eat one.

An accidental face detector

Disclaimer #1: I don't specialize in faces.  When it comes to learning, I like my objectives to be convex.  When it comes to hacking on vision systems, I like to tackle entry-level object categories.

Fun fact #1: Faces are probably the easiest objects in the world for a machine to localize/detect/recognize.

Note #1: I supplied the images, my algorithm supplied the red boxes.

Note #2: Sorry to all my friends who failed to get detected by my accidental face detector! (see below)

So I was hackplaying with some of my PhD thesis code over Thanksgiving, and I accidentally made a face detector.  oops!  I immediately ran to my screenshot capture tool and ran my code on my Mac desktop while browsing Google Images and Facebook.  It seems to work pretty well on real faces as well as sketches/paintings of faces (see below)!  I even caught two Berkeleyites (an Alyosha and a Jianbo), but you gotta find them for yourself.  The detector is definitely tuned to frontal faces, but runs pretty fast and produces few false positives.  Not too shabby for some midnight hackerdom.

Yes, I'm doing dense multiscale sliding windows here.  Yes, I'm HoGGing the hell outta these images. Yes, I'm using a single frontal-face tuned template.  And yes, I only used faces of myself to train this accidental face detector.

Note: If I've used one of your pictures without permission, and you would like a link back to your home on the interwebs, please leave a comment indicating the image and link to original.

Google Scholar, My Citations, a new paradigm for finding great Computer Vision research papers

I have been finding great computer vision research papers by using Google Scholar for the past 2+ years.  My recipe is straightforward and has two key ingredients. First, by finding new papers that cite one of my published papers, I automatically get to read papers which will be relevant to my own research interests.  The best bit is that by using Google Scholar, I'm not limiting my search to a single conference -- Google finds papers from the raw web.

Second, I have a short list of superstar vision researchers (Jitendra Malik, among others) and I basically read anything and everything these gurus publish.  Regularly visiting academic homepages is the best way to do this, but Google Scholar also lets me search by name.  In addition, nobody lists on their homepage their papers' citation counts.  This means if I visit a researcher's personal website, I have to make a decision as to what paper to read based on (title, co-authors, publication venue).  But highly-cited papers are likely to be more important to read first.  I believe that this is a good rule of thumb, and very important if you are new to the field.

I am really glad that Google finally let researchers make public profiles to view their papers and see their citations, etc.  See Google Scholar blog for more information.  I've been using statcounter to monitor my blog's visitors, and now I can use Google Scholar to monitor who is citing my research papers!  I'm not claiming that the only way for me to read one of  your papers is to cite one of my papers, but believe me, even if we never met at a vision conference, if you cited one one my papers there's a good chance I already know about your research :-)  I would love to see Google Scholar Citations pages one day replace "my publications" sections on academic homepages...

My Citations screenshot

My only complaint with Google Scholar is that I can't seem to get it to recognize my two most recent papers.  I have these papers listed on my homepage, so do my co-authors, but Google isn't picking them up!!!  I manually added them to my Google My Citations page, and using Google Scholar I was able to find at least one other paper which cites on of these two papers.

I read the inclusion guidelines, and I'm still baffled.  The PDFs are definitely over 5MB, but my older papers which were indexed by Google were also over 5MB.  Dear Google, are you seriously not indexing my recent papers because they are over 5MB?  It takes us, researchers, months of hard work to get our work out the door.  We see the sun rise for weeks straight when we are in deadline-mode, and the conferences/journals give us size limitations -- we work hard to make our stuff fit within these limits (something like 20MB per PDF).  And we, researchers, are crazy about Google and what it means for organizing the world's information -- naturally we are jumping on the Google Scholar bandwagon. I really hope there's some silly reason why I can't find my own papers using Google Scholar, but if I can't find my own work, that means others can't find my own work, and until I can be confident that Google Scholar is bug-free, I cannot give it my full recommendation.

Problematic papers for Google Scholar:

Abhinav Shrivastava, Tomasz Malisiewicz, Abhinav Gupta, Alexei A. Efros. Data-driven Visual Similarity for Cross-domain Image Matching. In SIGGRAPH ASIA, December 2011.

Tomasz Malisiewicz, Abhinav Gupta, Alexei A. Efros. Ensemble of Exemplar-SVMs for Object Detection and Beyond. In ICCV, November 2011.

If anybody has any suggestions (there's a chance I'm doing something wrong), or an explanation as to why my papers haven't been indexed, I would love to hear from you.

the fun begins at ICCV 2011

All the cool vision kids are going, so why aren't you?
http://www.iccv2011.org/

This will be my first ICCV ever! and my first trip to Spain!  Seriously though, if you need to find me over the next week, come to Barcelona. There are lots of great papers out this year and I'll be sure the write about the few which I find interesting (and haven't already blogged about).  If you want to learn more about the craziness behind ExemplarSVMs, or just to say 'Hi' don't hesitate to find me walking around the conference.  I'll be there during all the workshop days too.

If anybody has a favourite ICCV2001 paper they want me to look and perhaps write something about (hardcore object recognition please -- I don't care about illumination models), please send me your requests (in the comments below).

Ensemble of Exemplar-SVMs for Object Detection and Beyond

Over the next couple of days I will be announcing some very exciting news.  As many of you know, I defended my PhD this past Monday at CMU.  My family and friends came for the presentation as I defended 6 years of my life in front of Alyosha Efros, Martial Hebert, Takeo Kanade, and Pietro Perona.  You might be wondering what I've been up this past year -- what sort of new vision research have I produced since the Visual Memex paper.

Throughout the last year or so I have slowly abandoned the segment-then-recognize approach and fully embraced the exemplar-based component of my research.  Because once you go exemplar, you don't go back!  If only Nosofsky was here, he would be proud.  Once you have established a good exemplar-detection alignment, problems such as segmentation become trivial.  In fact, exemplar association enables a host of meta-data transfer applications.  Here is a quick overview of my recent ICCV 2011 paper with Alexei Efros and Abhinav Gupta (the super new and exciting professor at CMU who will likely revolutionize they way we, vision researchers, think about the interplay of geometric reasoning and object recognition).  

I will be defending my work to the ICCV crowd this fall in Barcelona.  Here is the paper.

Paper:

Tomasz Malisiewicz, Abhinav Gupta, Alexei A. Efros. Ensemble of Exemplar-SVMs for Object Detection and Beyond . In ICCV, 2011. [PDF] [Project Page]

Abstract:

This paper proposes a conceptually simple but surprisingly powerful method which combines the effectiveness of a discriminative object detector with the explicit correspondence offered by a nearest-neighbor approach. 

Exemplar Associations go Beyond Bounding Boxes

The method is based on training a separate linear SVM classifier for every exemplar in the training set. Each of these Exemplar-SVMs is thus defined by a single positive instance and millions of negatives. 

An ensemble of exemplars

While each detector is quite specific to its exemplar, we empirically observe that an ensemble of such Exemplar-SVMs offers surprisingly good generalization. Our performance on the PASCAL VOC detection task is on par with the much more complex latent part-based model of Felzenszwalb et al., at only a modest computational cost increase. 

Generalization from a single positive instance

But the central benefit of our approach is that it creates an explicit association between each detection and a single training exemplar. Because most detections show good alignment to their associated exemplar, it is possible to transfer any available exemplar meta-data (segmentation, geometric structure, 3D model, etc.) directly onto the detections, which can then be used as part of overall scene understanding.

This paper can be rightfully seen as a marriage of my older work on learning per-exemplar distances with the discriminative training method of Felzenszwalb et al.

Here are some summary pictures from my paper and a short description of each one:

1. Going beyond object detection (i.e., produce a category-labeled bounding box), we look at several meta-data transfer applications.  Meta-data transfer is a way interpreting an object detection in a way which transcends category membership.  The first task is that of geometry transfer.

Geometry Transfer

2. Segmentation is a well-known problem in computer vision -- generally tackled with bottom-up approaches which strive to produce coherent regions based on pixel-pixel appearance similarity.  We show that a recognize-then-segment is possible, and in particular an associate-then-segment approach based on transferring segmentations from exemplars onto detection windows. 

Segmentation Transfer

3. Object exemplar often show an interplay of objects, suggesting that it is possible to use the recognition of one object to prime the presence of another. 

Related Object Priming

P.S. Dr. Abhinav Gupta is looking for students, so if you are a 1st year CMU visionary (CMU visionary = robotics vision student@CMU), check out his presentation during the RI Immigration Course.

P.S.S. Anonymous Reviewer#3: Not only have you single-handedly saved my paper from the clutches of ICCV death, but you have resurrected a graduate student's faith in the justice of the vision peer review process.