Every time you upload a photo to Facebook, its deep-learning algorithms go to work, trying to ID things both incredibly specific (which of your friends is in this photo?) and general (is this photo outdoors or indoors?). But that information is largely hidden from users — until now.
Software engineer Adam Geitgey put together the snappily named Chrome extension “Show Facebook Computer Vision Tags,” which allows anyone to see what general information Facebook extracts from every photo that’s been uploaded. Install the extension and head over to Facebook, and you can start immediately seeing which objects Facebook can ID within pretty much any photo.
“The extension itself is super-simple,” says Geitgey, who estimates he built it in about an hour. “It has no features; it just lets you see what Facebook sees.”
In practice, it’s fascinating to see how accurate — or inaccurate — Facebook is at IDing stuff. Here is a selfie of me with two very good friends, including New York Mets pitcher Bartolo Colón:
Versus a (super-cool) picture of me in the office, where it comes close but doesn’t quite nail what I’m wearing on my face.
In this photo, Facebook is able to detect the difference between a tree and a Christmas tree:
But it has a harder time with other religions’ holiday iconography:
But it’s likely that if I were to use Geitgey’s extension on the same photos a year from a now, a lot more information would be pulled out — and some of those inaccuracies would be corrected. “When Facebook launched it in April, they could detect 100 keywords, but this kind of system grows as they get more data,” he says. “A year or two from now they could detect thousands of different things. My testing with this shows they’re well beyond 100 keywords already.”
Computer vision — in use by Google, Facebook, Apple, and scores of other companies — is largely aided by machine learning, a process in which a program is trained to do certain tasks by being exposed to large amounts of data. Google and Facebook are currently the best in the field, though not just because they have the best engineering teams working on it. “The point is that everyone knows how to do it,” says Geitgey. “The difference is Google and Facebook have the most data, so they can do the best job.”
Geitgey’s motivation for building the extension was to expose what information was already being collected by Facebook. “I think the feature that Facebook can detect objects is really interesting and really useful, especially if users are blind,” he says, “but a lot of users don’t realize how much info is being collected about them.” He compares it to the $2 fee you may pay when hitting up an ATM. “At the time, you think it’s worth the fee for the convenience. But if you look at end of year it might add up to a lot more than you might think.”
