Following my experiments around deep learning and automated media tagging (not only for fun in the music context, but also regarding YapMe that we’re launching very soon), I’ve build a tiny script which combines Clarifai‘s image recognition and Algolia‘s search-as-a-service to visually index my Instagram pictures, and find them in real-time!
You can browse a prototype here, and get the source on github to run it on your own Instagram feed. By using Python bindings of the aforementioned APIs, the full querying-tagging-and-indexing pipeline itself is less than 100 lines of code!
Tagging Instagram images with Clarifai
After previous experiments focusing on the music vertical, mostly using photograph-made pictures (album covers, video clips), I’ve finally tried Clarifai image recognition API on standard pictures to automatically tag my Instagram feed. The script I’ve build paginates the Instagram API and sends pictures by batch to Clarifai, which tags images immediately.
While there are a few oddities, the result are again very impressive. Here are the two things I enjoy the most about their tagging process:
– It tags at different levels. For instance, it might tag a picture with “beer” but would also add “beverage”, “alcohol” and “drink”. This is particularly useful if a tag which is low in the hierarchy (considering a taxonomy of concepts, e.g. “bottle > alcool_bottle > beer_bottle”) is wrong. A few of my whiskey picture where tagged “beer”, but searching for them with “drink” + “alcohol” allow me to find them in spite of a wrong tag.
– Representing concepts (feelings, abstract things) that are inferred from the elements tagged in the picture. This can go from high-level representations (“spectacle” or “concert” as you can see below) to feelings like “horror” or “scary” for doom-metal album covers. This high-level layer of knowledge representation can be useful in many cases, for instance filtering out scary pictures, without building your own corpus of words related to “scary”, “horror”, etc.
Indexing and searching with Algolia
But, similarly to using an image recognition API instead of setting up a GPU-cluster, what about using a simple powerful, Web-based API? So I naturally picked Algolia, the new kid on the block for search-API-as-a-service, already trusted by Periscope, Vevo, (Rap)Genuis and more.
At a glance, Algolia lets you index document and then search them through its API, taking care of all fancy features you can expect from a search engine (ranking, auto-complete, typos, faceting, etc.). It also has a very convenient UI to set-up those index details; for instance, below is how I’ve added a new facets for my images’ tags (extracted from the previous image recognition step).
Real-time search for Instagram pictures
The search results are displayed in real-time, and the facets (tags) also updated live. More interesting, Algolia supports typos and auto-complete results by default, so typing “cocner” will immediately show-up my concert pictures.
This can be sometimes annoying (if you search for “foot”, you might have “football” pictures first), but the facets help to narrow-down results (thanks @NicolasBaissas for the hint!), and after playing with it for some time, I had to admit the auto-complete by default is very relevant in most-cases.
Keep in mind that all those tags have been generated through image recognition, and that only a few lines of code have been used to (1) extract them from the images via Clarifia and (2) index and rank them via Algolia!
The code is available on github, so give it a try if you want to index your own pictures.