Browsing my stadium pictures

Visually index your Instagram pictures, and find them in real-time!

Following my experiments around deep learning and automated media tagging (not only for fun in the music context, but also regarding YapMe that we’re launching very soon), I’ve build a tiny script which combines Clarifai‘s image recognition and Algolia‘s search-as-a-service to visually index my Instagram pictures, and find them in real-time!

You can browse a prototype here, and get the source on github to run it on your own Instagram feed. By using Python bindings of the aforementioned APIs, the full querying-tagging-and-indexing pipeline itself is less than 100 lines of code!

Enjoyed this post?Read about related experiments in my #MusicTech e-book
Enjoyed this post? Read about related experiments in my #MusicTech e-book

Tagging Instagram images with Clarifai

After previous experiments focusing on the music vertical, mostly using photograph-made pictures (album covers, video clips), I’ve finally tried Clarifai image recognition API on standard pictures to automatically tag my Instagram feed. The script I’ve build paginates the Instagram API and sends pictures by batch to Clarifai, which tags images immediately.

While there are a few oddities, the result are again very impressive. Here are the two things I enjoy the most about their tagging process:

– It tags at different levels. For instance, it might tag a picture with “beer” but would also add “beverage”, “alcohol” and “drink”. This is particularly useful if a tag which is low in the hierarchy (considering a taxonomy of concepts, e.g. “bottle > alcool_bottle > beer_bottle”) is wrong. A few of my whiskey picture where tagged “beer”, but searching for them with “drink” + “alcohol” allow me to find them in spite of a wrong tag.

– Representing concepts (feelings, abstract things) that are inferred from the elements tagged in the picture. This can go from high-level representations (“spectacle” or “concert” as you can see below) to feelings like “horror” or “scary” for doom-metal album covers. This high-level layer of knowledge representation can be useful in many cases, for instance filtering out scary pictures, without building your own corpus of words related to “scary”, “horror”, etc.

Automatically tagging a concert picture
Automatically tagging a concert picture

Indexing and searching with Algolia

To index the tagged images, I had many options such as – among others – using a simple key-value engine like Redis (with lists of pictures associated to each tag), or solr and its facets.

But, similarly to using an image recognition API instead of setting up a GPU-cluster, what about using a simple powerful, Web-based API? So I naturally picked Algolia, the new kid on the block for search-API-as-a-service, already trusted by Periscope, Vevo, (Rap)Genuis and more.

At a glance, Algolia lets you index document and then search them through its API, taking care of all fancy features you can expect from a search engine (ranking, auto-complete, typos, faceting, etc.). It also has a very convenient UI to set-up those index details; for instance, below is how I’ve added a new facets for my images’ tags (extracted from the previous image recognition step).

Adding facets to Algolia search results
Adding facets to search results

Real-time search for Instagram pictures

After having indexed all images, I’ve used and updated Algolia’s instant search demo to build the UI depicted below, and that you can test live here.

Browsing my stadium pictures
Browsing my stadium pictures

The search results are displayed in real-time, and the facets (tags) also updated live. More interesting, Algolia supports typos and auto-complete results by default, so typing “cocner” will immediately show-up my concert pictures.

This can be sometimes annoying (if you search for “foot”, you might have “football” pictures first), but the facets help to narrow-down results (thanks @NicolasBaissas for the hint!), and after playing with it for some time, I had to admit the auto-complete by default is very relevant in most-cases.

Fixing typos and auto-completing results
Fixing typos and auto-completing results

Keep in mind that all those tags have been generated through image recognition, and that only a few lines of code have been used to (1) extract them from the images via Clarifia and (2) index and rank them via Algolia!

The code is available on github, so give it a try if you want to index your own pictures.

Enjoyed this post?Read about related experiments in my #MusicTech e-book
Enjoyed this post? Read about related experiments in my #MusicTech e-book

Leave a Reply

Your email address will not be published. Required fields are marked *

One thought on “Visually index your Instagram pictures, and find them in real-time!