Learn

The Arithmetics of Words or How Machines Describe Pictures

Ivan demonstrated how the meaning of words can be represented as vectors where every word is a point in a multi-dimensional space. This idea is applied when building the so-called “skip-grams” in which every word is a node in the input layer and the weights of the hidden layer correspond to the vector coordinates.

0
votes

Our event marked several “firsts” – for the first time on Monday, for the first time at a new venue and for the first time with a speaker that has worked for Google and Facebook. We were pleased to welcome our guests in Betahaus in Sofia as part of their skill sharing initiative. And it was an even greater pleasure to have Ivan Vergiliev as our speaker. Ivan has a stunning track record at some of the hottest tech ventures around the globe. After earning a bronze medal at the International Olympiad in Informatics, he studied Computer Science in Sofia University and secured two internships at Google and one at Facebook. His professional experience includes Musala Soft, Chaos Group and more recently SoundCloud, where he created recommendation engines. Lately Ivan is working with LeanPlum on mobile A/B testing.

Ivan shared some of the recent developments in applying deep learning to NLP problems. While neural networks have been around for a long time, recent innovations such as employing GPUs for faster computation and new ideas like convolutional networks have further advanced the field. The idea behind convolutional networks is to connect neurons only locally, as opposed to forming connections with each and every neuron from the neighbouring layers which is ineffective when training the network.

Ivan demonstrated how the meaning of words can be represented as vectors where every word is a point in a multi-dimensional space. This idea is applied when building the so-called “skip-grams” in which every word is a node in the input layer and the weights of the hidden layer correspond to the vector coordinates. Ivan showed how the distance between “man” and “woman” is very close to the distance between “king” and “queen”. Even more interesting was  the query engine that you can find on his blog. Playing with it can produce sometimes amusing and sometimes insightful results – start with entering “мъж”. To see how ‘adding and subtracting words’ can be turned into a phrase that actually makes sense, start with a relationship like “жена” – “мъж” = “кралица” – “крал”. Then if you move the word “крал” to the left-hand side, and enter the expression “жена” – “мъж” + “крал”, then the result will be close to “кралица”.

Formally, several neural language models exist. Ivan gave an example with the Feedforward neural network-based language model and with the recurrent neural network that is particularly good at capturing the global context. Finally, he delved into new developments such as the paragraph vectors and learning directly from raw data instead of using words as building blocks. The last concept relies on convolutional networks for representing the position of the letters.

A very interesting idea that Ivan presented is putting two languages in the same vector space in order to see the similarities in them. It turns out words with similar meanings have similar coordinates and stack closely when represented in a common space.

Ivan also revealed how deep learning can be utilized not only for text recognition but for image recognition and tagging as well. Go to this page to witness how an algorithm generates sentences describing the contents of pictures – some sentences are remarkably successful, some are on the funny side.

Finally, Ivan honestly discussed how neural networks can be broken – for example in image recognition, adding a little white noise to a picture completely changes the guess – from a panda to a gibbon.

Take a look at the presentation link below and the full video record from the event if you want to learn more.

The lecture was followed by the traditional networking drinks, this time conveniently in the cozy Betahaus bar. Don’t miss our great upcoming events and projects – stay tuned by visiting our website, following our Facebook page, LinkedIn page or following our twitter account.

Author: Vladimir Labov

Share this

Leave a Reply