This post follows a google's brilliant blog post
https://ai.googleblog.com/2017/05/efficient-smart-reply-now-for-gmail.html
Introduction
In 2016, they have launched Smart Reply alpha version, a feature for gmail to suggest replies using ML techniques. In fact, the number of users has significantly grown these days, so it is quite content for them.
So in this post, i would like to describe how did they come up with this idea and approach to the issues they got.
Hierarchical Structure of Languages
Do you remember one the finest book published in 2017, which is "How to Create a Mind: The Secret of Human Thought Revealed" by the inventor and futurist Ray Kurzweil. It became popular covered by many newspapers and magazines.
Inspired by this great book, they have adapted the concept that the language is structured in hierarchy. Namely, the content of the conversation or novel hugely rely on the context. As running example, they have given us this sentence, "That interesting person at the cafe we like gave me a glance.". The hierarchical chunks in this sentence are highly variable. For instance, considering the meaning of "a glance" in this context, we can say that this is good gesture because the fancy looking person and you had a glance each other, that is definitely Cool! On the other hand, some might say "hmm,, maybe she/he was annoyed because of your noise or something? or maybe she/he did just respond to the noise biologically..". Hence, we cannot simply determine the meaning belongs to the word "a glance" in this sentence, it is because potentially it is really ambiguous.
Modeling
Given the example, now we can consider how to model this complex phenomenon, which is a context in a language. Initially they have approached to this issues using LSTM encoder-decoder architecture though, it turned out that even with the google's sophisticated infrastructure it is massively computation expensive to follow emails word by word. Instead, they have come up with the hierarchical modeling, which is more computation efficient.
So first layer is for making features useful by embedding inputs. And in second layers, it is working on more abstract representation of the inputs. With this architecture, they succeeded to consider the context more efficiently and comprehensively comparing to the word-by-word approach.
Conclusion
In the end, they have evaluated the model and its capability of representation, it turned out that it was able to nicely detect the context and the generated sparse embeddings have meaningful format.