data.bythebay.io: Parsa Ghaffari, byte2vec: a flexible embedding model constructed from bytes
so my name is Parsa ghafari I'm CEO and Founder alien and today I'm gonna tell you a little bit about our company as well as a new embedding model that we've been working on called by Tuvok so in a nutshell we solve the problem of analyzing and understanding textual and visual content for developers and data scientists and we're essentially solving two problems that we face when dealing with content the first problem is the the natural and inherent complexity of textual and visual information so for instance in case of text you know even in a message that is only a few characters long you have lots of nuances and a noise and abbreviations and so on so as a human being when you read this you quickly realize that okay this is talking about the facebook Messenger app and this the tone is negative the language is informal English obviously and so on and so forth and the same could be said about images you know within a fraction of a second as a human you can sort of identify all the different objects and there Joe spatial orientation and so on and so forth but it's much harder obviously for a computer to do the same so understanding language and vision is difficult but it's at the core of many many user and content centric processes and so and the second problem that we're trying to solve is one of scale so there's simply too much content out there and it's going going like crazy and it's not going to stop and as conde nast recently reported more content was uploaded yesterday than any one human could consume in their entire lifetime so whatever solution we come up with for content analysis must be able to scale up to billions of pieces of content per day or so at a glance we are 13 people based in dublin ireland v remix of scientists engineers and sales and marketing people we have about fifteen thousand users on our platform which collectively pay us tens of thousands of dollars per month to use our services and then we've raised just over million dollars in venture funding so far in terms of products we have two products both of which are api's we have a text analysis API which is a package of about 14 different functions for extracting different types of information from a piece of text and then we have our news API which is an application of that text analysis technology to real-time streams of news content in terms of delivery their delivery either over the cloud or in a non pram fashion we have SDKs for seven different programming languages and we have integrations and bindings for various data analytics platforms such as rapidminer and more easy to use interfaces like google spreadsheets and so on just a brief overview of some of the functions that we have in the text API so sentiment analysis with different variations we have document level sentiment aspect based sentiment and entity level which I'll tell you more about in a bit we have categorization so categories in a piece of text according to the topics that are mentioned in it and so on entity extraction finding mentions of people places companies and so on in a piece of text summarization automatically summarizing a piece of text article extraction image tagging and so on and to give you a high-level idea of how the API works let's review some basic examples so the categorization we have a piece of text so it says Tesla beats expectations with the new model s we specify the language and we select the taxonomy so we have support for multiple industry specific economies and in this case i'm using IE b QA g which is an ad tech taxonomy and i want to categorize this piece of text and it comes back with a first and second-tier label for me and it says it's about the automotive industry and electric vehicles specifically and then in terms of sentiment analysis we have three variants as i mentioned we have document level aspect based on entity level the basic idea is that you have a message you have a tweet your comment a review and you want to know if it's positive or negative either as a whole or towards each of the entities that are mentioned in it or towards each of the aspects of the category of service or products that it it's talking about and then entity extraction is another well-known task in in NLP so we have a sentence like Apple was founded by Steve Jobs and we're going to identify mentions of people places organizations and so on so the first task is to sort of spot those entities and and assign the types of them so is this a person is it a company and then the second task is to link it to potentially linked into a knowledge base such as Wikipedia or dbpedia and that's when you need to perform disambiguation based on the context so Apple the fruit vs apple the company some of the things that we were working on right now so we adding support for more human languages and as I'll explain later we're becoming less and less language dependent so so that's that you're adding Scandinavian languages Russian Arabic Chinese and Korean and Japanese in the next month or so then I think the second thing is allowing our users to adjust the behavior of the API and the models based on their own data that's something we get asked for a lot so we're opening up some of the tools that we've built for training categorisation and entity extraction models and thirdly we're launching more vertical and domain-specific models for tasks such as aspect based sentiment analysis or categorization just briefly about the news API so it money like the some of the use cases and features that we have here media monitoring monitoring mentions of a company in across various channels then profiling authors force a semantic so authorship attribution and authorship analysis trying to sort of build a footprint for say each journalists out there or each user on social-media influencer ship analysis which is a combination of the the authorship profile as well as some metrics that indicate a influence across different channels and then they have a couple functions for detecting trans making prediction semantic search and so on just a quick example with the news API so you know this is the demo on our website you can set filters based on keywords with like boolean search options you can set a category to say give me all the stories that are about electric vehicles and mention Tesla is a company with the positive sentiment and you can set a time frame and different source metrics language and so on and it comes back with the analyze and enrich news content so this is a story from the Business Insider it has like tracked the performance on social media that's what you see on the first line and then an automatically generated summary of the of the article then you have the category so this is about electric vehicles again these are the entities that are mentioned in it and the sentiment the overall sentiment is is positive and then it also provides you with insights and predictions so various time series for volume of stories that meant mention this or match this query average sentiment polarity over time and so on word clouds based on entities keywords and so on breakdown of topics and categories and sentiment and finally histograms as well some of the new stuff that we're adding to our news API so it's obviously powered by our own text analysis technologies of whatever we out there in terms of domain-specific models or new languages is going to be reflected here more predictive analytics more sources and a more advanced query language or some of the things that we're adding to the news API and so as I mentioned I you know I want to focus a little bit more and one of the major functions or four tix API being sentiment analysis which essentially deals with quantifying people's opinions and thoughts and it can be applied to tweets reviews or any kind of user feedback relay textual user feedback and so you can look at sentiment at different levels and I'm going to explain the tree variations that we have in the inner api's so the most basic one is done a document level so you have a tweet and you want to know if it's positive neutral or negative that's the classical example of sentiment analysis and then you have aspect based sentiments were say you have a review about a hotel and you want to know what aspects are mentioned positively and and which aspects are mentioned negatively for so for instance here it says an arrival staff could have could not have been more helpful food was fantastic the place was spotless so this the first three aspects staff food and cleanliness or extracted from this sentence and the sentiment and then the second census says the only let down was the bed was like trying to sleep on a concrete floor it ruined our stay sorry so that's a obviously talking of the beds and it seems to be negative so that's aspect base ended we have domain specific models here for hotels restaurants cars airlines and so on then the third variation is entity level sentiment so you want to sort of understand the sentiment towards each of the entities that are mentioned in a piece of text so Apple easily tops Google Microsoft and list of valuable brands three entities are mentioned here all of which are companies or organizations and then the sentiment seems to be positive to ours apple and slightly negative two hours Google and Microsoft so that's the aim of entity level sentiment analysis you can run a two step process to detect entities first and then sentiment or you can actually run an end-to-end process that learns both both jointly in terms of approach so one of the key business requirements for us is for our algorithms to be as language agnostic as possible and we so in terms of architectures we use mostly CNN's LS CMS and variations and several attention mechanisms we use a board and character inputs for inputs for our models we use distance supervision very possible so in case of I don't know Twitter sentiment we look at emoticons anymore geez and so on to collect lots of weekly annotated data and in case of reviews we look at star ratings and and and things like that so any weak annotation that that we can retrieve for cheap essentially and then we pre use pre-trained embeddings which I'll talk about in more depth later and finally we've completely switched our back into tensorflow but as I mentioned one key area that I would like to focus on is is for the meetings for those of you who aren't familiar with Jordan beddings they're essentially these dense vector representations of Ford's regardless of how how they're trained and so meaning you're essentially your encoding each word as a set of digits so for instance these 100 digits represent the ward mutter and these honored digits represent award father and so that's the basic idea of Fort embeddings and then these vectors somehow capture a lot of interesting information both about the semantics so in the first example so you let say worth 2 micron on the entire English Wikipedia and you get these vectors back and if you look at some some basic examples so you know the classical examples man or yeah king- man equals queen- woman and it the model somehow captures a sense of gender or verb tense or even slightly more complex relationships such as countries and their capitals and all in a non supervised way which is obviously why or two back on board embeddings in general or so popular and then you can leverage this information as input to your deep neural nets for for various tasks so yes the warden bearings have obviously become the sort of de facto input layer in many NLP tasks but they obviously have problems too so some of these problems so they can't handle unknown words and out of vocabulary words that's I think one of the the most obvious issue so if you haven't encountered the board before you can sort of generative vector for it and so if you're in a downstream task you're sort of forced to use a randomized vector which is obviously obviously isn't great and can be particularly an issue in sparse and noisy domains like Twitter where you know a board must might might have you know you might see a word only a couple of times but and and so say it might get pruned if your say using port to back and you can't sort of have a meaningful representation for that word or phrase the second issue is that there's no shared representation at salt board level of it work to back so for instance you know when you and I encounter for it a new word that we haven't seen before we can somehow guess what it means based on say some morphological information so I mean so for instance if you see a word that ends in less even you know we can sort of guess that it's probably an adjective and it's probably indicating lack of something and thirdly I think as we as we discuss it's sort of in like in contradiction with our language independence requirements and I think finally if you if you have a model if you have a downstream application or a downstream architecture that requires character based models you cannot really leverage pre-training or pre-trained board embeddings true what vector and that's so because of these reasons we've come up with an enhancement to work to back which essentially creates a meting space on the rawest form of input being bites and so the motivation here is to overcome some of the challenges that that I just mentioned so quite awake is language independent because its constructed from bites and so it's a lot easier to actually have monolingual sorry multilingual models it's suitable for morphologically rich languages it is able to deal with out of vocabulary words a lot better and finally it's easily pluggable into state-of-the-art neural net architectures especially if you're taking say characters as input and so the the Ark attend sorry forgot to mention this is a work by 204 great research scientist Sebastian ruler and Nicholas pshoo and the architecture essentially looks like this so we have a sentence like the deep learning model is and then we convert that to a sort of bytes we create white vectors from them which are essentially of a very small vocabulary size of about 26 260 ins and then we over that we perform convolution and Max pooling we get autographic lee sensitive baseboard embeddings and then we perform projection we pass it through one or two highway layers and then we get semantically sensitive base world embeddings and the whole thing is trained with a skip gram objective so in the end you essentially get word embeddings for for each of the words and yeah so so that's a bite to work in a nutshell we are opening the code as well as submitting a publication right now based on this so i think in summary one of the things to have in mind is obviously this transition from like rule-based and linguistic space systems to classical machine learning to modern machine learning approaches and deep learning based approaches so I think say with a simple task like sentiment classification you know we've almost had these tree sort of distinct phases or eras you know where initially we had like rule-based systems and so say if you hate as or if I don't know if you say awful as an adjective in a treat then that too it must be negative and then we had the second phase with like classical machine learning so you would say these ten thousand tweets are positive and these ten thousand are negative and then you would do some feature engineering and extract features for you to an SVM or a logistic regression model and which is able to discriminate between the classes and then I think the third generation is kind of what what's happening right now with state-of-the-art deep learning approaches you know you can go pretty far by training world embeddings on large corpuses of on annotated data and then using distance of revision 2 or any kind of weak signal to to train your model which is obviously much less supervised than both of the previous approaches and then I think secondly there's a language independence so I think for such broad and high high volume domains such as Twitter and social media whatever representational model we come up with I think it must be able to deal with multiple languages only about a third of tweets or or in English and so yes I think that's the fundamental requirement for of our current like diverse fragmented world and then finally I'd like to reiterate that bite to it can be an effective supplement to work to whack and I think it will help us do better in noisier sparse of duane's and as I said we're releasing a publication in the coming weeks so so please stay tuned and yeah that's it any questions yep sure so in terms of leveraging this you can you can I think you can have two approaches the first approach is to sort of pre train Ward embeddings using the debate embeddings and then feed those word embeddings to your downstream task or the second way to do it would be to sort of like the decoder to into your architecture so if you have like I don't know CNN that takes a character input you can actually have the bite embeddings and then all the other stack on top that essentially allows you to fine-tune these these embeddings at a training time so i think that like you have to two general ways to leverage this in downstream tasks and but i think you like from what we found it works best in a multilingual setting where you pre-trained the word embeddings first get the report embeddings and then feed it to your neural net yes we use yeah we currently use a well I mean I'm not sure sorry civils sure I think Bellamy so we have a fixed window size of whites in the input and so and then so if you go back to the if you go back to this we so we get the white vectors we perform the convolution and then at this point so we just try to get it in in this shape and after we perform convolution on max bullying and then in terms of visualizing what's your said an Engram I mean yeah so we we don't have any like Engram i think is I mean it's something that you use in a ward context a white base for n gamma honestly not sure what it would look like I imagine that some of those combinations would indicate the defense's like this is like a normal ascii character like this there's a language yeah yeah sure you guys exactly so we've observed the same thing so with the test like nearest neighbors or nearest neighbors before like if you remove the highway layer what you get is like like the similarities or almost entirely or morphological but then after adding those highway layers somehow I don't have a good intuition for it honestly the start they start to get more and more semantically similar and yes we've run experiments with different numbers of highway layers as well as instead of say running convolutions using an lstm but I think the best best results we've we've if obtained so far are we to highway layers and convolutions yeah service and great was it happen quicker if we train on language acetylene and learn later sorry so sorry you are trained in the pen pals on language right murdered later Pete send the gradient password or right I don't know to be honest if it would be faster in terms of convergence but but like I mean since I mean the base idea was to have something that can extend to more than one language and then that's like that was sort of the objective from the get-go I think that's why we we didn't run many experiments in a multi monolinguals setting and certainly no like performance comparison between the two as far as I hi thanked 0