data.bythebay.io: Olivier Balbous, Visualizing big data
Recording: data.bythebay.io: Olivier Balbous, Visualizing big data
[Applause] thank you um okay so thanks very much for being here um I'm uh I'm Olivier so yeah as you said the software engineer I've been um building uh applications and systems for almost 20 years now and I've I'm um I would say Java and JavaScript uh specialist well I've been working with Java since the beginning and um I work for very uh small to big companies like lots of uh different expertise and uh for the last six years I've um a joined um Adrian's company the squid Solutions uh in Paris uh squid solution is a is a company doing um analytics Consulting and uh another part of the business is to build an analytics software solution so this is the reason why I'm here um I'm here to present this uh software that we call a bouquet and um and um we just released it as an open-source uh software so just this at the beginning of this year so this is why I I would like to show you because now you can access this software and use it for your own uh purpose um just uh no I'm not going to do the demo right now but just a word on on what we do uh with the open bouquet so open bouquet is is a what we call a toolbox it's uh a server that can um do analytics and it's also an SDK that you can use to build uh apps so I'm going to show you mostly the SDK part today um just as an example I've on yeah I'm just showing the the website is it okay um this is the kind kind of applications uh that we we built for our customers and I'm just going to focus on on one I I really uh really like so this like just to show you we we can do lots of kinds of different kinds of applications quite small okay um just so let's let's go to this this application I I would like to show you so this one we have uh we have built is uh we we we called it the the the user flow flow chart uh it's um it's something we did for one of our customers so yeah I didn't say that the customers we have are mostly American companies us companies uh so this is why we have offices in San Francisco as well um so yeah one of our customer uh wanted to uh have uh people in their company to access their data and to understand what are the key uh the key components of their business the key features and what makes the the business successful or not uh so they they wanted to have an application that well is visual like not just figures but have something that is interactive and you can play with so um talking to them we decided that yeah um the the problem is to analyze the flow of the users on their website and see how it performs um well of course uh analyzing where the do the user come from and what are the steps uh that perform well so it's a flow user flow diagram so we obviously use the D3 sun key diagram I don't know um yeah maybe a quick question to know how how many of you do know D3 or do um okay so most of and do you use it or just do you know it who who just uh who uses D3 okay good um so yeah so so you know what I'm talking about um uh so D3 is yeah we used it because it was um very tweakable in fact so we can integrate it uh easily into our application and uh so uh compared to the the original D3 s key diagram we have added lots of uh well features uh as you can like zoom out and zoom in to really like uh focus on some specific uh path on the well on the on the uh on your data um you can so it's just just I'm showing the the features like yeah you have like this uh this uh this little information a context information that shows uh more in details what's happening a little you know like a little pie chart inside and lots of like nice features um there there also we implemented like a drill down so I double click here and you see the the chart has changed because I added a new filter on my data here so I'm filtering uh on the starting page for instance um what and so um the the the goal is not to uh really talk about this application in particular but how we we made it um and U what's important to understand is that this application is made of widgets that uh we we provide with our JavaScript SDK so for instance the the date picker uh on top here is just a simple widget that you can reuse into another application uh the filters as well is is another widget so I can for instance um um well filter on a device let's say Linux okay I think that is so we we have widgets that communicates intercommunicate so the application is well interactive and reactive um so yeah I think this is um this is roughly what um what you have to to know about this uh application and uh now I would like to describe a little bit more on how we can do this using a bouquet um so I just have a slide here uh to yeah to show you the the bouet uh ecosystem okay so on the left you have the application I just shown you so it's a JavaScript application that uses a JavaScript SDK so all this is open source on top I will show you later it's an application we built to be able to tweak your data and to prepare your analysis and this uh application this SDK uh communicates to the bouet server uh through an API uh the bouet server is really the the um at the core of our work uh to provide um something that really uh um does uh in memory uh in database sorry in database analytics so this this is the the the core of uh of what we we did uh in database analytics um maybe uh I think you know what it is um uh is it okay for everyone or uh does does someone knows uh doesn't know uh in database analytics no everyone is okay with that so in database analytics is instead of like taking all the data from your database and Performing computations outside we use the power of the database to perform the computations and now with massively parallel databases like red shift uh well you can uh achieve um very good performance um so this is what what bouet does uh because if you like for instance if you want to achieve this you have to hand code your SQL queries you have to handle lots of boiler plate to uh code to uh to achieve this uh also open bouquet um um I'm going to show you briefly has uh caching capabilities to avoid uh hitting the database when the quer query has already been made and more Nifty features as well um and also has a very important feature which is uh the capability of indexing a specific specific uh Dimension Dimension I think you know what I'm talking about it's a column in a at a table so when you have a column with lots of values we bouet can automatically index those values to have like a full text search on those values um I will be able to show you this later um yeah so um so my presentation is about how to do that um so I'm not going to redo the the squid flow uh app I just shown you I'm going to do uh a simple a very simple app uh but I hope it's going to help you understand uh What uh bouquet is doing and why you should care about bouquet um so for this presentation I just took the stack of open data set uh it's so it's about stack Overflow business so it's post by users so just a uh just here what's the size of the data set and this data set has been loaded into a red shift uh cluster a quite small cluster um okay and so um let me show you yeah here is so yeah now I have this data um loaded into uh red shift I need to uh well access the data and perform an analysis uh the kind of analysis I would like just to to do is very simple analysis like to analyze the posts uh being made by users uh and analyze by age and location of those users so it's just dealing with two tables but two big tables of stock overflow so here I'll um I'm um showing you the the bouet Explorer uh user uh well interface that was on top top of my diagram I don't know if you remember but it's an application that provided uh open source with bouquet and this application is uh well I'm going to show you it's just like I'm going to create a new project so project is a a connection to a database so I just name it put my red shift uh jdbc URL user check the connection and so bouet read well reverses reverse Engineers the database schema uh so there's a public schema and then I'm going to just give it a short name that's can be useful for apis we'll see later okay so I have my project and now I'm going to just uh view the domains so the domains we call domains are in fact the tables of my database so here bouquet the server I'm connecting to uh the server on AWS is uh has just uh discovered the tables uh of my database so here I would like to uh to see the well I'm talking about the the age of the user and the location so obviously I'm going to deal with that so I'm going to configure so here yeah you see uh the dimensions the dimensions as I said before are well the columns basically I'm going to take well I'm interested in the age and location for this uh for these users um location could be interesting in uh being indexed so I'm going to just click here and uh when I save that uh bouet is going to trigger the automatic uh indexing of this location column in my database so it's going to get all the values and put it into a elastic search in fact and uh index those values so I will be be able to filter on like countries for instance like France okay uh and so as yeah the dimensions the okay I have this but now uh I'm not really uh what I would like to count is the number of posts uh so I'm going to go into the the posts table and here you see that bouet has so there's a little thing here I need to fix yeah B has discovered that there's a relation between post and users so he's proposing me the users relation in two Dimensions uh of of posts and what could be interesting also is to um be able to filter on the post uh creation date so this is an option also for Dimensions to be tagged as a period so it will be available in the date picker of the application so I'm just going to show you right right here see uh so I'm going to count the number of posts so count here is a m trick that has been automatically uh proposed by bouquet by looking at the the database well what the data are in in my database but of course I can create custom metrics uh assume something a because it yeah um and this is something yeah yeah okay yeah I'll do I do that later okay so I count my post by uh user age and if I click on preview here here I go okay um so now I was talking about filters uh so I can filter on the location so the location in in uh stack Overflow is is quite flexible so you can put any kind of location so there there are lot lots of values in location well just as an example I'm going to take like two two of them okay okay so now I have my analysis and uh maybe I could get rid of this uh these values where the where the the people didn't set their age so I I can create um a custom dimension um here so uh maybe it's better to name it h set and so I'm going to use the uh bouet as an expression mechanism where you can Define The Matrix and the dimensions if they were not discovered automatically because of course we we don't know everything so I'm going to just yeah filter well users age okay okay so what I did is just in fact create created a a new segment of my data so it's a new filter okay so now I have my users the users that have set an age okay it's the the the thing is just to illustrate what we can do with bouquet maybe the analysis is not exactly what you well not the most interesting one we can do but it's uh and so then maybe yeah I'm happy with this analysis so I would like to build uh a visual visualization around this database and maybe build a new app because here you well I'm in a a given app and uh I would like to be able to customize it uh so what we can do in bouet is to create what we call a bookmark a bookmark is a is a is a configuration that we we save uh to the server so I give give it a name I give it like an ID and yeah here I so this this bookmark I will be uh able to to share to other users so they can see my my current my current analysis uh and I can easily use it to uh build another app with this preconfigured uh analysis so this this is uh what I'm I'm going to do um and um maybe show you uh another tool that we have developed uh that will access this uh bookmark uh but be able to tweak uh the chart that will represent that uh so it's called Uh this is this is the one so it's another application that I have installed on my machine so it's when when I'm talking about applications I mean it's a it's a JavaScript uh client to to bouquet uh here I have the as you see I have the API URL uh which is the the my server I've just installed on AWS my bouquet server is this is the same for my um my bouquet uh Explorer I'm I'm just pointing those two uh clients to the same server okay because yeah I'm just to make sure you you really get uh the the the picture um so BBD 3 uh is um yeah this app is is made at um let me reload it at tweaking your your project so I've clicked on project so I I will find my project my project I just created in bouet Explorer because I'm pointing at the same server and I can use my my bookmark here okay and then well by default uh this application does like a bar chart so of course my data was uh really suited for a bar chart diagram but I can change it so what this application does is just like a JavaScript fi app so we call it uh D3 fiddle because we well it's D3 and we can fiddle with the app so here I have this um basic um code about um B chart just to illustrate we we we hard coded like something like we just have take the 21 rows of data but I can change that so yeah just remove and now I have all my data and uh we have like other examples like a pie chart or things like this so but really the idea of this one is to um to tweak the data so what happened and behind is that I have created my analysis and saved it as a bookmark to my Booker server and now this application is using the bouet SDK to get this bookmark to run the analysis that is this defined by this bookmark and get me the results and this JavaScript code is uh just uh doing this by it well the application is providing this analysis object by the way it's a backbone model object and uh I can get the results the rows the columns well the structure of this this object uh I can show you there's a Json view as well well we have put this simple code that just does a Jason stringify so you can view what's really return by the uh the API so we have the description of the The Columns and we have the values here for the rows okay where did you specify the name of the bookmark uh here it's uh yeah so this is just the the widget so this this is an an application that reuses because you see this is the the same widget as this one basically so I just put the widget so I can easily uh uh choose my project and choose my bookmark so this is one step uh but now I don't have yet a a complete application that I can customize I can just customize the the data V here so uh next step of my presentation is to go a little further and code uh an application that will use that so this is another way uh we can we can use the DS D SDK um so to to use this uh we also provide uh another very simple app it's like a skeleton app that does the the basics of setting well the API um provide being like the really basics of an an application and which will let you customize completely the application using the SDK does this also provide a local proxy to connect toost uh yeah this yeah this one is going to as it is using Grant uh I uh I can use the grand server uh feature so it's going to run a local server uh and my JavaScript code is going to be able to run in within this server I'm just going to show you that yeah but yeah this is yeah the goal of this application is also to provide a minimum uh framework I would say like to use grun B but you you're not oblig well you don't have to use it but this is the way we do um of course uh just K is U JavaScript and and then you have the API if you don't want to use JavaScript at all you can still use the API but it's more work of course um okay so so the the the the process here is well to clone this app so this is what I just uh I just did here so so uh here I just clone my uh bouet app simple and as I I run it so Grant Run Okay so it compiles and then it launches the the ground uh server so I can I can uh I can tweak my app so let me show you the server is a server or a no it's not a proxy it's just a local server like if you had an Apache uh yeah you don't well you don't need a proxy API it's a rest uh it's a rest API so it's HTTP it's just HTTP yeah um so here I open the the the my application code so this is my uh Simple app just just briefly what we do is we set up the API the from the GDK the dsdk uh we have this those controllers that will automatically uh uh compute the analysis and we have a login feature login widget and a status widget and then uh just this application doesn't do much because it's a skeleton application but it will take the results as before you've seen it will take the results of this uh uh of my analysis and display it uh so referring to your question before here I will specify my project and my bookmark ID so if I delete right my project is was S1 and bookmark B1 and I save it and so Grant has re rebuilt my application and I should be able to to view it yeah but it's okay sorry it's not what it ah I have a problem here this is not what I wanted to show you exactly because this one is okay not the right version okay um yeah I don't know yes okay that strange yeah okay so sorry it's it's a way to oh no it's not my app [Music] okay so in fact yeah so yeah sorry I don't know what's happening there uh it's it's it's showing the what I wanted to show you at the end uh so I'm just going to show you what uh should well in fact instead of this um uh this bar chart uh I should have had just a Json code as I do it here okay and what I well I wanted to show you is like to take this uh this code here and and replace this to have my my visualization in fact instead of just a Json output uh okay so I'm going to just show you how but you've already seen the result sorry okay um so what I what I'm just doing here is just um putting the code I was I had in BB3 in my GS fidu app just into this application to change the database in fact okay and well so I'm I'm just defining uh basically where my uh my database should fit into my HTML and uh and here I just have to well to put something else well to create a new div oh no it's not okay no it's okay well uh okay so well yeah you have already seen the result okay okay wow so now we have this uh in fact we have the data visualization that I had here in uh in BBD 3 but I have H here oh five minutes in um in a new application uh the rest of my uh talk was just to uh show you better how to add like this uh date picker uh to the application uh so I'm just because it is here so there's no surprise but this is how we do it we just uh add a new widget uh so data selection widget okay so this is uh if you know back backbone this is a backbone View and yeah uh I'm going to put this into a div date okay so by by just putting this code I added uh to my uh chart this uh date Pier widget so this is yeah okay I'm sorry um it shouldn't well you you should have had uh um I hope you understand what uh this is about but really the thing is that we provide the SDK to build rapidly uh an application and the JavaScript client so it's a JavaScript SDK but then we have an API uh uh a rest API so it's quite uh generic and we have the server features um Okay so um the rest is like uh it's open source it's really new it's just been released few months ago uh so we well we're looking for feedback for people uh trying it uh having issues of course and loving it and uh to well to make this product uh of course a better product that's it right um so when you install this you need to install on one single node of cluster or um so the bouet uh server the bouet server is uh can well it can be installed on one machine but also can support uh multiple machines so it's it's it's been made to uh to be uh uh scalable so the data is already spread out say har or Cassandra can it be used against already existing clusters um uh so there are two things there's the server and there's the database we we just we just we connect to database uh using uh well to uh SQL databases in fact so we we can only uh deal with SQL databases because we do in database analytics specifically jdbc data sources and yeah jdbc but yeah it's hard to find a non jdbc database but uh well SQL database I mean so like for instance Mong mongodb we cannot [Music] access yeah we do have a spark uh plugin but through uh Hive yeah yeah we need uh we need to to be able to do a SQL so let's consider that if we have 10de cluster right 10 cluster with spark installed so you have spark available so do you just need to install your open Bou on one Edge node to connect to your entire cluster or what is what is um well no it's it's a it's a separate thing well we we see your cluster as well just a a jdbc and endpoint and that's it so and we have our own cluster but yeah the the yeah open bouquet is you can install it in on your machine or on it can scale on uh on Multi machines as well but yeah it's independent really from the your database where the data is thanks okay so you do the through JBC are you moving data no uh we are not moving data you mean moving data uh from yeah your server oh yeah well of course we we the data goes through bouet server and we uh uh we do the query uh we try to optimize the query compared to what you have asked in terms of metrics dimensions and so on and we cache this query uh we try to be smart when we cat this query and then we serve the results uh to the through the API but that only do those simp Maybe Med complex you do more [Music] ADV your Ser well no the the point is to uh to rely on the on the database to perform this uh this computation so that means that you have have procedures or whatever analytics tools the numbers well it depends on the capacity capability of on of your database engine in fact yeah of the functionalities that it provides so this doesn't dos visual uh well it depends on what you call analytics but uh for machine learning some sorry machine learning H machine yeah okay no it's something else yeah no we yeah we do like uh what the database can do in fact okay so thank you very [Applause] much