Scale By The Bay 2021 : Taro Saito, Pure Scala Approach for Building Frontend and Backend Apps
okay i'm tell sergio uh actually michael lee introduces me about me so i can't skip this part but uh interesting part about my experience on oss is i've been using scala for 13 years it's already supplied only even for me i was using skull since 2.7 which was released in 2008 you know i didn't know i was using scuff for so long time the myocess activity includes airflame which will be explained in this talk and also created spd center plugin many people might have used this for publishing your project to maybe a center also created snappy java which is used by three for spark or key configuration and so on let me explain what i'm doing at my work on stranger data so twitter data is uh ready to use crowded platform so even non-engineers can use cloud data services with distributed sql engine so once you sign up you can start ingesting your data to twitter data and your data have its own cloud storage and you can make queries against your data using trino sql distributed engine or hypesberg you can also manage your jobs um using diglock which is another open source project for managing complex workflows and my world culture data is mostly at the background side for managing storage and operating distributed query engines but even background engineers like me also need to build front-line applications for monitoring services especially our trainer usage is really extreme we are learning more than one million sql queries every day we are we needed to create a ui for monitoring services and managing multiple adrenal clusters we built this ui using sketches this actually shows a list of queries running on our arena clusters and also i need to create a more better sql editor notebook like editor using sketches which is actually really nice because we can integrate visual studio course monaco editor with charges and you can integrate the power of visual studio and sketches inside the your browser so yeah if you use cultures you can like this kind of reach your applications but if you really need to build front and back applications there's too many things to learn so for front-end side you might need to learn javascript or v-act view js cz framework are very common you know this you may also need to learn how to use swift rather for building mobile applications you may also need to learn how to use electron which is used for building desktop applications based on web applications you may need to learn how to use typescript jquery or bootstrap css styles or the many things that you need to learn even for back inside there are many good libraries in scholar like arca play you may also need to use neti or java spring framework jt glpc there are many many good libraries on there but if you try to find scholar engineers mastering all of them it's almost impossible so we need to change our mindset just instead of learning everything how can we use scala for developing both kinds of applications without adding more knowledge about the existing frameworks so actually if you know much about scala you understand scala is already capable of implementing both front-end and back-end applications scala is functional and object-oriented language which is great for features for writing any application also if you're a front-end engineer you know people loves type secret over writing low javascript because typescript has type safety and skeletally type safe language sky also has a synchro's programming functionality used in future which is already yeah available both in scotland's cloud js so yeah if you if we think about the back-end or platform or learning skeletal scala is great because you can use we can use strong jvm ecosystem actually harco of enable already using neti which is built on java on grpc we can also use zfpc learning http 2 based efficient web applications for front-end side the major platform is web browsers softly firefox microsoft edge code we can use this browser another platform warning sky.js called a great thing about scratches is the major risk version 1.0 was released last year and it's still constantly improving the current budget is 1.7.1 which is already production ready to use and also surprisingly stable simply many people don't know the power of sketches yet so our goal here is how enabling scholar force development because color can be used both for front-end work and applications simply we didn't have good framework for regulating this power of scalar json scholar another good thing about doing scope first approach is the learning cost of building front-end application parking applications can be significantly low because you only need to learn scholar and learning scholar is relatively simple because you only need to read this book programming in skull written by martijn varsky and this book is a very comprehensive guide of skull glamour and like standard libraries once you read this book and probably you're very familiar with almost all functionality of scala and the the the key challenge here are how to render dome for the object for web browsers by using scotches another challenge is how to create a bridge between scholarships application scholar by using some sort of rpc framework and the solution here is using frame open source library which has 20 or more library modules so that we can use the basic for building now scholar oriented applications i'm going to explain in details but before that yeah let's go back to some history about the past actually similar protocols already exist like a google web toolkit which was released around 2006 almost 15 years ago google aimed to explore java first upload using java both for front-end background development and zwd can compile java code into javascript which is great because you don't need to learn anything about javascript for building front-end applications and dr gdp was used for building google inbox google groups disability and these services were built using zwd and basically you need to use ui widgets and defined in grpd for building composing user interfaces the thing is your non-javascript knowledge is required and also you don't need to care about the browser differences between internet explorer safari firefox chrome was once not present on that time 2006. so the biggest challenge from end development on that time was how to overcome the significant differences of brother behavior between internet explorer safari and so on and if you are writing front-end applications at that time probably internet explorer was the biggest enemy for your and development yeah many people hated the next explorer but nowadays the microsoft uh discarded the internet explorer and switch to microsoft edge which is based on open source chromium so the behavior is almost consistent with clone so you don't need to worry about this difference anymore that's a good progress in 2020 but yeah in 2006 many people are fighting with internet explorer dlt was a good frame uh but there are several disadvantages because if you use zwt everything looks like google's application so you customizing celsius it was actually difficult um so it was very compressed to define custom java objects for rpc services but yeah actually i used gwpd around 2017 then i wrote a dna rna sequence browser using dwt the background was javascript on front end was returning java dwd um html5 canvas this yeah even in 2017 we can write this kind of nice visualization of dna sequence data actually the reason i wrote this application was i was working academia doing genome science study so i need some visualization of dna sequence data okay so that was java's history so let's go back to the scala in scholar rendering though is there's best practices kind of best boxes is defining skull function for building number elements like table function or body function d function then you need to combine this function to form compostable elements for doing so there are two major um streams of methods one is like scatter or think is approach which is enclosing attributing the first parlance theses and enclosing child elements or nested elements inside second parentheses but it's a little bit complex to white because we need to like too many parentheses and also a formatting using scale format has been really challenging because it can be a layout is not always fully understood by the intel like scale format so the kind of best product is making no separation between dom elements and attributes which this support is taken by scottish real airframe rx which is most girlfriend of when we synthesize so you can easily see start of the end of the text just by using a single pair of parentheses then yeah the fm has a fmrx library for building compressed dom elements if you want to reuse domain elements you can define alex element interface which only has vendor method so that you can generate arbitrary dom elements and this example shows how to create a button or if you click the bottom showing some alert window then you can create you can use this code for changing button title or adding custom css style and then yeah by adding this kind of functions you can also write this kind of complex ui this is showing list of queries using a table html table so you can find table headers on here you can also see the iteration about the list of tables table rules then generator complex doms here so it's almost like a combining scale function and making change of function calls so if your sky engineer writing dom is also like like a scholar then uh unique point about content development is you need to do more reactive um story processing like you know so alex a e is an interface for defining such operation which is like a sort of a sequence of way or list of a type but it's like more likely because you can if you define a multiple beta or bit of data then it keeps receiving the input stream data from the present operators then produce the next events so you get one event at a time then produce next element now the usage is almost similar to scala collection you just need to use map filter these operations on them similar to skull collection then if you want to write more interactive though yeah you can also use alex variable for getting an event if the viable content is changed for example you can define rx variable of string world here then you can change reactive data so that you can print the message hello world then you need to keep subscribing the upcoming events the first example is showing the hello world because the initial value was rubbed the hollow string is generated here if you set a new variable here then the event is filed and this macbook data is be evaluated then hello our exhibition here so yeah actually fmrbs is gathering best practice from existing libraries live safety of though reactive streaming also cancelling the stream subscription is also necessary if you need to switch the web pages or you need to clean up uh already this is the most quick handlers and so on the next program is how to collect color on scottish applications so ideally if you can write the functional code that calls the demoscale function is better but in vrt we need to make an rpc code so that we can send the data through the network then receive the response from the server this carrier so many many billion million blocks are necessary um existing approach for implementing rpc uh basically two types rest or glbc now if this is basically defining functional interface for http endpoints using get method post methods like so um there are many rare frameworks for scala like play archived gtp financial free skinny framework can be used for building with west-based wave services in scala if you want to use zlpc you need to use google's photographer for schema language to define some kind of interface and generate hold for you know create steps based on the plot of files now there are many scholar wrappers over grpc like skype musical apache or pc can be used for creating such a pc based services but we need to revisit the value of the sky itself scanner is already very capable of defining rpc interface because we can use color functions rpc method definition scalar objects can be our pc's request and response type so why do we need to define invest services or gdp services if we already have scala so our new approach here is fmrpc which provides a scaffold rpc implementation you don't need to think about the rest of protocol buffer anymore you just need to define interface function interface using scala you can define model classes using case classes in scala then skype call can be a good office interface and also it's also can be a good api documentation then you can make a copy score between client servers applications so to actually implement rpc we need several building blocks like message sharealizer the serializer um so we also need a network-based data format so that we can send the data between several clients list of framework uses json protocol grpc uses protocol buffer by nine format in every rpc we can use message pack message pack or json json can be used as well then the interface language will be just color normalized program buffer we also need to have some code generator for building client server implementation then for serializing data we created f1 codec which is a message message-back-based objective organization library message pack itself is a sort of compact binary format of json so and after the object can be represented in both json or message bar by using the pack and unpack interface of airframe codec so scala object will be message back binary and for supporting compress object serialization we defined um many predefined codecs for payment primitive type values scalar collections and so on then if you have a complex case class object like creator 3 parameters like port name and timer you need to combine three codecs then create a message back value of these parameters but you don't need to write complex calls miss airframe codec now can be simply used just by writing message codec of a type then you can need a json data then produce scala object like class a like this you can also feed a message calculator here humpback message back is also supported then the next problem is how to call how to serialize function called arguments and actually it's not so difficult once we can serialize update data remember this method hello has two arguments personal object and message string then we are going to encode uh this function called argument as a map type value which are two keys also the message functions these are functional argument names and actual function arguments then we are going to serialize this map into message platforms by using lfm codec then next step is embedding this as a http request so we are going to use post request then the http pass will be the scholar package name on the follow by slash method name then we can automatically map our pc method called as https behind the scene then we can implement our pc servers just by extending um scatter trait which is a rotated object this is just a annotation there's no functionality just it's just for use for finding rpc interfaces from chrome's class files then you to implement the rbc server you just need to extend this rpc interface and define uh scala code this is not so difficult um a great thing about their rpc is you can plug arbitrary sub-outside applications server-side back-end like financial or glbc unquote information can be also pluggable if you uh want to use finego http1 based web server you can first create a lot of definition which had rpc interface implementation then you can use this center finagle to server with router by specifying port number of type of number can be used here and you can start the financial server so that's it so you now have rpc server implementation just by using scar codes no waste no protocol buffers are involved here you can also define glpc services using the same method you just need to define some of bc interface and you just need to extend it for employer so by actual implementation so creating uh web server based on dwbc is almost the same with financial you just need to call grpc server with router then start so you see there's no portable buffer dimension here because everything is in coupled behind the scene so the yeah let's see the performance difference between http one based services like finagle and http 2. um this just shows how many rpc requests can be made per second so higher bar is better value and blue cup bars sure um synchronous are physical which means making a b score one by one so if you use synchronous score yeah there's almost no difference between a financial or zero pc grpc is slightly faster but if you use a synchronous score where making confined obviously physical simultaneously then you can see significant gaps so if you use zero pc making glp making a pc request up to 40 000 to 80 50 000 is totally possible which is extremely fast it regardless what kind of implementation you use skype bb fmg pc pc general yeah if you can make something 40 to 50 000 specifications per second using uh this result is for my macbook pro machine single mastering example by this kind of performance so you just need to yeah add more mass into that scalability a key performance difference of mostly coming from the gear basis implementation which is multiplexing multiple of pc requests over http 2. so if you make a lot about this code the same connection will be reused for me processing multiple pc requests so that is a huge advantage of using glpc as a back end so if you want to know more details about how we are implementing glpc supporting every on pc um that you only need to look at two key points like descriptor on request to different response masters actually gspc itself is data format agnostic framework so not only protocol buffers you can also use json or message bug as about underlying data format updated format you can embed inside grpc framework but yeah writing this kind of code every time when you define rpc interface is extremely hard so we created a code generator called sbdf employing which basically reads rpc interface classes then generate rpc client code for accessing the servers applications this program can generate a different type of coding generator for scholar environments cultures environmental gsbc it can also generate open api yaml specification file so if necessary you know you can generate uh iphone-based client rpc client or you know other language a busy client based on the specification defined in open api you can also create a good api documentation now there is a good structure editor if you feed the yaml file into slogan that you can create api documentation at least so let's combine everything together so you need to define rpc interface you need to generate rpc client service synchronic client is a product of spkf and programming then you just need to use this query to make an rpc call clients my service hello mess of this code then you just need to pass object then getting a response from the server so that's it behind the scenes many things are happening for the serialization of skype object into message back by using fm codec is happening and also yeah interaction between client server is also happening but from the user point of view you don't need to care much about the details you just need to write the function call to the remote server the getting response so here is the last slide so airframe enables skull fast development so it makes possible to kick starting front-end and back-end applications with scala so you don't no longer need to think about this or political buffers or other frontend frameworks from the beginning if you are already a scholar engineer you can start developing both front-end and back-end applications just by using scala so that's a key point of the this talk now you can find more examples called github depository of fm you can dig into the example holder that you can find more complete examples of what i explained today okay that's it for my presentation okay thank you taro i'm not seeing any questions in the data discord channel so i think if anybody's got some we have somebody typing which should i see uh this is sbtb data okay tv data oh just thanks for the presentation okay uh so taro will be available briefly in the spatial channel channel for in the data q a room and i think that's it okay thank you maybe by joining you