Scale By The Bay 2020: Adam Warski, Project Loom? Better Futures? What’s next for JVM concurrent...
[Music] thank you very much uh thank you very much for coming my name is adam varsky i come from warsaw poland um it would be great to be in the bay area right now of course but we all know what the circumstances are and so let's keep our fingers crossed for next year and meanwhile uh what i would like to talk about today is pro is project loom and how it fits into and how it might impact the the landscape of concurrent asynchronous programming as we did today using futures and ios and other constructs like this so there will be this will be some speculation some overview of how how things might uh look like and so first it's always good to state what the actual problem is that we are trying to solve and the problem here is that we want to concurrently run multiple processes and when we say concurrently when we say concurrent multiple process it's good to have some concrete examples um in mind when thinking about the possible solutions so this might be a process so this might mean processing multiple http requests in parallel this might mean processing messages from a queue integrating with some external services or orchestrating workflows which integrate with these external services this might mean processing background jobs and there's also many other examples but uh i guess you could pick your your favorite one as we go through uh through product loom and and and futures and ios um so it's not only that you want to create and run concurrently running processes but you want to create a large number of them and we want to do it in a cheap way and so why is that why a large number and why why gp well first of all it might be necessary so it it might be the case that the business requirements that we are faced with um impose these these requirements right so it might mean that we need to process a very large number of er of incoming http requests over a large number of um messages from a message queue but it might also be the case that it's easier to model certain um to model solutions to certain pro to certain problems by working with fine grade tasks which are split into small chunks and so also from a programming model point of view it might be easier to actually work with small processes which might potentially run concurrently with each other so because of these two perspectives we are very often faced with a problem when we need to run a large number of processes and well if and they should be created of course in a cheap way so that our system is still performing well so uh well we are on the on the jvm so this that's our main constraint uh we are we are assuming that we are on the jvm and um we have to live with the constraints that the jvm brings and the basic unit of concurrency in in java into jvm is a thread a thread a maps one to one to a kernel thread so for each java thread there's a corner thread and these threads well they work well of course and we all better or worse know their characteristics but they're not the best abstraction for our processes they're expensive to create they are expensive to switch so switching our processing from one thread to another takes a lot a long time and we can only create a limited number of them we are limited by the stack memory so that's why we are looking for a better abstraction to actually solve our concurrency problem then the current java thread and so before we continue uh just a very short introduction of who i actually am and why i might be a good person to to talk about concurrence and loom um as mentioned i'm the co-founder and cto at software male we are a software consultancy and development company based in poland we've been remote for and distributed for 10 years so remote work is nothing new to us however we would also like of course the log down to end it's remote work nowadays it's not like is nothing like remote work when you can actually freely go outside and meet people um so i i've been developing mostly back-end applications starting with j2e through spring and various java frameworks scala for scala frameworks nowadays and i also do quite a lot of uh open source um as uh like a step client quick lens i also had an adventure with hibernate when i was younger um so yeah that's that's that's my background so now going back to to threading and concurrency so what are the current solutions to to the threading problem well one of them is to create thread pools this means that we create a pool of threads up front so we pay the creation costs upfront and so that's that's uh that's how we solve one of the problems but we are still stuck with the other problems uh we are still dealing with threats directly which so we can still create on a limited number of these threads and they are still expensive to switch and while we have to deal with threads meaning that um we have to program against them directly which is error prone and quite low level and it's not an easy task to actually create a correct concurrently concurrent program using threads uh directly and synchronizing them using logs semaphores queues and so on so thread pulls solve some of the problems but still we we were looking for something better and that's how people came up with the abstraction of a future or a computer a computable future as it's known in java and a future is essentially a value which is important that's a value like that you can store in a variable on a constant and it represents a computation that is already running in the background and that computation will eventually yield a value of type t or end with an error and these features have very nice properties so they are very cheap to create you can create an unlimited number of futures are limited only by the heap memory and they are cheap to switch so the trick here is that there's an executor which actually has a thread pool in the background and it schedules uh it it can schedule a large number of futures to run on a small number of threads and so this executor actually manages which future gets executed on which thread and when one future is done and another one can be can be scheduled to run there so we've got this thread pool we've enriched it with an executor and the future abstraction which is a value and we we are in a much better situation right now future is also nice because we can compose them in two basic ways so so the first way is a parallel composition so we can create two futures up front these two futures f1 and f2 they were the computations they represent will be running in the background the moment they are created and then we can compose the results so once both are ready we will get we will get the final results so this is power composition and we also have sequential composition where we first create one future and once this future is complete we create another future basing on the results of the first one so that's sequential composition we can sequence to computations one after another and um the our programming libraries and frameworks have evolved in a way that uh most of the time many of the modern frameworks now represent any potentially thread blocking or io operations using futures so in a way life used to be simple and so here we have a very simple example of a boring business logic code activating a user which is finding a user in the database which might be blocking of course it might involve iop operations and then conditionally activating the user in the database as well right so if we transfer and if we transform that to futures well the code doesn't look that nice right and we now because database interaction is a potential blocking operation it's now based on futures so we have to use here sequential composition to compose the activate and find futures uh conditionally right and futures also viral because now our activate user method needs to return a future as well and so our code is isn't as nice as it used to be um so yeah so so these are the three problems with futures uh that that we will try to solve the first one is lost control flow so the code isn't as readable so the business logic isn't as clearly visible as it used to be also there's the problem of lost context if you go back here if there's an exception happening over here in database that activate user the stack trace won't be that useful it won't include the fact that these methods were called before because the future will be scheduled to run independently of of of these other futures that have happened before and also it's virus so once one method returns the future all methods that that use it should return a future as well and that's that's where pro that's where project loom comes in if we go to the website we can see the mission of the project is to drastically reduce the effort of writing maintaining and observing high throughput concurrent applications that make the best use of available hardware so that's the mission what does it mean in practice so the project loom brings two major contributions the first one is changing the basic unit of concurrency in java so this used to be a thread now it's still a thread but it's a virtual thread so virtual threads are just like threads but they are very cheap to create and block so we still have the same interface the same base type here thread however we create it in a different way we created using the start virtual thread method and this is going to create a lightweight thread that doesn't map directly to a kernel thread we can create a lot loads of these in a cheap way and in in a very fast way just like with futures and that's not a new concept it's known in other languages as fibers go routines score routines processes green threads and so on with some important details of course that might uh differ between these these implementations and here what's important is that this concept will be uh present at the jvm level uh so we will so all of the java language of all of the jvm languages will be able to benefit from virtual threats so that's the first major contribution the second is retrofitting the existing jdk so that it works well with virtual threads so it means that any methods that use to block kernel threads will now become non-blocking so they they will still block but they will block virtual threads not kernel threads and so this means that if we have an input stream and if we reach from the input stream this will now this will be now an unblocking a synchronous operation if we have a writer a semaphore or even a stress sleep this will only block virtual threads and behind the scenes virtual threads work very similarly to futures as we have seen before so there's a thread pull and there's a scheduler which actually assigns which virtual thread will run on which colonel threats however because this abstraction is now lifted to the jvm level we don't have to implement it as a library on the uh using futures for example but it is present directly in the jvm right and a single virtual thread can uh can run on multiple kernel threads so here are some yield points so where the virtual trace does some blocking operations so then the scheduler actually will schedule other virtual threads to run run in its place um so long also contains continuations which are a lower level concept and it's planned to include staircal elimination at some point but that at a later stage however the main focus is on the virtual threads and on the retrofitting aspect so project loom started in 2017 so that's three years ago now as an exploratory of research project and there have been a couple of iterations into the approach of how this should be done currently there are early access builds available with loom implemented so we can actually see how java works with loom but i'm not aware of any specific dates or any specific uh jdk versions where lumu will be included so that's something to look out for um so we've seen the problems with is with the sequential code and what was wrong when we introduced futures to the sequential code so now we can actually go back to the old days go back to the old code and we can once again write sequential code using blocking operations so we don't have to be worried that we are blocking kernel threads uh because we are not the database.find user it might potentially block the virtual thread but it will still be performant because there's the scheduler behind the scenes which will actually make sure that the kernel threads the resources are used to their maximum but well is it too good to be true and in a way it is um a virtual thread is still a thread meaning that um the problems that we had with threads before such as communication between threads orchestration so when threats should start and stop how should the interruption of threats be managed these threats these problems still apply to virtual threats equally as they applied to threads as well and so we have some basic uh a couple of basic ways in which we can combine multiple strengths right so we can have computations which run in parallel and then are combined at the end into one final result we can have a race between two threads so the first result is taken and the second is interrupted and discarded we can have background jobs and we can have many other types of concurrently running processes which are combined in various ways in our in our in our system and we need a good way of managing that and threats might not be the best answer over here so we we can try using semaphores logs queues and channels actually communicate and synchronize between threads to manage these these various topologies of concurrently running processes but then well we will probably sooner or later encounter deadlocks race conditions and well all the reasons why concurrent programming is considered to be so hard and it is considered to be hard as far as i know at least um so while these primitives are great and it's great that they are in java uh they are probably best used and best wrapped in libraries which hide them and allow us to use something something better so maybe you know loom was great for this sequential code but what about concurrent programming so maybe let's think why did we use futures in the first place right so one obvious answer is for performance reasons asynchronous programming brings much better resource utilization of threats uh then then when compared to the blocking style and that is well studied and documented so i won't repeat it here but there's also the big gain of programming model of capturing a computation as a value which is equally important especially in concurrent programming which uh as we've said is quite hard so anything that might make it actually easier more readable and more comprehensible is valuable as well so the ability to capture computations as value here is very important and you know one way to reduce concurrency bugs is uh probably avoiding direct threat usage and that's what we will try to do so we will try to shift from using uh low-level concurrency primitives to some higher level operations and uh well our goal in general is to use more declarative concurrency then then imperatively using these low level primitives so futures gives as we've seen futures give us the basic building blocks which are sequential and parallel composition and using libraries we can probably some higher level combinators on top of that so as a very simple example of uh of what are the what's what's the difference that i'm talking about here let's consider fetching um data from two http endpoints in parallel and the first let's use virtual threads and concurrency primitives right we are in the lume world right now so we have a blocking a virtual thread blocking method which a calls an http get so if we need to run two of those in parallel what we need well we need to have a memory cell where the results will be stored we need to have a countdown ledge which will actually tell the main thread when the child's threads which actually do the parallel fetching are done then we need to start the virtual threads which will first invoke the http get then it will fill in the memory cell and it will count down on the ledge right so these are the child threats that happen in the background and in the main thread well we have to wait until uh these child threads are done and once they once they are done we can get the results so that's a lot of work and it's probably quite error-prone as well so how can we do a better how how would we do the same using futures well if we are we are now back to the current world using futures right so our http get method now returns a future well using futures is quite simple we create one future right it's a computation which runs in the background we create the second future and then we and then we compose the results of these two futures and we are done much simpler much less error prone and much more declarative as well so that's something that we definitely want to keep in our post loom concurrent world so how can we take the best of both worlds well it's it shouldn't be a surprise that no style fits all use cases right so sometimes it's better to write synchronous sequential blocking code right but sometimes it's still better to operate on futures even if we have these virtual threads at our disposal so maybe we could speculate a bit how could uh how could a more friendly api look like here so we are going back to the loom world right now so we have a blocking a virtual thread blocking http get method and we can imagine that we have a run method which takes a chunk of code and actually executes it in a in a in a virtual thread but still returns a future right so this so this this is a library method that we would need this run here but then we can actually block on the futures in the main thread and wait until their results are ready so this is already much better right than than what we've seen using the uh than what we've seen using the the concurrency primitives speculating even more maybe we'll have a library which has a run a parallel method which takes two chunks of code and a way to combine these results right so the this this method might be available in our hypothetical library as well so loom will still give us a lot of benefits when it comes to working with futures but these futures are still needed somewhere in our concurrency libraries um so yeah so these were the benefits of futures as we have seen so let's reiterate maybe why why would we use loom right so long gives us a great tool to write non-blocking code in in a blocking style and a synchronous style and it provides a retrofit of the blocking apis into non-blocking ones however it doesn't give us a complete replacement for concurrency libraries and and it's useful to view a concurrent processes in a dual way right one way is to look at the concurrently running process as code it as a code written using the blocking style but sometimes it's also beneficial to view it as a value which can in which can be composed and orchestrated so these are the two views these two perspective on a concurrently running process which are which can be used interchangeably and should be used interchangeably to write concurrent programs and so yeah so let's speculate a bit more like what kind of high-level operations might we expect on our futures in a loom world right so if like the some very easy ones are like retry which retries a computation until it's successful first which is uh racing to computations delay which delays the computation by a given amount of time and if you try writing this down it will turn out that all of these actually need lazily evaluated chunks of code like right retry needs to be provided with a chunk of code which can be evaluated multiple times and that's represented by a callable interface in in java and if we take a look at the executor service that is in room it's extended comparing to the current one you can see that quite a lot of methods actually do operate on callables um however um this executor service while it's a good basic building block it's uh in my opinion far from perfect so first of all we've got a couple of process abstractions present in that executive service so sometimes we have to work with callables sometimes we end up working with futures and sometimes we just get blocking methods which returns strict values right so invoke any is corresponds to to race and invoke all corresponds to uh run in parallel right so why does this one return a futures and why does this one return synchronous calls well i understand the historical reasons because we cannot break existing we we cannot change existing executive service but we can probably do better in terms of these abstractions as well there's also the with deadline operator over here which creates a new executor service where all virtual threats have a deadline until which they have finished shouldn't this rather be a property of a of a process not of the executor service so there are some design choices here which had to be made probably because of java conservative approach but still in our concurrency library maybe we can do better than that and that's where we come to ios which can also play an important role here ios and have frequently encountered in functional programming languages like scala haskell and and others and they differ slightly but in a very important detail from futures a future is a value representing a running computation so it's an eagerly evaluated computation while an io is a value representing a description of a computation or a lazily evaluated computation so this might sound like a small difference but it makes a really huge difference when you when when you get to use it so um these ios is something like com combining callable with the future let's say um so um but io offers much more than actually a callable and there's a number of benefits that that that we might uh that we might take so first of all um with ios we might we have the opportunity to get really good constellation semantics so cancellation interruption is a tricky subject to get right and currently loom doesn't offer a good solution here because with a cancellation it also comes the problem of defining uninterruptible regions if a process can be interrupted you will probably later want to define some cleanup logic so re releasing a database connection or cleaning up some resources right and this cleanup logic probably shouldn't be ever interrupted right so you need to define an entry uninterruptable region and while you can interrupt virtual threads you using thread interrupt you cannot define this under uninterruptable regions having only tricatch finally at your disposal and ios since they are externally coordinated offer a solution here ios also have a very nice property which allows you to separate the definition of a process from its sequencing right so you first describe the process it's only a description it's a lazily evaluated chunk of code so once you create it it doesn't run until it's uh until it's sequenced with other processes so you have to be very explicit as to how your concurrent program looks like as to which process runs concurrently with other processes actually when using futures it's quite easy to uh to do some extra uh to do some accidental concurrency right because once we create futures the computation starts running in the background um so so here we have to be very but with ios that's not true once we create ios it's only a lazy chunk of code right and only once we explicitly state that this io this process should run concurrently with this process they will actually do once this is run in the end so we can also get referential transparency and equation reasoning which are nice properties when you try to understand how how your code base works and you also get very nice composability properties so imagine having to define a process which retries a computation that is that has a timeout or a maybe we want to describe a computation which has a timeout but inside we do some retries right so these are er two combinators uh used in it in a different order so once retries first one timeout is first and another scenario timeout comes before retry so these are of course different computations and these are very important details into how you can how you describe your computations but um but using ios they but these io libraries give us actually the tools to to describe these things in a very composable and nice way something that isn't available in in in pure room and has to be built on top of it so yeah summing up where loom shines is a writing down sequential logic so that's probably the majority of our let's say business code well asynchrony is really a technical detail which is nice because it's faster and so on but it doesn't do any good for the readability of the code to actually introduce a synchrony in in that code and and here we represent our processes as code and now where ios or futures shine is orchestration of concurrently running processes right so this is probably more niche code but it's business it's a business critical it's quite hard to get right here asynchronous is a first-class citizen and we might actually want to take advantage of this asynchrony um directly in in in in the code and as before readability is paramount but it means a different thing over here right here by readability me we mean that we can see how these processes are orchestrated how these processes interact with each other communicate and so on and here very often it's very useful to have our process represented as a value and we've just scratched the surface of various topics that are impacted by by loom the structured concurrency there are threat locals there's reactive programming streaming actors all of these topics will in some way be impacted by low man we probably even can't imagine right now exactly how because you know once people get the two already they will actually probably start developing some really cool stuff with it as always happens uh but i guess the my important point here is that loom solves some of the problems with readability with sequential code but it doesn't really eliminate the need of the collaborative of the collaborative concurrency quite the opposite i think it will be easier and more uh program friendly to write down processes using either the code or the value point of views and using loom or or not and so a couple of reading points so i will post a link later uh with with the presentation so you can take a look there there are two blog posts by the loom team which i think are very good introduction to the subject uh also a couple of blogs which i've written trying to understand how loom might might fit into into these into the current world structured concurrency is a very uh is a very interesting subject i recommend uh reading these blogs as well as well as a couple of twitter threads which despite the very concise let's say nature of twitter still often have very useful information um so that's all that's all i had so uh thank you very much for for listening i hope this was valuable at least in some part and as i understand correctly we will now move to the spatial chart for q and a so [Music] you