What is Nix and how it can improve your Scala projects
Recording: What is Nix and how it can improve your Scala projects
Should we turn off our videos? It's up to you. It's It's up to you. I haven't got a policy on that, so whatever whatever you prefer. Great. Without further delay, everyone, let me introduce you to Gabriel Volpe who's going to do a talk on what is Mix and how it can improve your Scala projects. He's a very prominent He's a very prominent member of the Scala community and he has written a very good book on the functional programming with Scala, which I'll let him tell you more about it. Other than that, over to Gabriel. Thanks for coming to the talk at FOSDEM and FOSDEM Mix
Thank you all for organizing and for having me and yeah, without further ado, let me share my screen and hello to everyone there. Thank you for tuning in. Good morning, good afternoon, good evening. And yeah, let me share the screen. And yeah. I won't I won't ask the stupid the stupid question. Can you see my screen? I'll I'll assume that all of you can see my console and and if something's wrong, please let me know. Um so, the idea today is to introduce Mix, like very small introduction to Mix to to everyone who's not familiar with it and then see like three clear benefits that Mix has uh in Scala projects specifically aimed to the Scala community
But Mix is much more than that. It can It can do a lot of things. So, hopefully this is like the the entrance door where you can enter this magnificent world and then and you can take it from there. Um But basically, just to introduce Mix or at least it This is the the website of Mix OS. It's uh currently being reworked and redesigned. So, like hopefully it's like more beginner-friendly now. Uh it's basically this is all it says, right? Reproducible builds and deployments and introduces Nix and NixOS. Nix is normally um a power a package manager which allows you to to create reproducible builds and NixOS is basically uh a Linux distribution that uses Nix Nix the package manager
Um I'm actually giving this talk uh from NixOS right now and it's like any other Linux distribution. You can use whatever window manager you prefer. I'm using GNOME. Um but one thing I think it's the best way to introduce uh the benefits of using Nix, it's um basically installing software. Um and I I'll give an example because this is something that always happened to me in the past and before discovering Nix, it was like it's it's like a little bit messy the situation of how people expect you to have the right version of of Java or or whatever build tool you need to use to to run the software. Like let's say you you start on a new job and and then it it is required to to install a certain uh software then on a specific versions and there might be a readme file somewhere a confluence page or something that it documents that is probably outdated. Um So, like how do you guarantee that everybody every developer has a smooth onboarding process and I I I think uh this is one of the aspects that you're going to love from Nix. Um But let me get started with a very very simple example
Like this is all very hands-on. Uh actually this is kind of like a continuation of of a talk I recently gave last week at the next conf. Uh, it was mainly uh, presenting some slides with some examples, uh, but today we're just going to get hands-on, typing some commands, running some projects, so you can get the feeling of NYX in action. Um, so let's say we want to um, for example, I don't have I don't have any any Java command here. Like if I if I run Java, like you can you can see that it the the operating system is suggesting me to run any of these commands if I want to install Java. Um, I don't have SBT either. So, like it it actually suggests me how I can install SBT. Um, this is because I prefer not to install global software
Uh, like things like the Java version or the specific SBT version uh, for Scala programs or Scala projects, uh, I think they should be defined per project. Um, so that that is the only way that we can guarantee reproducibility across different teams. Um, so if I don't have SBT, I don't have Java, then uh, let's start this again. How do I install it? Well, one of one of the the ways to install NYX is using any of these commands, for example. And let's try it out. Uh, let's see we want to try the first. All right. Um, so it actually installed some version of Java
It's actually telling us here it's a 11.0.7. And we see some strange things here like a hash and something in the NYX store. Um, so we're going to talk a little bit about this. Uh, what does it mean? So, it apparently it's so if I run this command, I have Java right now. All right, that's great. Um but where does this version come from? What if you type this command in your machine? Is that reproducible? And and the answer is is actually no. Like the short answer is no. It it depends on what channel do you have
Like there's some there's a concept that is called next channels, which some people actually uh it's suggesting that this should be deleted deleted from from NYX. And and I I think I agree because NYX channels is something like um APT in in Ubuntu Debian or like something that it's upstream and and it's not reproducible um to install software in this way because it actually installs software kind of like globally. It's still in the store, but it is not reproducible if you want to share different Java versions. So um so I can actually delete this. And if I type Java version now, no, I didn't delete it. So what is the I think it's actually it's called JDK. Ah, that was the attribute name. I hope that works
Still there. Um So the attribute name Damn, I forgot how to delete stuff. Is that the right command? Okay, someone in the chat give me a hand because I forgot how to actually delete this stuff. Use the package name, not attribute name. Yeah, what is the package name? This one here? Yeah, AdoptOpenJDK HotSpot bin. Yeah, yeah. Ah, there we go. Cheers
Yeah. Okay, Java is no longer installed now. I should have practiced before giving talk. I'll forget forget that all of this existed. The the short answer anyway is don't use nix-env to install software. Uh we're going to see a better way to to actually provide software. So, we are good now. We don't have any Java version
Um so, the the best way is to actually use nix-shell. And I'll show you how it works. And nix-shell um basically you can uh run it packages like this. And let's say open shady cake. And it And it it'll give me the default shady cake version for whatever it's on on on my nix packages. What is the default? Um so, I run Java version here. It's there. If I exit nix-shell, um Java version is no longer there
So, this is like kind of like sandbox um in nix-shell. Um And right now we are actually saying nix-shell and we passed it a list of uh of packages here. We can actually use SBT as well. Um so, I I have some something that I actually shows me where it what are the packages that are available here in nix-nix-shell. So, I can run SBT version of of Java version. SBT always takes long because it does a lot of stuff on the background. So, if it takes long, I'll actually cancel. Okay, there it is
Um so, that's it. This is like special sandbox. Even if you have Java installed uh globally or like in your system, um you can enter a nix-shell with a specific version of the package. This applies to Python, Node.js, Ruby, Go, Rust, Haskell, any any other programming languages. Like it's very useful. Um So, sorry if it's boring to all the people that already know Nix, uh but I think this is a a nice introduction for like especially in the Scala community people are not super familiar with Nix, so we have to start from here, from zero. Um so, this is one way of uh creating a Nix shell like by by just using packages that are available here. This is of course also not reproducible if you do it if you run this in your machine, you actually use the version of Nix packages that you have in your machine
So, in order to share development shells, uh we can create a a shell.nix. Um so, if we if we create some some some file called shell.nix and we declare the dependencies and we pin the package's version, then we can have reproducible development shells. Um And and we can share that with a team. Um so, normally, let me see if I have some projects here, um we actually use a shell.nix and it has uh all I think they are here. While you're doing that, Gabriel, just to let you know, on the Twitch and Encode Panda said hi to you. I wanted to make sure that you know. Okay, who says hello? Sorry? Encode Panda is his username. Oh, yeah, yeah
Just Pavel. Another Polish friend. Very good. Um Let me see. I think in Redis for Cats there is something called shell.nix. It looks something like this. Like it could look differently. Uh but this is basically uh overriding the version of the SBT script and the Java version and pinning the NYX packages, creating a NYX shell here with these packages, make shell and a few packages that we want to have available
So, if you have this in in your in your projects, uh whatever, if it's Scala project or different language, it doesn't matter, you can share this with other developers and everybody can access the same software and the same version. Uh, that that, you know, everybody can can run the same the same thing. Um, in in the Scala community, like this is not standard yet, so it's like a shell.nix is kind of it's there, so if you want to use it, you can use it, but if not, you you can use the normal tools, you know, Java, SBT, and whatever installed globally, and that's fine, too, as well. But, uh I think it's it's a good practice to have a shell.nix. It doesn't bother anyone to have it there. Like, if you don't want to use it, that's okay. Uh, but you'll make all all the NYX people very happy if you provide one. Um, so, I recently started working on these projects, which is Well, it's not a project
It's kind of like a SBT template, this SBT-NYX.g8, which you can use as a as a opinionated um template to to have uh a Scala a small Scala project with Hello World with some of the stuff that you can do with NYX. And I tried to write stuff about it here and like all the use cases that I think are very clear, you can have clear benefits when using it in your Scala project. Uh, development shell, reproducible CI builds, reproducible and smaller Docker images. Uh, these are the three best use cases I could think of. There are many others and many other uses for Nix. If you start looking into Nix, you can see that this this whole universe of things that you can do. Um, so yeah, this this was like if you follow me on Twitter, you you probably seen this this Twitter poll I ran uh, for the Scala community and see how many people use Nix and how many actually know what Nix is. And yeah, only 7.2% people use Nix
And a high number don't actually know what Nix is. And the majority is actually not using Nix or not knowing what Nix is. So that's that's what actually motivated me to write uh, this guide and actually try to, you know, tell spread the word about Nix basically on on the Scala community. And and I also tried to give this talk at the in the Nix conf to spread the word, you know, that you know, we are also using Nix and together with Scala and there are very good use cases for that. So I invite you to check that out. There's a lot of documentation and well, we're going to go through some of these examples right now and I'll show you how that works. Well, there's a whole lot of stuff here that you can check. But this is like kind of like a tutorial or like kind of like a blog post in a readme file
But it's also a a template here. So if we want to create new project here, just go straight to the end. And you're only one command away assuming you already installed Nix. So if you install Nix, which is also one command, we're going to copy this and we're going to we're going to see what it does. Uh, base we're going to run this basically as a Nix shell with the P argument. It's since I don't have SBT installed globally, but I need SBT to create a new Scala project from a G8 template, then I need the SBT package. So, I can do that with Nick shell, and I can tell give me the the SBT in the Nick shell and run this command, and then exit the Nick shell. Uh so, that's that's what it's going to do
Basically, um it starts the default SBT stuff. And we're going to create the default project. So, get This is the name of the template, which is going to go with all the defaults. It's just a name of packages, the Scala version, SBT version, and so on. So, there you go. And you can see that we exit the Nick shell here. Uh now, we're going to go into this demo and and see see what what it's in there. Uh and this this is the opinionated structure that I that I talked about
Um you can see there's only one uh hello.scala, and then the rest is a bunch of like depends in Nick files, and also the build.sbt and dependencies.scala, plugins for SBT. Um Uh so, let's let's just go and check this out and see how it looks like. Um so, the hello.scala first of all, like everybody is going to be hopefully familiar with this. It's just a very very simple program. Um so, for example, uh I don't have SBT here, but if I if I run Nick shell with an SBT package, I can compile this project. I can run it. So, since this is taking a few seconds, is there any questions up till now? Nope. Okay
see any questions on the Twitch uh at the moment. Unless anyone on Zoom have any questions? Well, I say we we run the project here from a from a default SBT version that I got from next packages. Um I'm just going to exit this now. Um but yeah. Like we enter in next. I don't know if you pay attention, but I Yeah, I'm just going to I remind remember that I have to show this. So, when I when I enter next shell with the default SBT uh package that I have from next packages and I run SBT, pay attention to the first Yeah, that one. So, you can see uh welcome to SBT 1.3.13 and it utilizes this default Oracle Corporation Java 1.8
This is Java 8 version. Um so, this is depending on what next channel you have, you're going to have different SBT version and different Java version. Uh that's why it's not the recommended way to install things. But, I'm just going to show you first of all uh the next the next files that we have here. So, we're going to start with the shell.nix. Uh this is how it looks like. It's It is very minimalistic, very small um because it it is kind of like modularized. Uh we are importing this thing from next/packages.nix
Uh we're going to get into details of this before, but let's focus on this first. We say that we have uh packages.nix shell uh with a bunch of inputs. Um So, if I just run next shell without arguments, it's going to look for shell.nix in the same directory. And it's going to actually run and give me the packages that are defined in this file. And you can see that it unpacking something here and and this is because we are actually pinning the specific versions of next packages. That is actually something that guarantees reproducibility. So, I'll give you a few seconds, but I want you to try and run use this template in your machine. Basically, head over to the SBT next.g8 project on GitHub
And run this command and then enter the directory for the new project that you created and run next shell as well. And we're going to compare versions. So, I'll give you a few seconds and I'll drink some water. And yeah, so hopefully you can run this command and and we can compare the versions that we have. All right, hopefully everyone is everybody is catching up. So, we are in we already enter a next shell that that it is defined by this file shell.nix. So, if we compare what Java version do we have here, we have 11.0.8 from OpenJDK. Uh we also installed coursier
What version is that? 2.0.0 something. And let's see if you run SBT. Remember that before we have Oracle Java stuff and now we have the same script version SBT 1.3.13, but we you have a different version. Um and the the version we have here is exactly the same version as we have for Java. So, we have actually this the Java version we want. And also SBT utilizes the same Java version. What else? Um That's pretty much it. There's not much in this uh shell.nix
We have only a few packages. So, let's look into the other things. We will come back to this default.nix later because I think that that is the most complicated one. Uh so, we are importing uh nix packages here. So, let's see what is in there. Um It's also only a few lines. I tried to make it very modular, so it's easier to uh document. I'm actually writing more documentation now
This is not what you're going to get next time. I'm actually adding a lot of comments here in in current master. I haven't committed the changes yet. I didn't have time. But, the idea is like we have comments line by line and you'll understand what what what everything does. So, let's see what's going on here. Um This thing at the top it's actually an argument. And the whole thing it's just a function that takes uh this argument
It's um And this is like uh let bindings and then let in it's if you're familiar with Haskell, it's like pretty similar. But, basically we are assigning variables here like we're binding uh something to to a some values to these variables here. Um We have a few more imports. And whenever we are doing this inherit JDK or inherit config, that means that whatever like for example in this config.nix, it means that this expects a JDK argument in the same way as this does here. So, we can check this config.nix and we'll see that it also expects this JDK argument. Um so, let's go to the pin file. And this is where actually where defining uh the version exact version that we want to pin, so we can make everything in reproducible. Like, everybody will run this same exact version
And so, it's the only way to to guarantee reproducibility. This is the version of Nix packages. We are using something called fetchTarball, which fetches a tarball from the internet, in this case from GitHub. And it has a hash. Like, if if the content is altered uh in some way, then the hash is not going to match. Um And actually, and the the Nix shell will will fail to actually run. Um there's this something called last PT derivation, which we're going to talk in a little bit, something to actually uh build Scala projects using Nix 100% uh with SBT as well. Um but yeah, here I decided to put all this uh in all these packages here in this pin file, so we we can only change here, and then the rest should remain the same
Packages are actually importing these pin versions, and then we have this config. Config is something where we can actually override uh a few other packages, among other things. Um so, what I want to do in this case is override the JDK version, the Java version that runs with SBT. Remember, if we, you know, at the beginning we ran SBT uh and it by default it has this Java version. So, if you want to override it, it's not enough to create a shell.nix and say I want this Java version and I want SBT. You will actually have a Java version, but then when you run SBT, you will have a different Java version, which is the default that comes from SBT. So, if you want to actually override that, you have to do this. Actually, basically an override of the SBT packages
We override the attribute um JRE. Um it also It also takes an argument here. Um what else? Um In the shell.nix, we have this same argument, but we have a default value. It's a question mark and allows us to give us if if the JDK argument is not specified, then it initializes this JDK 11 by default and like we exit the next shell here, you can see that if we don't give any arguments, then we are using just the default version. But, if we want to change, let's say we want to use Java 8, then we just run nix-shell and we passed these arguments. Uh we call JDK and we use JDK 8. Let's start. I run Java version and see that we run the good one
You can say the same. JDK 14, for example. Actually downloading now right now. And the reason why we have this argument here, like normally in a I mean, in a project at work, um for an application, you normally compile your application with a single Java version. So, by default at work, for example, we use Java uh JDK 11. But, when you're testing a library or writing a library, you might want to test that it runs with different Java versions and different Scala versions, and you have a matrix of a Scala and Java versions, um to to actually verify that it compiles and it works with uh different versions. So, that's the reason why we have it here, and it's very useful for libraries. Uh we're using it using this approach in the neutral library, um somewhere here
Um I'll show that later. Um so, what else I want to show you? We haven't seen the SBT.nix. It's basically the same as shell.nix, except it only has the SBT package as a derivation, instead of a make shell. Uh you see that there's no make shell here. Uh we're going to get into this one. Um But, the real benefit, like, of having all of this Uh one thing I didn't mention, if you if you use GitHub, you can have GitHub actions and use Nix. Um and this template actually creates the the the YAML files for the workflows that you need to to run uh your project on on GitHub actions, and it looks something like this. Um um basically, we have a single job that runs on on a specific Ubuntu version, and we have a matrix of different Java versions, JDK 8, 11, and 14
You can add whatever Java version you want to test against. And only a few steps. And actually, the the cache x action here is fully optional. You don't need it if you don't want. Um but I'll talk about that later. Um this this is the the action that I installed next. There actually there are actually two different actions. One is by also by Cachix in the Cachix repository and this one is a different one
There's not much difference between the two right now, but I just have this one. I I tried both of them and in different projects I have one or the other. Um The interesting part is right here at the end. You can see that we have Nix shell and we pass these arguments string to the for the shady key argument that we have defined in our Nix shell. And we pass whatever values are defined in this matrix here. And then we run SBT compile and we utilize um CI.nix to run this this shell. So, what is this CI.nix? It is the same. It's just like shell.nix, but only has SBT
Uh because that's all we need all we need in the in the CI build. And I'll show you how this runs. Uh Uh for example, in Neutron we have something similar. Um So, you can see right here. I'll have something like this. Uh we have a few things a few more things here, but it it actually creates three different three parallel jobs and it runs them with different different uh Java versions. Uh we're actually also running Docker compose here and stuff to run Pulsar and stuff, but this actually using Nix shell. You can see it's the kind of like the same the same thing
Except this one is runs integration test and so on. It builds the micro the micro site as well. Um So, like I think this is a like a an a very good advantage of like using Nix because you have for development environments, Uh you can share this project anybody using NYX. You guarantee that they run the same SBT version, the same procedure version, the same Java version, even if they have installed it globally. Uh when you run when you have a shell.nix defined, you guarantee that they will be compiling your project with the same the same version. Uh there will be no discrepancies. That's the power of reproducibility. And and it the same applies for the CI build
We are not running a different version. Even even if it's a minor uh difference like in the in the Java version, there could be discrepancies. It happened to me in the past uh in production, especially runtime exceptions and stuff like very nasty stuff that it's very hard to catch. So, like having this peace of mind that it's like we are at least running and compiling with the same Java the same Java version, it's you know, you can sleep better at night. Um so, that's one thing. There's another thing here um that actually builds a derivation. So, we're going to talk about this, too. Um so far, we we only we've only seen uh mix shell
Uh let's talk about mix build. Um and see like we have the shell.nix has this uh packages.nix shell. And the ci.nix has the same. It actually runs using mix shell. Um but if we if we look at this one, which looks very similar, sbt.nix, it's very similar except uh like what it does is builds a derivation for the SBT package. So, what is a derivation for a SBT package? Well, um we can actually build it and run mix sbt.nix, and it will build this derivation here. Let's run it again. Um again, there's something in the next store with hash and with the version and name and version of package
So, the next store it's it's an an immutable uh kind of like a graph of dependencies that it lives on on your machine. And it's on your it's read-only. Um and we can actually have a look at how this derivation looks like. Um if you run this show derivation, I think we can see it in a JSON format. Oh. Not that one. I wanted to see this one. So, next show derivation
And we're going to copy this link here. I know that's not the derivation, I think. That is actually already the the binary. I'm sorry. Um Uh what about the next instantiate? Yeah, instantiate. That SBD. Ooh. Okay
Yeah. So, next build actually builds the binary here. If we you you have a a symlink uh to to the the derivation that has been built. Um I think it's next instantiate. Next SBD. Yeah. So, this actually builds the derivation, but it doesn't build the the package itself. So, we can Thank you for the pointer
Uh we can uh we can inspect how this derivation looks like. And it's basically a bunch of uh uh you know, like the where is the where is this located in the next door, the outputs, uh, the inputs derivation, what it depends on, and you can see the the Java version here, JDK and and other stuff. Um, but it's basically the same SBT version we define and we override here in our config. We are overriding the Java version. And the interesting part is like the win when we run a NYX build, uh, SBT this generates a binary, so we can access by default under result. It's a simlink. So, if you run SBT here, it'll work. It's a binary
Um, actually cancel that. And because this is a derivation and when we can also override, this actually runs with Java 11. But if you wanted to run with SBT with Java Java 14, for example, we can do the same we do with NYX shell. We pass an argument, say say JDK 14, and it'll build this derivation. But you can see it has a different hash. It's different from this one. This one is the default using JDK 11. This one is JDK 14
So, if you go and run the the new result that has been simlinked again, you can see that it's using Java 14 again. So, because in the CI build we use three different Java versions, and and this is the binary that we are building, we can the there are three different binaries generated for SBT. This This is something that we can we can cache. And so, like nobody else has to rebuild this again. Uh, for example, we are sharing this project with other teammates, uh, they can use a binary cache to pull the binary that we already built or that the CI build has built by default and and we don't have to build that again. But you're free to not use the binary cache, but it you will have to build this again in your machine. And for SBT specifically, it it's it's there there there's not a big difference, but if you're if you are a Haskell developer, uh it makes a huge difference in Haskell when you have to build a huge amount of packages. Sometimes it can take hours, no kidding
Um so it's very very common practice to use the binary cache and there's service called Cachix that that I'm using here and and it's also provided in this template. Um Yeah, it's a binary cache for Nix and so whenever we build something like uh we run this and I can say Cachix uh push and the name the name of the name of my of my cache. Uh you need to to actually log in with Cachix in your machine and and and have a few keys, but then it actually works. Um only to push, but to to actually pull uh to to use a cache, you can only say Cachix.use and the name of the the name of the cache, for example. Like I have one for Neutron. And that's it. So, this is like why we have this SBT.nix uh um the YAML file for the GitHub workflows. Every time we change anything here under the Nix folder, so let's say we update the Nix packages version, we want to use the latest unstable from today, for example
Um um and we will try to build this derivation. So, that would actually happen in the CI build and then it would push um to the to the binary cache by default. So, then we when we use uh if you use this binary cache, we don't have to build it in our machine start. That's something that is going to happen in the CI build. Um All that's all about uh building SBT as a derivation and shell.nix. So, if you have any questions, like interrupt me at any time. But, if not, I'm just going to continue and talk about the other stuff that is interesting here. Um The basically building Docker images uh using Nix
Do you have a question there or not? I hear noises. Yeah, if you use this the public the cache server like cache.nix, then what what is your strategy to handle the secret? The To handle the secret? Uh we like in if you use it in GitHub Actions, you can just use a GitHub secret uh and set set the the token in there like I for example here. Uh if I go to settings, there's a secret section here. And you can see I have a cache.nix signing key here. Mhm. Um you basically put it in there and actually it you can write it once, but you cannot read it back. So, anybody with admin permissions cannot read it. Um So, it it's actually pretty safe to use it here
But, that's only if you want to push um binaries from GitHub Actions. But, if you don't need to do that, then you don't even need to do that. Uh it it only suffices with one person pushing to the binary cache, and then the rest will just benefit from pull from pulling packages the the the binaries or that we already built. Um so, they don't have to rebuild it in their machines. So, like mostly either the CI build or one person only has the secrets and uh it's a good practice to have it in the CI build because if you are in a company in a team and one person has the secrets in their machine and they leave the company and then you're all screwed up, right? So, it's it's better to to share it uh uh to have it in the CI build and and having that managed somehow. Like there are different different strategies to to to manage secrets, but that that is very much out of the scope uh of the talk. Uh But in the same way you manage different other secrets, you can put it whatever. Just make sure that you like it's accessible to the team at least to at least to two persons with admin rights and access to the infrastructure and they can they can decode it from somewhere
Uh but yeah, that's pretty much it. Thank you. No worries. And So, let's get at it. GitHub Actions. I actually really like GitHub Actions. I It's a pity that we cannot use it at work. We are using GitLab GitLab runners and it's a nightmare
It's very slow. Uh Uh not saying that GitHub Actions is perfect either, but uh I really like it. Uh okay. Docker. There is a docker.nix file here that we haven't talked about yet. Uh what is interesting about building Docker images using Nix. First of all, it's very easy. It only takes a few lines of code
Doesn't take longer than this. Um What we can do is base build a base Docker image uh which only contains the Java version that we want to run. And the Java version can be specified here. Um if we want to have the same exact Java version that we used to compile and run the projects in the in the CI build, then we can use JDK. But, if you want to make it a little bit smaller, we can uh just use the the share RE because we don't need to compile stuff in Docker. We only need need to run it. So, this is a declarative way of saying this is the exact Java version, and whatever is declared in the pin mix packages that I have declared here. So, it's all very much reproducible
Every time you run this, you're going to get the same Java version, no matter when or where you run it. It this this should work everywhere. Um so, the methodology to actually build the Docker image is running also mix build. I haven't run this in a while, um but I think uh running mix Docker run mix should do it. Yeah, so this creates uh a tarball, and we can use the the Docker daemon to load this into the into the the So, if I look into the result, the the symlink to the result is actually pointing to this tarball. So, if I run uh Docker load result, it should load load that into our Docker images into our Docker daemon. So, I have a bunch of things here, a lot of stuff. But, it's right here at the end
I'll try that again. It's this one right here. Um It actually weighs 163 MB and has some kind of hash. And you see this one, it says it's been built 50 years ago. And like everybody used to Docker, um you know, it's like what the hell is that? Why it's like 50 years ago? And the answer it's like um that every time you build a Docker image the result is a different binary um because of that timestamp. So, we can say that the the binaries generated by Docker build are not reproducible because of that timestamp. But, if you want the same the same thing, we can do this, for example, create create it now and and we're going to have um the created tag. You see, it's rebuilding the whole thing again
And you can see interesting how it builds from different layers. And every layer points to some package in the in the next store. So, if we run the local load again and images and now we see it's been built 25 seconds ago. So, this is kind of like a trade-off. Um if you want to have um to have this uh timestamp here then when when it's been built uh but lose the reproducibility of the binary, then that is a trade-off that, you know, you can accept or not. I personally don't really care about that timestamp, so uh normally just remove that and it's always 50 years ago. Um So, yeah, that's how we build Docker images. Um basically, this only has the this only has the the Java version
Now, we're going to see how we can build the application to be deployed using this Docker image. But, is there any question there? I keep the microphone. Uh I was just about to ask what you just said, so Okay, yeah, yeah. Uh cool. I read your mind there. Um well, let's go to the to the build.sbt. We've seen enough Nix, so this is like a little bit familiar stuff for for the Scala developers. Uh Scala version, version organization, whatever
These are all the defaults, and we have a single project. As we've seen before, there's only a hello.scala there. And hold on a second. Um so, we are we are using a plugin here for the default plugin in in SBT, which is called SBT Native Packager. Uh I also added the SBT Global Add Flags for the Scala compiler. But, I think uh SBT Native Packager is kind of like the the facto uh building tool for for for Scala, and when you whenever you use SBT as the building tool. So, I think it's very important to show how it integrates with Nix, and how how you can still keep using the same tools, uh but actually try to have that smaller starting Docker image with Java version that you want as the starting point. And this is all it takes
Basically, we have enabled the plugins, Java packaging, and the Docker plugin, and we apply this bunch of settings. So, in these settings, what we are doing, we we overriding the the Docker commands, and we say we going to use this base JRE Docker image that we created, uh that is actually right here. And then, we just going to copy the JAR, all the the libraries, the dependencies, and we going to copy our application stuff. And this is the entry point. The entry point is basically just going to run Java uh invocating our our main. So, let's try and build this. For that, we need to run I need to enter mix shell. Uh so, we need to run SBT
I think it's it's the usual command. It's something like docker stuff. Uh It's like Uh Ooh. What is the command? I can't remember. It's not called docker. Uh I completely forgot. If someone there familiar with SBT native packager remembers what is the docker command to build with SBT native packager, completely forgot about it. It's No
It's not reverse so. No. It's just to build it. Uh let's just go to the template. In the template, I think it's that is documented. Somehow. Uh SBT assembly. What else we using next build here? Uh docker.publishLocal
There you go. Yeah. So, docker.publish local. Yeah, that's the one. Found it. Uh so, that that basically runs the steps that we had we had defined in our docker commands right here. And it actually tagged a new version called SBT nix demo with the snapshot version that we have. We can effectively check that
Let's say SBT nix next And it's right there. It's created 20 seconds 21 seconds ago. Um so, we can give it a give it a try. Uh let's see. Let's BT. inix.demo If we run this, we can we can see the hello message. So, we actually made Nix and SBT native packager becomes friends here. Uh we still use native packager to, you know, to build the Docker images using SBT as we always do, but we make sure that the Java version we use it's actually defined uh using Nix, so we can guarantee reproducibility right there
And another cool surprise is that uh at least like what I what I found so far, if if you look in Docker Hub and all these Alpine Docker images using Java and the light versions, even the light versions are even heavier than the one generated by Nix. So, in addition to have reproducible Docker images, you have uh a smaller Docker images, which is also a nice surprise as well. So, there's double reason there to to use this approach. Um So, this is the approach using um SBT native packager. This is the default that I actually recommend um for most people getting started with Nix. Um There there are a few other examples here. For example, at work we use the Mill build tool. We don't use SBT, so, you know, we're missing out from all this stuff
We don't have SBT native packager or stuff. Um Uh but there's an example here that shows how you can use SBT Nix assembly uh and and packages like how you can use SBT assembly, basically, instead of SBT native packager. SBT assembly, for those who don't know, it just creates a fat jar, basically an uber jar with all the dependencies embedded in the same jar. So you have a single jar to run. And that's very easy to package using NYX. And And yeah, there there is an example here in the in the template and it is not all only a template, there's also a lot of different um examples here using assembly, native custom and and a few other things. Um So yeah, check that out. Like just download the project
In addition to play around with with with the template, you can also clone the whole repository here and run the project there under modules here and and see how they are configured. Um there's a bunch of examples there. And yeah, if it is if it is something wrong, if you see something that doesn't make sense, make sure you raise an issue on GitHub or ping me somewhere else. And you know, let's let's try to make this better. And this is one example which I'll SBT native packager using uh the docker images built using NYX. Um there is the final one here that I'm going to show you which is a little bit more complex, but it utilizes a project that is called um SBT derivation by Francesco Zanni. Actually, quite new, but it's uh uh it works pretty nicely. And yeah, you can read you can read about it here, but I'll show you just a quick example how it works
And actually, if you use the template, you're going to have this default.nix that actually utilizes SBT derivation. Uh we've seen before in our pin.nix that we have this project here. So we are actually pinning the version of SBT derivation we use, with a specific commit hash and and a shot 256. But, let's have a look at this file. This is uh again, we have uh a shady key argument here with a default value. And this is actually building the whole Scala project using Nix, nothing else. Uh well, it runs SBT under the hood, but the way we build it, we don't even have to run SBT. Uh Right, so has a few stuff here, but instead of like just going through let let let's just run it first
Basically, if we run nix build without argument, it's going to look for this default.nix in your in our directory and it's going to try to build that. So, this is what what we're going to do now. Like, the first time if you if you didn't run it and it actually downloads a few packages if you are new and and actually storing that in the nix store, but you know, subsequent times it's going to be much faster. Uh so, we can if it takes a while, we can leave it running there. Um another important stuff is that um for all the people that are using Nix in a different distribution or in Mac OS and stuff, it's very important to remember to uh to clean the the nix store because it actually grows very quickly and it consumes a lot of space and like um we have to remember remind people to actually run the the garbage collector. There's something called uh can't even remember the command, but it's something like uh nix It's the nix collect garbage. collect garbage or stuff. I have it I have it running I run it automatically, so I don't Yeah, I mix like garbage and stuff
Um We have it um running automatically for me uh every Saturday, so I don't have to to remember about this this stuff. Uh but if you use NYX OS, that's that's like one of the perks that you that you have, right? So, you can um I'll show you how that looks and uh Gabriel Gabriel on the Twitch Twitch chat said uh people are asking how you would use uh NYX with IntelliJ. Uh for example, if you want to use different uh uh ver- versions per project, how do you switch in IntelliJ to the different versions? Is there a clever way of doing that? Um I'm not really familiar with IntelliJ. So, how how does it pick up the the Java version? Does it take it from from the environment variable? Or how does it so? Like like most of the people at work use uh uh VS Code with Metals or or different text editors, so I'm not really familiar with that. My personal experience with IntelliJ and using NYX has been to just uh manually point it to the correct Java home. Um the Java home or or just use the environment variable. But I haven't used anything fancy like there and or anything like that with IntelliJ, so I can't comment for them to say that. I just thought I'd mention it
Yeah, like to be honest, uh yeah, I I don't know. Um I I I'll need to know how how IntelliJ picks up the Java version and stuff, but uh other than that, I don't know. But if you if you if you can actually fire uh like spin up a IntelliJ IDEA from the terminal, like let's say that you are in a NYX shell, so like you have the Java versions right there, and you can just like I don't know, run IDEA IDEA, Yeah. And it picks up the the version from there, then I that could work, but I don't know what IDEA does. I remember back in the days when I was using it that it it actually had it it its own Scala compiler and stuff. So, we had a lot of red herrings that you know, things that actually were working on the SBT terminal didn't work on on IntelliJ IDEA and that was complicated. Just just just thinking out loud, you could probably create a shell.nix or default.nix for your project, and then in there make sure that the Java home is set appropriately, and then IntelliJ could pick up that Java home from the environment variable. And then you start Right
Yeah. start it from the command line like you said, yeah. This is our Right. So, like that's one of the things that you can do here in the next shell. You can declare environment variables here. Let's say my my and variable equals blah blah blah, and that will be picked up in the environment, too. Exactly. And then you just And then after you've done nix-shell, you can just start IDEA from the command line and hopefully have it pick up that environment variable as the overriding Java home
Right. Yeah. Yeah, that could work. I think it's a question of experimenting and see how that works. Yeah. Sorry to interrupt, but I just want to make sure that the question doesn't go on unanswered. But I I think like what we recommend, like even because of all these red herrings with IntelliJ IDEA in the past, I don't know. I think it improved right now, but I don't know if it's perfect yet
And even with Metals and VS Code, sometimes it's not perfect. So, like we embraced, you know, just compiling the project from the terminal always. So, like from the next shell, you run the SBT and run the commands from SBT, basically. At least before pushing. Like you know, make it make make sure that it compiles with with the SBT and Java version that you have defined there. But then, as you're using for editor, like make Yeah. You will probably work with any any Java version stuff. Um yeah, so we run next build
You could see it took quite a while because uh I had to pull a lot of dependencies and stuff. Uh and it finally created this this thing here, like there it should be uh pointing here, result. So, I think if you run this result, we should have a binary there. Yeah. And it's called So, if I run next build again, this is already built, so it won't actually rebuild it. So, but let's modify something in the code. Um Hello. Uh say world
Oops. What am I doing? Uh if I run next build again, it actually detects the changes in it and it it builds the project again. So, this project is not is not super perfect. There are a few things that I they could take advantage of, like for example, uh caching the the SBT version and and utilizing the cache of dependencies as well, I think. But at the second time, for sure it's going to be faster. And now we're going to have a look at how how it is defined this default.nix. But you see, like it compiled pretty much pretty fast right now. So, like if you run this again, you can see that it changed again
So, back to this default.nix, let's have a look at what it does. It it it starts pretty much the same importing packages, the pin version, the configuration. And we are using this SBT derivation, which is the the stuff that we download. The the the thing that actually lets us build the Scala projects. Um we have a few overlays is some other way of overriding packages as well of like including packages in our version of NYX packages overriding the shell version here. And here we have we declare the main class for our project and then we make these packages available below. So, after the the in class. So, we have this SBT package
Everybody knows what it does. Make wrapper is something that uh allows you to build a binary and and a few utilities here. So, this SBT.make derivation is defined by this SBT derivation project. Uh it requires a name, a version. And this is the first interesting part. It it builds two derivations. One one for the the dependencies, one for the the Scala project. So, instead of like um trying to map every every dependency because you know, like in the JVM and in shell versions, like there's a whole universe of dependencies and especially from Scala it's even more complex because you have cross compiling Scala version that you have 2.12, 2.13
We have Dotty right now, Scala 3. And plus different shell version. It's actually a nightmare. There there there were a few attempts to to build uh SBT projects using NYX, but both projects are abandoned. Another Another approach that I haven't tried yet, but I I want to probably try it. Um there is a project called uh Maven to NYX by Farit. I think he was on the on the call. Not sure if he's still around
And to He had to drop off, unfortunately. Okay. Uh like the people There's one thing that I want to try because uh uh Maven to NYX is actually the this project by by Farit. Yeah, Maven to NYX. Yeah, this one here. It basically allows you to build uh a Maven project using NYX, it's called Maven to NYX. Um and because it embrace utilizes Maven and and Maven is very much the standard in in the Java world. And if you run SBT, we can actually generate a POM for every dependency or for every module in our SBT
So, I think we could make it work. There is a command, native command from SBT, which is called make make POM. So, if we have a look at this, um there is probably a way to make it work with the uh Maven to NYX. See, it's just a standard Maven POM uh file. It's an XML. So, that's one of the things that I wanted to try, but I haven't had the time yet. Um but then, yeah, this one is not very important. It's basically a warm-up command to download dependencies and hash them
There's a question? How do you get that the SHA, the the actual string? The dependency SHA. Very, very good question. Uh there is something in the in the NYX community which is called um TOFU. uh which means trust on first use. Okay. Uh Well, I'm not in the slide. Uh I wish I wish I had known about that sooner. Very good
So, like normally, when whenever you start a new project, you don't know this hash. And specifically for SBT derivation, it's uh it's quite complicated. But, for example, uh if you want to know the hash for for this package, which is a SHA-256, it's very easy to get it. Um for that we just run something called next prefix. I unpack because this is tarball and if you run that, we should get the same the same hash. And you can see it here, right? So, this is how we get it for for packages. Um the same with next packages or any other tarball. If it's a binary or it's not a tarball, you can remove this unpack and you will get the different hashes as well
Um but specifically specifically for these dependencies, um we normally do um something like um enter anything like something that is valid like that, right? So, like a bunch of zeros. It has to match the the the the length the length of the of the thing and then we try to compile it. And then next is going to tell us that we got this hash but you expected this one. So, it's not going to compile. And well, pretty much what we're going to do, we're just going to um copy paste it from the terminal and replace it there and we're going to use that one. So, that's that's the concept of trust on first use. Um for some things it's very easy to compute but for specifically SVID derivation, it's uh it's not very easy. Um so, that that's the recommendation
But then I'll I'll let it compile down there. In other words, uh make it build inputs is something that we're going to need need for uh building our our binary. And this is pretty much actually uh declaring where our where uh declare where we have our source files, you know, where are the relevant source files that we want to include because all of these is going to be taken uh taken into account for the to their So, if you have a Git ignore or a few other files here, if you want to include the readme and stuff, then that that generates a different a different hash, a different derivation. So, uh if you want to make it reproducible and make sure that there's nothing else, you know, in the directory, that you have some specific things for IntelliJ IDEA or VS Code or for whatever, um then we want to make sure that that is excluded from our source. So, we specifically declare what is exactly the stuff that we need to build. And And it's here, everything on the project, everything on the source, and the build SBT, nothing else. And this will actually make a reproducible derivation for our project. Um next, we have the the build phase, which is runs SBT stage
That means that we depend on SBT package manager again. So, this this SBT derivation project assumes that we use SBT package manager. And then, finally, the the install phase, it's basically picking up that that uber jar that was like No, that that actually doesn't create an uber jar. Basically, creating the jar and the dependencies. And then, calling make wrapper that actually invokes the Java binary and runs runs our our jar of the application uh with the specific Java version. So, that that's what it does. So, here here it is. Like you see, we we got this hash mis- mismatch uh message from Nix
Uh what we wanted and then what we got. So, uh trust on first use, basically just copy-pasted from here, and trust that this is going to work. Uh that's pretty much all we do. So, that's all about SBT derivation. Uh I don't recommend it to uh to people just getting started with Nix because, you know, there are a lot of concepts, and it's actually a little bit a little bit slower than chassis in SBT. So, uh uh this is still I think I consider this is still experimental or it could work for a small projects, but not for big ones. Um So, but the the things that are actually Sorry? Can I ask a question? Yeah. Right
Yeah, so I'm wondering if where this puts the dependencies when when it downloads them. Does it use that the global SBT cache or does it set up a one specific to this build to this product? Um I think it it it utilizes the same SBT cache. Um you can see here. Uh but it creates it creates a a a normal um a specific dot nix folder. Uh to be honest, I haven't looked much in how how it actually works. So, I can't really tell, but it might maybe hard to find out what it does. Um but you know, I just consider it experimental. I wouldn't be using this anytime soon
Like but it's good to play around with it. Um But what I recommend is basically what I what I what I said here in the in the guide. Uh what are the killer use cases? Because I want people to use nix and to feel uh good with it, right? So, I I just defined a smaller subset of what nix can do and can do really really well. So that you know, because even if I recommend people to use SBT derivation, they're going to have they're going to find a lot of different issues and their their experience with nix is going to be ruined and like we don't want that. We want people to say, "Wow, this this is really really amazing, right?" And uh I think it's it's very important to sell uh whatever nix does well. Um And nix does these three things very well. Creating a development shell as we've seen with next shell, creating reproducible CI builds because we share the same configuration for the development shell to run in the CI build, and creating reproducible and smaller Docker images that you can actually integrate with SBT native package. So, I think these three things uh are things that mix does very well uh within the Scala community
Um so, I want to I want to promote and and embrace people to use this just for these two use cases. If you then get more enthusiastic with mix and you want to try something else, then pretty sure you're going to have fun, but it's like mix is not perfect either. There there are a few things that are really hard and you know, error messages sometimes are cryptic and and it's hard as a beginner to get started with when you get stuck. But, there is a there is a wonderful community behind it and and there are Discord channels, Slack channels, free notes, so it's I really encourage everyone to take a look at this, but this is a very good starting point if you want to use it with Scala. Are there any questions so far? Okay. Um yeah, I think I'm just going to wrap up one more thing um that I have I forgot to mention um So, um like sometimes it's very important especially with people that have installed Java and SBT and other software globally uh you know, to make sure that actually the Java version you're using is not the one that you have installed globally, but the one that is defined in in the next shell. And you have to remember to enter this next shell in order to have that Java version, right? Um so, there is something that we embrace at work which is called direnv. And and we make it optional for everybody in the company, but it is actually very much recommended
So, it only takes this .envrc file and there are a few different values that you can use. In this case, I'm using next gear and we actually caches the derivations. So, if I do this, um I can have I'm I'm actually in a in a next shell right now. And if I say um I'm here like right now, I don't have Java here. But if I enter the the directory, I say Java and I have the version. Is it Whenever you enter a directory, you have already all the dependencies defined in your shell shell.nix, which is it's amazing, right? You have you forget about running next shell. Um it's basically everything there. So, uh something that I wasn't asked to it's like you have to run direnv allow the first time like uh I wasn't asked because I did it before
But, for example, I don't have Java here. You know? If I go here, you see that I don't have the the the I'm not in a next shell. I don't have Java because you have to to allow it. You have to run this command first and then it runs that. So, this one is very useful to me and I think I recommend its use, but whenever you're getting started, it takes an extra step to actually install direnv and set it up in your machine. So, we make it optional, but yeah, I I recommend its use. If I'm not mistaken, if I'm if I'm not mistaken, direnv is actually created by Jonas who's on this Zoom call with us. Ah, he's still there
Yeah, thank you. Thank you very Thank you very much, uh Jonas, for that. Yeah, I was I was I was having a hard time associating your real name with your your handle on the internet. So, it's it's good to make to make the connection. Uh Okay. Um Yeah, I think that that's all I have to say. If there is any questions, I'll be happy to to try and answer them. But, I'll look at the guide and Yeah, and have some with me
Oh, I have a general question. How do you debug NYX expression when some problem happen? Uh well, the problem Yeah, I think it's depending on what what what it's the actual issue you're having. There are many Like as as a beginner as a beginner, NYX is actually hard. Um now, my my experience is in NYX is also like, you know, getting started constantly bumping into issues. It's not it's not uh very easy, especially when you find errors. So, um that's why I only like for it specifically for Scala people, I only recommend this subset. Because uh nothing can go wrong. If all like if something goes wrong, it's just a very small stuff
Um another thing that I didn't mention is at work, we have people using Mac OS and Linux. Um so, we share the same shell.nix, and some packages are actually specific specific for Mac, specific for Linux. So, for that you have to declare it. But, um we have all a few packages for that. And yeah, like I don't have a Mac, so I cannot test that. But, I have some teammates and say, "Hey, can you run the NYX shell or the NYX build and see that works on your machine?" And you know, try to make it reproducible. But, um that is the the the only difference we have. Like some people have a Mac on Linux, but we try to make it work for everyone
But, coming back to your question, yeah, whenever it's something goes wrong, what do you do? Oh, uh I think with time you start to to learn how to read error messages. And um Okay, let me share my screen again. Um Like one thing like because you when you when you're starting with NYX you you want to play around with uh you want to get familiar with the NYX NYX the language, right? So, if you um you want to Wait, I have it up in some other Yeah. If you if you want to be able to get familiar with the syntax, you know, and if something goes wrong here and um you know, if you are used to auto-completion and and and and text editors and and yeah, mainly to IDEs that actually have auto-completion and and type checking or there's nothing like that on NYX, unfortunately, yet. So, you're going to have to learn the syntax. Um that's why I'm I I don't actually recommend that right now to you know, unless you really want to get into it, then it's by all means, it's go go ahead and learn it, but there's going to be a lot of walls you're going to hit. But, something very useful is the NYX REPL. NYX REPL you can just come around and and and run NYX commands here
Uh we can run NYX packages, for example, and have the packages available here. And you can run anything that you run here, assign variables, and and so on. So, whenever you want to to play around and see how recursion works in NYX or how stuff you can evaluate something very quickly on the REPL. Um but then yeah, let's say we want to Let's generate an error here and see Let's see what kind of uh Oh, this one is very complex, so I'm not going to show it, but uh uh let's say that we we forget we forget the semicolon here. I have some some linter here, so it will actually let's me know here, but uh um Okay, um so, this is the NYX shell. Let's try to run next shell. And on this case the error is very nice, right? So, syntax error, unexpected bracket. So, this one is very easy to to actually figure it out, right? So, like I but you need to get familiar with next the language in order to know that it actually expects a semicolon here
There's no other way. You need to learn it. It's like you want to write Scala and you get a compiler error. In order to fix it, you need to understand a little bit the Scala syntax, right? It's very similar with next the language. Um what else? Right, let's say that we forget to pass this this argument. We pass an empty argument. Let's see if that's going to work. And this one is a little bit cryptic, but still understandable, right? We say anonymous function
Anonymous function in package.nix called without required argument shdk. I think we can all make sense of that, right? If we go to nix packages here, we have an argument. It is a function that expects an argument and we didn't we actually call this function without passing that argument. So, I think nix got a lot better now at the error messages, but it used to be worse and then kind of hard to follow. Um so that's pretty much like that. This these are very small examples, but yeah, I think we're running out of time to go into more details. Uh but yeah. Yeah, this is a great answer
Thanks. All right, we're just going to wrap it up here and yeah, I guess thank you thank you to all to all of you, all the organizers and all the people showing in on Twitch. Yeah, it was really a fabulous talk. I'm even more excited about using nix. You know, using nix it really was an really thank you for taking the time to join us and And everyone on the on the on the call and on Twitch and Zoom as well. Thank you for joining us. Before we go, just two two messages. Firstly, Alexey you want to have a few words
Alexey is my partner in crime with us at Scala meetup of course, yeah. Over to you Alexey.