SF Scala: Chris Stewart, Bitcoin in Scala
Recording: SF Scala: Chris Stewart, Bitcoin in Scala
[Music] you have a Bitcoin protocol implementation called Bitcoin s so back when I got into Bitcoin back in 2013 2014 there was one primarily implementation for the JVM called Bitcoin J and a person learning about Bitcoin you know I wanted to get into some proto or get into some project that I actually could maybe like start to understand how this like crazy currency works you know whenever anyone gets into Bitcoin I I think most technical people I talk to have this you know initial gut reaction that's like this is a scam or a Ponzi scheme or some pyramid scheme or whatever another one of these you know 2008 financial crisis tricks that are getting played on you so I mean like I wanted to go in and like look at this stuff and looking through Bitcoin J was it was it was tough and I thought I could bring some value to the ecosystem by creating this kind of robust implementation in Scala so talking about like the philosophy behind the development of Bitcoin s is I'm very influenced by this guy named Yaron Minsky and he actually talked at SF Scala I think about a month ago he's a Wall Street trader they worked for a firm called Jane Street they used oh camel is their primary programming language and he has these old YouTube videos on there where he talks about doing development for things that handle money and his biggest takeaway from like working with things that directly act interact with money is you should be terrified that at any moment you know you can like write this tight little loop to light yourself on fire and lose like you know a ton of money so whenever you're doing development with things that um you know directly interact with money in this case Bitcoin um you need to be careful and you need me make sure you're taking certain procedures to hopefully you know be as safe as possible so the case of Bitcoin it's a bearer asset so what that just means is you know once you send the Bitcoin to say a malformed contract or a wrong address it's it's gone forever there's no way to redeem it I mean you can't go to the bank and say hey I screwed up and giving my money back so the stakes in Bitcoin are pretty high and I think Scala is a good choice of a programming language to hopefully avoid those mistakes as you guys all know you know Scala statically typed that's a huge win in itself over some programming languages like JavaScript which terrified the living daylights out of me for handling Bitcoin stuff but it's also you know offers things like immutability and you know how to break datatypes and stuff which is very nice so another thing that Bitcoin has tries to do is encode invariance into the type system so as we all know Scala has a very expressive type system that allows you to do all sorts of cool things but it seems like at the end of the day it's you know encoding the simple things into the type system that I found to be some of the nicest stuff so for instance a Bitcoin private key needs to be 32 bytes in length if you don't have the 32 bytes there's there's something off about it and you could possibly do something wrong with it so you know putting a little require statement inside of the private key implementation saying that you know you need to follow these specific formatting rules can you know be just a simple tool to make sure you don't do something wrong with your money another thing I've found extremely useful and this isn't necessarily specific to Scala but it's property based testing so Scala has a library called Scala check that I use very extensively in Bitcoin s and the idea behind property based testing frameworks is you can kind of specify these mathematical properties that should always hold true so going back to the example of a private key again it's an example of a property for this would be like for every private key that I generate they must follow this specific rule set if it doesn't follow this specific rule set I must be generating this wrong in some way shape or form so I need to go back and look to see what the heck is going on and figure out where the bug is my implementation and then what's really nice about creating these properties is you can kind of build libraries of them and have them propagate up your system to deliver stronger and stronger guarantees and invariants about you know different layers in your system and the end goal is to use these uh with the end goal T is to use these things to allow other people to use these things in their own testing projects in Bitcoin s I want to provide a robust set of what's called generators for other application developers to use if they're building a Bitcoin application so for instance say if you're building a wallet and you want to you know handle all these different Bitcoin transaction types you can use the set of generators that I've given you in the Bitcoin s library to test whatever your application functionality is instead of having to think up all these corner test cases that are you know kind of maddening to sit at your keyboard and really think about so hopefully that's and then there's a whole like package in Bitcoin s that provides these generators so how many people follow Bitcoin fairly closely fairly closely yeah anybody own Bitcoin okay cool so Bitcoin is going through a political battle right now for those that are unaware and I not even thought when I got into cryptocurrency that it was a political so that means you know there's no way you can bring politics into you know monetary policy then you go back and look at uh then you go back and look at also the history of about every other internet protocol and there's like you know knock-down drag-out fights and the not you know not actual fights but you know these long winded battles on forums and in real life conferences I guess in the IETF like you know debating parameters on a protocol but anyway so Bitcoin is going through one of those and one of the features that is kind of a mer from this political fight is called segregated witness and this is just a new feature in the protocol to make sure that we eliminate something called transaction malleability and trans and actually this allows for cool things like what Fabrice is going to talk about a little bit with the Lightning Network is this wasn't possible before well it wasn't easy before we had this thing called seg what and what seg what does is that if effectively removes this thing called transaction malleability so people you know bitcoins this distributed peer-to-peer network where you propagate transactions around the network that means any node that you propagate your transaction to has a opportunity to modify it and they can modify it and clever little ways that'll like change the identifier for this so as it's passed around the network maybe you modify it in one way then you modify it in another way then the guy and the back modifies it another way and so like which one's the like real one they can do this without invalidating the transaction actually which is frustrating for tracking features since you don't really have a you know a canonical identifier that you can follow it's couldn't constantly change but when being passed around the network so some a group of guys smarter than me figured out how to do this and implement it in Bitcoin core the C++ reference implementation and now I've got it in Bitcoin ass so this is like a really cool thing that allows a lot of layer 2 solutions to be built on top of it so again Bitcoin s has all these generators which are what is used to conjure up the data for property based testing so for these generators we have some crypto primitives so that's like your private key your public key you have scripts which are the smart contracts that actually lock Bitcoin up on the Bitcoin blockchain you have transactions in blocks so I'm going to show you what some of these generators actually look like and the very most basic generator is the private key yeah so the you know the most important thing in Bitcoin is probably you know the cryptographic private key that actually can be used to unlock your bitcoins and send them somewhere so using Scala check we've created a little little generator that says okay any time you call this just create a new lip to curve private key and you know it's it's very simple you know it's just a call to a constructor that you know get some random bytes and you know puts them in a little case class but you know using this you can like build more complex generators so for instance here is the public key generator so you know here we're just calling flatmap on the private key generator you're getting this private key and then calling the public key method on it so now we have generators for private and public keys so now why not a generator for a digital signature or a double-shot 256 hash that is the unique identifier that I was talking about earlier that bitcoins use or Bitcoin transactions use as a unique identifier so building this like little suite of libraries is relatively simple to start but as it keeps growing you can do very powerful things with it so another feature offered in Bitcoin us is what is called Lib SEC P 256 K 1 for any crypto nerds out there that's a lichte curve that Bitcoin uses so there's different curves you can pick for various use cases in cryptography and bitcoins founder Satoshi Nakamoto chose this curve I mean I think it was a relatively arbitrary choice I don't know if yet any reasoning behind it or if it was just something he went with but anyway with this lib SEC P 256 k1 is like a sea library that's optimized within an inch of its life to process these bitcoin transactions to be able to run a Bitcoin full node you have to validate the entire history of the blockchain and this is extremely expensive computationally it takes a while to actually get all this information from the blockchain and then you need to validate every digital signature that occurs in this blockchain to make sure the transfer of ownership from one coin to another from one person owning a coin to them sending it to another person is valid if you don't process all these digital signatures which I I it's in the millions and millions and millions of digital signatures at this point you can't make a straw you you don't know who owns what basically so before we had this Lib SEK P 256 K 1 it would take I think was like three or four days to be able to sync up a full Bitcoin node and that reduced it down to like I think it was like 12 hours or something at the time when it was released in Bitcoin we had this kind of ongoing problem though that the ledger keeps growing and growing and growing and I think currently it's 160 170 gigabytes in size and you know a blockchain is just a fancy append only data structure where you someone needs to keep around the entire transaction transactional history to make sure we can figure out who owns what anyway bitcoin s supports this libtech P to 256 k1 functionality to be able to process these digital signatures faster and hopefully validate the entire blockchain state someday so the next feature that I've got listed up here is types to imitate C's unsigned integers and this is uh so again the Bitcoin protocol is written in C++ as you guys know the JVM has signed integers C++ has unsigned integers so you get all these weird cases when you're dealing with you know the Bitcoin protocol because you know you know 32 bits and a signed number or with 32 bits an unsigned number is 31 bits in a sign number I think I have that correct so there I decided when I started getting really into all this stuff I was initially trying to keep all this in my head and it was very frustrating to figure out when to do these conversions and when I needed to do these conversions so I decided to create you know little types to represent these things and they've been very very useful for me and I've provided you know like simple arithmetic operations on these types but they could use some work too and this is a you know one of the reasons I'm giving this talk is you know a plea for you know people willing to write free software you know on the on this open source project so for instance if you don't know much about Bitcoin but are interested in it this would be a great place to start contributing is you know something as simple as an unsigned integer and figuring out if there's any bugs in like this implementation or looking through it to see if maybe we we should support like overflowing 2-0 or whatever maybe we shouldn't maybe we should be returning try types and encode a failure if an overflow happens it's all like you know I guess food for thought and like what's the best way to have failure scenarios whenever an implementation diverges so another thing that has been interesting here with the Bitcoin protocol lately and the guys that maybe own a little bit of Bitcoin might have heard about this is a new address type is coming for Bitcoin so just like an email you have email address that is in Bitcoin you have Bitcoin addresses they usually start with a 1 or a 3 and then a bunch of gibberish after that and that's just you know a cryptographic hash I guess there's been problems with the previous addresses where people will for instance if they're transcribing it right at and then if that happens they lose money forever right there's a pretty high stakes here if you don't write these addresses right so the Bitcoin protocol developers decided to get together and try and come up with a new address type or address format to be able to catch some of these bugs and make sure you know we don't have people sending a million dollars to an address that no one can ever unlock because you know what's losing a million dollars right so anyway I've got a pull request open for this and I'm gonna show you guys a demonstration of what these things kind of look like here in a little bit but the pull request is at this URL here and this is a an interesting thing about Bitcoin in general is there's a large demand for peer review in Bitcoin going back to what you're on Minsky said is like if anytime you're writing code you should be terrified because you know you have an opportunity to lose money Bitcoin core has an excellent peer review process the C++ implementation has you know developers that have been contributing now for 4-5 to 7 years I guess in some cases in every pull request gets reviewed by multiple people before it gets merged into the codebase and that's like what really needs to be done because you know this is a huge financial application that if something wrong gets merged in you know a bunch of people can lose a lot of money or there can even be developers with malicious intent and you don't realize I mean maybe it's even obfuscated if anyone's seen the obfuscated C contest before you know people can do extremely clever things with a C and C++ but peer review is really a cornerstone to any open financial system or blockchain in my opinion and this is another one of those I you know I'm gonna give you guys homework or whatever it's like you know if you're interested in this stuff take a look at this pull request the Bitcoin stuff might be a little hard to you know understand at first but you got to start somewhere or you gotta kind of crawl before you can walk and you know they always say with Bitcoin or anything else in life you know the best time to start with six years ago and then the next best times to start today right so so yeah let's check out some examples of Bitcoin s and we will open up our friendly s okay can you guys seen the back this large enough you good okay we're gonna import a few things to start so this is kind of where the entire Bitcoin s core project resides and we are going to import the crypto package so let's start with the basics this creates a new private key okay so this is you know is it exactly as it looks this long hex string here with a boolean at the end I guess is a 256 bit private key that would be stored in a Bitcoin wallet so these private keys are used to secure ownership on a Bitcoin network and as you can see with Bitcoin s you know you just have this little constructor here you call to generate this new private key is everyone with me so far good okay now the next thing to receive a Bitcoin payment you need to be able to derive the public key associated with that private key so underneath the hood here there's some fancy math happening and you know there's a crypto library that I call but we will just call the simple public key method so again we generate the public key there so this is when the Bitcoin protocol started Satoshi only had one contract type and that contract type was called pay to pub key so to receive a payment from somebody else you would just send them this big old hexadecimal string here and then your counterparty would actually create a contract and send you this public key so we can actually look at what this contract would look like so you know the big buzz is like smart contracts or whatever you know there's one of the buzz words in the crypto space anyway I think they might be a little misleading I think more like dumb contracts cuz you can like actually execute them on a computer you know computers aren't really that smart they do exactly what you tell them to do smart contracts for like you know you get all these in bit ambiguities like legal contracts and you take them to court the judges got to figure it out what's going on these are you know dead simple contracts that a computer can execute so we'll call this pay to pub pay to pub key okay cool so what is in this string here it's a contract that is encoded in hexadecimal so Bitcoin inside of it has a little programming language that you're gonna run these instructions against to figure out if the right person signed this transaction so I'll print out what these instructions look like okay so this is what you need you need to get this program right here to evaluate to true to be able to send bitcoins from one address to the other address I'm gonna give you a little bit of details on what this program actually says so with this with the Bitcoin contracting language you need it's kind of similar to like assembly languages if you've ever used those before I think that real definition is like it's like a fourth like language which I'm not familiar with fourth but I mean that's I guess what it is so the very first instruction here is specifying how many of the next bytes to push onto the stack so this the shared state in this program execution is the stack so anything you need to do or any variable you need to store needs to go onto the stack for you to pop off later and have an opcode use it so the very first thing is pushed 33 bytes onto the stack these next 33 bytes here is this public key that I derive for my private key earlier so this public key will be pushed onto the stack and then the final operation here is up check sig and this is one of the heavy lifters in a Bitcoin protocol that says you need to check that this transaction is signed by the private key that course mathematically corresponds to this public key so this is how you get these ownership properties in Bitcoin so this is one of the oldest like contract types that Satoshi eventually began with I kind of have like little helper functions to be able to like translate it to more of a human readable form because this like nasty hex definition at the top doesn't really tell anybody anything so this actually has gone out of style though no one uses pee to pub key anymore because you can't derive an actual Bitcoin address from and actually I don't know do you know why they never specified an address format for pay to pub key because I just know we don't anymore yes segue so it's not this never was used like widely outside of like the very beginning people in Bitcoin because it was very hard to exchange this public key and if this public key is embedded in the blockchain for the entirety it's basically embedded in the blockchain right away so actually quantum computers could possibly break the discrete logarithm problem that's called which is kind of the problem that makes cryptography really hard and reverse engineer the private key so I think that might have been one of the big reasons I they didn't go with this contract format so they created a new contract format called paid to pub key hash so I'm gonna create a new script pub key pass in the pub key from before that corresponds to the private key I first generated and now I get this long nasty hacked hex string and we'll print out the instructions for it so here is the contract that still corresponds to that exact you still need to provide so you still need to sign the transaction with a private key that's corresponds to that public key we had before but there's a few other things you need to satisfy and namely you need to get you need to provide something on the stack that hashes to this value right here so a way that the protocol developers figured out to get around this problem of quantum computing possibly having a big breakthrough and like everyone's bitcoins can be stolen right away was hashing that public key and saying okay when you want to spend these bitcoins a you need to provide the public key that hashes to this value here and then B you need to sign the transaction with the private key that corresponds to that public key so this is the in the Bitcoin contracting language is how you go about achieving this you duplicate what's on the stack top you hash that stack top value that you just duplicated you check to see here with this OP equal verify if the two things on the stack are equal if that's true then you check the signature on the transaction to see if it was signed with that public key so a lot of a lot of steps there and this is why some people have gripes about bitcoins contracting language they'll say there's other programming languages that have you know simpler or more traditional programming idioms so like you know for loops or while loops or you know syntax that doesn't look like this but it's done this way for good reason and you need to be careful what you can allow in your smart contracts too I mean we're all programmers here I mean how many of you have written bugs before it's it's super easy with a project that as a cotton competitor to Bitcoin like aetherium I mean there's already been multiple instances of people writing smart contracts and having them hacked for 30 40 million bucks this is you know the dual edged sword with you know these contracts is you're writing a program that if you can get this program to evaluate to true you can send money wherever you want so you have to be very very cautious with doing this so now we have this paid a pub key hash contract and now let's encode this into a Bitcoin address that some of you have seen before so using Bitcoin s we're gonna import a couple things now we're gonna create a value called Paita pub key hash of adder and call the constructor and we'll have to specify the network that this goes on so some people don't know this but Bitcoin has multiple networks there's main net which is the one we've all heard of before that he has like a hundred billion dollars worth of value on it but then there's a test net the developers used to you know tinker around you know that's where you can really have fun I'm going to specify the main net as the parameter parameter in this address because I will show you it's nothing to do with the actual contracts itself it's for humans so if we drive mm-hmm yes so in this case you notice how you have the one on front which is if you've ever received Bitcoin for payment before you'll see usually they have a one on the front now if I would have changed this to tests net and see now this one starts with an M so that's like an easy way if you're a human you're like I don't want to actually send money real money if I'm you know tinkering with my application or whatever so now you can like send this over your friend and do some crazy contract that maybe I'll lose your money so yeah now again I was talking about bit you know these address types have problems namely there's kind of weird characters and there's mixed case so say if I'm talking to you and I'm trying to relay what an address you need to pay me at is I need to be like it's lower case M lower case R capital case W lower case V you know blah blah blah it's super easy to mix one of these up so one of the features the that these new address types have is like they're all the same case but to be able to illustrate this and this is another powerful contracting thing is that was in yeah I've added with segregated witness is versioning of these contracts so in this old system I was showing you here there's no way to version on this just you know this is the smart contracting language so you know that's the way it's going to be and doing what we call hard Forks which is a backward incompatible change to Bitcoin it's very controversial again we're sharing in a hundred billion dollar currency here and you can imagine getting everybody to agree on like you know what the color of the carpet is is hard enough rather than changing rules and a hundred billion dollar currencies so we try to avoid or actually bitcoin almost has avoided hard Forks all together but the bitcoin developers figured out a clever way to create a versioning system for this smart contracting language that was talking about so we can you know we maybe we find some cool new cryptography that we want to use and that's a lot of what this conference I was at this weekend was talking about we have to have an upgrade path to you know say something breaks to maybe uh the cryptographic algorithm the Bitcoin uses currently is out of date because computers got super powerful or the discrete logarithm problem that I was talking about earlier broke we need a way to upgrade and so I'm gonna show you what these new contract types look like so we have these new things called witness script pub keys and there's just a new format of these contracts we're gonna pass in this pay to pub key hash contract to this and we again get one of these long nasty hex strings but let's print out what the actual instructions look like so we again this looks a lot like the very first one I showed you guys but the most important thing here is this OP zero and what this means is this is a witness version zero this is the you know first version of these contracts so whenever we figure out something new we can upgrade and be like okay you know now we're talking about version one and we can use this cool new cryptographic algorithm or some crazy new smart contract and feature that's hopefully safe but then you know it checks the witness version and then pushes this hash on to the stack and then the next thing must that means that program that's nested inside of this must run hash to valuate to this exact same hash here and then execution continues like before and I'll show you guys what the new address type looks like so this is a little bit of a preview the Bitcoin protocol is going to be upgrading to this 0.15 dot two I think is what it is within a couple months I think is when they're rolling that out at least in Bitcoin core so this is what the ANU address types will look like when they come out and you notice here we have this separator value one so everything on the left-hand side of one signifies what the network is and then everything on the right-hand side of the ones signifies what the actual witness program is and then if we were to replace that with test net we got a TBH here that signifies and then actually when Fabrice gets up here I don't know if you have any examples but Lightning Network that you know layer above Bitcoin has their own address scheme so it's this new address scheme is like pretty flexible - which is really cool that you know we can build layer two stuff with it and do all sorts of crazy things okay let's get back on track so the future you know I mean I've already plead with you guys to you know throw me a pull request or some code review but we're actually where I want Bitcoin has to go in the future is we currently don't have any native wallet support so that means storing private keys storing smart contracts that you need to spend coins and then an easy transaction builder type so that's to build a transaction right now in this library it's involved we'll say there's no way you can expect third party developers to get that involved into it and actually it might be a little unsafe for them to do it so a nice way to you know say spend coins to this address sign it with this key maybe use this subset of coins to fund this transaction and then maybe do a sanity check at the end to make sure you're not sending an absurd amount of money somewhere so another thing is an S PV node so what s PV means is simple payment verification and that is a lightweight way to process Bitcoin payments so full node is when you process every single transaction on the blockchain and you you can't do that on your smartphone right it's just not practical you need a lot of you know you need a beefy computer to be able to do it without it taking forever so what an SPV node does is it uses cryptography in a clever way to check if a transaction was included in a block and then you're making an assumption that if that transaction was included in a block the minor must have thought it was valid so another way of saying that is you're trusting the minor that they did transaction verification for you which is a little bit against the ethos of Bitcoin because everyone should try verify their own transactions but then there's you know practically the practical side of it too and then maybe roll that out to a full node eventually but writing a full node in a different programming language can be dangerous there is always going to be discrepancies between a code base written in one language and a code base written in another language there's you know implementations of various different internet protocols like say HTTP SMTP you know they all have their own different implementations but in this case we have you know again money at stake and the key innovation behind any blockchain is coming it's converging on who owns what so that means if I have a scholar implementation that thinks transactions are valid under these certain cues Kait are certain cases while this C++ implementation thinks it's valid under a different set of circumstances you have divergence of the ledger you have divergence of who owns what which can be dangerous because that means you could accept a payment that a bunch of the rest of the network doesn't think is valid so that means you really haven't accepted a payment but your node thinks you do which is know a really bad thing it's like someone handing you a hundred dollars that you think is real and everybody else looking around he's like that's fake money so again there you know there's questions about it and maybe it would be interesting to do but you have to caveat that with the risks around it so an RPC client and what this RPC client would do is you know stand in between Bitcoin D and any Scala application built on top of it and call the C++ implementation get the message back relay it and you know converted into native Scala types and then your application can do whatever you want with it and then I mean the biggest thing with Bitcoin us right now is keeping up with Bitcoin R&D it is moving at a brac breakneck pace people and I don't think it's gonna stop anytime soon with the laser latest conference I've been at there's a lot of interesting stuff in the horizon and it's it's it's the exciting future but as someone you know that's developing it you know you gotta make sure you have support for all these cool features or else no one's going to use them at the end of the day right so I think I'm gonna oh yeah so where can you contribute again another shameless plug I'm not shilling enough enough for Bitcoin us anyway so I I have some issues on like Bitcoin the Bitcoin s core repository and what I wanted to kind of try and do is like link to link issues to you guys that aren't Bitcoin related but they're Scala related so you don't necessarily have to be a Bitcoin expert to go through some of these issues you just need to know the Scala programming language and hopefully once you you know dip your toes in the water you can like get comfortable like this is even what I did when I learned Bitcoin originally was like you know you start this little subset that you're really comfortable with and then you know you move out a little bit further when you get comfortable with that and then you know you keep going and keep going and keep going and that's how you learn anything in my opinion so like I've got some issues on this issue tracker that they're not they're not Scala I mean they're not a Bitcoin related they're just you know me being a terrible scallop programmer or you know not doing things the right way so there's some some things on there that you can kind of dive into and hey guys thanks for having me I really appreciate you guys taking the time to listen to me talk about this project and it's something I'm very passionate about and I think you know Bitcoin it's come a long way in just the time I've been involved in it and you know there's people that keep on coming up with brilliant new use cases for it and you know it really is this like kind of unexplored frontier in my opinion here is my Twitter handle up at the top here my email and then again the bitcoin house repository for you to send all your pull requests to I can see some people already have one typed up right but yeah thanks guys I really appreciate it [Applause] [Music]