SF Scala: Magnus Rønager, Tradeshift Go
Recording: SF Scala: Magnus Rønager, Tradeshift Go
[Music] so I have this presentation also but this mainly about treasure should go a little bit of our trade shift we company founded nine years ago started up in in Quebec and has later expanded to San Francisco so China so Bucharest's France England and I think now we have 20 locations or more so so we have expanded quite a lot always looking for new people we are in the in the business business area primarily working with invoices Ian's vices and end a lot of other areas including trade shift goal which I will talk about now all right start a little bit on myself I'm manos Vanna I'm from Denmark I just relocated to San Francisco four months ago and I love it so far a penetration ship for almost three years and started out as a site reliability engineer and I kind of still am a little bit but also transition is a software development I've been very heavily involved in setting up the trade shift infrastructure both in globally but also in China and we have scales from few hundred servers to thousands of servers within a short period of time okay so trade ship Co let me start by asking a semi random question was in the last year how many has had their debit cards canceled by the bank for fraud alright I thought thought of Livermore myself I had one last month actually someone misuse my Danish debit card for almost $1000 that was pretty annoying and haven't gotten one yet so trade ship go will actually this is one of the main killer features for me at least solves this problem so essentially you you create a bunch of virtual credit cards to have one source card and then based on that you can you can request a new virtual card with with a fixed limit of money so for example I want a credit card with $15 which I'll use for umbrella then you go in and requested someone else your manager prove it and you can use it for that purpose right and for that inside edge of goal you will also see a list of transactions for that card so so you can kind of verify it and it works very well with the expense department so that's trace of go what we are working on with with Scala is kind of the bag and part of it so this is this is tray of cards so trace of Cass is acting as an interface for multiple card providers and here you can see that that we have one interface so so when you when you register a virtual card it's kind of agnostic of which which care provider you're using we will identify that and and send the call to to the existing bank generating the card giving the feedback back to the user and the user doesn't have to worry about that and all consumers of the cards doesn't have to worry about that because of because we're working with with with credit cards and debit cards we have to be PCI compliant so there are a lot of regulations for how we treat the data and where we could put all data coming from the side reliability role I think this is super interesting also all right so going further into a trash of cards we are using cqs and event sourcing so command query response segregation so just quick how many are using this for any application currently yeah that's not many I thought it was more anyway the the whole principle about it is that you kind of segregated up into commands and events so an example could be that that you use it as a create account command that is just an issue and then it will it will take that and and do something about it produce an event that is then persisted in some event stall right so so based on that you can then you have a stream of events that you can just continuously pass you can roll back and replay the events if you have some data problems of course this this means you have have eventual consistency with with your application but one of the strong sides about this is that you can actually decoupled the writes and the reads so if you have a traditional Postgres database it's super good for reads but but you can very easily be limited by it by the amount of write you can make so here you can you can use another data store for writes which could be Cassandra or Kafka or any other queueing system that is actually optimized for for heavy write loads and then just do it from a stream which is also super optimized for this and then even though you fall behind we'll just continuously pick them up and materialize it in in a in a rich strong database for example Postgres or elasticsearch so this is how we have implemented sakura-san event sourcing in in cots so if you start by by an actor so that could be treasured goal or user since an API requests for example create an account that will then be passed into either a write or read command and if it's a write command it will go to to the akka access to its magic there and then persist the event in Cassandra which could be account creatives the center will then temple is a stream of new events to back to to the application and then materialize that in in both the database and also we are using a data dock for for matrix right so once it persisted in in Postgres we can actually start reading stuff so when when you do the the next API request for example get all accounts then instead of going directly to the Excel we can actually read it from from the database instead and this is where it becomes eventual consistent because there will be a delay from from persisting in in the active state and all the way through the Cassandra and enrich your last any questions about this so okay so so the question was about eventual consistency and and the banking model so we don't we don't store any source of truth in our application so we don't have actual financial data yeah but but if you look at at the reads if you get one account for example then it will be directly from from the AK estate and then it's not eventual consistency but but directly except but but it's mostly if you get lists and and stuff where you have to read from the materialized views and then it's eventual consistent but it's it's just it's it's mostly for display the stack we're using is so so for our service we are using Ubuntu and for all the components we are wrapping in in a target container which means that so so for chip cards we are using Scala of course so so we have the target contain a specialized for this we could have other components using note or a Java anything like that and they have specialized containers for that we using Scala akka and Farakka we're using the the modules access HTTP stream evasion spray for adjacent passing and then persistence and clustering for the message sterilizations like commands and events we are we are sterilizing them with protobuf so we can send them around to the different axis because we are clustering it like mission we are using Postgres for the materialized views and cassandra for for the events tours and at the database access libraries where you see slick and then recently we have converted from SPT to to maven which will give us benefits for better integration with our CI infrastructure gives us a more streamlined for the rest of the components and phase shifts and will give us some other capabilities for doing some component testing for the note discovery of someone was burning up the cluster we are using no discovery by DNS and the DNS is resolved from console so how do we develop and trade shift we use github and we support model so basically we have we have to make master ripple from from trace shift and then you take for get out branch off and then create a pull request back to put to the master branch get up here to review it make sure that it lives up to the standards be sure they have small pull request so this is a review and then we just iterate fast everything is being build with Jenkins we have Jenkins files of that so so we can kind of explain how how is it's a component being built and then we don't need to have backups of Jenkins job and stuff like that we have both compute unit tests we have quite extensive unit tests and we have recently written component tests and we defined components as here by spawning up the entire stack of cows and mark all the dependencies of that and just hammering all the API requests we can to verify that the request and response are or what we expect we have remote specifications for for documenting the the API so we can give it to to tresh of go who is the primary consumer of cars right now we try to deploy it several times a week to small iterations so so we don't risk building up a lot of code that that could put everything in jeopardy we encourage everyone to deploy so whenever anyone merges anything just just can't apply as soon as possible everything is fully monitored we come from the right background I I care a lot about this so so I monitor everything all the time we have hooked up to today to dart which is and this has a bunch of alerts set up to automatically alert on slack and pages duty and and will notice immediately if something goes wrong for exceptions we use something called century which space is just aggregating all the exceptions we are getting and then we have it hooked up to so slack also so we can see whenever a new exception is and then we'll tackle tackle it immediately and fix it and deploy there are some interesting questions our problems with with cats so one of them is PCI compliance which is there's a lot of rules here and we need to have audits every single year so one of them and this shouldn't scare anyone off the mean this it's just handing it then and creating resolutions around it so one of it is how we handled all the sensitive data and make sure that that we don't risk leaking any of it at all how do we handle it how to start encryption segregation of network and so on lots of interesting is a real problems in that and problematic of course now if the problem so we do integrate with with different copywriters and and this is of course a problem deal with with external API is going being slow or suddenly being dead they could break the contract by producing different results and expect us and this happens and another interesting one is developing feature parity with the different card providers so if we already have a specific set of features for one or two care providers whenever one bought new ones we need to make sure that that they can do the exact same as the first one or we have to to implement this at our end that was just a quick introduction of cards and and how we're using it in trade shifts any questions yes so actually the the program was it was selected when this project started more than two years ago but it it's it's it's a good series Asian format that is it's fast to pass and and has an overhead so that makes sense so in in cards we we don't use it we have we have some initiatives and to achieve where we are trying to explore the areas but but for now not in in coworkers so for compliance the question was do I encode them in tests so so yeah the question is how do we test for for compliance we do have a lot of testing for example if if we have to deal with with any credit card numbers or any sensitive data we are verifying that everything is masked probably and and we're not writing anything to locks other than that we just ensure that the network is segregated and and that we live up to took the compliance rules we do get some modest every year like mentioned and if there's any violations that will ensure that it's patched yes so what I characterize our Scylla is more object oriented or functional now I would say it's it's is of course a mixture of both it's probably more in the object oriented in this fashion but we do a lot of functional awesome we are hiring we have job openings both for Scala and if you're exploring other areas like Java or Co note less less job opportunities thank you [Applause]