scale.bythebay.io: Thayne McCombs, An introduction to Xtract
Recording: scale.bythebay.io: Thayne McCombs, An introduction to Xtract
good morning I'm seeing McCombs I'm a software engineer at lucid software which is a located in Salt Lake City today I'm gonna be talking about an XML library for Scala to deserialize XML documents into Scala structures a little bit of background a couple of summers ago we worked on implementing indesign import where we would import idml file indesign idml files into lucid presses JSON based document format so idml files are essentially just a zip file that contains a whole bunch of XML files inside of it continued actual data for the documents so we needed to convert from XML into Scala objects and then perform some transformations on that and then serialize it into JSON so for our first attempt we simply use the built in Scala XML library to access the elements of the XML document and then we would convert those strings into whatever types we needed in form validation on that [Music] which worked okay but we start notice that we were started doing the same things over and over again and wanted to consolidate that so the sort of things that we wanted to make better is validating the content so making sure that the XML was what the way we expected it to be handling optional values being able to convert to non string values into or convert non string values from the strings easily and then also an important thing for us as we want to be lenient with bad XML but still keep track of the errors so that we if the user gave us a malformed document we could give the SIL converted into a lucidpress document but tell them what errors we encountered so they can know that it maybe didn't import correctly so at least that all of our services are play Scala services so we had some experience with the play JSON library and we sort of wanted something similar to that but for him working with XML instead of JSON so after looking around and not really finding anything that really met our needs we decided to create it ourselves and we've open-sourced that and it's available on github and maven central so extract has four main parts to it the first part is at the XPath class which represents paths in XML it's a way to address specific nodes or sequences of nodes in an XML document parsed result which is omona that contains a result or a list of errors it's very similar to like try or option for representing the result of a operation the XML Reader which is really the heart of the library which is just an object that describes how to how to parse a chunk of XML into a scala object and then finally some combine it or syntax which allows you to combine a whole bunch of smaller XML readers into a more complex XML reader yep so XPath okay there you go it's based on the XPath syntax but it's a little bit different partly because of limitations of Scala syntax so we can't use double forward slash as an operator because that is the comment delimiter in Scala and then we use parenthesis instead of square brackets because that's how you do indexing in Scala so with a single backslash you can access child elements a double backslash you can access descendants with parentheses you can access the element of a list of elements backslash at allows you to access an attribute of an element the with outer methods can be used a filter based on an attribute of an element so either elements that contain an attribute or contain a specific value for an attribute children just gets a list of all of the children of an element and then finally the read method allows you to take a an XML path and then pass it an XML reader and it will apply that reader at that path so it will basis entually find the element at the path that you specify and then apply the reader to transformation at that specific path you can get an overall reader that converts the dog it will parse that specific path the parsed result has is a sealed trait with three case classes success failure and partial success so success and failure are pretty straightforward a success wraps a success a successfully parse value a failure it contains a list of errors that were encountered and then a partial success has both a successful value and a list of errors that were encountered so you can keep track of errors but still give a successful value and luckiness before similar to an option or try you can convert it to option with the two option method and it's it also implements methods like filter flat map map etc so you can operate on it like that XML reader is essentially just a function that converts a node sequence which is the base type for elements or a list of elements in the Scala XML library so it's a function that takes a node sequence and returns a part results but the object also defines additional methods like map flatmap compose etc and then the extract library provides many predefined readers for things like int long option boolean sequences basically all your basic Scala types so here's some sample XML now we'll be using in the sample code and I'm not to show you so it's just a some XML representing a blog post so it has metadata in the head we have some attributes and some tags all right our sample project so in our in this mean method all I'm doing is I call XML reader up can you see that they make it bigger not good enough okay [Music] so this XML reader of blog will find will try to find an implicit value of the type XML reader blog and then we calling read on it is the methane which will actually transform the XML into the blog type and so this here returns a partial results of type blog and then we just print that out here in a real application user Poly's pattern matching to extract the actual value that you wanted or handle the errors if that's the case maybe no not that one all right so not on the companion object for the blog case class I would defined a implicit reader or implicit Val of type XML reader blog and this is how the excellent the external reader of method found the enclose the object so a typical use case is to define an implicit reader on the companion object for the type that you want to read so that it can be found by scholars implicit lookup so there's a couple of imports that are important to have when you're defining the readers so the external reader class objects obviously so you can define the types also on this double underscore here similar to in plays JSON library the double underscore is used to represent to the root of the XML document and you'd so you can create XML pass by starting from double underscore and then traversing down importing underscore from XML reader gives you all of the predefined XML reader instances which are mostly implicitly implicit values and then finally importing play API libs functional dot syntax to underscore provides some implicit deaths that allow you to use some special syntax to create XML readers so if we look at the reader for blog yeah in basically what idea is I create a whole bunch of smaller XML readers and then combine them by using this and operator and the so combining them with the and operator creates a builder object which then has an apply method on it which you pass a function which takes a number of parameters of and that has account and type equal to the providers Earth XML readers that are provided with the and keyword and then converts those parameters into the desired type so in this case we just pass it the apply method on the companion object for blog which is essentially the constructor for blog so it takes the values for each property and blog and creates a blog object from them so we use XPath syntax here so for example the first one we take the root of the document then we get the head element of that and then the title element of that and then we call dot read of type string and this will find an implicit XML reader of type string and use that to parse the title we there's if you call the optional on it then if the element is missing then it will return in none and then of course it wraps in the sum if it is there similarly the default method will provide a default value if it an element is missing this attribute method allows you to specify you want a specific attribute it's a sort of a shorthand for you for using double underscore back slash at and then the type and then you pass it a XML reader and and the ANU method allows you to parse an enumeration using the with name method you can also use for comprehension syntax so here we are create using an XML reader for the first name and the last name and then since we use this for comprehension it will create succeed if both the first and last names are parsed successfully but if either of them fails then it will fail and then it concatenates the T together and then here we're defining a name reader of type string so we explicitly use a string name reader a string xml reader here rather than using the implicit one that's predefined you can also use the sequence method treat a sequence of elements and again this is will implicitly find an XML reader of type section and then apply them to each element and a list of elements so here we're getting all of the sections and applying the section reader to each of them and then we call a map here so we take each so that we can convert all those sections into or we convert the sequence of sections into a content which is just wrapping a sequence of sections any questions on that okay so finally for a partial parse or for a partial par success the typical way that you create it is with recover which you where you pass it a default value and the sequence also by defaults will create a partial par success and you can use a strict sequence instead if you want it to fail if any of the sub elements are invalid so and recover it works a lot like the default method but it returns will keep it will store an error if the parsing fails and continue with that if you want to define a XML reader for a recursive type then you use lazy read which uses by name semantics to enable you to define our reader in terms of itself and finally moving forward I'm planning on changing it to use cats instead of the play XML library or play functional syntax library because it's a lot better documented and better understood so any questions yeah I need to receive a sequence right yeah so the way you would do that is you define a reader for the type for like the seal trait you could provide a single reader for that or yeah you have write us or you could provide separate readers for each of the case classes but typically you define a an XML reader for the sealed trait and then you would if it was implicit we could automatically find that when you use the sequence or you could explicitly pass it into the sequence function so so you so you can also define XML readers as as a function that converts it gets passed in a note sequence and returns the press results so depending on how you just determined which instance of or which taste class to use for that trait you you could put that logic in the function and then use that to figure out which treat that class to use right thanks