SBTB 2023: Pardis Noorzad, Cross-company data exchange for the cloud.

SBTB 2023: Pardis Noorzad, Cross-company data exchange for the cloud.

Recording: SBTB 2023: Pardis Noorzad, Cross-company data exchange for the cloud.

[Music] I'm Parise nurad I'm the founder of General folders and today I want to talk about how and why uh cross company data exchange happens um you know why does data leave the boundaries of one company into another and how is this done um I worked as a data scientist data engineer ml engineer and a manager of data teams at various companies across various Industries and over and over uh my team and the devops team and the security and legal teams engineering teams we were all involved with resolving this one challenge uh you know businesses exchange data for a variety of reasons that's how they collaborate and uh they do this quite often often um but to make this more clear um I'd love to go through some scenarios outlining why businesses exchange data um one example um that I've seen over and over and it's kind of uh similar to the first job I held at a a retail analytics and AI platform company uh is that AI companies analytics companies prefer SAS deployment over on Prem deployment uh they need to bring data to their uh from their customers to their own Warehouse um and here are some of those reasons you know um it's simpler you know all all the reasons why um any kind of company uh would rather SAS deployment over anything else uh also in the case of AI there's additional Hardware optimizations that might be necessary and so uh you know an AI company might might work with a retailer um and they need to transfer uh transaction data and return uh data um retailers might also want to send uh return data to their suppliers or uh their inventory data to their Logistics Partners um or marketing data to their advertisers uh you know this would include uh kind of customer demographics purchase history uh that sort of thing in the banking uh and financial services uh and insurance sector um an AI company for example one building a fraud detection engine uh similar to work I've done at paytm um might be working with insurance companies um and the data would include policy information customer information claims data uh Banks might exchange data with credit bureaus um to assess the credit worthiness of barrows uh and or lenders exchange data with appraisers determine the value of a property and so on um and my last job uh we wanted to work uh in healthc care uh we wanted to work with a real estate analytics company to find out where to buy our next clinics um and so we send uh booking and appointments data to that company as as we kind of identified new areas for for um new clinics um Healthcare Providers also work with pharmacies to send prescriptions uh patient information medication refill information uh they also exchange data with labs in order to send and receive results uh this data includes patient information test orders and and things like that um so now that we kind of have seen some of the motivations for uh data exchange uh between uh different organizations and companies let's go through some of these methods that uh companies use um I've had a lot of conversations with companies about this topic and so this is kind of an overview of those conversations um and the goal is to kind of um enumerate some of the very many methods that uh companies use to exchange data and what they have available today um and here's one of the most common ways that companies uh exchange data so they extract uh kind of uh data in CSV file uh as a CSV file and then set up an SFTP server or a Dropbox of box uh or send this through slack or Gmail um and uh so but like regardless of uh that kind of intermediary use the steps are similar in that uh kind of data is extracted from the cloud uh where it usually resides uh as a CSV file then moves into an SFTP server box or Dropbox or Gmail or something like that and then the other side of the transaction the other company needs to download that CSV file and then also find a way to upload it back into the cloud data warehouse data lake or wherever they may meet that data um and uh there's also other ways you know some companies set up in uh you know S3 bucket um but there are cases where uh one company wants to like control the end to-end Pipeline and so they will ask for their customers credentials to connect directly to their uh kind of S3 or their database um and to build and monitor that end to-end pipeline themselves um usually more uh kind of techsavvy newer companies will do something like this and then even set up de beum do uh log based CDC replication from let's say a database into a warehouse and things like that um another very popular uh method for data transfer today is um uh you know uh if you're a usually ass sass vendor where you know kind of what data your customers want uh you will expose an API and it's um there's a lot of benefits to this because consumers customers can choose which data they want um out of all that's available and there are standards to apis everyone can Implement and knows um apis are database independent and Tech independent which is great they're open not tied to a vendor um an API is a contract and so there is no need for additional uh description of schemas and things like that um which is also great and then apis are testable by both sides so both sides know one what they're sending they know what they're receiving they know what to expect um and um Additionally you know as a kind of um SAS vendor you can also uh or SAS vendor partner you could also Implement an API um to sorry um to pull data from uh vendors sorry I missed okay um kind of talked about the um uh sorry I missed my train of thought but uh so um you know SAS vendors Expose apis and then um you know as uh when you're kind of working with those SAS vendors and you want access to your data uh you can Implement those apis and pull pull that data um and then there are tools right now uh ETL tools that help you pull this data uh in a very reliable way um you can also move data out of the warehouse and into operational systems like MailChimp and net Suite uh using reverse ETL tools as well and so this is again another uh kind of in the opposite direction uh of still using apis for data transfer um yet another method that companies use for for data transfer is uh by uh exposing an SDK um or building an SDK that your customers can Implement in their apps and and things like that um it's not easy Integrations take time um but the data is very reliable um every change kind of in the SDK requires uh D you know the app to be tested deployed and shipped again um and then there's uh these uh new methods uh of sh sharing data where you're not actually transferring data but only exposing a view of the data um but it works in like specific scenarios uh it's good for sharing data when it's for read purposes only um and you can do this on Snowflake and red shift uh the great thing about it is that it's instant access uh because there's no movement and data never leaves your servers so you can get uh access to usage metrics as well um and you can restrict access whenever you need to and one example is of this is companies like stripe uh right now use Snowflake and red shift to ex uh to share their data without the need for API integration of course they have their apis available as well um but they are offering more data through these which is probably kind of simpler for them uh as compared to man managing and maintaining uh their API ensuring that it's up to dat with all their newest uh features and things like that um Salesforce segment amplitude all use this zero copy zero ETL kind of data sharing strategy as well and um CL this is kind of uh a diagram um by AWS red shift um that they use for stripe and their customers uh and then um when you're on AWS you can use Federated queries in red shift and query and analyze data across your red shift and S3 and other places uh that you have um and then one really cool application of data sharing is data market places um it's a one to many kind of share um and with AWS data exchange data vendors can provide easy secure access to the data and reach AWS customers and consumers get read access through red shift sharing capabilities and so on uh uh snow snowflake has a very similar kind of um uh feature as well um so we talked about these uh kind of available uh tooling and methods that companies use um and now fragmentation aside and that you know there's all these various methods when when you're faced with um a data exchange task um we want to see uh kind of what are the challenges of these existing kind of tools and scenarios um so CSV suppresses type information that needs to be inferred later um and uh when data is being uploaded back to the database SFTP you know has limits and so sometimes requires chunking and things like that um it's also hard to keep data up to date you Ma you need to manage that logic when a row changes um of something that you sent before it's hard to communicate that and and for knowing that your kind of uh client can kind of manage those uh updates uh in the right way um and there's also no data validation or handshake that happens and so we see over and over people say hey I received pii I didn't ask for um and now I have to find a way to scrub it or things like that or I sent pii by accident I send data from another customer to this customer or things like that um sometimes uh you also U you know some customers uh ask for obscure formats um and so you need to do additional transformations to get to those um new formats um ultimately you know as we talked about uh this method is kind of like throwing data over the fence you really don't know what's happening at the other side you don't know when that other company has time to build the rest of the pipeline how will they manage it how will they ensure data integrity and security and and things like that um and and you don't know uh kind of uh you know uh if if that data has been received on time you just don't have visibility over the end to-end pipeline um one story here is that uh we needed to send data to uh one of our uh kind of Partners and uh they said that on their side they have a manual process to download the data from the SFTP and upload to their warehouse and they can only do that once a month um but um in the kind of interim they you know they said that the exec team needs access to metrics top level metrics they need that uh to be up uh updated you know pretty regularly daily hourly and things like that and so so uh we would need to uh create dashboards for all their Topline metrics and uh and so we we're kind of like uh acting as their kind of you know data arm uh in the interim until they would uh get access to their raw data at the end of the month um you know with with uh SFTP you speed is a challenge it's very difficult to kind of uh come up with SLA just because of uh what we just spoke about um and so because of that speed issue you know we talked about how some companies share their credentials and say okay this other kind of client will build out the entire end to-end pipeline they'll monitor it and uh kind of uh have control over the speed and and all of that um however um you know deciding on the tech to use how to to connect to the database um who will build the pipeline um you know who gets access to what and and those things kind of take some time um the other thing is only one side now has full visibility and the other side kind of has no visibility over what is happening with their data um and so they don't control when data is sent how it is sent or anything like that um and then the nature of Partnerships really changes a lot you know um and so uh you kind of need to maintain all of these various pipelines uh over time as well new data needs to be sent old data needs to be revoked uh maybe the partnership is um for whatever reason severed and now you need to remember to make sure all of those pipelines are off um another challenge is pricing you know now that you have two companies in the mix uh which side of the transaction should be paying for what it's uh not always going to be the same it's not always the sender who pays for things it's not always the uh data consumer who's paying for uh everything and so uh with with this approach let's say in credential sharing it's not really clear who should pay for egress costs and who should pay for the pipeline and other things um and so there's also you know security issues as we talked about you really don't know kind of how what uh access um what are uh areas that your partner uh needs access to and and so kind of coming to an agreement o over that will take time and require a lot of trust um and then uh you know also what are the challenges with apis it's a really great method but there are some challenges um not all data consumers Expos an API um and connectors for all possible apis Under the Sun don't exist um and so um and it's also difficult to manage costs uh as well because if uh some you know if you require data transfer over sfdp uh one side has to uh in you know spend the cost of actually exposing the API another side has to pay for ETL kind of uh tools to implement that and um you know there is no other way uh to kind of um change this kind of dynamic um and then sdks also uh require a long time to both um kind of build and then actually Implement as well and then there's not always support for all programming languages um with zero ETL tools it's temporary access it's not great for all scenarios and usually works across the same Cloud same region um and so uh sometimes you need to transfer data to the same region to be able to use some of those kind of sharing capabilities um and then with marketplaces again similar reasons as before but also it uh kind of uh really is built for the one to many use cases and it's not clear on like who should pay for that transaction um you know when when you have all of these pipelines auditing is a challenge um you know pipelines all over the place if you ever need to audit it's a lot of work um and you know because there's all of these me methods there's this uh you know there are too many options and it takes so much time uh to um you know set up a pipeline and the sales proc and this kind of slows down the sales process and onboarding and time to Value um what should we look for in a solution for this uh we'd want something with endtoend coverage that manages the entire pipeline we want something that's Cloud agnostic and Tech agnostic uh we want a solution that makes it easy to monitor uh and test all pipelines and manage them in one place uh we want it to be easy to calculate uh costs and do accounting and also um you know charge the right side um a good solution uh should uh have an easy way to like manage evolving contracts it should maintain a ledger of all transactions that happen and most important a good solution would make it easy for companies to stay secure and compliant um and finally easy to set up uh because we we hear over and over companies lose sales because Integrations take too long um and if this stuff resonates I'd love to uh you know talk further um and uh also wanted to mention that we're now part of Tech Stars uh have almost less than a month to go uh and U let's connect please follow General folders on the various apps we love to talk more thank you thank you paries so we do have one minute if anyone has a question uh so very briefly you end your your note with the your what Your what your wish list is do you have any clues as to what the solution is yeah thank you for asking uh we're building this solution and uh it's a working on General folders for um B2B data Exchange to manage all of this uh the things that I've mentioned uh we connect to both sides of the transfer um and replicate data on a Cadence um add testing and validation as well um on both sides of the [Music] transfer