Enterprise Architecture Meets Data Architecture
Transcript
all right you're listening to another episode of the CTO advisor podcast we have another returning guest Matt Wallace CTO of faction who provides multi-cloud data services and this is a timely episode got engaged with Matt as a result of my AWS everyday Series where I'm reviewing an AWS product every day we got the EMR which is a Hadoop based product service offering by AWS and there's some Nuance there it's not it's it it's Hadoop compatible but it's not Hadoop in this sense that you know you're
scaling out notes really cool service from AWS and Matt was troubleshooting Hive related issues if you don't know what any of that means that's fine we're going to walk through this but first Matt welcome back to the program yeah hey and thanks for having me I'm super excited and uh it's good to be chatting again I've been watching the podcast like caught your uh your episode with Rob the other day that was great that's a fun one right the uh platform group the one I know
it's another one of your passion topics as well uh and we're going to get into a little of that we're going to have two personas which are pretty accurate I think the only caveat is that Matt could carry both personas but that's okay I'm going to have the Persona of the Enterprise architect whose basic job it is to Bro is to provide a platform to data architects and Matt is going to play the role of the data architect and there's a place where the two roles
bleed and we need to understand from a Enterprise infrastructure architecture perspective and from an Enterprise data architecture perspective how do we communicate between the groups so that the end of the day when they when a internal customer or even external customer says they want to collaborate on a data product or project and one customer says hey we use Hadoop And Hive and the other one says we use Hadoop And Hive there becomes a problem that is not as clear as it would seem or even between
internal groups so Matt let's talk about let's let's do some basic definitions because I need them myself when we're talking about these Frameworks what are the fundamental building blocks of all of them in general okay well I mean I think that's Broad and I think from the things that I've seen you know and we're talking about like large data sets right everybody's got their databases and so on but from this big data perspective the things that kind of live in that ecosystem especially things like you
know EMR you know you have all these things that are built in the Hadoop ecosystem so you had Hadoop and Hadoop included hdfs and over time hdfs which was a great way to take a bunch of commodity hardware and big drives and make it you know resilient and performant you know got replaced a lot as those things went to the cloud with object storage because object and htfs look very similar in terms of the way that they see the world and so it was pretty easy
to put this abstraction for object in the hood but then you had all these Services over time that got built up on that right you've got a hive you've got hbase you've got spark and so over time you went from um Hadoop being built sort of as a data warehouse where a lot of the users were they were bi people they were running SQL Aquarius it was just a way to do these you know map reduced jobs to process massive amounts of data and it started
getting used for all kinds of other use cases like streaming like analytics like machine learning and so I think there's been this really this flowering of this ecosystem and so you have those personas where you've got you know classic bi uh types of people you've got the machine learning people who are kind of you know split into that data engineered data scientist right where the data scientist people are really good at linear algebra and you know all kinds of calculus and the data engineer people are
much more Innovative they understand some level of infrastructure they know how spark and high spark and Pi torch work and all those things and they're trying to bring those things together and as data has become more important you even literally have people who are like data product managers where their whole job is to understand like where's the data live how can we use it you know to improve the business you know where's the value come from you know what's the next thing they're treating data as
a product and it becomes really interesting to see the interplay of these and of course over time like EMR is not a new service it's one of Amazon's very first things like you know EMR redshift Athena they all feel like they're you know quote-unquote ancient in terms of cloud years it's like dog years and um and I think you see these the evolution of these services has kind of tracked to some extent the evolution of these personas right glue which is a much newer service looks
feels a lot more like it was designed for those newer access patterns so it's really interesting across the board so let's walk through a typical business Journey that happens that I've I've seen so I have this huge amount of data in a Hadoop data Lake uh the hdfs and a data scientist sees it I'm like oh wow that is awesome repository I want to run some real time curies against that data set [Music] you come in to advise on this uh setup traditional whether we're talking
about EMR or uh atfs built out cluster and the number one complaint is I I can't get real-time queries done the the queries are taking their batch queries and they're taking forever to get back results what's the next step in the data Journey yeah I mean that's interesting because I think you're you're skipping into a part where you're a little bit talking about performance and this is actually one of the things that was really magical in my mind about the way I saw that evolution of
the hotel ecosystem because you had the the data that was there for that bi kind of scaled SQL for the mapreduce jobs right and when you saw spark layered on top spark catered to that data scientist Persona and if you're using you know something like Pi torch and Pi spark you can come in and and Spark understands living on top of Hadoop and it can just go in and access parts of hdfs they can use it as an abstracted file system and your you know your
machine learning people are Off to the Races now I think the Things become complicated when you start to ask the question and you mentioned earlier you know this idea of EMR being a place where you can have a hadoop-like workload and not necessarily worry about scaling infrastructure because ultimately you know your data science people don't necessarily want to worry about you know that question now of course what's interesting is all these tools you know when they when you run a query against say Pi Spark it's
going to have this ability to basically farm that out into a set of jobs that then get parallelized across cluster so I mean you know much like anything performance the answer to hey it's not running fast enough is we'll scale out and I think at least this ecosystem around mapreduce it's literally that's what it was designed for right no matter how you're slicing it whether it's classic you know map reduce on hdfs or whether it's spark and it's accessing you know objects in S3 you can
scale out to kind of do a certain amount of that work and of course there's questions about how much can you scale out and you know if you start looking at things like uh you were doing say training uh you know like an llm you know like the chat GPT that everybody's talking about it's not that simple right you know the question of how those things work there's oh there's a limit to how much you can parallelize certain parts of that process so you know performance
wise I think scale horizontally but but yeah I think one interesting thing that I've seen and you kind of said hey there's this data here and I want to run with it a lot of Enterprises can't get that far necessarily because they've got their data teams and their data teams are stuck answering the question of hey what data do we have and where is it what's the format what columns does it have how much of it is there am I allowed to use it and even
that question can be hard to answer before you even get to running the query and I think even and the reason why I asked the question around performance is that when you know pre-emr I know EMR has been around forever but pre me adopting EMR my organization adopting EMR if I started my journey in Hadoop Hadoop in putting spark on top of Hadoop from an infrastructure architecture perspective engineering perspective someone has to think about that like someone has to add the spark nodes and and can
do the connecting EMR makes that magical I can just run the job this the to an extent from the plumbing of needing to do the cluster configuration all of that kind of stuff I'm told it's easy okay I haven't I haven't touched uh Hadoop cluster in like seven years I don't know it could be easy or not I don't know but there's this and this is where it gets interesting like there's people the people that think like me I think block storage object storage uh replication
uh uh data protection I think at that level of of data interaction and then there's people like you who have to think about okay how is this data stored so that well one how is this data classified so that I can determine how it's best stored so that I can determine how it's best transacted against and I think there's a a separation between people who provide the infrastructure and that level of understanding and my understanding literally stops that I provided your OS I've provided you maybe
even a EMR type environment to go run with it what am I missing in my requirements Gathering to better serve you so you have the agility that you need to provide data services to the data scientists the machine learning folks Etc yeah I mean it's so interesting because there's these layers and and there's a certain layer that has to do with the mindset of what you're going to do with the data and what data it is and it really lives completely those people are not at
all concerned about anything that looks like storage and so we started talking about block or object even hdfs it's like it barely is on the radar whatsoever in fact they almost want like a cookbook or a Playbook that says this is how I get to my data source let me worry about you know queries and you know if it's the machine learning people you know they're asking what do I need to do to add a feature you know how do I avoid overfitting a model that
I want to talk about storage at all um and it's so fascinating too because these are both distributed systems talking distributed systems like if you think about like any more cluster that's accessing S3 as the back end for its data like on one hand you have the scale out compute layer and you can create jobs those jobs can scale horizontally and at the end of the day the the execution um is something probably your data people whether they're bi people or data science people don't want
to worry about they want to write a query or or things of that nature or they want to load a file you know from from a Target they want to load a parquet file directly into a data frame say and then they want to work with it they don't want to worry about what's under the hood of that your data engineer is is asking the question of okay what is underneath the subtraction layer right what is it how am I storing it where does it go
how scalable is it and so your your data engineering team may be worried in a way that your data science team is not about okay how am I going to partition this table what are the partition you know keys look like do I need multiple copies um and those things are questions they have to answer to make it work and those things still live another layer above that abstraction like I can run spark and actually Mount NFS and literally use a classic NFS based file system
and load data directly into say a pandas data frame right out of a file from NFS but is that the thing I want to do right that's not necessarily infinitely scalable the way that object is and so there's a lot of questions you have to ask about how big is the data set going to be who's going to access it how much parallelization you need is it going to move around and incidentally I mean there's also a whole ton of other abstractions if you look at
something like um you know nowadays it's Starburst which is based on trino which came from Presto SQL it's this system that does distributed querying where the data doesn't even necessarily live in the same spot you run a query and you might have Agents from Starbursts that are out querying your data on the west coast to your data on the east coast and your day in Europe and they're all actually completely different locations and it's treating those almost like partitions and pulling together to answer that question
or something like alexio for example that can abstract like different types so you actually can have one common interface for your data science people and whether it lives on NFS or S3 or a host of other things maybe even your postgres database they can abstract that away and treat that as a you know unified data source um and all those things I think are separated from the infrastructure concern and how fast is you know S3 versus you know classic hdfs versus you know maybe if you're
doing super performant things in AWS you know I knew some folks that work in one of the really big tech companies that were doing a bunch of data science work inside of Amazon and and when they were had active data sets they actually loaded it all into EFS because it was so performant so they were using the file system abstraction because you know waiting for objects to load was just not quick enough for them and so sure you have to have people to answer all of
those questions but of course if people don't know where the data is and they can't use it effectively which is more like a catalog metadata governance question than it almost doesn't matter what's under the hood from an infrastructure layer and of course as you know this is like the classic EA dilemma if you don't give people things they need to do their job effectively they kind of go solve it themselves and then you end up with the sort of like weird sprawl and everybody does their
own thing and then pulling it back into governance can be really tricky so let's talk about that problem because I think that's where the core of my audience is at is this EA problem which is I do understand that I need to know uh where the data is at and not only do I have to know whether data is that I have to give my data consumers and data you users interface in which they can identify and discover the data because I don't want from an
infrastructure perspective I don't want uh them to go out and get the data a second time or a third time this has happened to me I've had people ship snowballs two or three times of the same data because they can't identify where the data is within the organization so talk about this kind of data cataloging data Discovery problem yeah I mean it's interesting and I'm obviously like I have my own lens here because a lot of the folks that I work with you know at faction
have that really super oriented like multi-cloud bent and sometimes that's because they had a pattern that really worked in a cloud and then they're realizing they really want to extend their practice to to include some data that's on Prem still or they have a subsidiary or a partner that's in a different cloud and they realize the pattern doesn't work there because they can't use whatever service and I made a great example here right is you look at AWS glue it's got this embedded catalog it's a
you know it's fundamentally began as sort of ETL as a service right scalable ETL jobs and it could kind of consume data from things like Kinesis you could feed it in to do ETL and then store that in S3 and then every other service like you know you've been talking like that EMR and Athena they can then consume that data even using glue as a catalog right looking at glue's catalog in order to say like that way Athena can just show you what tables are available
because it's asking blue what's available but then you know that doesn't necessarily easily extend outside those four walls and so they're trying to find those patterns that make sense right and this is where you start to see things like databricks has their I think they call it Unity catalog right which spreads across workspaces and keeps track of things or stuff like Starburst where they can have like a centralized catalog and they can act as that broker for all of that information I think the commonality is
everybody really loves it when they can make the the catalog and that metadata more Universal there is definitely not that I've seen um you know from at least my sample size any sort of like Universal agreement on the right way to do things it really just depends on where you are and that Journey with your data what kind of data you have and what your willingness is right I mean there is a trade-off for those things if you can tell your teams we're just going to
use AWS for everything and you know that everything is always going to tightly couple that your glue and your EMR and your Athena and your redshift and your S3 will always work together there's so many problems you don't deal with like credentials so seamless to have like glue have permissions to access S3 pockets and yet by the same token you start you know getting outside of those four walls and you start asking a question of well how do I even if I have a connector for
glue how am I going to manage credentials for this external data source so that glue can consume it and how do I do that securely and you start getting into you know a whole host of other problems sometimes it feels like you have to trade one problem for another right because making that seamless for your data people is a lot more work for people that have to kind of set those environments up and yet you know I think in a lot of cases the results are
worth it um but there's definitely uh some immaturity around all of these practices I feel like no one feels like they've got it all right yet yeah I think you're making the argument for a data platform team and I think this leads into this conversation around maturity and what happens when an organization a mature will uh well-disciplined organization that has maybe not strict rules but they have Frameworks around data data governance and they meet a team whether through acquisition or collaboration that quite isn't there that
hasn't thought it all the way through so where's the mismatch like if I'm an organization and I acquire an organization let's keep it relatively simple I acquire organization because they have the data that I want and I'm just going to outright acquire them but they don't have the this hygiene and I get in and this in the data catalog is all over the place what's the journey to reconcile this challenge yeah you know I think there's probably this pattern that you recognize right which is when
you have these two kind of separate Estates and by the way it doesn't even have to be as messy as you're talking about right like you might have an acquisition and maybe just because you want to merge the businesses hit synergies but then after you acquire you're like Hey listen we have our customer data you have your customer data but we're now the same or we got to put these together because these data sets it's not one plus one equals two it's one plus one equals
five right it's more valuable together um and it could be that they have similar patterns right just one was in Amazon the other was in Azure and they have to normalize that somehow but it turns out you know you have those separate states even when there's super similarity you're still forced to do one of two things you have to kind of put them together which usually means migration from one to the other or you have to connect them in some way which might be like real
time or again you have to abstract them right and in a similar way that you can abstract all kinds of other things right you know load balancers to abstract you know Regional endpoints or things like that you know to to play into the infrastructure you know folks mindset um you might have this is where something like if you were to stand up um you know Hive server and this is kind of how we got started on this conversation originally you can have the Hive server that
can actually as long as it's got connectivity to these different data sources you can actually have it act as a master repository for these things and it's by the way that is by no means the the single way to uh you know to have it like obviously what Amazon is doing you know data zones or what Azure is doing with purview or what databricks does with the unity catalog a lot of these things are they're relatively recent Services I mean in terms of cloud time frames
even super recent services and a lot of them are all there to address this question like how do we unify data I think actually what is really fascinating too is it's kind of part and parcel of that Trend that you see among Cloud providers to start taking services that really extend outside their four walls right they're acknowledging that data lives in other clouds it lives on Prem and this one is specific to that challenge of like how do I unify all those data pieces but I
mean going back to your question it's like at the end of the day abstraction layer you know real-time access like that's where something like Starburst comes into play or migration to merge them you just say listen you guys have two ears or whatever the time frame is get all the data over here and I think what's really challenging for the data people is even as this is happening they're these like real seismic shifts in what you can do with these data sets and I think I'm
thinking about things like the Delta Lake technology that databricks has that's open source the Apache Iceberg which is very similar these things that are taking the format of data away from being vanilla parquet and taking them into these world you know worlds where it's a lot easier to abstract them a little bit more and have different views of the same data or point in time copies of data without having to make true copies right where they can actually you could freeze certain data sets or present
certain columns that are abstractions of the whole data set to other teams when they're not supposed to see everything because there's so many use cases around that I mean that's a great example I mean you know if you look at the Amazon side what they're doing with the data room a side of things where it's specifically meant to help organizations share data safely and securely with other organizations where they have a relationship that they may not be able to share all of that data and so
they need compliance right because they've got a table that's got 20 or 20 Columns of data but three of those they have to redact and remove to share it because they're not allowed to share it so all those concerns I think kind of come into play and maybe the path is different depending on what your requirements are yeah we talked about many of these uh challenges whenever I talked to Farmer customers and there's a desire to share data with Academia and you want to keep the
the proprietary bits of your data to yourself but you want Academia to see you know the genome sequencing and and the the less undifferentiated data but when there's a ton of it you can get a lot of information out of and we so far we've been talking at the the data layer itself well we haven't even touched is the application layer the and what what makes this really complicated you know as a data person a pure data person you can see that an organization has the
data that you want but that data is not given that it's in you know a Hadoop or a hive type of cat catalog it could be sap Hana it could be sap ECC it could be a Oracle wreck uh database and when you talk about the reconciliation making sure that you can either merge data and use the data transforming the data is not always an option not you know not in place I can't just take sap Hana database merge it with a hive data set and
now I have one data source not easily anyway and that you know the the transfer to to you know extract the data and transform it and leave it in in one single system I may need a hybrid I may need something on top of the two systems to transact because I can't go back and change my business processes that are built on sap yeah so you know we have all of these complexities around how does how do we enable the business to Leverage The this important
asset without disrupting the business I think I think you're right and I think this is why those products like the alexios like the Starbursts like I think almost from the beginning everybody who's thinking about data practice should be asking themselves like how do we set up a process this is completely away from the technology how do we have a process that helps us Define as an organization where data will live how we approve the paradigms for accessing that data because you don't want you can't you
just made the point that you can't centralize everything in a single way 100 normalization will not happen I think that's probably true for everybody who has any variety of data a variety of use cases but on the other hand like the cost of just going wild west and saying well everybody you just figured out that makes no sense at all that's like everybody doing their own cloud architecture right and so you kind of have to centralize to the point where you know there's going to be
different requirements and you have to put that substrate of these are the patterns that we approve of and if you want to deviate from these you're going to have to do a lot more in terms of getting those approved and things of that nature and then with those patterns in mind you can start implementing you know some tools right and I think you know there's some popularity and some growing popularity behind those platforms that normalize things and I keep mentioning things like Hive like Star Wars
like Alexia because those are examples in my mind of those tools that normalize like if you're if you're a data scientist and you're you're connecting into a Luxio it can literally normalize querying data out of you know S3 and off a file system and on a postgres server for you to the point where you can like write queries that join you know tables as long as you understand the schema from those sources the fact that the underlying you know storage for those the underlying where the
data lives doesn't even matter to you because you can join the schemas you know it's a whole different problem I think and this is where you know some of the life science use cases start to be completely different from the data science use cases so you mentioned life science and genomics you know I've obviously played around a whole bunch with some of those genomic use cases um and one of the funny things about that is I mean there's certainly there's tons of information about the data
but at the end of the day a genome is you know one or more you know it's 66 gigabytes compressed roughly of gzip data is that's a human genome you know the one I was using split into two parts and that's the that's the raw version right the thing that comes off the thing that reads your DNA and then you can create the aligned file the BAM file and then from there you can do all this variant calling but now you have to ask the question
of like there's so much to keep track of because with that level of compliance you're constantly asking like it's your point if we partnered with an organization to get this how long are we allowed to have access it are there certain things we can do and not do you know think about the gdpr perspective somebody's got a right to be deleted and if they you know put in and say get rid of everything you've got you got to find all the data that that person you
got from that person or derived from that person and scrub it you know that lineage and that could be complicated and so you know it's a completely different problem than the sort of like a quasi-structured data where you know you could kind of apply a SQL schema and it's you know csvs or parquet files or things but you still have to keep track of tons of data and if you're an organization that does both of those because if you're you know doing Pharma you might have
one team that's responsible for understanding how your billion dollar drug does in the market how many doctors are using it you know is there some sort of like cohort that won't you know prescribe it and why but then you've got a whole set of scientists who are going over like a genomic data and they're literally they're getting all this data from a study and they go listen I've got a thousand participants and I want to analyze which genes of my participants indicate that this drug will
be successful so I can kind of understand um you know how it will fit as I as I take it to Market you know same business two wildly different use cases wildly different data and of course you probably have some kind of I.T and Cloud team and Enterprise architecture team that in some way touches both of those isn't that crazy yeah and I'll add one last point to it and we'll end on this notice there's a cross team that's trying to find new indications for the
same drug so they're trying to figure out what are doctors you know what are the existing indications and the indication is what the drug can be used for as authorized here in the US by the FDA and what our doctors actually prescribe driving the drug for an example is ozympic is being used as a weight loss drug doctors can prescribe it as such but uh the farmer company cannot advertise it as such so how do how do the uh actual scientist the bioscientists and Pharma scientists
use the data or can they use the data from the prescription uh information that they have from their Erp and Health Care Systems to find new and Test new indications of uh indicator indications for the drug so it's it's a fascinating problem and something that I've given a ton of thought through there's a ton of stuff that I just functionally don't have a grasp of I've never managed a hive uh framework I've never solved the data problem in that way so there's some things that I
that haven't quite grocked but I hope this has helped you folks kind of get a sense of the skill of the problem the different directions that you can go to Matt let's get in in the plug for faction if people want to learn more about faction what do you do and where can they find you sure well factioninc.com is the URL yeah faction8.com is the URL and we do multi-cloud data services and I think you know when we start talking about these cases of hey this
teams and Amazon and this Sub City areas in Azure or you know how do we Interlink like on-prem with these Cloud environments we get you know it hip deep in that and a lot of the reason I've been so interested and playing around with so many things like Hive is because that same problem of you know how do I have a catalog that touches everything it's even more interesting when you're starting to talk about cloud and so like one of the things that we're able to
do is get that data centralized right very close to all the public clouds and then you can do something like have a bunch of data that lives on Faction where you can access it equally as if it looks like it's in the public Cloud but you can get to it from Amazon from Azure from Google from Oracle all at the same time and if you want to you know replicate it to and from on-prem or access it from on-prem or from your co-environment you can do
that too but of course for us that begged the question of how are you keeping track of that and if 100 of your data is not on Faction how do you normalize between your faction data and your Amazon data and your Azure data that's where we started seeing Hive and so you know there is something pretty awesome when you're running a SQL query you know you're literally joining table data faction with table data at Amazon and table data at Azure but a lot of our customers
really are are in that same boat right they've got different teams and different places in different environments and they're asking how do I normalize this not only because they don't want to pay for two or three copies but because there's more to having more than one copy and this applies not just to data faction but universally if you start trying to make copies then you have to ask like how do I know my version is up to date you know if I did insert at Amazon
and somebody else did insert in Azure that I think and so you know we kind of help with that problem but but dealing with all that governance concern becomes even more interesting because you're explicitly going after enabling those multi-cloud uses but yeah Reach Out all right and if you want to find out more about the CTO advisor you can find us on the web the ctoadvisor.com Matt thanks for joining talk to you uh on the uh on the twitters yeah absolutely much appreciated Keith okay
