The cloud community still is on the fence about data lakes. On the one hand one could argue it’s a fancy word for something that has existed since humans started using computers. On the other hand, with the advent of Cloud computing, it is finally possible to store, distribute and process (or mine) unstructured data in a safe, governable way that can actually create business value. So when should a Cloud-native data lake be considered, and how can you make it work?

On the fence about data lakes

From a business perspective, cloud-native data lakes can be tricky. In the recent past, large enterprises have attempted to connect and mine (old) datasets with the expectation to gain new insights and ultimately create value. Too often, such projects are guided by a misconception of what ‘Big Data’ is or can do for an organization. Up until this day the internet continues to be littered with promises about AI, machine learning and the secrets that can be unlocked with these technologies in datasets. And while there are many examples of successful cloud-native data lake deployments that add value, the opposite is also true. As was pointed out during the CAA Cloud-native data lake event, senior leadership on occasion becomes enamored by the (real or imagined) possibilities of data, requiring IT to make it happen. Unfortunately, there is a real risk of a waste of time, resources and goodwill.

The challenge for the Cloud Architect

The Cloud Architect thus faces an interesting challenge. He or she needs to be able to make or unmake the case for a Cloud-native data lake, depending on the feasibility of such a project. If there is sufficient ground for architecting a Cloud-native data lake, he or she of course needs to know how to do it. There are numerous variables to take into account that will eventually make a Cloud-native data lake add value for the organization while avoiding the operational and Governance, Risk and Compliance risks that accompany it.

How then should the business case for a cloud-native data lake take shape? We could probably write a book about this (and maybe will someday), but these are the 5 questions a Cloud Architect needs to consider from a business and operational perspective when planning a Cloud-native data lake:

1. Is there a business oriented problem that can be solved with acquirable data?
2. Is there sufficient scale in the data sets, is there enough data to justify the required resources?
3. Does the eventual solution touch the core business?
4.  Are the right tools available, do business owners and the team understand there needs to be a code base instead of a webportal or GUI?
5.  Is there a team that knows what to do and can autonomously engineer the data lake, with limited oversight?

Getting these things right will make the difference between a misguided approach and getting the desired results. Those results in turn are highly dependent on the way the data lake is architected.

How a cloud-native data lake can add business value

A useful perspective on architecting a data lake is how the data in the lake should be made available to consumers of this data. Given the fact the data lake holds unstructured data, consumers will have to know which is what, and whether they have access. To see how this would work in practice, it is useful to use an example.

Consider a data lake that holds data sets on mortgages and relevant variables, spanning about two decades. As one can imagine, an institution that is in the business of providing mortgages wants to know how many mortgage holders historically default on them, what the average returns on these mortgages are and will also want to know how mortgage defaults relate to historical interest rates and the historical health of the overall economy in a given country. The data involved has its origin in different data sets, and some of the data is probably privacy sensitive. Also, the data can be sensitive in a competitive sense, justifying tight controls of who has access.

Given the age of the data sets, its highly probable compatibility issues will arise when pushing this data into the data lake. These type of issues can take a lot of time to mitigate, unless an unified data model is issued that is able to assign metadata in a consistent way and making the data available through a publisher-subscriber model. This will also help to control access, and enable teams to discover data without extracting data from the data lake. It is advisable to create an automated workflow which offers a potential subscriber a sample from a data lake, giving him the opportunity to decide whether the data in the lake is useful to him. This workflow will need to automatically provide or deny access based on predefined rules, such as clearances for specific types of data. In this way, the use of this data can be aligned with organizational goals, creating real business value. 

Make a data lake work for you

All in all, the Cloud-native data lake requires a lot of thought, from its conception to its actual use. And even though there is some (contextually justified) skepticism about data lakes, with the right business case and right architecture, innovation can be accelerated with a thoroughly automated data lake.

In the run up to the much anticipated Cloud Architect Alliance diner and award show on January 30th, our speakers give a sneak peak into what they have to say that night. Data Solution Architect René Bremer and Senior Cloud Solution Architect Sarath Sasidharan, both currently at Microsoft, share their take on architecting secure data lakes that add business value.

The challenge of scattered data

Many enterprises are considering setting up an enterprise data lake. They want (and often need) to get more business value out of the data that is available within the organization. But even though the potential value of data is recognized, actually using data in this way can be a challenge.

According to René Bremer and Sarath Sasidharan, enterprises need to centralize data first before they can use it to create business value. “Large enterprises face a problem when data needs to be accessible to different teams, departments and offices.” Bremer says. “The last thing you want is teams exchanging data bilaterally, as this data will be ‘marginalized’ and most likely is not accessible to the rest of the organization. It’s difficult to control data that does not have a single source that is used by the whole organization. Centralizing data therefore is the first step to creating a data lake that adds business value.”

Controlling accessible and useful data in a data lake

Centralizing data is a first step, but it certainly should not be the only one. For data in a data lake to become useful, it should be clear who owns it and who is allowed to access it. A pub-sub service should be in place to ensure asynchronous use of data. “Both metadata and a clearly defined pub-sub paradigm are key to creating a functional data lake” Sasidharan explains. “Any data that is put into a data lake needs to be stored according to a proper methodology, with proper metadata and scheming validation for pub-sub to work as it is intended to.”

Adding metadata may seem a fairly straightforward affair, but when terabytes or petabytes of data are involved, a strict policy for assigning metadata fields and labels makes the difference between a functional data lake and a dysfunctional data swamp, Bremer says. “You need to think long and hard about the way metadata is added to data. Business metadata, technical metadata and operational metadata need to be compulsory additions to data that is put in a data lake. Without them, it will be difficult to determine the source and value of data. And even worse, it will be difficult to have tools use the data at all. Ideally, metadata can be interpreted by a variety of tools, for example Bricks or Sequel.” 

And that’s not all, according to Sasidharan. “You will also need to think about the data lake from the consumer’s perspective. You need to adhere to certain standards for API’s, streaming and other ways data in a data lake can be published. It should be easy for a data consumer to access the desired data while at the same time controls should be in place to ensure only authorized users can access certain data for a predefined period of time. A single pane of glass to manage consumption of data in a data lake is indispensable to a secure and efficient enterprise data lake.”

Learnings from real customer cases

To eliminate data siloes between application and business departments, Microsoft has created an open source data model (called Common Data Model) in collaboration with SAP and Adobe. During the upcoming Cloud Architect Alliance event on January 30th, Bremer and Sasidharan will discuss two customer cases and will elaborate on pub-sub flows, metadata and the Common Data Model, and general principles that need to be considered when implementing a data lake. To get guests started, they will even share some generic patterns on metadata and pub-sub. Make sure you don’t miss out and claim your free ticket right now! Don’t wait too long, there’s only a few spots left.

In the run up to the much anticipated Cloud Architect Alliance diner and award show on January 30th, our speakers give a sneak peak into what they have to say that night. First up is Maurits van der Drift, mission critical cloud engineer at Schuberg Philis. Van der Drift will talk about the sense and nonsense of cloud-native data lakes and how data can help to enhance business models.

Nothing more than a database?

Gone are the days when tens or hundreds of on premises or colocated servers were clustered with Hadoop to store and process unstructured data. Cloud-native data lakes offer a far more flexible way to create and maintain databases without the need to manage the physical hardware. Another major advantage is that cloud-native data lakes are not bound to a specific location, which makes planning and executing migrations (from data center to data center, for example), a thing of the past. Even though a cloud-native data lake may in its core function like a data warehouse or data base, its cloud-native nature offers far more possibilities.

“One might think the term ‘cloud-native’ is just marketing lingo for what is essentially a database”, Van der Drift says. “And even though that would be a legitimate take on the cloud-native data lake from a purely conceptual point of view, recent technological developments have really changed how data can be leveraged to add business value. The cloud is indispensable in this sense, because it offers an efficient, tailor made solution for data analysis and value extraction.”

Know what to look for before you take a dive

That being said, the promise of data lakes has not yet been fulfilled, Van der Drift observes. “When the idea of data lakes started to become more popular, there seemed to be no limit to the business value unstructured data could generate. Combined with AI and machine learning, many organizations were told they were sitting on massive amounts of useful data and they needed to tap into this new possible revenue stream. The truth is however that in many cases, the unstructured data organizations have is tough to leverage from a business perspective. In addition, even if value can be extracted from unstructured data, it can only be successfully leveraged if you know what you are looking for. In other words: you need to have made the business case for a cloud-native data lake before you actually start creating one.”

Even though Van der Drift has some reservations when it comes to ‘the promises of Big Data’, he notes there are many ways to create business value with data lakes. And it has become easier to do so. ‘A major advantage of cloud-native data lakes is the razor-sharp efficiency they bring. You don’t have to invest in hardware anymore, you don’t have to keep developers on your payroll to maintain your clusters and you only pay for what you use. Infrastructure costs for cloud-native solutions usually are only 10% of the total costs of a project. This means that a small increase in efficiency can rationalize a large increase in infrastructure costs. Needless to say, the threshold to use data analysis is lowered, but it remains important to know why you want to set up a cloud-native data lake and what the desired results are.”

Claim your free ticket for our upcoming event

Do you want to see Maurits explain when and how a cloud-native data lake can add business value? And would you like to enjoy the company of your peers and a free three course dinner while doing so? Join us January 30th 2020 during our next event: ‘How to build a cloud-native data lake & annual Election Night’. During this diner show, you will learn everything there is to know about cloud-native data lakes and can also enjoy the annual Cloud Architect of the Year election. There number of available tickets is limited so make sure to claim yours right now!