Monetary and Know-how Heavyweight Lowering Time Spent on Information Discovery by One-third with Atlan

When one thinks of Nasdaq, their eponymous change in New York Metropolis possible involves thoughts. It’s the world’s second-largest, with $18 trillion in market capitalization; house to the world’s most prestigious know-how corporations.

Michael Weiss, Product Supervisor at Nasdaq, joined Atlan on the 2023 Gartner Information & Analytics Summit to share how Nasdaq is a lot greater than its flagship change, and the way an organization that has run on information for over 5 many years is changing into a frontrunner in information know-how, as properly.

“It could be a shock simply how massive Nasdaq is,” Michael defined. “Nasdaq operates over 30 exchanges throughout a wide range of asset lessons throughout the globe. We’re additionally house to over 4,100 publicly listed corporations globally. Within the U.S., we’re house to 73% of all publicly listed tech corporations, and the 4 largest by market cap. Nasdaq is really the house of know-how.”

A Decade of Cloud Expertise

Contemplating their place within the cautious and infrequently slow-moving monetary sector, it’d come as a shock to many who Nasdaq started their cloud journey with AWS way back to 2012. Initially, they constructed inside merchandise specializing in regulatory reporting, then income administration, supported by the cloud. “From there, we went to information warehousing and analytics. Ever since then, we’ve been systematically transferring towards the cloud,” Michael shared.

With greater than a decade of expertise working in AWS, Nasdaq is starting to maneuver the majority of its essential workloads to the cloud, and is defining how the following era of capital markets will function.

Regardless of this deep expertise, Nasdaq sees ample alternative to enhance. First, buying and selling system information is difficult in measurement, and in complexity, with as many as 140 billion occasions per day processed within the U.S., alone. “While you take a look at buying and selling system information, it’s essential to comprehend it’s optimized for [operational] efficiency, not for analytics,” Michael defined.

Nasdaq’s course of for making ready and presenting information was one other focus, with their legacy ETL instruments unable to maintain up when each the varieties of information, and the demand for information, scaled considerably. “They have been a bit inflexible,” Michael shared. “They didn’t actually modify to the place we have been attempting to get to on the time.”

Between the complexity of their information, legacy instruments that have been gradual to develop to Nasdaq’s ambitions. A workforce that had its palms full sustaining their technical panorama, Nasdaq’s information workforce realized a possibility to raised help their enterprise companions. “If you happen to have been a product proprietor or enterprise consumer who wished to get a brand new set of insights into your information, it might take fairly a while to have somebody construct it out on the present platform,” Michael shared. “We weren’t actually enabling our enterprise customers to get what they wanted.”

Consequently, parallel groups started to emerge, every with a novel strategy to creating information options. “If you happen to have been a enterprise unit, you may go to one among 4 groups to attempt to get a solution. As a matter of truth, they might go to all 4 groups to see who would win first,” Michael defined.

It could take a compelling imaginative and prescient and technique, backed by a extra fashionable information stack and workforce construction, to raised help Nasdaq’s rising inside demand for information.

A Information Chief’s Information Stack

“It was solely about 18 months in the past that we determined to take a step again and re-evaluate our tech stack,” Michael defined. And in that brief time, Nasdaq has made a number of key investments.

First was an implementation of dbt, accelerating their workforce’s means to construct information fashions. “We actually wished a software that was versatile, and will actually allow individuals who had SQL expertise already to reap the benefits of that know-how,” Michael shared. Nasdaq continued to function on AWS, embracing Redshift as their main information retailer, alongside a wide range of AWS companies corresponding to S3 as their information lake, Glue for information integration, and QuickSight for enterprise intelligence. 

This new stack additionally included Monte Carlo, adopted for information observability. “A number of our information monitoring on the time was very reactive. The pipeline may succeed however we wouldn’t know if the information was proper, fallacious, or detached,” Michael shared. “With a software like Monte Carlo, we’re capable of catch a few of that earlier on.”

However the closing piece of the puzzle was Atlan. Atlan was adopted as Nasdaq’s window to their modernizing information stack, and as a vessel for an information governance apply that was shortly maturing.

That is how we’re actually going to take ourselves to the following degree.

Michael Weiss, Product Supervisor

Evolving Groups and Processes

Past taking a leap ahead in technical maturity, Nasdaq’s information workforce understood that their workforce construction and working mannequin would wish to make the same leap in the event that they have been to fulfill, then exceed enterprise wants. “You’ll be able to usher in new instruments and to maintain the identical methodology and the identical cultural approaches you don’t actually clear up the core drawback,” Michael shared. “How do you deliver a data-first tradition to your complete group?”

Nasdaq lengthy had a centralized construction for reporting; applicable for his or her legacy know-how and obligations, however a mannequin that struggled to allow their enterprise counterparts to make use of new instruments and applied sciences. 

“We appeared on the talent units inside totally different groups to strive to determine one of the best ways to deliver forth change at a extra basic organizational degree,” Michael defined. 

This strategy led to fastidiously focusing Nasdaq’s information group into extra particular roles. First is the Platform Staff, liable for sustaining core know-how and decreasing cognitive load on analysts and shoppers. Subsequent is the Financial Analysis workforce, targeted on information science and serving as a middle of experience for enterprise items conducting deeper evaluation.

Whereas a less-centralized mannequin was a greater match for the rising variety of information shoppers, it required a extra mature engagement and onboarding mannequin as customers started to take advantage of new capabilities and information. 

“We began taking a look at how we essentially embed information into every one among our core companies to allow them to self-serve,” Michael shared. “So we got here up with an engagement mannequin the place we’d go to one among our organizations. We do a survey to start out. We have been attempting to get an evaluation of the place they have been with their expertise, after which from there we might truly embed with them a set of knowledge initiatives they wished to pursue and get them up and working.”

This new course of, contemplating the distinctive wants of Nasdaq’s enterprise items, and guiding them by means of necessities, joint growth of enterprise fashions, the creation of a metrics layer, and enablement to transition accountability, meant that relatively than performing as a service heart, Nasdaq’s central information workforce might concentrate on bigger initiatives, teaching and consulting when vital.

We didn’t wish to simply say, ‘Hey, right here’s the platform. Have enjoyable, good luck. Come again inside six months and tell us what you consider it.’ We discovered that this enabling actually has helped drive success getting groups on board.

Michael Weiss, Product Supervisor

Selecting Atlan for Lively Metadata Administration

With core information know-how in place, a workforce construction that matched Nasdaq’s information service technique, and an onboarding and engagement mannequin to make sure enterprise companions have been well-equipped to make use of new capabilities, Michael’s workforce turned to energetic metadata administration as a path for training, collaboration, and discovery.

“This isn’t the primary time we’ve launched new know-how at Nasdaq for information. I’ve been at Nasdaq for 10 years. That is the third or fourth time we did it. Every time you study somewhat one thing totally different,” Michael shared. “After we launched this one of many key issues that our government workforce introduced up was ‘What’s going to be totally different this time?’ ‘What different issues are you going to herald the combo to scale back friction between information producers and shoppers?’”

Essential to Nasdaq’s government workforce was the flexibility for customers to know what information and instruments they’ve at their disposal, and energetic metadata administration grew to become a vital know-how for answering this query. And not using a platform like Atlan, information and capabilities would stay undiscoverable, decreasing the potential return on funding Michael had hoped for at first of Nasdaq’s journey.

“To get key stakeholders to purchase into all the things, it was essential we had a plan in place. We as soon as discovered ourselves targeted on BI and ETL however forgot about how one can scale back friction between producers and shoppers of knowledge. It was front-of-mind for all stakeholders that we deliberate to resolve this in any new information strategy. With out this the remainder of our mannequin can be DOA,” Michael defined.

Amongst Nasdaq’s stringent necessities for energetic metadata administration was a SaaS mannequin – avoiding the necessity to handle their very own infrastructure – in addition to the flexibility to fastidiously mannequin entry factors throughout customers and roles, and a confirmed file of safety aligning with Nasdaq’s place as a extremely regulated enterprise entity.

Maybe most essential was cultural alignment, a deciding issue for why Atlan proved to be the perfect match for Nasdaq.

It’s not simply the know-how, however the group behind it. We actually wished a associate who would are available in and assist us actually perceive the ability and good thing about this software. We discovered that.

Michael Weiss, Product Supervisor

Scoping the Information Service Alternative

An essential a part of Nasdaq’s path to implementing Atlan was surveying information shoppers to raised perceive how they might enhance the invention and utility of knowledge.

Almost 100 finish customers responded to the survey, and what Michael’s workforce discovered was intriguing. 75% of respondents reported spending time attempting to know the context round information. And energy customers, who spend six or extra hours per week on information, spent two of these hours attempting to know the context round what they have already got entry to.

Take into consideration that! A 3rd of their time each week is spent simply attempting to know what’s there. Think about if we might deliver a product in that helps scale back that effort and actually permits them to get proper to the center of the issue — to drive information merchandise from insights into the enterprise. And that’s what we’re attempting to get to.

Michael Weiss, Product Supervisor

These findings have been constant throughout Nasdaq’s various information and capabilities. Whether or not uncooked information from their matching engine, or ready metrics and visuals, important time was spent on accumulating context, slowing supply on key insights and information merchandise.

Looking for a beyond-the-numbers understanding of stakeholder sentiment, the survey requested for qualitative suggestions, resulting in the belief that siloed and misplaced information have been key issues to deal with. With out sturdy possession of knowledge belongings, or constant documentation, Michael’s stakeholders discovered it troublesome to choose up the place one other member of Nasdaq’s group left off.

“It was not unusual that you’d have somebody working in a single a part of the group, realizing one a part of one thing. Then an information product proprietor would are available in, and doesn’t perceive why selections have been made, and even what was obtainable, in order that they’re again to sq. one,” Michael defined.

Driving Widespread Understanding of Nasdaq’s Information

“A few of the early outcomes have been actually astonishing,” Michael shared. Regardless of being early of their Atlan adoption journey, a follow-up survey reveals indicators that Nasdaq’s alternative of Atlan for energetic metadata administration is yielding optimistic outcomes.

“The one remark that retains arising is ‘That is like having Google for our information,’” Michael defined. “Letting them begin with column names and baseline descriptions has enabled our customers to actually perceive what’s in our lakes and warehouses already, after which begin asking the appropriate questions.”

Having made the appropriate know-how selections, and with a workforce construction that helps their enterprise items distinctive information wants, the introduction of Atlan into the Nasdaq ecosystem has catalyzed a typical understanding of their information and the instruments at their disposal. And with a rising array of knowledge, insights, and capabilities at shoppers’ fingertips in Atlan, confidence in Michael and his workforce’s information technique is enhancing.

“Though we’re early into this journey there may be already a little bit of a shift within the sentiment of our workforce. Whereas we nonetheless have lots to do, this journey is already proving invaluable and has been essential for the chief workforce having confidence we are able to proceed to ship on our promise.”