Amgen, the world’s largest impartial biotech firm, has lengthy been synonymous with innovation. For 40 years, we have pioneered new drug-making processes and developed life-saving medicines, positively impacting the lives of hundreds of thousands around the globe.

Knowledge and AI are pivotal to our enterprise technique. Recognizing the abundance of knowledge inside our enterprise, our imaginative and prescient was to determine a data-driven group the place knowledge analytics is made accessible via self-service governance capabilities. In our pursuit of modernization, we rigorously chosen the Databricks Lakehouse Platform because the bedrock of our digital transformation journey. This strategic determination has enabled us to unlock the true potential of our knowledge and AI throughout numerous departments, leading to streamlined operational effectivity and accelerated drug discovery. As we repeatedly enrich our knowledge lake with various domains, together with restricted and delicate knowledge, our affect expands even additional.

Moreover, we acknowledged the necessity for enhanced knowledge governance to enhance our efforts. Our earlier knowledge governance answer proved complicated, difficult to handle, and lacked fine-grained entry management. To handle these obstacles and facilitate widespread adoption of our governance functionality throughout the enterprise, we now have not too long ago built-in the Databricks Unity Catalog into our governance processes. This integration represents a major milestone in our journey, bolstering knowledge governance by offering a strong answer that’s each user-friendly and simplifies administration whereas providing granular entry management.

In the present day, we’re sharing our progress and success thus far within the hopes that others can be taught from our journey and apply it to their very own enterprise methods.

Utilizing IAM roles for governance was troublesome to handle and lacked fine-grained entry controls

Amgen operates inside a extremely regulated trade the place compliance is the cornerstone of our operations. We acknowledge the important significance of correct governance and auditability for any restricted or delicate knowledge. Knowledge democratization was the unique goal of our Enterprise knowledge lake initiative, making certain that every one Amgen customers have entry to the obtainable knowledge. Nonetheless, the inclusion of delicate knowledge within the knowledge lake highlighted the necessity for extra sturdy knowledge entry governance.

Beforehand, we relied on AWS Glue as an enterprise knowledge catalog and AWS’s id and entry administration (IAM) for role-based entry controls. This concerned creating separate IAM roles and associating them with particular clusters to cater to distinctive use instances. Nonetheless, managing quite a few teams and their related cluster assets independently posed important challenges. Furthermore, IAM roles solely ruled entry to storage, leaving metadata accessible to all. The absence of fine-grained entry controls made auditing a fancy activity, hindering our skill to audit knowledge entry and executed queries successfully.

To handle these challenges, we acknowledged the necessity to transition to user-level entry and consumer attribute-based entry controls. For instance, customers could be assigned attributes similar to price facilities, and knowledge inside Finance could be managed primarily based on the assigned price heart. Nonetheless, implementing user-attribute-based entry management via IAM roles would have required the creation of an enormous variety of roles, posing a major administration burden.

We evaluated a number of off-the-shelf governance instruments. Whereas among the instruments met rapid necessities, similar to managing tables on the database stage, they proved insufficient for extremely restricted knowledge domains like EDW (Finance) and Workday (HR). Furthermore, we had considerations about bypassing these instruments on the Databricks cluster, creating potential vulnerabilities and making certain complete protection throughout all clusters, and scaling the answer.  Moreover, sustaining plugins on selective clusters posed challenges when it comes to script consistency and ongoing upkeep.

Migrating to Unity Catalog simplified entry administration and eradicated noncompliance and safety incidents

Presently, 90 % of our use instances are on Databricks. Provided that, we felt we would have liked a Databricks native governance answer for the long run. To start shifting in that route, we turned to Unity Catalog.

Adopting the Unity Catalog resulted in a number of rapid advantages.

• First, we did not should create or handle a minimum of 120+ IAM roles. We will management entry via Unity Catalog and the APIs Unity Catalog offers. Every little thing is managed via entry management lists (ACLs) or dynamic views. Consequently, we went from a whole lot of IAM roles to only one or two principal IAM position.
• The second profit we realized is simple auditability. Modifying Unity Catalog ACLs is far simpler than parsing IAM insurance policies after which figuring out who has what entry. This reduces the audit effort for the operate by 50%. The question historical past provides us the power to see who accessed what knowledge at what cut-off date.
•  Unity Catalog is simple to handle. It is allowed us to maneuver away from devoted cluster-based entry to a shared cluster pool with the consumer and role-based entry controls, lowering Databricks price by 10-20%.
• It unifies every little thing at a central place and allows seamless cross-functional knowledge analytics and the tight integration with the Databricks ecosystem offers true differentiation.

Presently, we now have round ~500 objects mapped in Unity Catalog (and rising) and ruled via its ACLS. Since shifting to Unity Catalog we have a lot greater confidence in our knowledge governance and adherence to compliance. As soon as we begin onboarding extra features, we anticipate these advantages to multiply.

Constructing additional on our Databricks Unity Catalog success

That is solely the preliminary stage of our journey. We’ve got an even bigger imaginative and prescient forward and are diligently crafting a method that can propel us towards our objective of migrating nearly all of our knowledge belongings from AWS Glue to the Unity Catalog. As our enterprise knowledge panorama encompasses quite a few knowledge domains, hundreds of databases, and hundreds of thousands of objects, Unity Catalog is poised to turn out to be our default catalog. This strategic shift will streamline and unify our knowledge ecosystem, enabling seamless administration and exploration of our intensive knowledge assets.

We’ll use Unity Catalog’s knowledge lineage options to reinforce observability, construct confidence in our knowledge creation, and monitor delicate knowledge utilization throughout our knowledge property. Moreover, we’re captivated with using Delta Sharing in Unity Catalog for exterior knowledge sharing. Whereas we at the moment share knowledge internally, we’re actively exploring the gathering and sharing of exterior knowledge with a number of distributors via Delta Sharing.

In conclusion, the combination of the Unity Catalog has enhanced our skill to implement exact and complex governance insurance policies for Amgen’s restricted knowledge units, together with Finance and Workday. This exceptional achievement has sparked immense enthusiasm inside our knowledge engineering division, resulting in elevated funding in our knowledge platform, with Unity Catalog serving because the central Metastore and entry administration service. Looking forward to the following yr, we anticipate that Unity Catalog will facilitate over 80% of software knowledge consumption at Amgen, benefiting our huge consumer base of over 10,000 energetic customers. With this shift, we’re poised to attain effectivity enhancements of 60-80% in auditing and entry administration, firmly positioning our firm for achievement as we proceed to increase our analytics choices.

Watch our presentation at Knowledge and AI Summit 2023 to be taught extra.