Prospects use Amazon Managed Workflows for Apache Airflow (Amazon MWAA) to run Apache Airflow at scale within the cloud. They need to use their present login options developed utilizing OpenID Join (OIDC) suppliers with Amazon MWAA; this enables them to offer a uniform authentication and single sign-on (SSO) expertise utilizing their adopted id suppliers (IdP) throughout AWS providers. For ease of use for end-users of Amazon MWAA, organizations configure a customized area endpoint to their Apache Airflow UI endpoint. For groups working and managing a number of Amazon MWAA environments, securing and customizing every setting is a repetitive however essential process. Automation via infrastructure as code (IaC) can alleviate this heavy lifting to realize consistency at scale.

This publish describes how one can combine your group’s present OIDC-based IdPs with Amazon MWAA to grant safe entry to your present Amazon MWAA environments. Moreover, you need to use the answer to provision new Amazon MWAA environments with the built-in OIDC-based IdP integrations. This method means that you can securely present entry to your new or present Amazon MWAA environments with out requiring AWS credentials for end-users.

Overview of Amazon MWAA environments

Managing a number of person names and passwords will be tough—that is the place SSO authentication and authorization is available in. OIDC is a broadly used customary for SSO, and it’s doable to make use of OIDC SSO authentication and authorization to entry Apache Airflow UI throughout a number of Amazon MWAA environments.

While you provision an Amazon MWAA setting, you may select public or personal Apache Airflow UI entry mode. Personal entry mode is usually utilized by clients that require limiting entry from solely inside their digital personal cloud (VPC). While you use public entry mode, the entry to the Apache Airflow UI is out there from the web, in the identical method as an AWS Administration Console web page. Web entry is required when entry is required exterior of a company community.

Whatever the entry mode, authorization to the Apache Airflow UI in Amazon MWAA is built-in with AWS Id and Entry Administration (IAM). All requests made to the Apache Airflow UI must have legitimate AWS session credentials with an assumed IAM function that has permissions to entry the corresponding Apache Airflow setting. For extra particulars on the permissions insurance policies wanted to entry the Apache Airflow UI, check with Apache Airflow UI entry coverage: AmazonMWAAWebServerAccess.

Completely different person personas corresponding to builders, knowledge scientists, system operators, or architects in your group might have entry to the Apache Airflow UI. In some organizations, not all workers have entry to the AWS console. It’s pretty widespread that workers who don’t have AWS credentials might also want entry to the Apache Airflow UI that Amazon MWAA exposes.

As well as, many organizations have a number of Amazon MWAA environments. It’s widespread to have an Amazon MWAA setting setup per software or staff. Every of those Amazon MWAA environments will be run in numerous deployment environments like growth, staging, and manufacturing. For giant organizations, you may simply envision a situation the place there’s a must handle a number of Amazon MWAA environments. Organizations want to offer safe entry to all of their Amazon MWAA environments utilizing their present OIDC supplier.

Answer Overview

The answer structure integrates an present OIDC supplier to offer authentication for accessing the Amazon MWAA Apache Airflow UI. This enables customers to log in to the Apache Airflow UI utilizing their OIDC credentials. From a system perspective, because of this Amazon MWAA can combine with an present OIDC supplier moderately than having to create and handle an remoted person authentication and authorization via IAM internally.

The answer structure depends on an Utility Load Balancer (ALB) setup with a completely certified area identify (FQDN) with public (web) or personal entry. This ALB offers SSO entry to a number of Amazon MWAA environments. The user-agent (internet browser) name movement for accessing an Apache Airflow UI console to the goal Amazon MWAA setting contains the next steps:

1. The user-agent resolves the ALB area identify from the Area Identify System (DNS) resolver.
2. The user-agent sends a login request to the ALB path /aws_mwaa/aws-console-sso with a set of question parameters populated. The request makes use of the required parameters mwaa_env and rbac_role as placeholders for the goal Amazon MWAA setting and the Apache Airflow role-based entry management (RBAC) function, respectively.
3. As soon as it receives the request, the ALB redirects the user-agent to the OIDC IdP authentication endpoint. The user-agent authenticates with the OIDC IdP with the prevailing person identify and password.
4. If person authentication is profitable, the OIDC IdP redirects the user-agent again to the configured ALB with a redirect_url with the authorization code included within the URL.
5. The ALB makes use of the authorization code obtained to acquire the access_token and OpenID JWT token with openid e-mail scope from the OIDC IdP. It then forwards the login request to the Amazon MWAA authenticator AWS Lambda operate with the JWT token included within the request header within the x-amzn-oidc-data parameter.
6. The Lambda operate verifies the JWT token discovered within the request header utilizing ALB public keys. The operate subsequently authorizes the authenticated person for the requested mwaa_env and rbac_role saved in an Amazon DynamoDB desk. The usage of DynamoDB for authorization right here is optionally available; the Lambda code operate is_allowed will be personalized to make use of different authorization mechanisms.
7. The Amazon MWAA authenticator Lambda operate redirects the user-agent to the Apache Airflow UI console within the requested Amazon MWAA setting with the login token within the redirect URL. Moreover, the operate offers the logout performance.

Amazon MWAA public community entry mode

For the Amazon MWAA environments configured with public entry mode, the person agent makes use of public routing over the web to connect with the ALB hosted in a public subnet.

The next diagram illustrates the answer structure with a numbered name movement sequence for web community reachability.

Amazon MWAA personal community entry mode

For Amazon MWAA environments configured with personal entry mode, the person agent makes use of personal routing over a devoted AWS Direct Join or AWS Consumer VPN to connect with the ALB hosted in a personal subnet.

The next diagram reveals the answer structure for Consumer VPN community reachability.

Automation via infrastructure as code

To make organising this resolution simpler, we have now launched a pre-built resolution that automates the duties concerned. The answer has been constructed utilizing the AWS Cloud Improvement Package (AWS CDK) utilizing the Python programming language. The answer is out there in our GitHub repository and helps you obtain the next:

• Arrange a safe ALB to offer OIDC-based SSO to your present Amazon MWAA setting with default Apache Airflow Admin role-based entry.
• Create new Amazon MWAA environments together with an ALB and an authenticator Lambda operate that gives OIDC-based SSO assist. With the customization supplied, you may outline the variety of Amazon MWAA environments to create. Moreover, you may customise the kind of Amazon MWAA environments created, together with defining the internet hosting VPC configuration, setting identify, Apache Airflow UI entry mode, setting class, auto scaling, and logging configurations.

The answer affords quite a few customization choices, which will be specified within the cdk.context.json file. Observe the setup directions to finish the mixing to your present Amazon MWAA environments or create new Amazon MWAA environments with SSO enabled. The setup course of creates an ALB with an HTTPS listener that gives the person entry endpoint. You could have the choice to outline the kind of ALB that you simply want. You’ll be able to outline whether or not your ALB can be public going through (web accessible) or personal going through (solely accessible throughout the VPC). It is strongly recommended to make use of a personal ALB together with your new or present Amazon MWAA environments configured utilizing personal UI entry mode.

The next sections describe the particular implementation steps and customization choices for every use case.

Stipulations

Earlier than you proceed with the set up steps, be sure to have accomplished all stipulations and run the setup-venv script as outlined throughout the README.md file of the GitHub repository.

Combine to a single present Amazon MWAA setting

When you’re integrating with a single present Amazon MWAA setting, comply with the guides within the Fast begin part. You should specify the identical ALB VPC as that of your present Amazon MWAA VPC. You’ll be able to specify the default Apache Airflow RBAC function that every one customers will assume. The ALB with an HTTPS listener is configured inside your present Amazon MWAA VPC.

Combine to a number of present Amazon MWAA environments

To connect with a number of present Amazon MWAA environments, specify solely the Amazon MWAA setting identify within the JSON file. The setup course of will create a brand new VPC with subnets internet hosting the ALB and the listener. You should outline the CIDR vary for this ALB VPC such that it doesn’t overlap with the VPC CIDR vary of your present Amazon MWAA VPCs.

When the setup steps are full, implement the post-deployment configuration steps. This contains including the ALB CNAME report to the Amazon Route 53 DNS area.

For integrating with Amazon MWAA environments configured utilizing personal entry mode, there are further steps that must be configured. These embrace configuring VPC peering and subnet routes between the brand new ALB VPC and the prevailing Amazon MWAA VPC. Moreover, you could configure community connectivity out of your user-agent to the personal ALB endpoint resolved by your DNS area.

Create new Amazon MWAA environments

You’ll be able to configure the brand new Amazon MWAA environments you need to provision via this resolution. The cdk.context.json file defines a dictionary entry within the MwaaEnvironments array. Configure the small print that you simply want for every of the Amazon MWAA environments. The setup course of creates an ALB VPC, ALB with an HTTPS listener, Lambda authorizer operate, DynamoDB desk, and respective Amazon MWAA VPCs and Amazon MWAA environments in them. Moreover, it creates the VPC peering connection between the ALB VPC and the Amazon MWAA VPC.

If you wish to create Amazon MWAA environments with personal entry mode, the ALB VPC CIDR vary specified should not overlap with the Amazon MWAA VPC CIDR vary. That is required for the automated peering connection to succeed. It could take between 20–half-hour for every Amazon MWAA setting to complete creating.

When the setting creation processes are full, run the post-deployment configuration steps. One of many steps right here is so as to add authorization information to the created DynamoDB desk to your customers. You should outline the Apache Airflow rbac_role for every of your end-users, which the Lambda authorizer operate matches to offer the requisite entry.

Confirm entry

When you’ve accomplished with the post-deployment steps, you may log in to the URL utilizing your ALB FQDN. For instance, In case your ALB FQDN is alb-sso-mwaa.instance.com, you may log in to your goal Amazon MWAA setting, named Env1, assuming a selected Apache Airflow RBAC function (corresponding to Admin), utilizing the next URL: https://alb-sso-mwaa.instance.com/aws_mwaa/aws-console-sso?mwaa_env=Env1&rbac_role=Admin. For the Amazon MWAA environments that this resolution created, you could have acceptable Apache Airflow rbac_role entries in your DynamoDB desk.

The answer additionally offers a logout function. To sign off from an Apache Airflow console, use the conventional Apache Airflow console logout. To sign off from the ALB, you may, for instance, use the URL https://alb-sso-mwaa.instance.com/logout.

Clear up

Observe the readme documented steps within the part Destroy CDK stacks within the GitHub repo, which reveals the right way to clear up the artifacts created by way of the AWS CDK deployments. Bear in mind to revert any handbook configurations, like VPC peering connections, that you simply may need made after the deployments.

Conclusion

This publish supplied an answer to combine your group’s OIDC-based IdPs with Amazon MWAA to grant safe entry to a number of Amazon MWAA environments. We walked via the answer that solves this drawback utilizing infrastructure as code. This resolution permits completely different end-user personas in your group to entry the Amazon MWAA Apache Airflow UI utilizing OIDC SSO.

To make use of the answer to your personal environments, check with Utility load balancer single-sign-on for Amazon MWAA. For added code examples on Amazon MWAA, check with Amazon MWAA code examples.

Concerning the Authors

Ajay Vohra is a Principal Prototyping Architect specializing in notion machine studying for autonomous automobile growth. Previous to Amazon, Ajay labored within the space of massively parallel grid-computing for monetary threat modeling.

Jaswanth Kumar is a customer-obsessed Cloud Utility Architect at AWS in NY. Jaswanth excels in software refactoring and migration, with experience in containers and serverless options, coupled with a Masters Diploma in Utilized Laptop Science.

Aneel Murari is a Sr. Serverless Specialist Answer Architect at AWS based mostly within the Washington, D.C. space. He has over 18 years of software program growth and structure expertise and holds a graduate diploma in Laptop Science. Aneel helps AWS clients orchestrate their workflows on Amazon Managed Apache Airflow (MWAA) in a safe, price efficient and efficiency optimized method.

Parnab Basak is a Options Architect and a Serverless Specialist at AWS. He makes a speciality of creating new options which can be cloud native utilizing fashionable software program growth practices like serverless, DevOps, and analytics. Parnab works carefully within the analytics and integration providers house serving to clients undertake AWS providers for his or her workflow orchestration wants.