AWS Glue is a serverless knowledge integration service that makes it simpler to find, put together, and mix knowledge for analytics, machine studying (ML), and software improvement. You should utilize AWS Glue to create, run, and monitor knowledge integration and ETL (extract, remodel, and cargo) pipelines and catalog your belongings throughout a number of knowledge shops.

Some of the widespread questions we get from prospects is the best way to successfully optimize prices on AWS Glue. Over time, we now have constructed a number of options and instruments to assist prospects handle their AWS Glue prices. For instance, AWS Glue Auto Scaling and AWS Glue Flex may also help you scale back the compute value related to processing your knowledge. AWS Glue interactive classes and notebooks may also help you scale back the price of creating your ETL jobs. For extra details about cost-saving finest practices, consult with Monitor and optimize value on AWS Glue for Apache Spark. Moreover, to know knowledge switch prices, consult with the Value Optimization Pillar outlined in AWS Properly-Architected Framework. For knowledge storage, you’ll be able to apply basic finest practices outlined for every knowledge supply. For a value optimization technique utilizing Amazon Easy Storage Service (Amazon S3), consult with Optimizing storage prices utilizing Amazon S3.

On this publish, we deal with the remaining piece—the price of logs written by AWS Glue.

Earlier than we get into the price evaluation of logs, let’s perceive the explanations to allow logging in your AWS Glue job and the present choices accessible. Once you begin an AWS Glue job, it sends the real-time logging data to Amazon CloudWatch (each 5 seconds and earlier than every executor stops) in the course of the Spark software begins working. You may view the logs on the AWS Glue console or the CloudWatch console dashboard. These logs offer you insights into your job runs and assist you optimize and troubleshoot your AWS Glue jobs. AWS Glue gives quite a lot of filters and settings to scale back the verbosity of your logs. Because the variety of job runs will increase, so does the amount of logs generated.

To optimize CloudWatch Logs prices, AWS not too long ago introduced a brand new log class for occasionally accessed logs known as Amazon CloudWatch Logs Rare Entry (Logs IA). This new log class gives a tailor-made set of capabilities at a decrease value for occasionally accessed logs, enabling you to consolidate all of your logs in a single place in a cheap method. This class supplies a cheaper choice for ingesting logs that solely should be accessed sometimes for auditing or debugging functions.

On this publish, we clarify what the Logs IA class is, the way it may also help scale back prices in comparison with the usual log class, and the best way to configure your AWS Glue assets to make use of this new log class. By routing logs to Logs IA, you’ll be able to obtain important financial savings in your CloudWatch Logs spend with out sacrificing entry to essential debugging data once you want it.

CloudWatch log teams utilized by AWS Glue job steady logging

When steady logging is enabled, AWS Glue for Apache Spark writes Spark driver/executor logs and progress bar data into the next log group:

If a safety configuration is enabled for CloudWatch logs, AWS Glue for Apache Spark will create a log group named as follows for steady logs:

<Log-Group-Title>-<Safety-Configuration-Title>

The default and {custom} log teams shall be as follows:

• The default steady log group shall be /aws-glue/jobs/logs-v2-<Safety-Configuration-Title>
• The {custom} steady log group shall be <custom-log-group-name>-<Safety-Configuration-Title>

You may present a {custom} log group title by way of the job parameter –continuous-log-logGroup.

Getting began with the brand new Rare Entry log class for AWS Glue workload

To achieve the advantages from Logs IA in your AWS Glue workloads, it is advisable full the next two steps:

1. Create a brand new log group utilizing the brand new Log IA class.
2. Configure your AWS Glue job to level to the brand new log group

Full the next steps to create a brand new log group utilizing the brand new Rare Entry log class:

3. On the CloudWatch console, select Log teams underneath Logs within the navigation pane.
4. Select Create log group.
5. For Log group title, enter /aws-glue/jobs/logs-v2-infrequent-access.
6. For Log class, select Rare Entry.
7. Select Create.

Full the next steps to configure your AWS Glue job to level to the brand new log group:

1. On the AWS Glue console, select ETL jobs within the navigation pane.
2. Select your job.
3. On the Job particulars tab, select Add new parameter underneath Job parameters.
4. For Key, enter --continuous-log-logGroup.
5. For Worth, enter /aws-glue/jobs/logs-v2-infrequent-access.
6. Select Save.
7. Select Run to set off the job.

New log occasions are written into the brand new log group.

View the logs with the Rare Entry log class

Now you’re able to view the logs with the Rare Entry log class. Open the log group /aws-glue/jobs/logs-v2-infrequent-access on the CloudWatch console.

Once you select one of many log streams, you’ll discover that it redirects you to the CloudWatch console Logs Perception web page with a pre-configured default command and your log stream chosen by default. By selecting Run question, you’ll be able to view the precise log occasions on the Logs Insights web page.

Issues

Take into accout the next issues:

• You can’t change the log class of a log group after it’s created. It’s worthwhile to create a brand new log group to configure the Rare Entry class.
• The Logs IA class gives a subset of CloudWatch Logs capabilities, together with managed ingestion, storage, cross-account log analytics, and encryption with a decrease ingestion value per GB. For instance, you’ll be able to’t view log occasions by way of the usual CloudWatch Logs console. To be taught extra concerning the options provided throughout each log lessons, consult with Log Courses.

Conclusion

This publish supplied step-by-step directions to information you thru enabling Logs IA in your AWS Glue job logs. In case your AWS Glue ETL jobs generate massive volumes of log knowledge that makes it a problem as you scale your functions, the very best practices demonstrated on this publish may also help you cost-effectively scale whereas centralizing all of your logs in CloudWatch Logs. Begin utilizing the Rare Entry class together with your AWS Glue workloads immediately and revel in the price advantages.

In regards to the Authors

Noritaka Sekiyama is a Principal Huge Knowledge Architect on the AWS Glue group. He works based mostly in Tokyo, Japan. He’s chargeable for constructing software program artifacts to assist prospects. In his spare time, he enjoys biking on his highway bike.

Abeetha Bala is a Senior Product Supervisor for Amazon CloudWatch, primarily targeted on logs. Being buyer obsessed, she solves observability challenges by way of modern and cost-effective methods.

Kinshuk Pahare is a frontrunner in AWS Glue’s product administration group. He drives efforts on the platform, developer expertise, and large knowledge processing frameworks like Apache Spark, Ray, and Python Shell.