It’s frequent to retailer the logs generated by buyer’s functions and companies in varied instruments. These logs are vital for compliance, audits, troubleshooting, safety incident responses, assembly safety insurance policies, and lots of different functions. You possibly can carry out log evaluation on these logs to know customers’ software habits and patterns to make knowledgeable selections.

When operating workloads on Amazon Net Providers (AWS), it is advisable to analyze Amazon Digital Non-public Cloud (Amazon VPC) Circulation Logs to trace the IP visitors going to and from the community interfaces for the workloads of their VPC. Analyzing VPC circulate logs helps you perceive how your functions are speaking over the VPC community and acts as a principal supply of data to the community in your VPC.

You possibly can simply ship information to supported locations utilizing the Amazon Kinesis Knowledge Firehose integration with VPC circulate logs. Kinesis Knowledge Firehose is a completely managed service for delivering near-real-time streaming information to numerous locations for storage and performing near-real-time analytics. With its extensible information transformation capabilities, you can even streamline log processing and log supply pipelines right into a single Kinesis Knowledge Firehose supply stream. You possibly can carry out analytics on VPC circulate logs delivered out of your VPC utilizing the Kinesis Knowledge Firehose integration with Datadog as a vacation spot.

Datadog is a monitoring and safety platform and AWS Accomplice Community (APN) Superior Know-how Accomplice with AWS Competencies in AWS Cloud Operations, DevOps, Migration, Safety, Networking, Containers, and Microsoft Workloads, together with many others.

Datadog allows you to simply discover and analyze logs to realize deeper insights into the state of your functions and AWS infrastructure. You possibly can analyze all of your AWS service logs whereas storing solely those you want, generate metrics from aggregated logs to uncover, and ship alerts about tendencies in your AWS companies.

On this publish, you learn to combine VPC circulate logs with Kinesis Knowledge Firehose and ship it to Datadog.

Resolution overview

This resolution makes use of native integration of VPC circulate logs streaming to Kinesis Knowledge Firehose. We use a Kinesis Knowledge Firehose supply stream to buffer the streamed VPC circulate logs to a Datadog vacation spot endpoint in your Datadog account. You should utilize these logs with Datadog Log Administration and Datadog Cloud SIEM to investigate the well being, efficiency, and safety of your cloud assets.

The next diagram illustrates the answer structure.

We stroll you thru the next high-level steps:

1. Hyperlink your AWS account together with your Datadog account.
2. Create the Kinesis Knowledge Firehose stream the place VPC service streams the circulate logs.
3. Create the VPC circulate log subscription to Kinesis Knowledge Firehose.
4. Visualize VPC circulate logs within the Datadog dashboard.

The account ID 123456781234 used on this publish is a dummy account. It’s used just for demonstration functions.

Stipulations

You must have the next stipulations:

Hyperlink your AWS account together with your Datadog account for AWS integration

Comply with the directions offered on the Datadog web site for AWS Integration. To configure log archiving and enrich the log information despatched out of your AWS account with helpful context, hyperlink the accounts. Whenever you full the linking setup, proceed to the next step.

Create a Kinesis Knowledge Firehose stream

Now that your Datadog integration with AWS is full, you may create a Kinesis Knowledge Firehose supply stream the place VPC Circulation Logs are streamed by following these steps:

1. On the Amazon Kinesis console, select Kinesis Knowledge Firehose within the navigation pane.
2. Select Create supply stream.
3. Select Direct PUT because the supply.
4. Set Vacation spot as Datadog.
   

5. For Supply stream title, enter PUT-DATADOG-DEMO.
6. Preserve Knowledge transformation set to Disabled beneath Remodel information.
7. In Vacation spot settings, for HTTP endpoint URL, select the specified log’s HTTP endpoint primarily based in your Area and Datadog account configuration.
   
8. For API key, enter your Datadog API key.

This enables your supply stream to publish VPC Circulation logs to the Datadog endpoint. API keys are distinctive to your group. An API key is required by the Datadog Agent to submit metrics and occasions to Datadog.

9. Set Content material encoding to GZIP to cut back the dimensions of knowledge transferred.
10. Set the Retry period to 60.You possibly can change the Retry period worth if it is advisable to. This relies on the request dealing with capability of the Datadog endpoint.
    
    Below Buffer hints, Buffer measurement and Buffer interval are set with default values for Datadog integration.
    

11. Below Backup settings, as talked about within the stipulations, select the S3 bucket that you just created to retailer failed logs and backup with particular prefix.
12. Below S3 buffer hints part, set Buffer measurement to five and Buffer interval to 300.

You possibly can change the S3 buffer measurement and interval primarily based in your necessities.

13. Below S3 compression and encryption, choose GZIP for Compression for information information or one other compression technique of your alternative.

Compressing information reduces the required cupboard space.

14. Choose Disabled for Encryption of the information information. You possibly can allow encryption of the information information to safe entry to your logs.
    

15. Optionally, in Superior settings, choose Allow server-side encryption for supply information in supply stream.
    You should utilize AWS managed keys or a CMK managed by you for the encryption kind.

16. Allow CloudWatch error logging.
17. Select Create or replace IAM position, which is created by Kinesis Knowledge Firehose as a part of this stream.
    

18. Select Subsequent.
19. Evaluation your settings.
20. Select Create supply stream.

Create a VPC circulate logs subscription

Create a VPC circulate logs subscription for the Kinesis Knowledge Firehose supply stream you created within the earlier step:

1. On the Amazon VPC console, select Your VPCs.
2. Choose the VPC that you just to create the circulate log for.
3. On the Actions menu, select Create circulate log.
   

4. Choose All to ship all circulate log information to the Firehose vacation spot.

If you wish to filter the circulate logs, you possibly can alternatively choose Settle for or Reject.

5. For Most aggregation interval, choose 10 minutes or the minimal setting of 1 minute when you want the circulate log information to be out there for near-real-time evaluation in Datadog.
6. For Vacation spot, choose Ship to Kinesis Knowledge Firehose in the identical account if the supply stream is about up on the identical account the place you create the VPC circulate logs.

If you wish to ship the information to a unique account, check with Publish circulate logs to Kinesis Knowledge Firehose.

7. Select an choice for Log document format:
8. For those who go away Log document format because the AWS default format, the circulate logs are despatched as model 2 format.
9. Alternatively, you may specify the customized fields for circulate logs to seize and ship it to Datadog.

For extra data on log format and out there fields, check with Circulation log information.

10. Select Create circulate log.
    

Now let’s discover the VPC circulate logs in Datadog.

Visualize VPC circulate logs within the Datadog dashboard

Within the Logs Search choice within the navigation pane, filter to supply:vpc. The VPC circulate logs out of your VPC are within the Datadog Log Explorer and are mechanically parsed so you may analyze your logs by supply, vacation spot, motion, or different attributes.

Clear up

After you take a look at this resolution, delete all of the assets you created to keep away from incurring future costs. Check with the next hyperlinks for directions for deleting the assets:

Conclusion

On this publish, we walked via an answer of how you can combine VPC circulate logs with a Kinesis Knowledge Firehose supply stream, ship it to a Datadog vacation spot with no code, and visualize it in a Datadog dashboard. With Datadog, you may simply discover and analyze logs to realize deeper insights into the state of your functions and AWS infrastructure.

Do this new, fast, and hassle-free manner of sending your VPC circulate logs to a Datadog vacation spot utilizing Kinesis Knowledge Firehose.

Concerning the Writer

Chaitanya Shah is a Sr. Technical Account Supervisor(TAM) with AWS, primarily based out of New York. He has over 22 years of expertise working with enterprise prospects. He likes to code and actively contributes to the AWS options labs to assist prospects resolve advanced issues. He supplies steering to AWS prospects on finest practices for his or her AWS Cloud migrations. He’s additionally specialised in AWS information switch and the information and analytics area.