lambda-kinesis-to-s3

baikonur-oss/lambda-kinesis-to-s3/aws

Terraform Module HCL AWS ⏱ stale 5y 5mo

Terraform module and Lambda for Kinesis to S3 logs transfer

Install

module "lambda-kinesis-to-s3" {

source = "baikonur-oss/lambda-kinesis-to-s3/aws"

version = "2.1.1"

}

⭐ Source on GitHub 📦 Registry page

README

Amazon Kinesis to S3 log transfer Terraform module Terraform module and Lambda for saving JSON log records from Kinesis Data Streams to S3. !terraform v0.12.x ![Language grade: Python](https://lgtm.com/projects/g/baikonur-oss/terraform-aws-lambda-kinesis-to-s3/context:python) Prerequisites 1. Records in Kinesis stream must be valid JSON data. Non-JSON data will be saved with unknown prefix. 1. gzipped JSON, CloudWatch Logs subscription filters log format are supported. 2. Logs without either of necessary keys listed below will be saved as unknown as well. 2. JSON data must have the following keys (key names are modifiable via variables): 1. log_type: Log type identifier. Log data will be saved by this key: %log_type%/YYYY-MM/DD/. 2. log_id: Any unique identifier. Used to avoid file overwri

Inputs (20)

Name	Type	Description	Default
batch_size	string	Maximum number of records passed for a single Lambda invocation	required
log_path_prefix	string	Log file path prefix	required
lambda_package_url	string	Lambda package URL (see Usage in README)	required
kinesis_stream_arn	string	Source Kinesis Data Streams stream name	required
name	string	Resource name	required
log_bucket	string	Target S3 bucket to save data to	required
log_type_unknown_prefix	string	Log type prefix for logs without log type field	`"unknown"`
log_retention_in_days	string	Lambda Function log retention in days	`30`
timezone	string	tz database timezone name (e.g. Asia/Tokyo)	`"UTC"`
timeout	string	Lambda Function timeout in seconds	`60`
starting_position	string	Kinesis ShardIterator type (see: https://docs.aws.amazon.com/kinesis/latest/APIR	`"TRIM_HORIZON"`
log_type_field	string	Key name for log type	`"log_type"`
log_type_field_whitelist	list(string)	Log type whitelist (if empty, all types will be processed)	`[]`
runtime	string	Lambda Function runtime	`"python3.7"`
log_timestamp_field	string	Key name for log timestamp	`"time"`
tags	map(string)	Tags for Lambda Function	`{}`
memory	string	Lambda Function memory in megabytes	`256`
handler	string	Lambda Function handler (entrypoint)	`"main.handler"`
log_id_field	string	Key name for unique log ID	`"log_id"`
tracing_mode	string	X-Ray tracing mode (see: https://docs.aws.amazon.com/lambda/latest/dg/API_Tracin	`"PassThrough"`