lambda-kinesis-to-fluent

baikonur-oss/lambda-kinesis-to-fluent/aws

Terraform Module HCL AWS ⏱ stale 5y 9mo

Terraform module and Lambda for sending logs from Kinesis to Fluent endpoint.

Install

module "lambda-kinesis-to-fluent" {

source = "baikonur-oss/lambda-kinesis-to-fluent/aws"

version = "0.1.0"

}

⭐ Source on GitHub 📦 Registry page

README

Amazon Kinesis to Fluent log transfer Terraform module Terraform module and Lambda for sending JSON log records from Kinesis Data Streams to a Fluent endpoint. !terraform v0.11.x Prerequisites 1. Records in Kinesis stream must be valid JSON data. Non-JSON data will be saved with unknown prefix. 1. gzipped JSON, CloudWatch Logs subscription filters log format are supported. 2. Logs without either of necessary keys listed below will be saved as unknown as well. 2. JSON data must have the following keys (key names are modifiable via variables): 1. log_type: Log type identifier. Failed logs will be saved by this key: %log_type%/YYYY-MM/DD/. 2. log_id: Any unique identifier. Used to avoid file overwrites on S3. Also is useful to search for a specific log record. 3. time: Any timestamp supported

Inputs (24)

Name	Type	Description	Default
batch_size	string	Maximum number of records passed for a single Lambda invocation	required
security_group_ids	list	VPC Security group IDs	required
failed_log_s3_bucket	string	S3 bucket name for saving failed logs (ES API errors etc.)	required
kinesis_stream_arn	string	Source Kinesis Data Streams stream name	required
subnet_ids	list	VPC Subnet IDs	required
name	string	Resource name	required
lambda_package_url	string	Lambda package URL (see Usage in README)	required
failed_log_s3_prefix	string	Path prefix for failed logs	required
fluent_sink_endpoint	string	Target Fluent sink endpoint (IP or hostname)	required
fluent_tag	string	Second-level fluent tag name (first level is log type)	required
log_type_field_whitelist	list	Log type whitelist (if empty, all types will be processed)	`[]`
timeout	string	Lambda Function timeout in seconds	`60`
runtime	string	Lambda Function runtime	`"python3.7"`
tracing_mode	string	X-Ray tracing mode (see: https://docs.aws.amazon.com/lambda/latest/dg/API_Tracin	`"PassThrough"`
tags	map	Tags for Lambda Function	`{}`
log_retention_in_days	string	Lambda Function log retention in days	`30`
timezone	string	tz database timezone name (e.g. Asia/Tokyo)	`"UTC"`
memory	string	Lambda Function memory in megabytes	`256`
log_type_field	string	Key name for log type	`"log_type"`
log_type_unknown_prefix	string	Log type prefix for logs without log type field	`"unknown"`
log_id_field	string	Key name for unique log ID	`"log_id"`
log_timestamp_field	string	Key name for log timestamp	`"time"`
handler	string	Lambda Function handler (entrypoint)	`"main.handler"`
starting_position	string	Kinesis ShardIterator type (see: https://docs.aws.amazon.com/kinesis/latest/APIR	`"TRIM_HORIZON"`