s3.md

Amazon S3 plugin for Fluentd

Overview

s3 output plugin buffers event logs in local file and upload it to S3 periodically. This plugin splits files exactly by using the time of event logs (not the time when the logs are received). For example, a log '2011-01-02 message B' is reached, and then another log '2011-01-03 message B' is reached in this order, the former one is stored in "20110102.gz" file, and latter one in "20110103.gz" file.

Example: S3 Output Deployment

Example output configurations

spec:
 s3:
   aws_key_id:
     valueFrom:
       secretKeyRef:
         name: logging-s3
         key: awsAccessKeyId
   aws_sec_key:
     valueFrom:
       secretKeyRef:
         name: logging-s3
         key: awsSecretAccesKey
   s3_bucket: logging-amazon-s3
   s3_region: eu-central-1
   path: logs/${tag}/%Y/%m/%d/
   buffer:
     timekey: 10m
     timekey_wait: 30s
     timekey_use_utc: true*/

Configuration

Output Config

Variable Name	Type	Required	Default	Description
aws_key_id	*secret.Secret	No	-	AWS access key id Secret
aws_sec_key	*secret.Secret	No	-	AWS secret key. Secret
check_apikey_on_start	string	No	-	Check AWS key on start
grant_read	string	No	-	Allows grantee to read the object data and its metadata
overwrite	string	No	-	Overwrite already existing path
path	string	No	-	Path prefix of the files on S3
grant_write_acp	string	No	-	Allows grantee to write the ACL for the applicable object
check_bucket	string	No	-	Check bucket if exists or not
sse_customer_key	string	No	-	Specifies the customer-provided encryption key for Amazon S3 to use in encrypting data
sse_customer_key_md5	string	No	-	Specifies the 128-bit MD5 digest of the encryption key according to RFC 1321
compute_checksums	string	No	-	AWS SDK uses MD5 for API request/response by default
warn_for_delay	string	No	-	Given a threshold to treat events as delay, output warning logs if delayed events were put into s3
use_bundled_cert	string	No	-	Use aws-sdk-ruby bundled cert
s3_endpoint	string	No	-	Custom S3 endpoint (like minio)
ssekms_key_id	string	No	-	Specifies the AWS KMS key ID to use for object encryption
s3_metadata	string	No	-	Arbitrary S3 metadata headers to set for the object
force_path_style	string	No	-	If true, the bucket name is always left in the request URI and never moved to the host as a sub-domain
auto_create_bucket	string	No	-	Create S3 bucket if it does not exists
index_format	string	No	-	sprintf format for %{index}
signature_version	string	No	-	Signature version for API Request (s3,v4)
enable_transfer_acceleration	string	No	-	If true, S3 Transfer Acceleration will be enabled for uploads. IMPORTANT: You must first enable this feature on your destination S3 bucket
ssl_verify_peer	string	No	-	If false, the certificate of endpoint will not be verified
proxy_uri	string	No	-	URI of proxy environment
grant_read_acp	string	No	-	Allows grantee to read the object ACL
check_object	string	No	-	Check object before creation
sse_customer_algorithm	string	No	-	Specifies the algorithm to use to when encrypting the object
use_server_side_encryption	string	No	-	The Server-side encryption algorithm used when storing this object in S3 (AES256, aws:kms)
s3_region	string	No	-	S3 region name
acl	string	No	-	Permission for the object in S3
grant_full_control	string	No	-	Allows grantee READ, READ_ACP, and WRITE_ACP permissions on the object
hex_random_length	string	No	-	The length of %{hex_random} placeholder(4-16)
s3_object_key_format	string	No	%{path}%{time_slice}_%{index}.%{file_extension}	The format of S3 object keys
s3_bucket	string	Yes	-	S3 bucket name
store_as	string	No	-	Archive format on S3
storage_class	string	No	-	The type of storage to use for the object(STANDARD,REDUCED_REDUNDANCY,STANDARD_IA)
aws_iam_retries	string	No	-	The number of attempts to load instance profile credentials from the EC2 metadata service using IAM role
buffer	*Buffer	No	-	Buffer
format	*Format	No	-	Format
assume_role_credentials	*S3AssumeRoleCredentials	No	-	Assume Role Credentials
instance_profile_credentials	*S3InstanceProfileCredentials	No	-	Instance Profile Credentials
shared_credentials	*S3SharedCredentials	No	-	Shared Credentials

Assume Role Credentials

assume_role_credentials

Variable Name	Type	Required	Default	Description
role_arn	string	Yes	-	The Amazon Resource Name (ARN) of the role to assume
role_session_name	string	Yes	-	An identifier for the assumed role session
policy	string	No	-	An IAM policy in JSON format
duration_seconds	string	No	-	The duration, in seconds, of the role session (900-3600)
external_id	string	No	-	A unique identifier that is used by third parties when assuming roles in their customers' accounts.

Instance Profile Credentials

instance_profile_credentials

Variable Name	Type	Required	Default	Description
ip_address	string	No	169.254.169.254	IP address
port	string	No	80	Port number
http_open_timeout	string	No	-	Number of seconds to wait for the connection to open
http_read_timeout	string	No	-	Number of seconds to wait for one block to be read
retries	string	No	-	Number of times to retry when retrieving credentials

Shared Credentials

shared_credentials

Variable Name	Type	Required	Default	Description
profile_name	string	No	-	Profile name. Default to 'default' or ENV['AWS_PROFILE']
path	string	No	$HOME/.aws/credentials	Path to the shared file.