s3.cn-north-1.amazonaws.com.cn€¦ · web viewwe can use aws lambda to replicate s3 bucket...

S3 Bucket Cross Account Replication CloudFormation Templates - User Guide

CloudFormation Templates - User Guide

Solution BackgroundBased on AWS Security and Audit Account best practice, we need to store security logs in a centralized bucket. For example, we have a production account, a development & test account and an audit account. We need to store CloudTrial log, AWS Config log and VPC flows log to a central S3 bucket which belongs to audit account. The auditor (a user in audit account) can have read-only access right to visit the central S3 bucket.

We can use AWS lambda to replicate S3 bucket automatically. The benefit of the solution is we can have a full copy of all protected S3 bucket. We only replicate new created objects in source S3 bucket, so an object is deleted in source S3 bucket, it will not be deleted in target S3. The solution diagram is illustrated below:

The whole idea of these CloudFormation template coming from Nitin Prakash’s blog (http://www.smartshifttech.com/cross-account-s3-data-copy-using-lambda-function) , I made the following modifications:

- Change policy documents and lambda functions to support China Region (BJS and ZHY)

- Change lambda function naming method to support multi-sources replicated to the same target

- Add a bucket policy to destination bucket to enhance security

2

http://www.smartshifttech.com/cross-account-s3-data-copy-using-lambda-function


- Change bucket policy of source bucket to allow AWS services (CloudTrail, AWS Config, etc) to write log files in source bucket.

There are 3 templates:1. 1source.json

You can use this template to create an IAM role in source account, this role will be used by lambda function via assume role to read objects from source S3 bucket.

Input parameters: NoneOutput: The name and ARN of the IAM role in source account

2. 2target.jsonYou can use this templet to create an IAM role in destination account, this role will be used by lambda function to create objects in destination S3 bucket. A bucket policy is also created in this template to protect destination S3 bucket. This policy will allow lambda function to read and write objects, the auditor in destination account can only have read-only right.

Input parameters: Source S3 bucket name Destination S3 bucket name The ARN of source IAM role (this is created via 1source.json) The security auditor’s user name in destination account

Output: The name and ARN of the IAM role in destination account3. 3source.json

This is the major part of the whole solution. This template should be executed in source account. This template will create the following components:- A SNS topic which will be used to trigger Lambda function- A bucket policy for source S3 bucket. This policy will authorize lambda

function have the proper privilege to read object from source. This policy can also make sure that CloudTrail and AWS Config services have the privilege to write log files to the source.

- A lambda function (S32S3Function) which is used to replicate new objects from source to destination.

- A lambda function (S3BucketConfiguration) which is used to attach a bucket notification to the source bucket. When a new object is created in source bucket, this bucket notification will send a message to the SNS topic, then S32S3Function will be triggered.

Input parameters: The first stack name (the stack created by 1source.json) Source S3 bucket name Destination S3 bucket name

3


Destination account ID number The ARN of destination IAM role (this is created via 2target.json)

How to Deploy the TemplatesPerquisites:

- Source and target S3 bucket is exist- The IAM user for security auditor is created in destination account

Deployment steps:There are three steps. First, login to the source account, execute CloudFormation using 1source.json. Second, login to the destination account, execute CloudFormation using 2target.json. Third, login to the source account, execute CloudFormation using 3source.jsonStep-1: Source account

1. Login and switch to CloudFormation service and click create new stack button

2. Upload the template to S3

3. Click Next and enter a stack name, Click Next twice, select the following:

4. Click Create5. Wait for the template to finish before going to step-2. You will see the following

screen if AWS deploy this template successfully.

4


Step-2: Destination account1. Login to destination account and switch to CloudFormation service and click

create new stack button


3. Click Next, enter a stack name, destination bucket name, auditor user name, source bucket name, source IAM role ARN (this can be got from the output of the first stack)

5


4. Click Next twice, select the following:

5. Click Create.6. It will take a while to deploy the template. Once finished, you will see the

following screen.

Step-3: Source account1. Login and switch to CloudFormation service and click create new stack button


6


3. Click Next, enter a stack name, destination bucket name, destination account ID number, source bucket name, destination IAM role ARN (this can be got from the output of the second stack), the first stack name.

4. Click Next twice, select the following:

5. Click Create.6. If everything is OK, you will see the following screen.

7


I also create a bash script to run these CloudFormation template automatically. But you need to install and configure AWS CLI.You need to edit ~/.aws/credentials file and ~/.aws/config file to add 2 profiles for both source account and destination account. For more information, you can reference: https://docs.aws.amazon.com/cli/latest/userguide/cli-multiple-profiles.html.The following parameters you need to edit before you run s32s3.sh:

SourceBucketName DestinationBucketName DestinationAccountId DestinationSecurityUserName SourceProfile TargetProfile Step1StackName (the stack name to execute 1source.json) Step2StackName (the stack name to execute 2target.json) Step3StackName (the stack name to execute 3source.json)

LimitationsThese CloudFormation templates can only be run in China Regions (BJS and ZHY), you need to modify names of AWS service endpoint if you try to use these templates in other AWS regions.

Appendix-Source Code1. file s32s3.sh:

#!/bin/bashSourceBucketName=" "DestinationBucketName=" "DestinationAccountId=" "DestinationSecurityUserName=" "SourceProfile=" "

8

https://docs.aws.amazon.com/cli/latest/userguide/cli-multiple-profiles.html


TargetProfile=" "

Step1StackName="stack1"Step2StackName="stack2"Step3StackName="stack3"

SourceAccountId=$(aws sts get-caller-identity --output text --query 'Account' --profile $SourceProfile)echo "Start to create the first stack"

aws cloudformation create-stack --profile $SourceProfile --capabilities "CAPABILITY_IAM" --stack-name $Step1StackName --template-body file://1source.jsonaws cloudformation wait stack-create-complete --profile $SourceProfile --stack-name $Step1StackName

SourceS3CpRole=$(aws cloudformation describe-stacks --profile $SourceProfile --stack-name $Step1StackName --query 'Stacks[].Outputs[?OutputKey==`SourceS3CpRoleName`].OutputValue' --output text)SourceS3CpRoleARN=$(aws cloudformation describe-stacks --profile $SourceProfile --stack-name $Step1StackName --query 'Stacks[].Outputs[?OutputKey==`SourceS3CpRoleARN`].OutputValue' --output text)

echo "Source S3 CP Role: ${SourceS3CpRole}"echo "Source S3 CP Role ARN: ${SourceS3CpRoleARN}"echo "Start to create the second stack"

aws cloudformation create-stack --profile $TargetProfile --capabilities "CAPABILITY_IAM" --stack-name $Step2StackName --template-body file://2target.json --parameters ParameterKey=SourceBucketName,ParameterValue=$SourceBucketName ParameterKey=DestinationBucketName,ParameterValue=$DestinationBucketName

9


ParameterKey=SourceS3CpRole,ParameterValue=$SourceS3CpRoleARN ParameterKey=DestinationSecurityUserName ParameterValue=$DestinationSecurityUserNameaws cloudformation wait stack-create-complete --profile $TargetProfile --stack-name $Step2StackName

DestinationS3CpRole=$(aws cloudformation describe-stacks --profile $TargetProfile --stack-name $Step2StackName --query 'Stacks[].Outputs[].OutputValue' --output text)

echo "Destination S3 CP Role: ${DestinationS3CpRole}"echo "Start to create the third stack"

aws cloudformation create-stack --profile $SourceProfile --capabilities "CAPABILITY_IAM" --stack-name $Step3StackName --template-body file://3source.json --parameters ParameterKey=Step1StackName,ParameterValue=$Step1StackName ParameterKey=SourceBucketName,ParameterValue=$SourceBucketName ParameterKey=DestinationBucketName,ParameterValue=$DestinationBucketName ParameterKey=DestinationAccountId,ParameterValue=$DestinationAccountId ParameterKey=DestinationS3CpRole,ParameterValue=$DestinationS3CpRoleaws cloudformation wait stack-create-complete --profile $SourceProfile --stack-name $Step3StackName

echo "Done!"

2. file 1source.json

{ "AWSTemplateFormatVersion": "2010-09-09", "Description": "First step to setup cross account S3 copy",

"Resources": {

10


"SourceS3CpRole": { "Type": "AWS::IAM::Role", "Properties": { "AssumeRolePolicyDocument": { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": { "Service": "lambda.amazonaws.com" }, "Action": "sts:AssumeRole" }] } } } }, "Outputs": { "SourceS3CpRoleARN": { "Value": { "Fn::GetAtt": ["SourceS3CpRole", "Arn"] }, "Export": { "Name": { "Fn::Sub": "${AWS::StackName}SourceS3CpRoleARN" } } }, "SourceS3CpRoleName": { "Value": { "Ref": "SourceS3CpRole" }, "Export": { "Name": { "Fn::Sub": "${AWS::StackName}SourceS3CpRole" }

11


} } }}

3. file 2target.json

{ "AWSTemplateFormatVersion": "2010-09-09", "Description": "Second step to setup cross account S3 copy", "Parameters": { "SourceBucketName": { "Description": "Source S3 Bucket Name", "Type": "String" }, "DestinationBucketName": { "Description": "Target S3 Bucket Name", "Type": "String" }, "SourceS3CpRole": { "Description": "SourceS3CpRole ARN", "Type": "String" }, "DestinationSecurityUserName": { "Description": "The user name in destination account who need to access destination S3 bucket", "Type": "String" } }, "Resources": { "DestinationS3CpRole": { "Type": "AWS::IAM::Role", "Properties": { "AssumeRolePolicyDocument": { "Version": "2012-10-17", "Statement": [{

12


"Effect": "Allow", "Principal": { "Service": "lambda.amazonaws.com", "AWS": { "Ref": "SourceS3CpRole" } }, "Action": "sts:AssumeRole" }] } } }, "DestinationS3CpPolicy": { "Type": "AWS::IAM::Policy", "Properties": { "Roles": [{ "Ref": "DestinationS3CpRole" }], "PolicyName": "destination-s3-cp-policy", "PolicyDocument": { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Action": [ "s3:GetObject", "s3:ListObject" ], "Resource": { "Fn::Join": [ "", [ "arn:aws-cn:s3:::", { "Ref": "SourceBucketName" }, "/",

13


] ], "Fn::Join": [ "", [ "arn:aws-cn:s3:::", { "Ref": "SourceBucketName" }, "/*", ] ] } }, { "Effect": "Allow", "Action": [ "s3:ListObject", "s3:PutObject", "s3:GetObject" ], "Resource": { "Fn::Join": [ "", [ "arn:aws-cn:s3:::", { "Ref": "DestinationBucketName" }, "/", ] ], "Fn::Join": [ "", [ "arn:aws-cn:s3:::", { "Ref": "DestinationBucketName"

14


}, "/*", ] ] } } ] } } }, "TargetBucketPolicy": { "Type": "AWS::S3::BucketPolicy", "Properties": { "Bucket": { "Ref": "DestinationBucketName" }, "PolicyDocument": { "Statement": [{ "Sid": "Target Bucket Policy", "Effect": "Allow", "Principal": { "AWS": { "Fn::Join": [ "", [

{ "Fn::GetAtt": ["DestinationS3CpRole", "Arn"] } ] ] } }, "Action": ["s3:*"],

15


"Resource": [{ "Fn::Join": [ "", [ "arn:aws-cn:s3:::", { "Ref": "DestinationBucketName" } ] ] }, { "Fn::Join": [ "", [ "arn:aws-cn:s3:::", { "Ref": "DestinationBucketName" }, "/*" ] ] } ] }, { "Effect": "Allow", "Principal": { "AWS": { "Fn::Sub": "arn:aws-cn:iam::${AWS::AccountId}:user/${DestinationSecurityUserName}" }

}, "Action": ["s3:List*", "s3:Get*"], "Resource": [{

16


"Fn::Sub": "arn:aws-cn:s3:::${DestinationBucketName}" }, { "Fn::Sub": "arn:aws-cn:s3:::${DestinationBucketName}/*" } ] } ] } } }, }, "Outputs": { "DestinationS3CpRole": { "Value": { "Fn::GetAtt": ["DestinationS3CpRole", "Arn"] }, "Export": { "Name": { "Fn::Sub": "${AWS::StackName}DestinationS3CpRole" } } } }}

4. file 3source.json

{ "AWSTemplateFormatVersion": "2010-09-09", "Description": "Third step to setup cross account S3 copy", "Parameters": { "Step1StackName": {

17


"Description": "The first step name which is used to create SourceS3CpRole", "Type": "String" }, "SourceBucketName": { "Description": "Source S3 Bucket Name", "Type": "String" }, "DestinationBucketName": { "Description": "Taeget S3 Bucket Name", "Type": "String" }, "DestinationAccountId": { "Description": "Taeget S3 Account ID", "Type": "String" }, "DestinationS3CpRole": { "Description": "The ARN of DestinationS3CpRole", "Type": "String" } },

"Resources": { "S3CpTopic": { "Type": "AWS::SNS::Topic", "Properties": { "Subscription": [{ "Endpoint": { "Fn::GetAtt": ["S32S3Function", "Arn"] }, "Protocol": "lambda" }] } }, "SourceBucketPolicy": {

18


"Type": "AWS::S3::BucketPolicy", "Properties": { "Bucket": { "Ref": "SourceBucketName" }, "PolicyDocument": { "Statement": [{ "Effect": "Allow", "Principal": { "AWS": { "Ref": "DestinationS3CpRole" } }, "Action": ["s3:List*", "s3:Get*", "s3:PutBucketNotification", "s3:PutBucketPolicy", "s3:DeleteBucketPolicy"], "Resource": [{ "Fn::Sub": "arn:aws-cn:s3:::${SourceBucketName}" }, { "Fn::Sub": "arn:aws-cn:s3:::${SourceBucketName}/*" } ] },

{ "Effect": "Allow", "Principal": { "AWS": { "Fn::Sub": "arn:aws-cn:iam::${AWS::AccountId}:role/${LambdaExecutionRole}" } },

19


"Action": ["s3:List*", "s3:Get*", "s3:PutBucketNotification", "s3:PutBucketPolicy", "s3:DeleteBucketPolicy"], "Resource": [{ "Fn::Sub": "arn:aws-cn:s3:::${SourceBucketName}" }, { "Fn::Sub": "arn:aws-cn:s3:::${SourceBucketName}/*" } ] }, { "Effect": "Allow", "Principal": { "AWS": { "Fn::ImportValue": { "Fn::Sub": "${Step1StackName}SourceS3CpRoleARN" } } }, "Action": ["s3:List*", "s3:Get*", "s3:PutBucketNotification", "s3:PutBucketPolicy", "s3:DeleteBucketPolicy"], "Resource": [{ "Fn::Sub": "arn:aws-cn:s3:::${SourceBucketName}" }, { "Fn::Sub": "arn:aws-cn:s3:::${SourceBucketName}/*" } ]

20


}, { "Effect": "Allow", "Principal": { "Service": "lambda.amazonaws.com" }, "Action": ["s3:List*", "s3:Get*", "s3:PutBucketNotification", "s3:PutBucketPolicy", "s3:DeleteBucketPolicy"], "Resource": [{ "Fn::Sub": "arn:aws-cn:s3:::${SourceBucketName}" }, { "Fn::Sub": "arn:aws-cn:s3:::${SourceBucketName}/*" } ] }, { "Sid": "AWSCloudTrailAclCheck", "Effect": "Allow", "Principal": { "Service": "cloudtrail.amazonaws.com" }, "Action": "s3:GetBucketAcl", "Resource": [{ "Fn::Sub": "arn:aws-cn:s3:::${SourceBucketName}" }] }, { "Sid": "AWSCloudTrailWrite", "Effect": "Allow", "Principal": { "Service": "cloudtrail.amazonaws.com"

21


}, "Action": "s3:PutObject", "Resource": [{ "Fn::Sub": "arn:aws-cn:s3:::${SourceBucketName}/*" }], "Condition": { "StringEquals": { "s3:x-amz-acl": "bucket-owner-full-control" } } }, { "Sid": "AWSConfigBucketPermissionsCheck", "Effect": "Allow", "Principal": { "Service": [ "config.amazonaws.com" ] }, "Action": "s3:GetBucketAcl", "Resource": [{ "Fn::Sub": "arn:aws-cn:s3:::${SourceBucketName}" }] }, { "Sid": " AWSConfigBucketDelivery", "Effect": "Allow", "Principal": { "Service": [ "config.amazonaws.com" ] }, "Action": "s3:PutObject", "Resource": [{

22


"Fn::Sub": "arn:aws-cn:s3:::${SourceBucketName}/*" }], "Condition": { "StringEquals": { "s3:x-amz-acl": "bucket-owner-full-control" } } } ] } } }, "SourceS3CpPolicy": { "Type": "AWS::IAM::Policy", "Properties": { "Roles": [{ "Fn::ImportValue": { "Fn::Sub": "${Step1StackName}SourceS3CpRole" } }], "PolicyName": "SourceS3CpPolicy", "PolicyDocument": { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Action": [ "logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents" ], "Resource": "arn:aws-cn:logs:*:*:*" }, { "Effect": "Allow",

23


"Action": [ "sts:AssumeRole" ], "Resource": "arn:aws-cn:iam::*:role/*" } ] } } }, "S32S3Function": { "Type": "AWS::Lambda::Function", "Properties": { "Handler": "index.handler",

"Role": { "Fn::ImportValue": { "Fn::Sub": "${Step1StackName}SourceS3CpRoleARN" } }, "Description": "copy source bucket to target bucket.", "Code": { "ZipFile": { "Fn::Join": [ "", [ "import urllib\n", "import boto3\n", "import ast\n", "import json\n",

"print('Loading function')\n", "def handler(event, context):\n", " DestinationAccountId = ", "\"", { "Ref": "DestinationAccountId"

24


}, "\"", "\n", " DestinationS3CpRole = ", "\"", { "Ref": "DestinationS3CpRole" }, "\"", "\n", " target_bucket = ", "\"", { "Ref": "DestinationBucketName" }, "\"", "\n", " s3 = boto3.client('s3')\n", " sns_message = ast.literal_eval(event['Records'][0]['Sns']['Message'])\n", " source_bucket = str(sns_message['Records'][0]['s3']['bucket']['name'])\n", " key = str(urllib.unquote_plus(sns_message['Records'][0]['s3']['object']['key']).decode('utf8'))\n", " copy_source = {'Bucket': source_bucket, 'Key': key}\n", " print 'Copying %s from bucket %s to bucket %s ...' % (key, source_bucket, target_bucket)\n", " sts_client = boto3.client('sts')\n", " assumedRoleObject = sts_client.assume_role(RoleArn=DestinationS3CpRole,RoleSessionName='AssumeRoleSession1')\n", " credentials = assumedRoleObject['Credentials']\n",

25


" s3 = boto3.client('s3',aws_access_key_id=credentials['AccessKeyId'],aws_secret_access_key=credentials['SecretAccessKey'],aws_session_token=credentials['SessionToken'],)\n", " s3.copy(copy_source, target_bucket, key)\n" ] ] } }, "Runtime": "python2.7", "Timeout": "300" } }, "LambdaInvokePermission": { "Type": "AWS::Lambda::Permission", "Properties": { "Action": "lambda:InvokeFunction", "Principal": "sns.amazonaws.com", "SourceArn": { "Ref": "S3CpTopic" }, "FunctionName": { "Fn::GetAtt": ["S32S3Function", "Arn"] } } }, "S3CpTopicPolicy": { "Type": "AWS::SNS::TopicPolicy", "Properties": { "PolicyDocument": { "Id": "S3CpTopicPolicy", "Version": "2012-10-17", "Statement": [{ "Sid": "S3CpTopicPolicy", "Effect": "Allow",

26


"Principal": { "AWS": "*" }, "Action": "sns:Publish", "Resource": { "Ref": "S3CpTopic" }, "Condition": { "ArnLike": { "AWS:SourceArn": { "Fn::Join": [ "", [ "arn:aws-cn:s3:::", { "Ref": "SourceBucketName" } ] ] } } } }] }, "Topics": [{ "Ref": "S3CpTopic" }] } }, "BucketConfiguration": { "Type": "Custom::S3BucketConfiguration", "DependsOn": [ "SourceBucketPolicy" ], "Properties": { "ServiceToken": {

27


"Fn::GetAtt": [ "S3BucketConfiguration", "Arn" ] }, "Bucket": { "Ref": "SourceBucketName" }, "NotificationConfiguration": { "TopicConfigurations": [{ "Events": ["s3:ObjectCreated:*"], "TopicArn": { "Ref": "S3CpTopic" } }] } } },

"S3BucketConfiguration": { "Type": "AWS::Lambda::Function", "Properties": { "Description": "S3 Object Custom Resource", "Handler": "index.handler", "Role": { "Fn::GetAtt": ["LambdaExecutionRole", "Arn"] }, "Timeout": "30", "Runtime": "nodejs4.3", "Code": { "ZipFile": { "Fn::Join": [ "\n", [ "var response = require('cfn-response');", "var AWS = require('aws-sdk');",

28


"var s3 = new AWS.S3();", "exports.handler = function(event, context) {", "var respond = (e) => response.send(event, context, e ? response.FAILED : response.SUCCESS, e ? e : {});", "process.on('uncaughtException', e=>failed(e));", "var params = event.ResourceProperties;", "delete params.ServiceToken;", "if (event.RequestType === 'Delete') {", "params.NotificationConfiguration = {};", "s3.putBucketNotificationConfiguration(params).promise()", ".then((data)=>respond())", ".catch((e)=>respond());", "} else {", "s3.putBucketNotificationConfiguration(params).promise()", ".then((data)=>respond())", ".catch((e)=>respond(e));}};" ] ] } } } }, "LambdaExecutionRole": { "Type": "AWS::IAM::Role", "Properties": { "AssumeRolePolicyDocument": { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": { "Service": [ "lambda.amazonaws.com" ]

29


}, "Action": [ "sts:AssumeRole" ] }] }, "Path": "/", "ManagedPolicyArns": [ "arn:aws-cn:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole" ], "Policies": [{ "PolicyName": "S3Policy", "PolicyDocument": { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Action": [ "S3:*" ], "Resource": "arn:aws-cn:s3:::*" }] } }] } } }, "Outputs": { "S3CpTopic": { "Value": { "Fn::GetAtt": ["S3CpTopic", "TopicName"] }, "Export": { "Name": {

30


"Fn::Sub": "${AWS::StackName}S3CpTopic" } } } }}

31

s3.cn-north-1.amazonaws.com.cn€¦ · web viewwe can use aws lambda to replicate s3 bucket...

Documents