Transfering files from AWS S3 bucket to another AWS S3 bucket from another account.
Scenario requirements:
- Environment A: S3 bucket
- Environment B: S3 bucket and EC2 : The EC2 is not necessary, aws cli can be installed locally.
- AWS IAM user creation on both accounts with permissions: PowerUserAccess and AmazonS3FullAccess
Create environment A S3 bucket
Update S3 bucket permissions > Bucket policy
{
"Version": "2012-10-17",
"Id": "<autogenerated>",
"Statement": [
{
"Sid": "<id>",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::<iam-user-id-B>:user/<username-B>"
},
"Action": "s3:*",
"Resource": [
"arn:aws:s3:::<s3-bucket-name-A>",
"arn:aws:s3:::<s3-bucket-name-A>/*"
]
}
]
}
Scenario B
- Create a role so that EC2 can access S3 bucket with the policy
AmazonS3FullAccess
- Create an S3 bucket
- Creating EC2
Assign IAM role
Assign to the same subnet as S3 bucket.
change EC2 security to open ssh(port 22) and https(port 443) to your IP
Security group rule ID Port Protocol Source groups
sgr-<id> 22 TCP <IP> default
sgr-<id> 443 TCP <IP> default
Connect to the EC2 via ssh
ssh -v -i "./<cert-name>.cer" ubuntu@<server-public-ip>
Inside the EC2 install AWS cli
apt update
apt install unzip
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install
aws configure
Insert your AWS account B: Access Key ID and Secret access key
To test the connection to the S3 bucket from Account A and/or account be use amazon cli
aws s3 ls s3://<S3 bucket B name>
aws s3 ls s3://<s3 bucket A name>
Once you have the right to acces both buckets then you can start copying over files using two methods:
AWS cli
aws s3 cp s3://<Bucket A name> /<filename+extension> s3://<bucket B name/<filename+extension>
Python (BOTO3)
pip install boto3
Create a file
import boto3
#Creating Session With Boto3.
session = boto3.Session(
aws_access_key_id='account B',
aws_secret_access_key='account B'
)
s3 = boto3.resource('s3')
copy_source = {
'Bucket': '<bucket A name> ',
'Key': '<filename + extension>'
}
s3.meta.client.copy(copy_source, '<bucket B name>', '<filename + extension>')
then you can execute the file
python3 <filename>.py