AWS S3 Sync Command – A Beginners Guide To File Synchronization 

In today’s fast paced cloud world, managing and syncing the files between the local environment and the cloud is a common instance. When you are backing up a system, the AWS S3 Sync command is a powerful tool that can help with the process. 

This guide will help you familiarize yourself with how the AWS S3 sync command works through the basic syntax, real-world examples, and debugging options. 

What is the AWS S3 Sync Command?

The AWS S3 sync command is a powerful tool of the AWS Command Line Interface (CLI) that is used to synchronize the files between a source and a destination. This source and destination can be: 

• A local directory and an S3 bucket
• Two S3 buckets
• An S3 bucket and a local directory

Unlike other copy commands, the sync command compares the content of the source and the transfer location and then only syncs the new files or the modified one, which makes it quite efficient for larger groups. 

Key benefits of using AWS S3 sync:

• Saves the destination bandwidth and the time by skipping on the unchanged files. 
• Automatically syncs files recursively through subdirectories. 
• Supports file filtering with the include/ exclude patterns. 
• Allows secure testing using the –dryrun mode
• Can mirror deletions using the –delete flag

Basic Syntax of AWS CLI S3 Sync

aws s3 sync <source> <destination> [options]

• <source> can be a local path or an S3 bucket URL (e.g., s3://my-bucket)
• <destination> can also be local or S3

Syntax Example 1: Sync Local Folder to S3 Bucket

aws s3 sync ./my-folder s3://my-bucket-name

Syntax Example 2: Sync S3 Bucket to Local Directory

aws s3 sync s3://my-bucket-name ./local-folder

Commonly Used Options:

• –delete: Removes the files from the destination that do not exist in the source anymore
• –dryrun: Stimulates the sync operation without making any external changes 
• –exclude / –include: Filters files to sync
• –exact-timestamps: Compares both file size and modified time

Related Article: AWS Monitoring Tools: A Complete Guide to Observability on the Cloud

Common AWS S3 Sync Command Examples

The aws s3 sync command supports multiple different practical use cases. Here are some common scenarios that you might encounter. 

1. Sync a Local Directory to an S3 Bucket

When you need to upload all the files and subdirectories from the local folder to an S3 bucket. 

aws s3 sync ./project-files s3://my-startup-bucket

2. Sync an S3 Bucket to a Local Directory

Download the files from the s3 bucket to the local directory.

aws s3 sync s3://my-startup-bucket ./backup-folder

3. Sync Between Two S3 Buckets

Move or copy the files between different buckets for migration or redundancy. 

aws s3 sync s3://source-bucket-name s3://target-bucket-name

4. Use –delete to Remove Extra Files

Ensure that the destination mirrors the source exactly by deleting all the unmatched files. 

aws s3 sync ./project-files s3://my-startup-bucket –delete

5. Test First with –dryrun

Preview the sync results without actually deleting or uploading the actual changes. 

aws s3 sync ./project-files s3://my-startup-bucket –dryrun

Recursive Syncing with AWS S3 Sync

One of the most useful features of the AWS S3 sync command is that it syncs recursively by default. This means that you do not need to specify the flag to include subdirectories, they are automatically processed in the command. 

What It Means:

• All the subdirectories are scanned and synced. 
• Only the new or modified files are synced. 
• Saves the time and space bandwidth. 

Example:

aws s3 sync ./website-files s3://my-bucket-name

This command will upload all the new and modified files from the website-files directory, including all the subfolders to the S3 bucket. 

Using Filters with Include and Exclude Options

Sometimes, you only need to sync a certain type of files or exclude the unnecessary ones. These –include and –exclude flags let you define exactly what gets transferred.

Syntax Rule:

All the files are excluded by default if you use the –exclude “*” — then selectively included using –include.

Example 1: Sync Only .jpg Files

aws s3 sync . s3://my-bucket –exclude “*” –include “*.jpg”

Example 2: Exclude Temporary Files

aws s3 sync . s3://my-bucket –exclude “*.tmp”

Example 3: Include Specific Subfolderaws s3 sync ./ s3://my-bucket –exclude “*” –include “images/*”

Tips:

• –exclude is evaluated before –include
• Use wildcards like * for flexible matching
• You can combine multiple –include/–exclude rules

Dry Run and Debugging Options For AWS S3 Sync Command

Before you execute the full sync, it is best to dry run, so that you can view the preview before the operation. The AWS CLI provides options like –dryrun and –debug to help verify, monitor, and troubleshoot sync behavior.

• Using –dryrun

The –dryrun flag enables you to simulate the sync operation without making any changes, which means that you preview the changes before making it. It will list the files that would be changed after execution. 

Example:

aws s3 sync ./local-dir s3://my-bucket –dryrun

This is a safe manner to verify how the command will behave before making the changes. 

• Debug Mode for Troubleshooting

If something goes wrong, then you can always use the –debug flag to get detailed logs. This includes the information on the HTTP requests, authentication, and file comparisons. 

Example:

aws s3 sync ./local-dir s3://my-bucket –debug

Other helpful options:

• –profile <profile-name> – Use a specific AWS CLI profile
• –region <region-name> – Specify the target region
• –only-show-errors – Suppress normal file output and display only errors

Handling Deleted Files and Versioning

By default, the aws s3 sync command does not delete the old files from the destination if they no longer exist in the source. To ensure that you carry out a full synchronisation, you can use the delete flag. 

• Syncing with –delete

This command makes sure that the destination looks exactly like the source after synchronisation. 

Example:

aws s3 sync ./website s3://my-bucket –delete

• Working with S3 Versioning

If you have enabled versioning on the destination in the S3 bucket, then the deleted files will not be removed permanently. Instead, the previous versions will remain accessible with a  marker.

This is particularly useful for:

• Protecting against accidental deletions
• Restoring previous file versions if needed

To enable versioning, use:

aws s3api put-bucket-versioning –bucket my-bucket –versioning-configuration Status=Enabled

Performance and Cost Optimization Tips

Using s3 sync can help improve the performance and reduce the extra costs, especially when you are dealing with larger datasets or frequent sync jobs. 

• Avoid Unnecessary File Transfers

The sync operation will compare the file size and modified timestamps by default. For more control, you can use –exact-timestamps to exclude the overwritten files with the same size and different timestamps. 

Example:

aws s3 sync ./dir s3://bucket –exact-timestamps

• Exclude Unnecessary Files

Use –exclude and –include flags to stop syncing the files that are not required, such as logs: 

Example:

aws s3 sync . s3://bucket –exclude “*.log” –exclude “node_modules/*”

• Use File Compression

If you are syncing multiple small files, then you can consider compressing them into a .zip or .tar.gz archive to reduce transfer time and overhead.

• Schedule Sync During Off-Peak Hours

Automate syncs using cron jobs or AWS Lambda functions during periods of low network activity to reduce impact on performance and cost.

Conclusion 

The aws s3 sync command is a super useful and versatile tool for managing the file synchronisation between local environments and Amazon s3. By mastering the command, you can easily sync and transfer the files practically. 

FAQs