Amazon S3

What is Amazon S3 #

Amazon S3 is one of the main building blocks of AWS.

It is advertised as “infinitely scaling” storage.

• Many websites use Amazon S3 as a backbone
• Many AWS services use Amazon S3 as an integration

Amazon S3 Use Cases #

• Backup and Storage
• Disaster Recovery
• Archive
• Hybrid Cloud storage
• Application hosting
• Media hosting
• Data lakes & big data analytics
• Software delivery
• Static Website

Amazon S3 - Buckets #

• Amazon S3 allows people to store objects (files) in “buckets” (folders)
• Buckets must have a globally unique name (across all regions and all accounts)
• Buckets are defined at the region level
• S3 looks like a global service but buckets are created in a region

S3 Bucket naming convention #

• No uppercase
• No underscore
• 3-63 characters long
• Not an IP
• Must start with lowercase letter or number
• Must NOT start with the prefix “xn–”
• Must NOT start with the suffix “s3alias”

Amazon S3 - Objects #

• Objects (files) have a Key

• The key is the FULL path:

  • s3://my-bucket/my_file.txt
  • s3://my-bucket/my_folder/another_folder/my_file.txt

• The key is composed of prefix + object name

  • s3://my-bucket/my_folder/another_folder/my_file.txt

• There is no concept of “directories” within S3 buckets (although UI will suggest there is)

  • Just keys with very long names that contain slashes ("/")

• Object values are the content of the body

  • Max Object size is 5TB
  • If uploading more than 5GB, must be “multi-part upload”
  • Metadata (list of text key / value pairs - system or user metadata)
  • Tags (Unicode key / value pair - up to 10) - used for security / lifecycle

Amazon S3 - Security #

• User-based
  • IAM Policies - which API calls should be allowed for a specific user from IAM
• Resource-based
  • Bucket policies - bucket-wide rules from the S3 console - allows cross account
  • Object Access Control List (ACL) - finer grain (can be disabled)
  • Bucket Access Control List (ACL) - less common (can be disabled)
• IAM Principal can access an S3 object if
  • The user IAM permissions ALLOW it OR the resource policy ALLOWS it AND there is no explicit DENY
• Encryption - encrypt objects in Amazon S3 using encryption keys

S3 Bucket Policies #

• JSON based policies

  • Resources: buckets and objects
  • Effect: Allow / Deny
  • Actions: Set of API to Allow or Deny
  • Principal: The account or user to apply the policy to

• Use S3 bucket policy to:

  • Grant public access to the bucket
  • Force objects to be encrypted at upload
  • Grant access to another account (Cross Account)

Examples #

• Public access - Bucket Policy
• User access to S3 - IAM permissions
• EC2 instance access - IAM Roles
• Cross-Account access - Bucket Policy

Bucket settings for Block Public Access

• Those settings were created to prevent company data leaks
• If you know your bucket should never be public, leave them

Applying Bucket Policy #

S3 > General purpose buckets (or other) > your-bucket > Permissions > Ensure Allow public access is enabled

Edit Bucket policy (follow Policy examples OR Policy Generator)

Policy Generator #

• “arn” can be found in the bucket settings itself
• “/*” can / should be added to propagate to all objects in the bucket

Add Statement and copy the generated policy to Bucket policy

Object(s) in that bucket should now be available from the internet via URL

S3 - Static Website Hosting #

Bucket must be made public (S3 Bucket policy, see above) in order for the static website to work. If it isn’t then 403 Forbidden error appears.

Enable static website hosting #

S3 > General purpose buckets (or other) > your-bucket > Properties > Static website hosting

Voila! #

Although no HTTPS enabled! - check “AWS Certificate Manager (ACM)” section in Security and Compliance on how to use ACM for SSL / TLS certificate management.

Amazon S3 - Versioning #

• Versioning can be enabled at the bucket level
• Versioning buckets:
  • Protect against unintended deletes (ability to restore the version)
  • Easy roll back to previous version
• Notes:
  • Any file that is not versioned prior to enabling versioning will have version “null”
  • Suspending versioning does not delete the previous versions

Enable S3 Versioning #

S3 > General purpose buckets (or other) > your-bucket > Properties > Bucket Versioning > Edit > Enable

In order to restore the file, toggle “Show Versions” switch and DELETE unwanted object (destructive, will permanently delete).

If “show versions” toggle is off, object can be safely deleted. It will NOT be permanently deleted, only “Delete marker” will be applied. Object can be easily restored when “Show versions” is ON. #

S3 - Replication #

• CRR - Cross Region Replication
• SRR - Same Region Replication

For S3 replication to work, Versioning must be enabled on both - source and destination buckets. Buckets can be different AWS Accounts.

Copying is asynchronous, proper IAM permissions must be applied to S3.

Use Cases: #

• CRR - compliance, lower latency access, replication across accounts
• SRR - log aggregation, live replication between production and test accounts

Enabling S3 Replication #

1. Create 2 new buckets and enable Versioning in both

   • rk-test-replica-london-origin
   • rk-test-replica-irl-dest

2. On the origin bucket create the Replication rules

   • Enable replication
   • Select source and destination buckets
   • Create IAM role or select an existing one (Create)
   • Select any other options (encryption, destination storage class, delete marker replication and so on)

S3 Storage Classes #

Standard #

• Amazon S3 Standard - General Purpose
  • 99.99% Availability
  • Used for frequently accessed data
  • Low latency and high throughput
  • Sustain 2 concurrent facility failures
  • Use cases: Big Data analytics, mobile & gaming applications, content distribution

Infrequent access #

For data that is less frequently accessed but requires rapid access when needed. Lower cost than S3 Standard.

• Amazon S3 Standard-Infrequent Access (IA)
  • 99.9% Availability
  • Use cases: Disaster Recovery, backups
• Amazon S3 One Zone-Infrequent Access
  • For data that is less frequently accessed but requires rapid access when needed
  • High Durability (99.999999999% Availability in a single AZ)
  • 99.5% Availability
  • Use cases: Secondary backup copies of on-prem data, data can be recreated

Glacier #

Low-cost object storage for archiving / backup.

Pricing includes storage price + retrieval cost.

• Amazon S3 Glacier Instant Retrieval
  • Milliseconds retrieval, great for data accessed once a quarter
  • Minimum storage duration of 90 days
• Amazon S3 Glacier Flexible Retrieval
  • Expedited (1 to 5 mins), Standard (3 to 5 hours), Bulk (5 to 12 hours) - free
• Amazon S3 Glacier Deep Archive
  • For long-term storage
  • Standard (12 hours), Bulk (48 hours)
  • Minimum storage duration of 180 days

Intelligent tiering #

Moves objects automatically between Storage Tiers based on usage for a small monthly monitoring and auto-tiering fee.

There is no retrieval charges in S3 Intelligent Tiering.

• Frequent Access tier (automatic): default tier
• Infrequent Access tier (automatic): objects not accessed for 30 days
• Archive instant Access tier (automatic): objects not accessed for 90 days
• Archive Access tier (optional): configurable from 90 to 700+ days
• Deep Archive Access tier (optional): configurable from 180 to 700+ days

Objects can be moved between classes manually or using S3 Lifecycle policies.

More:

• S3 Storage classes: https://aws.amazon.com/s3/storage-classes/
• S3 Pricing: https://aws.amazon.com/s3/pricing/

S3 Durability and Availability #

• Durability
  • High durability (99.999999999, 11 9’s) of objects across multiple AZ
  • If you store 10,000,000 objects with Amazon S3, you can on average expect to incur a loss of a single object once every 10,000 years
  • Same for all storage classes
• Availability
  • Measures how readily available a service is
  • Varies depending on storage class
  • Example: S3 standard has 99.99% availability = not available for 53 minutes a year

More:

• https://docs.aws.amazon.com/AmazonS3/latest/userguide/DataDurability.html
• https://aws.amazon.com/s3/storage-classes/

S3 Encryption #

• Server-Side Encryption (Default)
• Client-Side Encryption (Encrypted by user at the client side)

IAM Access Analyzer for S3 #

• Ensures that only intended people have access to your S3 buckets
• Example: publicly accessible bucket, bucket shared with other AWS account
• Evaluates S3 Bucket Policies, S3 ACLs, S3 Access Point Policies
• Powered by IAM Access Analyzer

AWS Snowball #

• Highly-secure, portable devices to collect and process data at the edge and / or migrate data in and out of AWS
• Helps to migrate up to Petabytes of data

It is recommended to use AWS Snowball devices if it would take more than a week to transfer over the network. #

More:

• https://aws.amazon.com/snowball/
• https://aws.amazon.com/snowball/pricing/

AWS Storage Gateway #

Amazon S3 File Gateway connects on-premises applications to the cloud to store and access archive repositories, application data, database backups and so on.

S3 File Gateway is used for on-premises data intensive applications that need file protocol access to objects in S3.

More: https://aws.amazon.com/storagegateway/file/s3/

Types of Storage Gateway: #

• File Gateway
• Volume Gateway
• Tape Gateway

» Sources « #

• S3 Documentation: https://docs.aws.amazon.com/s3/
• S3 Availability and Durability: https://docs.aws.amazon.com/AmazonS3/latest/userguide/DataDurability.html
• S3 Storage Classes: https://aws.amazon.com/s3/storage-classes/
• S3 Pricing: https://aws.amazon.com/s3/pricing/
• S3 AWS Snowball: https://aws.amazon.com/snowball/
• S3 File Gateway: https://aws.amazon.com/storagegateway/file/s3/

Full YouTube Rahul’s AWS Course: https://www.youtube.com/playlist?list=PL7iMyoQPMtAN4xl6oWzafqJebfay7K8KP

» References « #

• Storage
• Security and Compliance

» Table of contents (CLF-C02) « #

» Disclaimer « #