Block and File Storage
- Block storage operates at a lower level - the raw storage device level and manages data as a set of numbered, fixed size blocks. Block storage is accessed directly or via a SAN using protocols such as iSCSI and Fiberchannel. Directly attached or network attached, block storage is very closely associated to the server and operating system that is using the storage.
- File storage operates at upper level - at the OS level which manages data as a named hierarchy of files and folders. This type is storage is accessed directly or over a NAS using protocols such as CIFS and NFS. This type of storage is also closely associated with the server and OS that is using the storage.
- Amazon S3 storage is different from file or block storage as it is a cloud object storage. It is not associated to the server or OS that is using it. It functions as an independent storage.Instead of managing data as files or blocks using iSCSI, Fiber channel or NFS, it manages objects using API built on standard http verbs.
Amazon S3 is -
- An object based strorage (to store pics, files, documents - OS cannot be installed on this type of storage).
- Size of files that can be uploaded - 0 bytes to 5TB
- Storage capacity of an S3 bucket is - Unlimited
- It is a reliable storage as data is spread replicated across multiple devices and facilities
- Files are stored in "Buckets" which are like folders on the cloud (web folders). Buckets are flat and have to file hierarchy. Buckets cannot be nested (i.e bucket within a bucket is not allowed).
- Nomenclature for an S3 bucket - https://"bucketname".s3.amazon.aws.com/"name of object". It is best practice to use bucket names that contain your domain name and conform to rules of DNS names. This ensures that your bucket names are your own, can be used in all regions and can host static websites.
- Even though the namespace for an Amazon S3 bucket is global, each Amazon S3 bucket is created in a specific region that you choose. You can create buckets in a region closer to your users to reduce latency, to fulfill data localization and compliance issues or you can create them away from your primary DC facilities for DR and business continuity needs. Data in an Amazon S3 is stored within the region unless you explicitly do cross region replication.
- HTTP code 200 is received after successful upload of an object to S3
- Built for 99.99 availability
- Amazon guarantees 99.9999999% (11 nines) of durability.
- Provides tiered storage
- Provides lifecycle management, versioning, encryption
- It is a simple key value store which is object based. An object has the following attributes -
  1. Name
  2. Value
  3. Version ID
  4. Metadata - Is of two types -
    1. System metadata - created and used by AWS. will include date last modified, object size, md5 digest, http content type (see pic below)
    2. User metadata - is optional and can be specified by the user only at the time when the object is created. e.g - tags. It must begin with "x-amz-meta-"
  5. Sub-resources namely
    - ACLs
    - Torrents
Nearly every AWS service uses S3 directly or indirectly. For example, Amazon S3 serves as a durable target storage for Amazon Kinesis and Amazon elastic mapReduce (Amazon EMR), it can be used as storage for Elastic block storage (EBS) and Amazon RDS snapshots or can be used as data staging and loading mechanism for Amazon RedShift and Amazon DynamoDB. Some common use cases can be -
- Back up and archiving on premise and cloud data
- Content, media and software storage and distribution
- Big data analytics
- Static website hosting
- Cloud native mobile and internet application hosting
- Disaster recovery
Data can be secured in an S3 using/ Ways to control access to S3 buckets -
- ACLs (to secure/restrict access to individual objects or complete buckets) - not used very widely these days
- Bucket policies (to secure/restrict access to entire buckets) - recommended. Bucket policies include an explicit reference to the IAM principal in the policy. This principal may be associated with a different AWS account so Amazon S3 bucket policies allow you to assign cross-account access to Amazon S3 resources. Using a bucket policy, you can specify -
  - Who can access the bucket
  - from which subnet (CIDR block) or IP address
  - during what time of the day
Difference between a normal folder and an S3 bucket is that an S3 bucket has a unique namespace that can be accessed by a URL and resolved by a DNS (so the namespace should be globally unique i.e bucket names are global)
Objects inside a bucket DO NOT inherit the tags for the bucket.
Data consistency model for S3
- Read after write consistency for PUTS of new objects
- Eventual consistency for overwrite PUTS and DELETES (i.e can take some time to propogate). In all cases updates to a single key are atomic, i.e you will either get new data or old data, but never an inconsistent mix of data.
Amazon S3 API supports the following operations -
- Create/Delete a bucket
- Write an object
- Read an object
- delete an object
- List keys in a bucket
Native interface for Amazon S3 is the REST (Representational State Transfer) API that supports the operations mentioned above.
Prefixes and Delimiters - While amazon S3 uses a flat structure in a bucket, it supports the use of prefix and delimiter parameters while listing key names. This feature helps you organize, browse and retrieve objects within a bucket hierarchically. / and \ are used as delimiters. eg. logs/2016/January/Server1.log.

Used with IAM or S3 bucket policies, prefixes and delimiters also allow you to create equivalent of department subdirectories or user home directories within a single bucket, restricting or sharing access to these subdirectories as needed.

S3 Storage Tiers/Classes
1. Standard Tier
  - Availability SLA - 99.99%
  - Durability - 11 nines
  - Low latency and high throughput
  - Objects are stored redundantly across multiple devices and multiple facilities
  - Designed to sustain loss of two facilities concurrently
  - Used for frequently accessed data (short term or long term)
2. S3 Infrequently accessed (S3-IA)
  - Cheaper than standard tier and used for objects that need to be accessed less frequently
  - Low latency and high throughput
  - Availability SLA - 99.9%
  - Durability - 11 nines
  - Objects are stored redundantly across multiple devices and multiple facilities
  - Users are charged for retrieval fee (on a per GB basis) over and above the storage fee. There is also a minimum object size of 128 KB and minimum duration of 30 days, so it is best suited for infrequently used data that is stored for longer than 30 days.
3. S3 Infrequently accessed One Zone (also called Reduced Redundancy Storage or RRS)
  - Cheaper than S3 IA, but least redundant
  - Used when you do not require resiliency across multiple zones. Data is stored in only one AZ
  - Used for objects that do not need to be accessed frequently or data that can be easily reproduced/replicated such as image thumbnails.
  - Availability SLA - 99.5%
  - Durability- 11 nines
  - Users are charged for retrieval fee (on a per GB basis) over and above the storage fee
  - 20% cheaper than S3 IA
4. Amazon Glacier
  - Used for archival and long term backup. Can be used as a storage class of S3 (eg. when used for lifecycle management) or as an independent archiving service.
  - Cheaper than above
  - Durability - 11 nines. Data is stored on multiple devices on multiple AZs.
  - To retrieve an Amazon Glacier object, you issue a restore command using one of the S3 APIs and 3-5 hours later, the Amazon Glacier object is copied to RRS . The retore just creates a copy and the original object still stays on Glacier till explicitly deleted.
  - Works in 3 models -
    - Expedited - When you need data retrieval in minutes - Most expensive flavor of Glacier
    - Standard - Less expensive than expedited, but data retrieval takes 3-5 hours
    - Bulk - Least expensive, but data retrieval takes 5-12 hours.
  - Not available in all regions
  - Glacier allows upto 5% data to be retrieved from it for free every month. Anything over that that is retrieved from Glacier is charged on a per GB basis per month.
  - Use cases - replacement of tape based backup solutions and for storage/archiving of data for audit and compliance.
  - Data stored in Glacier is in the form of TAR (Tape archive) and ZIP files.
  - Amazon Glacier terms -
    - Archives - Data is stored in Glacier in the form of archives. Each archive can be upto 40 GB and glacier can have multiple archives. Archives are automatically encrypted and are immutable - i.e cannot be modified once created.
    - Vaults - Containers for archives. Each vault can contain upto 1000 archives. Access to vaults can be controlled using IAM policies or vault access policies.
    - Vault locks - Vault lock policies help in deploying and enforcing compliance controls for Amazon Glacier locks.
If you need additional file or block storage in addition to Amazon S3, use
- Elastic Block storage (EBS) for EC2 instances
- Elastic File Storage (EFS) for network attached shared file storage using NFS protocol.
S3 charges are based on -
- Storage (per Gig basis)
- No. of requests to access data on the storage
- Storage management pricing (charges for maintaining tags, metadata)
- Data transfer pricing (cross region replication of data)
- Transfer acceleration
S3 Versioning -
- Used to keep versions of objects in the S3 bucket. Protects data from accidental or malicious deletion.
- Once turned on, it cannot be turned off - can only be suspended. Versioning is turned on at each bucket level.
- Stores all versions of the object (including all writes, even if you delete the object). Each version has a unique ID.
- Integrates with Lifecycle management rules
- Multi factor authentication can be added to versioning to provide an additional level of security. With this, users need additional level of authentication (temporary OTP on a hardware or virtual MFA device) to delete a version of an object or change versioning state of a bucket. Note that MFA delete can only be enabled by the root account
- Amazon S3 preserves existing objects anytime you perform a PUT, POST, COPY, or DELETE operation on them
Pre signed URLs - All Amazon S3 buckets are private by default i.e only the creator can access their objects. The owner can share the objects with others using a pre signed URL using their own security credentials to give time limited permissions to download the objects. Pre signed URLs are valid only for a specific duration and can be used to protect against "content scraping" of web content such as media files stored on S3.
You can upload files much faster to S3 by enabling multipart upload - it fragments the file, uploads fragments in parallel and assembles them together again. It provides ability to pause and resume uploads as well. Helps in improving network utilization. It is a three part process - initiate, upload parts and abort/complete. It should be generally used when the object size is over 100Mb and you must use this feature if the object size is over 5 GB. You can set an object liecycle policy on a bucket to abort incomplete multi part uploads after a specified number of days. This will minimize the storage costs associated with multi part uploads that were not completed.
Range GETs - Used to download (GET) only a portion of the object in Amazon S3 or Amazon glacier. You can specify range of bytes of the object that you need to download.
Cross Region replication (CRR) - Amazon Simple Storage Service (Amazon S3) offers cross-region replication, a bucket-level feature that enables automatic, asynchronous copying of objects across buckets in different AWS Regions. Buckets configured for cross-region replication can be owned by the same AWS account or by different accounts. Cross-region replication is enabled with a bucket-level configuration. You add the replication configuration to your source bucket. Any ACLs and metadata associated with the bucket also gets replicated. In the minimum configuration, you provide the following:
1. The destination bucket, where you want Amazon S3 to replicate objects
2. An AWS IAM role that Amazon S3 can assume to replicate objects on your behalf
3. Versioning must be enabled on source and target buckets to do CRR
  - When you enable CRR, only new objects created after enablement of CRR and replicated.
  - Regions must be unique for CRR to work
  - You cannot replicate to multiple buckets or use daisy chaining
  - Delete markers are replicated. However individual versions with delete markers are not replicated
  - Cross region replication is commonly used to -
    - Reduce latency for users who are far from the original S3 bucket
    - Store data far away from the original source for DR and compliance purposes.
  - With CRR, you can set up replication at a bucket level, a shared prefix level, or an object level using S3 object tags.
Lifecycle management -
- Can be used in conjunction with versioning
- Can be applied to current as well as previous versions
- Lifecycle configurations can be applied to the entire bucket or only to a few objects specified by a prefix.
- Following actions can be done -
  1. Transition from standard to S3-IA (min. 30 days from creation date, size should be min 128Kb)
  2. Archive to glacier from S3-IA (default is 30 days from S3-IA or if not initially moved to IA, can be straightaway moved from S3 standard to Glacier on day 1)
  3. Delete the objects permanently
- Within a lifecycle rule, the prefix field identifies the objects subject to the rule. To apply the rule to an individual object, specify the key name. To apply the rule to a set of objects, specify their common prefix (e.g. “logs/”).
Logging - S3 server access logs can be enabled (disabled by default) and logs can be stored in the same or a different bucket. Logs can include -
1. Requester account and IP address
2. Bucket name
3. Request time
4. Action (GET, PUT, LIST etc.)
5. Response status or error code
Event Notifications - Can be sent in response to actions taken on objects uploaded/stored on S3. These notifications enable you to run workflows, send alerts or perform other actions in response to changes in objects stored in S3. Notifications are set at bucket level and can be applied to objects in the bucket as well. Notification messages can be sent through the amazon Simple Notification Service (SNS) or amazon Simple Queuing Service (SQS) or delivered directly to AWS Lambda to invoke Lambda functions. You can also set up event notifications based on object name prefixes and suffixes. For example, you can choose to receive notifications on object names that start with “images/."
Cloudfront - Amazon CloudFront is a fast content delivery network (CDN) service that securely delivers data, videos, applications, and APIs to customers globally with low latency, high transfer speeds, all within a developer-friendly environment.
- Edge location - Location where content will be cached. These locations are not just read only. You can write/PUT an object too
- Origin - Origin of all files that the CDN will distribute. These can be S3 buckets, EC2 instances, Elastic Load Balancer, Route 53
- Distribution - Name of the CDN which consists of a collection of edge locations. There are two types of distributions -
  1. Web distribution - Used for websites (using http)
  2. RTMP distribution - Real Time Messaging Protocol distribution is used for media streaming using RTMP (eg. Adobe flash)
- To use AWS Cloudfront, you first need to create a distribution, which is identified by a DNS domain name such as gsehgal81.cloudfront.net. To serve files from AWS cloudfront, you simply use the distribution domain name instead of the actual website domain name.
- Cloudfront also works seamlessly with any non-AWS origin server which stores original, definitive versions of your files. eg. an on premise wed server.
- Objects are cached for the duration of the TTL. By default, the TTL timer is 24 hrs, but you can change this timer. Minimum timer is 0 sec for web distribution and 3600 sec for RTMP.
- You can clear the cache before expiring of the TTL, but this service is charged. This can be done by calling the "invalidation" API which will clear the cache from all Cloudfront edge locations regardless of whether the expiration timer/TTL has expired or not.
- You can have multiple origins for the same cloudfront distribution
- You may want to restrict access to content on cloudfront to only selected users such as paid subscribers or to applications/users in your company. To restrict access to your cloudfront you can use three methods -
  1. A signed URL - The URL is valid only for a certain time period and can optionally be accessed from a certain range of IPs
  2. Signed Cookies - Require authentication via public and private key pairs
  3. Origin access identities (OAI) - To restrict access to an S3 bucket only to a special cloudfront user associated with your distribution. This is the easiest way to ensure that content in a bucket is only accessed by an Amazon cloudfront user.
- Use cases of cloudfront-
  - Serving static assets (images, CSS, javascript etc. that make up a bulk of the requests) of a website
  - Serving a whole website or web application
  - Serving content to users who are widely distributed geographically
  - Distributing software or other large files
  - Serving streaming media
- When NOT to use Cloudfront -
  - When most requests come from a single location
  - When most requests come via a corporate VPN
S3 Security and Encryption
- All newly created S3 buckets are private. They can be made public manually. All objects in a bucket are also private by dafault.
- S3 buckets can be configured to create access logs. These logs can be saved on another S3 bucket or another AWS account.
- Different encryption methods -
  1. In transit - via SSL or TLS
  2. At rest - Two types -
    1. Server Side - Three types
      1. S3 Managed keys (SSE-S3) - Encryption is managed by AWS. Each object is encrypted with a unique key, applying strong multi factor encryption and then the key itself is encrypted with a master key that is regularly rotated. Uses AES256 encryption. The encrypted data, encryption key and the master key are all stored on separate hosts to provide additional security.
      2. Key management server (KMS) managed keys (SSE-KMS) - Here AWS handles your key management system and protection for S3 while you manage the keys. Also uses AES256 encryption. It is more expensive than SSE-S3, but has additional benefits
        
        There are separate permissions for using the "envelope key" (master key) which provides additional level of control
        
        Provides an audit trail of who tried to access your S3 and when. Also provides information on failed attempts when unauthorized users try to access your S3/decrypt the data
        
        Also, AWS KMS provides additional security controls to support customer efforts to comply with PCI-DSS, HIPAA/HITECH, and FedRAMP industry requirements.
      3. Encryption with customer provided keys (SSE-C) - Customer manages the encryption keys and AWS manages the encryption/decryption
    2. Client Side - To encrypt data on the client before sending it to AWS. When using client side encryption, customer retains end to end control of encryption process including management of encryption keys. Can be done in two ways -
      1. AWS KMS managed customer master key
      2. Client side master key
S3 Transfer acceleration - Utilizes the Cloudfront edge network to accelerate uploads to S3. Enables fast, secure and easy transfers of files over long distances between end users and S3 buckets. With transfer acceleration enabled on AWS, you get a unique URL where you can upload objects at the edge location which are later transferred over the AWS backbone to the S3 bucket. URL eg. cloudguru.s3-accelerate.amazonaws.com. Transfer acceleration has additional charges.
- To get started with S3 Transfer Acceleration enable S3 Transfer Acceleration on an S3 bucket using the Amazon S3 console, the Amazon S3 API, or the AWS CLI. After S3 Transfer Acceleration is enabled, you can point your Amazon S3 PUT and GET requests to the s3-accelerate endpoint domain name.
- S3 Transfer Acceleration is designed to optimize transfer speeds from across the world into S3 buckets. If you are uploading to a centralized bucket from geographically dispersed locations, or if you regularly transfer GBs or TBs of data across continents, you may save hours or days of data transfer time with S3 Transfer Acceleration.
S3 Storage gateway - Connects an on-premise software appliance with a cloud based storage and provides a seamless and secure integration between the on-premise IT environment and the AWS storage infrastructure. It provides low-latency performance by caching frequently accessed data on premises, while encrypting and storing all your other data in Amazon S3 or amazon glacier.
- It supports industry standard storage protocols to work that work with your existing applications.
- Once you have installed the storage gateway and associated it with your AWS account through the activation process, you can use AWS management console to create the storage gateway option to suit your needs.
- It is a virtual appliance that you install in the hypervisor of your data center. It is available as a virtual machine image for VMWare ESXi or Microsoft Hyper-V.
- Three types of storage gateways -
  1. File Gateway -
    - Used for flat files only which are directly stored on S3 (file based storage)
    - Files are stored in your S3 bucket and are accessed through NFS mount points
    - Ownership, permissions and timestamps are durably stored in S3 in the user metadata of the object associated with the file.
    - Once objects are transferred to S3, they can be managed as native S3 objects and bucket policies can be applied directly to the objects (such as cross region replication, versioning, lifecycle management).
  2. Volume Gateway -
    - Present your applications with disk volumes using iSCSi block protocol (block based storage)
    - These are similar to virtual hard disks with OS or databases
    - There are two types of volume gateways -
      1. Cached volumes -
        
        Allows customers to expand their local storage capacity to Amazon S3.
        
        All data stored in the gateway cached volume is moved to amazon S3. Only recently accessed data is retained in local storage to provide low latency.
        
        Maximum size of a single volume - 32PB. However a single gateway can support upto 32 volumes for a max storage of 1PB
        
        Point in time snapshots can be taken to backup your storage gateway. The snapshots are incremental backups that only capture changed blocks. They are compressed to minimize storage charges.
        
        All gateway cached volume data and snapshot data is transferred to Amazon S3 over SSL connections. It is encrypted at rest in S3 using S3-SSE. However, you cannot directly access this data via Amazon S3 API or the S3 console. You must access it via the storage gateway.
        
        Use case is when customers want to increase their storage capacity without increasing their storage hardware or changing storage processes.
      2. Stored Volumes -
        
        Allow storing data on on-premise storage and asynchronously backing up data to Amazon S3.
        
        Provides low latency access to data and provide off site backup
        
        Data is backed up as EBS snapshots on S3.
        
        Maximum size of single volume - 16PB. Single gateway can store upto 32 volumes with max storage of 512TB.
        
        Same concept of snapshots and incremental backups applies here, but they are stored as EBS snapshots
        
        All gateway cached volume data and snapshot data is transferred to Amazon S3 over SSL connections. It is encrypted at rest in S3 using S3-SSE. However, you cannot directly access this data via Amazon S3 API or the S3 console. You must access it via the storage gateway - same as for cached volumes.
  3. Tape gateway (Gateway Virtual Tape Libraries or VTL) -
    - Durable and cost effective archival solution
    - VTL interface lets you leverage your existing tape based application infrastructure to store data on virtual tape cartridges that you create on your gateway VTL. A virtual tape is analogous to a physical tape, except that it is stored on the cloud.
    - A gateway can contain 1500 tapes/ 1PB of total tape data. Virtual tapes appear in your gateway's VTL.
    - Virtual tapes are discovered by your backup application using its standard media inventory procedure. Supported by applications such as Netbackup, Backup Exec, Veeam etc.
    - When your tape software ejects a tape, it is archived on a virtual tape shelf (VTS) and stored on Amazon glacier.
Snowball (Was called Import/Export earlier)- Uses AWS provided shippable storage appliances shipped through UPS. Each snowball is protected through AWS KMS encryption and made physically rugged to protect and secure your data in transit. Use cases can be DC migrations to cloud where you have more data to move than you can move through an internet connection. There are three variants -
1. Snowball (standard) -
  - Petabyte scale data transfer in and out of AWS
  - Addresses challenges such as high network cost of large scale data transfers, long transfer times, security concerns.
  - Data transfer through snowball can be done at approx. 1/5 cost of using internet bandwidth.
  - 80 TB snowball is available in all regions
  - Provides multiple level of security -
    - Tamper resistant enclosures
    - AES256 encryption
    - industry standard trusted platform module (TPM) designed for security of data in custody
    - After completion of transfer, AWS performs software erase of the snowball appliance
2. Snowball edge -
  - 100 TB data transfer device
  - Has onboard storage and compute
  - Use cases
    - To transfer large amounts of data in and out of AWS
    - Temporary storage for large local datasets
    - Supports local workloads in remote or offline locations
3. Snowmobile -
  - Petabyte or Exabyte levels of data transfer
  - Upto 100PB per snowmobile transferred in a shipping container pulled by a truck
  - It takes about 6 months and 10 snowmobiles to transfer 1 exabyte of data
Snowball can -
- Import from S3
- Export to S3
- To move data from glacier, you will have to first restore data to an S3 and then import S3
Legacy import/export disk -
- Has an upper limit of 16TB
- Customers send data to AWS using their own storage devices (compatibility used to be an issue)
- Customers buy and maintain their own hardware devices
- Encryption is optional
- Jobs cannot be managed by AWS console
- Has been discontinued by AWS now
Types of connectivity for snowball appliances
1. Ethernet/RJ45
2. STP+ copper
3. STP+ fiber
Static Websites hosting - You can use S3 to host static web content.
- It is serverless
- Very cheap and scales automatically
- Static only, CANNOT host dynamic content (only html - no PHP or .net)
- URL format - "S3 bucket name".s3-website-"region name".amazonaws.com
- You need to create a DNS name in your own domain for the website using CNAME or an Amazon Route53 alias that resolves the Amazon S3 website URL.
- Static websites use S3 in GET intensive mode. For best performance, use amazon Cloudfront for caching layer in front of your S3 bucket.
Amazon S3 doesn't automatically give a user who creates a bucket or object permission to perform other actions on that bucket or object. Therefore, in your IAM policies, you must explicitly give users permission to use the Amazon S3 resources they create.
S3 Vs Glacier
- S3 storage - max size of object is 5TB, Glacier max size of archive is 40TB
- Archives in Glacier are identified by system generated IDs. Objects in S3 can be given friendly names by users.
- Glacier archives are automatically encrypted, while encryption is optional for objects in S3.
A common pattern is to use S3 as "Blob" storage for data while keeping an index to that data in another service such as Amazon dynamoDB or Amazon RDS.
S3 Restore Speed Upgrade for S3 Glacier is an override of an in-progress restore to a faster restore tier if access to the data becomes urgent.
If the Amazon S3 buckets that hold the data files do not reside in the same region as your cluster, you must use the REGION parameter to specify the region in which the data is located. If the Amazon S3 buckets that hold the data files do not reside in the same region as your cluster, you must use the REGION parameter to specify the region in which the data is located.
FAQs -
How much data can I store in Amazon S3?
The total volume of data and number of objects you can store are unlimited. Individual Amazon S3 objects can range in size from a minimum of 0 bytes to a maximum of 5 terabytes. The largest object that can be uploaded in a single PUT is 5 gigabytes. For objects larger than 100 megabytes, customers should consider using the Multipart Upload capability.
How is Amazon S3 data organized?
Amazon S3 is a simple key-based object store. When you store data, you assign a unique object key that can later be used to retrieve the data. Keys can be any string, and they can be constructed to mimic hierarchical attributes. Alternatively, you can use S3 Object Tagging to organize your data across all of your S3 buckets and/or prefixes.
How reliable is Amazon S3?

Amazon S3 gives any developer access to the same highly scalable, highly available, fast, inexpensive data storage infrastructure that Amazon uses to run its own global network of web sites. The S3 Standard storage class is designed for 99.99% availability, the S3 Standard-IA storage class is designed for 99.9% availability, and the S3 One Zone-IA storage class is designed for 99.5% availability. All of these storage classes are backed by the Amazon S3 Service Level Agreement.
Q: Where is my data stored?

You specify an AWS Region when you create your Amazon S3 bucket. For S3 Standard, S3 Standard-IA, and S3 Glacier storage classes, your objects are automatically stored across multiple devices spanning a minimum of three Availability Zones, each separated by miles across an AWS Region. Objects stored in the S3 One Zone-IA storage class are stored redundantly within a single Availability Zone in the AWS Region you select.
Q: What is an AWS Availability Zone (AZ)?

An AWS Availability Zone is an isolated location within an AWS Region. Within each AWS Region, S3 operates in a minimum of three AZs, each separated by miles to protect against local events like fires, floods, etc.

Amazon S3 Standard, S3 Standard-Infrequent Access, and S3 Glacier storage classes replicate data across a minimum of three AZs to protect against the loss of one entire AZ. This remains true in Regions where fewer than three AZs are publicly available. Objects stored in these storage classes are available for access from all of the AZs in an AWS Region.

The Amazon S3 One Zone-IA storage class replicates data within a single AZ. Data stored in this storage class is susceptible to loss in an AZ destruction event.
Q: How do I decide which AWS Region to store my data in?

There are several factors to consider based on your specific application. You may want to store your data in a Region that…
- ...is near to your customers, your data centers, or your other AWS resources in order to reduce data access latencies.
- ...is remote from your other operations for geographic redundancy and disaster recovery purposes.
- ...enables you to address specific legal and regulatory requirements.
- ...allows you to reduce storage costs. You can choose a lower priced region to save money.
Q: How much does Amazon S3 cost?

With Amazon S3, you pay only for what you use. There is no minimum fee. You can estimate your monthly bill using the AWS Simple Monthly Calculator.

We charge less where our costs are less. Some prices vary across Amazon S3 Regions. Billing prices are based on the location of your bucket. There is no Data Transfer charge for data transferred within an Amazon S3 Region via a COPY request. Data transferred via a COPY request between AWS Regions is charged at rates specified in the pricing section of the Amazon S3 detail page. There is no Data Transfer charge for data transferred between Amazon EC2 and Amazon S3 within the same region or for data transferred between the Amazon EC2 Northern Virginia Region and the Amazon S3 US East (Northern Virginia) Region. Data transferred between Amazon EC2 and Amazon S3 across all other regions (i.e. between the Amazon EC2 Northern California and Amazon S3 US East (Northern Virginia) is charged at rates specified on the Amazon S3 pricing page
Q: How am I charged if my Amazon S3 buckets are accessed from another AWS account?

Normal Amazon S3 pricing applies when your storage is accessed by another AWS Account. Alternatively, you may choose to configure your bucket as a Requester Pays bucket, in which case the requester will pay the cost of requests and downloads of your Amazon S3 data.
Q: How can I control access to my data stored on Amazon S3?

Customers may use four mechanisms for controlling access to Amazon S3 resources: Identity and Access Management (IAM) policies, bucket policies, Access Control Lists (ACLs), and Query String Authentication. IAM enables organizations with multiple employees to create and manage multiple users under a single AWS account. With IAM policies, customers can grant IAM users fine-grained control to their Amazon S3 bucket or objects while also retaining full control over everything the users do. With bucket policies, customers can define rules which apply broadly across all requests to their Amazon S3 resources, such as granting write privileges to a subset of Amazon S3 resources. Customers can also restrict access based on an aspect of the request, such as HTTP referrer and IP address. With ACLs, customers can grant specific permissions (i.e. READ, WRITE, FULL_CONTROL) to specific users for an individual bucket or object. With Query String Authentication, customers can create a URL to an Amazon S3 object which is only valid for a limited time
Q: Can I allow a specific Amazon VPC Endpoint access to my Amazon S3 bucket?

You can limit access to your bucket from a specific Amazon VPC Endpoint or a set of endpoints using Amazon S3 bucket policies. S3 bucket policies now support a condition, aws:sourceVpce, that you can use to restrict access.
Q: What is Amazon Macie?

Amazon Macie is an AI-powered security service that helps you prevent data loss by automatically discovering, classifying, and protecting sensitive data stored in Amazon S3. Amazon Macie uses machine learning to recognize sensitive data such as personally identifiable information (PII) or intellectual property, assigns a business value, and provides visibility into where this data is stored and how it is being used in your organization. Amazon Macie continuously monitors data access activity for anomalies, and delivers alerts when it detects risk of unauthorized access or inadvertent data leaks.
Q: What checksums does Amazon S3 employ to detect data corruption?

Amazon S3 uses a combination of Content-MD5 checksums and cyclic redundancy checks (CRCs) to detect data corruption. Amazon S3 performs these checksums on data at rest and repairs any corruption using redundant data. In addition, the service calculates checksums on all network traffic to detect corruption of data packets when storing or retrieving data.
Q: Can I setup a trash, recycle bin, or rollback window on my Amazon S3 objects to recover from deletes and overwrites?

You can use Lifecycle rules along with Versioning to implement a rollback window for your Amazon S3 objects. For example, with your versioning-enabled bucket, you can set up a rule that archives all of your previous versions to the lower-cost Glacier storage class and deletes them after 100 days, giving you a 100-day window to roll back any changes on your data while lowering your storage costs.
Q: What is S3 Intelligent-Tiering?

Amazon S3 Intelligent-Tiering (S3 Intelligent-Tiering) is an S3 storage class for data with unknown access patterns or changing access patterns that are difficult to learn. It is the first cloud storage class that delivers automatic cost savings by moving objects between two access tiers when access patterns change. One tier is optimized for frequent access and the other lower-cost tier is designed for infrequent access.

Objects uploaded or transitioned to S3 Intelligent-Tiering are automatically stored in the frequent access tier. S3 Intelligent-Tiering works by monitoring access patterns and then moving the objects that have not been accessed in 30 consecutive days to the infrequent access tier. If the objects are accessed later, S3 Intelligent-Tiering moves the object back to the frequent access tier. This means all objects stored in S3 Intelligent-Tiering are always available when needed. There are no retrieval fees, so you won’t see unexpected increases in storage bills when access patterns change.
Q: What performance does S3 Intelligent-Tiering offer?

S3 Intelligent-Tiering provides the same performance as S3 Standard storage.
Q: How durable and available is S3 Intelligent-Tiering?

S3 Intelligent-Tiering is designed for the same 99.999999999% durability as S3 Standard. S3 Intelligent-Tiering is designed for 99.9% availability,
Q: Is there a minimum duration for S3 Intelligent-Tiering?

S3 Intelligent-Tiering has a minimum storage duration of 30 days, which means that data that is deleted, overwritten, or transitioned to a different S3 Storage Class before 30 days will incur the normal usage charge plus a pro-rated charge for the remainder of the 30-day minimum.
Q: Is there a minimum object size for S3 Intelligent-Tiering?

S3 Intelligent-Tiering has no minimum billable object size, but objects smaller than 128KB are not eligible for auto-tiering and will always be stored at the frequent access tier rate.
Q: Can I have a bucket that has different objects in different storage classes?

Yes, you can have a bucket that has different objects stored in S3 Standard, S3 Intelligent-Tiering, S3 Standard-IA, and S3 One Zone-IA.
Q: What charges will I incur if I change the storage class of an object from S3 Standard-IA to S3 Standard with a COPY request?

You will incur charges for an S3 Standard-IA COPY request and an S3 Standard-IA data retrieval.
Q: Is there a minimum storage duration charge for S3 Standard-IA?

S3 Standard-IA is designed for long-lived but infrequently accessed data that is retained for months or years. Data that is deleted from S3 Standard-IA within 30 days will be charged for a full 30 days.
Is there a minimum object storage charge for S3 Standard-IA?

S3 Standard-IA is designed for larger objects and has a minimum object storage charge of 128KB. Objects smaller than 128KB in size will incur storage charges as if the object were 128KB. For example, a 6KB object in S3 Standard-IA will incur S3 Standard-IA storage charges for 6KB and an additional minimum object size fee equivalent to 122KB at the S3 Standard-IA storage price.
Q: How am I charged for using S3 One Zone-IA storage class?

Like S3 Standard-IA, S3 One Zone-IA charges for the amount of storage per month, bandwidth, requests, early delete and small object fees, and a data retrieval fee. Amazon S3 One Zone-IA storage is 20% cheaper than Amazon S3 Standard-IA for storage by month, and shares the same pricing for bandwidth, requests, early delete and small object fees, and the data retrieval fee.

As with S3 Standard-Infrequent Access, if you delete a S3 One Zone-IA object within 30 days of creating it, you will incur an early delete charge. For example, if you PUT an object and then delete it 10 days later, you are still charged for 30 days of storage.

Like S3 Standard-IA, S3 One Zone-IA storage class has a minimum object size of 128KB. Objects smaller than 128KB in size will incur storage charges as if the object were 128KB. For example, a 6KB object in a S3 One Zone-IA storage class will incur storage charges for 6KB and an additional minimum object size fee equivalent to 122KB at the S3 One Zone-IA storage price.
: Are there differences between how Amazon EC2 and Amazon S3 work with Availability Zone-specific resources?

Yes. Amazon EC2 provides you the ability to pick the AZ to place resources, such as compute instances, within a region. When you use S3 One Zone-IA, S3 One Zone-IA assigns an AWS Availability Zone in the region according to available capacity.
: Can I use the Amazon S3 APIs or Management Console to list objects that I’ve archived to Amazon S3 Glacier?

Yes, like Amazon S3’s other storage classes (S3 Standard, S3 Standard-IA, and S3 One Zone-IA), S3 Glacier objects stored using Amazon S3’s APIs or Management Console have an associated user-defined name. You can get a real-time list of all of your Amazon S3 object names, including those stored using the S3 Glacier storage class, using the S3 LIST API
Q: Can I use Amazon Glacier direct APIs to access objects that I’ve archived to Amazon S3 Glacier?

Because Amazon S3 maintains the mapping between your user-defined object name and Amazon S3 Glacier’s system-defined identifier, Amazon S3 objects that are stored using the S3 Glacier storage class are only accessible through the Amazon S3 APIs or the Amazon S3 Management Console.
Q: How can I retrieve my objects that are archived in Amazon S3 Glacier and will I be notified when the object is restored?

To retrieve Amazon S3 data stored in the S3 Glacier storage class, initiate a retrieval request using the Amazon S3 APIs or the Amazon S3 Management Console. The retrieval request creates a temporary copy of your data in the S3 RRS or S3 Standard-IA storage class while leaving the archived data intact in S3 Glacier. You can specify the amount of time in days for which the temporary copy is stored in S3. You can then access your temporary copy from S3 through an Amazon S3 GET request on the archived object.

With restore notifications, you can now be notified with an S3 Event Notification when an object has successfully restored from S3 Glacier and the temporary copy is made available to you. The bucket owner (or others, as permitted by an IAM policy) can arrange for notifications to be issued to Amazon Simple Queue Service (SQS) or Amazon Simple Notification Service (SNS). Notifications can also be delivered to AWS Lambda for processing by a Lambda function.
Q: How much data can I retrieve from Amazon S3 Glacier for free?

You can retrieve 10GB of your Amazon S3 Glacier data per month for free with the AWS free tier. The free tier allowance can be used at any time during the month and applies to Amazon S3 Glacier Standard retrievals.
Q: How am I charged for deleting objects from Amazon S3 Glacier that are less than 90 days old?

Amazon S3 Glacier is designed for use cases where data is retained for months, years, or decades. Deleting data that is archived to Amazon S3 Glacier is free if the objects being deleted have been archived in Amazon S3 Glacier for 90 days or longer. If an object archived in Amazon S3 Glacier is deleted or overwritten within 90 days of being archived, there will be an early deletion fee. This fee is prorated. If you delete 1GB of data 30 days after uploading it, you will be charged an early deletion fee for 60 days of Amazon S3 Glacier storage. If you delete 1 GB of data after 60 days, you will be charged for 30 days of Amazon S3 Glacier storage.
Q: What does it cost to use Amazon S3 event notifications?

There are no additional charges for using Amazon S3 for event notifications. You pay only for use of Amazon SNS or Amazon SQS to deliver event notifications, or for the cost of running an AWS Lambda function.
How secure is S3 Transfer Acceleration?

S3 Transfer Acceleration provides the same security as regular transfers to Amazon S3. All Amazon S3 security features, such as access restriction based on a client’s IP address, are supported as well. S3 Transfer Acceleration communicates with clients over standard TCP and does not require firewall changes. No data is ever saved at AWS Edge Locations.
Q: What if S3 Transfer Acceleration is not faster than a regular Amazon S3 transfer?

Each time you use S3 Transfer Acceleration to upload an object, we will check whether S3 Transfer Acceleration is likely to be faster than a regular Amazon S3 transfer. If we determine that S3 Transfer Acceleration is not likely to be faster than a regular Amazon S3 transfer of the same object to the same destination AWS Region, we will not charge for the use of S3 Transfer Acceleration for that transfer, and we may bypass the S3 Transfer Acceleration system for that upload.
Q: Can I use S3 Transfer Acceleration with multipart uploads?

Yes, S3 Transfer Acceleration supports all bucket level features including multipart uploads.
Q: How should I choose between S3 Transfer Acceleration and AWS Snow Family (Snowball, Snowball Edge, and Snowmobile)?

The AWS Snow Family is ideal for customers moving large batches of data at once. The AWS Snowball has a typical 5-7 days turnaround time. As a rule of thumb, S3 Transfer Acceleration over a fully-utilized 1 Gbps line can transfer up to 75 TBs in the same time period. In general, if it will take more than a week to transfer over the Internet, or there are recurring transfer jobs and there is more than 25Mbps of available bandwidth, S3 Transfer Acceleration is a good option. Another option is to use both: perform initial heavy lift moves with an AWS Snowball (or series of AWS Snowballs) and then transfer incremental ongoing changes with S3 Transfer Acceleration.
Q: Can S3 Transfer Acceleration complement AWS Direct Connect?

AWS Direct Connect is a good choice for customers who have a private networking requirement or who have access to AWS Direct Connect exchanges. S3 Transfer Acceleration is best for submitting data from distributed client locations over the public Internet, or where variable network conditions make throughput poor. Some AWS Direct Connect customers use S3 Transfer Acceleration to help with remote office transfers, where they may suffer from poor Internet performance.
Q: What are S3 object tags?

S3 object tags are key-value pairs applied to S3 objects which can be created, updated or deleted at any time during the lifetime of the object. With these, you’ll have the ability to create Identity and Access Management (IAM) policies, setup S3 Lifecycle policies, and customize storage metrics. These object-level tags can then manage transitions between storage classes and expire objects in the background.
Q: Why should I use object tags?

Object tags are a tool you can use to enable simple management of your S3 storage. With the ability to create, update, and delete tags at any time during the lifetime of your object, your storage can adapt to the needs of your business. These tags allow you to control access to objects tagged with specific key-value pairs, allowing you to further secure confidential data for only a select group or user. Object tags can also be used to label objects that belong to a specific project or business unit, which could be used in conjunction with S3 Lifecycle policies to manage transitions to other storage classes (S3 Standard-IA, S3 One Zone-IA, and S3 Glacier) or with S3 Cross-Region Replication to selectively replicate data between AWS Regions.
Note that all changes to tags outside of the AWS Management Console are made to the full tag set. If you have five tags attached to a particular object and want to add a sixth, you need to include the original five tags in that request.
Q: How much do object tags cost?

Object tags are priced based on the quantity of tags and a request cost for adding tags. The requests associated with adding and updating Object Tags are priced the same as existing request prices.
Q: What is S3 Inventory?

The S3 Inventory report provides a scheduled alternative to Amazon S3’s synchronous List API. You can configure S3 Inventory to provide a CSV, ORC, or Parquet file output of your objects and their corresponding metadata on a daily or weekly basis for an S3 bucket or prefix. You can simplify and speed up business workflows and big data jobs with S3 Inventory. You can also use S3 inventory to verify encryption and replication status of your objects to meet business, compliance, and regulatory needs.
Q: How much does it cost to use S3 Lifecycle management?

There is no additional cost to set up and apply Lifecycle policies. A transition request is charged per object when an object becomes eligible for transition according to the Lifecycle rule. Refer to the S3 Pricing page for pricing information.
Q: Can I use CRR with S3 Lifecycle rules?

Yes, you can configure separate S3 Lifecycle rules on the source and destination buckets. For example, you can configure a lifecycle rule to migrate data from the S3 Standard storage class to the S3 Standard-IA or S3 One Zone-IA storage class or archive data to S3 Glacier on the destination bucket.
Q: Can I use CRR with objects encrypted by AWS Key Management Service (KMS)?

Yes, you can replicate KMS-encrypted objects by providing a destination KMS key in your replication configuration.
Q: Are objects securely transferred and encrypted throughout replication process?

Yes, objects remain encrypted throughout the CRR process. The encrypted objects are transmitted securely via SSL from the source region to the destination region.
Q: Can I use CRR across AWS accounts to protect against malicious or accidental deletion?

Yes, you can set up CRR across AWS accounts to store your replicated data in a different account in the target region. You can use CRR Ownership Overwrite in your replication configuration to maintain a distinct ownership stack between source and destination, and grant destination account ownership to the replicated storage.
Q: What is the pricing for S3 Cross-Region Replication?

You pay the Amazon S3 charges for storage (in the S3 storage class you select), COPY requests, and inter-Region data transfer for the replicated copy of data. COPY requests and inter-Region data transfer are charged based on the source Region. Storage for replicated data is charged based on the target Region.
S3 bucket properties include -
1. Versioning
2. Server access logging
3. Static Website hosting
4. Object level logging
5. Tags
6. Transfer acceleration
7. Events
8. Requester pays
S3 Object level properties include the S3 storage classes. This means that You can choose a storage class for an object as well when you are trying to upload it to an S3 bucket. This storage class could be different from the storage class of its bucket i.e a standard S3 bucket can have objects stored as different storage classes like standard-IA, glacier, standard -IA one zone etc.You can choose a storage class for an object as well when you are trying to upload it to an S3 bucket. This storage class could be different from the storage class of its bucket i.e a standard S3 bucket can have objects stored as different storage classes like standard-IA, glacier, standard -IA one zone etc.
Cross Origin Resource sharing - Cross-origin resource sharing (CORS) defines a way for client web applications that are loaded in one domain to interact with resources in a different domain. With CORS support, you can build rich client-side web applications with Amazon S3 and selectively allow cross-origin access to your Amazon S3 resources. Cross-origin resource sharing (CORS) defines a way for client web applications that are loaded in one domain to interact with resources in a different domain. With CORS support, you can build rich client-side web applications with Amazon S3 and selectively allow cross-origin access to your Amazon S3 resources. To configure your bucket to allow cross-origin requests, you create a CORS configuration, which is an XML document with rules that identify the origins that you will allow to access your bucket, the operations (HTTP methods) that will support for each origin, and other operation-specific information.

The following are example scenarios for using CORS:
- Scenario 1: Suppose that you are hosting a website in an Amazon S3 bucket named website as described in Hosting a Static Website on Amazon S3. Your users load the website endpoint http://website.s3-website-us-east-1.amazonaws.com. Now you want to use JavaScript on the webpages that are stored in this bucket to be able to make authenticated GET and PUT requests against the same bucket by using the Amazon S3 API endpoint for the bucket, website.s3.amazonaws.com. A browser would normally block JavaScript from allowing those requests, but with CORS you can configure your bucket to explicitly enable cross-origin requests from website.s3-website-us-east-1.amazonaws.com.
- Scenario 2: Suppose that you want to host a web font from your S3 bucket. Again, browsers require a CORS check (also called a preflight check) for loading web fonts. You would configure the bucket that is hosting the web font to allow any origin to make these requests.
Types of URLs to access S3 bucket -
- Virtual hosted style - In a virtual-hosted–style URL, the bucket name is part of the domain name in the URL. For example:
  - http://bucket.s3.amazonaws.com
  - http://bucket.s3-aws-region.amazonaws.com.
- Path style URL - In a path-style URL, the bucket name is not part of the domain (unless you use a Region-specific endpoint). For example:
  
  US East (N. Virginia) Region endpoint, http://s3.amazonaws.com/bucket
  
  Region-specific endpoint, http://s3-aws-region.amazonaws.com/bucket
  
  In a path-style URL, the endpoint you use must match the Region in which the bucket resides. For example, if your bucket is in the South America (São Paulo) Region, you must use the http://s3-sa-east-1.amazonaws.com/bucket endpoint. If your bucket is in the US East (N. Virginia) Region, you must use the http://s3.amazonaws.com/bucket endpoint.
Delete marker - A delete marker is a placeholder (marker) for a versioned object that was named in a simple DELETE request. Because the object was in a versioning-enabled bucket, the object was not deleted. The delete marker, however, makes Amazon S3 behave as if it had been deleted. Delete markers accrue a nominal charge for storage in Amazon S3. The storage size of a delete marker is equal to the size of the key name of the delete marker. The object named in the DELETE request is not actually deleted. Instead, the delete marker becomes the current version of the object. (The object's key name (or key) becomes the key of the delete marker.)

A delete marker has a key name (or key) and version ID like any other object. However, a delete marker differs from other objects in the following ways:
- It does not have data associated with it.
- It is not associated with an access control list (ACL) value.
- It does not retrieve anything from a GET request because it has no data; you get a 404 error.
- The only operation you can use on a delete marker is DELETE, and only the bucket owner can issue such a request.
Using S3 Object Lock, you can prevent an object from being deleted or overwritten for a fixed amount of time or indefinitely. S3 Object Lock provides two ways to manage object retention: retention periods and legal holds. A retention period specifies a fixed period of time during which an object remains locked. During this period, your object will be WORM-protected and can't be overwritten or deleted. A legal hold provides the same protection as a retention period, but has no expiration date. Instead, a legal hold remains in place until you explicitly remove it.
S3 bucket contents are not public by default. If you want to make some objects in the S3 as public, use the object permissions to give read/write or both access to everyone (or just click on make public). However if you want to make the entire bucket or specific directories public, use a bucket policy.
If you mention a principle as "*", this means any user, role, group
Bucket policy editor is the text editor where you enter the bucket policy commands. Policy generator is the user friendly tool that helps you generate the JSON code to be pasted in the policy editor.
When you have multiple versions of an object and you have used the GetObject permission in the bucket policy, you will only be able to view the latest version of the object. If you try to access the previous versions using the S3 object links to previous versions, you will get access denied. To get access to all previous versions, use the GetObjectVersion permission.
Read Exam Essentials from book

	Created by Gaurav Sehgal almost 6 years ago

Note on AWS Certified Solutions Architect Associate notes, created by Gaurav Sehgal on 16/11/2018.

AWS Certified Solutions Architect Associate notes