Skip to main content

AWS Architect Associate Notes - S3

    1. S3 is a simple key based object store.
    2. S3 has globally unique namespace i.e. bucket name has be unique across all region but S3 bucket is created in a region. The data is stored on multiple devices spanning at least 3 AZs
    3. S3 has three URL scheme
      1. Global (legacy)
      2. Virtual Domain Based
      3. Path Based
    4. File size 0 bytes to 5 TB. Largest PUT 5GB
    5. Successful uploads produces a HTTP 200
    6. Consistency
      1. Read after write for PUTS of new objects
      2. Eventual consistency for overwrites PUTS and DELETES
    7. S3 as objects and has following properties
      1. Key
      2. Value
      3. Version id
      4. Metadata
      5. Sub resources
        1. Access control list
        2. Torrents
    8. 99.99% availability, Amazon guarantees 99.9% and  99.999999999% durability (unlikely it will be lost)
    9. Tiered storage
      1. Standard
      2. S3 - IA
      3. S3 - one Zone - IA
      4. S3 - Intelligent Tiering
        1. Objects > 128 KB
        2. Minimum 30 days before an object Tiering is applied
      5. S3 - Glacier
      6. S3 - Glacier Deep Archiver
    10. Lifecycle management
      1. Automates moving object between different storage classes
    11. Versioning
      1. Once enabled, it can't be disabled, only suspended
      2. Versioning is at bucket level
      3. Stores all version
      4. MFA Delete with versioning provides extra security
      5. Uploading a new version reset the permissions and make it private again. Older version permissions don’t change
      6. Size of S3 bucket is sum of version
      7. Deleting an object puts a delete marker over. Previous versions are still there. To restore the file, delete the delete marker
    12. Encryption at Rest - server side or client side
      1. SSE-S3 - Amazon managed keys
      2. SSE-KMS - Key managed service provided for AWS Key management
      3. SSE-C - Customer managed keys
      4. KMS has quota based on region - for every download and upload you will use KMS. Quota based on region is either 5500, 10000 or 30000 request per second. You cannot request to increase quota
    13. Secure data using
      1. Access Control List
        1. Read, Write, Full_Control to specific users
      2. Bucket Policies
        1. Fine grain access
      3. IAM Policies
      4. Query string Authentication - URL valid for limited time.
    14. Charges based on
      1. Storage size
      2. No of request
      3. Storage class
      4. Data transfer
      5. Transfer acceleration - takes advantage of edge location to transfer data
      6. Cross region replication pricing
    15. By default all buckets and objects are private.
    16. Object lock modes  (WORM - Write once Read Many)
      1. Governance mode - cannot delete or overwrite until you has special permission.
      2. Compliance mode - cannot be deleted or modified by any user for the duration of retention period
      3. Legal hold - no retention period, prevents object from being overwritten or deleted until removed
      4. Object lock can be enabled ONLY at the time of bucket creation
      5. S3 Glacier vault lock policy - once the policy is locked it cannot be changed.
    17. S3 Prefixes are basically subfolders in the bucket. 5500 request per prefix. More prefix, more requests
    18. Sharing buckets
      1. Bucket policies & IAM - Bucket level - Programmatic  only
      2. ACL & IAM - Object level _ programmatic access only
      3. Cross account IAM roles - Programmtic and console access
    19. Cross Region Replication
      1. Only new objects or new versions of exiting objects are replicated
      2. CRR requires versioning to be enabled on source and destination
      3. Deleting a object from original bucket wont delete from replicated bucket
    20. S3 Transfer acceleration
      1. You upload file to edge location rather than directly to S3
      2. Amazon backbone network transfers the file to S3
      3. https://s3-accelerate-speedtest.s3-accelerate.amazonaws.com/en/accelerate-speed-comparsion.html - tool to compare upload speed to various regions compared to edge locations
    21. DataSync
      1. Tool used to sync large file from on prem to AWS
      2. NFS or SMB file system compatible
      3. You can also sync EFS to EFS
    22. CloudFront - Content Delivery Network service
      1. Origin  - content that will be delivered - S3, EC2, ELB or Route 53
      2. Distribution - collection of edge locations
      3. Web distribution - typically for web sites
      4. RTMP - used for media streaming
        1. This is being replaced with
          1. HLS - HTTP Live Streaming
          2. HDS- HTTP Dynamic streaming
          3. MSS - Microsoft Smooth Streaming
          4. DASH - Dynamic Adaptive Streaming over HTTP
      5. Signed URL for scenario 1 URL = 1 file
      6. Signed cookies for scenario 1 cookie = multiple files
      7. CloudFront Signed URL
        1. Client Use AWS SDK to generate signed URL
        2. You attach a policy to each signed URL
      8. S3 Signed URL
        1. Limited lifetime
        2. Issues a request as IAM user who creates the presigned URL
    23. Snowball
      1. Petabyte level storage solution for transferring data in and out of AWS
      2. 50 and 80 TB
      3. Snowmobile is 100PB, 45 feet long container
    24. Storage Gateway
      1. Virtual or physical device to connect on prem application to cloud storage
      2. Three types
        1. File gateway
        2. Volume gateway
        3. Tape Gateway
    25. Athena vs Macie
      1. Athena is serverless services, provides SQL like interface to data stored in S3
      2. Macie uses machine learning and NLP to identify is S3 objects contain PII
    26. Access Analyser
      1. Evaluates bucket access policies, enables you to discover and swiftly remediate
      2. Needs to be enabled from IAM Access Analyser
    27. Access Points
      1. Used for shared data
      2. Instead of using single bucket policy, you create access points for each application
      3. Provide unique path
      4. You can create multiple access points for same bucket
    28. Querying S3 data
      1. S3 Select - SQL like queries for data stored as CSV, JSON and Apache Parquet
        1. S3 Select is used to fetch a subset of data rather than entire object. E.g. You can query data from a zipped CSV rather than download entire CSV
      2. Athena - interactive queries for data stored as CSV, JSON, ORC, Apache Parquet and Avro
      3. Redshift spectrum - run queries against S3
    29. S3 Batch
      1. Automation of an operation
    30. Notification - send notification through following channels for changes to objects
      1. SQS
      2. SNS
      3. Lambda
    31. Read S3 FAQ - https://aws.amazon.com/s3/faqs/
Labels:

Comments

Post a Comment

Popular posts from this blog

How to detect HTML5 support for a browser?

HTML5 has introduced lots of new cool  tags . Not all the browsers support all tags and also the implementation of these tags may be different for each browser. HTML5 specification defines the functional aspects of these tags and not the implementation. Also the general concensus is that by 2022 all browsers will support all new features of HTML5. Of all the modern browsers, Chrome seems to have implemented most, if not all, featutes of HTML5. IE9 supports few. Firefox sits in between. So as a developer how do you make use of the cool HTML5 features without causing any compatibility issues with existing browsers? Traditionally developers have used User Agent to detect browser type and use the features accordingly. However these days, you can easily change a User Agent by using addons in your browser. So you need a more robust way to detect the features supported by the browser as the same engine of two different versions of a browser mig...
Post a Comment
Read more

Searching Unicode characters in Oracle table

Oracle implementation of Regular expression has no support for using hexadecimal code to search for Unicode characters. The only way to search for Unicode character is it use the character itself. Normally with Regular expression, you can use \x or \u followed by hexadecimal code to search for any character. E.g. \x20 will match space. But REGEXP_LIKE in Oracle does not support \x. You need to use unistr function to convert the code to equivalent character and then use it with REGEXP_LIKE. E.g. REGEXP_LIKE(source,'[' ||unistr('\0020')|| ']');
Post a Comment
Read more

System.Configuration in .Net Framework 2 onwards

Often application need custom configuration section. System.Configuration namespace includes classes for reading and writing configuration settings. There is a slight difference in how you use this namespace depending on the Framework version you are using Prior to .Net Framework 2.0, the .Net Framework included System.Configuration namespace, but that version of the namespace is now outdated. If you simply add the System.configuration namespace to your project (using in C#), your application references the outdated namespace. To refer to the updated namespace, follow these steps 1. In VS, open the project that requires System.Configuration namespace. 2. Click on the Project menu and then click Add Reference 3. On the .Net tab, Select System.Configuration as shown in following figure, and click OK 4. Now add the System.Configuration namespace to your project normally using Imports (in VB) or using (in C#) and your application will reference the correct version of the namespa...
Post a Comment
Read more