Abdullah Şamil Güser

Databases on AWS

249. AWS Databases - Introduction, Benefits, and Types

Introduction

• AWS offers a diverse range of database services, catering to various use cases and requirements.
• Focus: Flexibility, ease of management, and integration capabilities.

Key Features Across AWS Databases

• Network Isolation: Deployment in Virtual Private Cloud (VPC).
• Security Groups: Control access to databases.
• Encryption: At-rest and in-transit encryption options.
• High Durability and Availability: Replication across devices and Availability Zones.
• Automated Backup: Standard across services.
• Multi-Region, Multi-Master Setups: Ideal for global applications and disaster recovery.

Database Services Overview

1. Relational Database Service (RDS)

• Purpose: Quickly set up relational databases.
• Variety: Supports multiple engines (e.g., Aurora, SQL Server).
• Use Cases: Traditional applications, ERP, CRM, e-commerce.

2. Amazon Redshift

• Data Warehousing: Petabyte-scale, columnar storage, massively parallel processing.
• Integration with S3 Data Lake: Runs queries across structured and semi-structured data.
• Use Cases: Business Intelligence, analytics.

3. NoSQL Databases

• Options: DynamoDB, Cassandra, DocumentDB.
• Characteristics: Unlimited scaling, consistent low-latency.
• Use Cases: High-traffic web applications, e-commerce, gaming.

4. Amazon ElastiCache

• In-memory Data Store: Supports Memcached and Redis.
• Performance: Sub-millisecond latency.
• Use Cases: Caching, session management, gaming leaderboards.

5. Amazon Neptune

• Graph Database: Analyzes highly connected datasets.
• Use Cases: Social network analysis, recommendation engines.

6. Amazon Timestream

• Time Series Database: Optimized for IoT, industrial telemetry, DevOps.
• Efficiency: Cost-effective compared to relational databases.

7. Amazon Quantum Ledger Database (QLDB)

• Ledger Database: Uses blockchain technology for immutable, verifiable transaction logs.
• Use Cases: Systems of record, supply chain, banking transactions.

8. Amazon Elasticsearch Service

• Search and Analysis: Builds powerful search and log analytics solutions.
• Use Cases: Infrastructure monitoring, application monitoring, security event management.

Database Migration Service (DMS)

• Function: Migrates data to and from AWS.
• Flexibility: Supports one-time and continuous replication.
• Versatility: Handles both homogeneous and heterogeneous migrations.

Summary

• AWS offers a wide range of managed database services, enabling selection based on specific business needs.
• Each service is designed to handle particular data management and processing requirements, providing flexibility and scalability.

250. Relational Database Service (RDS) - Features and Benefits

Introduction

• AWS RDS simplifies database management, ideal for applications requiring a relational database.
• Offers multiple database engines: Aurora, MySQL, PostgreSQL, MariaDB, Oracle, and SQL Server.

Key Benefits of RDS

• Administrative Automation: Manages server provisioning, database setup, patching, and backups.
• Scalability: Easy scaling of database’s compute and storage resources.
• High Availability: Multi-AZ deployment for increased availability.
• Backup and Restore: Automated backups to S3, point-in-time restores, and manual snapshots.
• Read Replicas: Offload read traffic from primary database to improve performance.

High Availability and Disaster Recovery

• Multi-AZ Deployments: Synchronous replication to standby instances in different Availability Zones.
• Failover Process: Automatic failover to standby in case of primary instance failure.
• DNS Name Pointing: Applications use a consistent DNS endpoint, updated during failovers.
• Maintenance and Patching: Applied with minimal downtime, using failover mechanisms.

Backup and Snapshot Features

• Automated Backups: Daily full backups, transaction logs to S3, up to 35 days retention.
• Snapshots: User-initiated backups, stored indefinitely in S3.
• Restoration: Create new instances from backups or snapshots for data recovery.

Scaling and Storage

• Compute Scaling: Change instance types with minimal downtime.
• Storage Scaling: Increase storage capacity without service interruptions.
• Maximum Storage Capacity: Varies by database engine (16 TB for SQL Server, 64 TB for Aurora, 32 TB for others).

Network and Security

• VPC Deployment: Deploy in private or public subnets as per requirements.
• Security Group Control: Manage network access to the database.
• Encryption: At-rest using KMS keys, in-transit with SSL/TLS.

Access and Permissions

• Control Plane Access: Managed via AWS Identity and Access Management (IAM).
• Data Plane Access: Database user accounts with specific privileges.

Customization and Monitoring

• DB Parameter Groups: Customize database engine settings.
• Cost Management: Reserved instances for cost savings.
• AWS Config: Track and manage configurations.
• Operational Monitoring: Amazon CloudWatch for metrics and logs.

Summary

• RDS simplifies the complexity of setting up and managing a relational database.
• Provides essential features like high availability, backup, scaling, and security.
• Enables teams to focus more on application development and less on database administration.

251. Aurora and Aurora Serverless Relational Database

Introduction

• Amazon Aurora is a relational database service with unique features and performance enhancements.

Aurora Storage Subsystem

• Automatic Replication: Data is replicated across 6 copies in 3 Availability Zones.
• Quorum Writes: Write acknowledged after 4 out of 6 copies are stored.
• Read Replicas: Supports up to 15 low-latency read replicas.
• Primary Instance Failure Handling: Rapid failover, typically under 60 seconds.

Compatibility and Performance

• Compatibility Modes: MySQL and PostgreSQL.
• Performance: Up to 5 times faster than standard MySQL and 3 times faster than standard PostgreSQL.
• Cost Efficiency: Claims 1/10th the cost compared to commercial databases.

Global Database

• Cross-Region Replication: Replicates across multiple regions for low latency reads and disaster recovery.
• Fast Local Reads: Provides low-latency reads in each region.

Endpoints

• Cluster Endpoint: Points to the current primary instance for read/write operations.
• Reader Endpoint: Load balances connections across read replicas.
• Instance Endpoint: Direct connections to individual instances.

Aurora Serverless

• Decouples Processing and Storage: Automatically scales processing based on load.
• Idle State Handling: Removes processing instance during inactivity, reducing costs.
• Auto Scaling: Scales based on the load within configured Aurora compute units.

Use Cases

• Ideal for intermittent or unpredictable workloads.
• Suitable for applications that require high availability and fast failover capabilities.

Certification Importance

• Understanding Aurora is crucial for AWS certification exams.

Summary

• Amazon Aurora offers high performance, scalability, and cost efficiency.
• Unique features like automatic replication, serverless options, and global databases set it apart from traditional relational databases.

252. DynamoDB - Primary Key, Partitions, and Features

Introduction

• DynamoDB is a managed NoSQL service on AWS, crucial for AWS certification exams.
• Key-value store: Stores data as JSON documents with primary key and dynamic attributes.
• Schema-less: Only primary key is mandatory, flexible in storing new attributes.
• Performance: Consistent single-digit millisecond latency.

Data Storage Comparison

• Relational Database: Requires predefined tables and columns.
• DynamoDB: Stores all movie details in a single JSON document, handling one-to-many relations easily.

Primary Keys

• Simple Primary Key: Single attribute key (e.g., movie title).
• Composite Primary Key: Consists of a partition key and a sort key (e.g., year and movie title).

Data Distribution and Partitions

• Partition Key: Determines data partitioning; should be chosen to evenly distribute data.
• Example Scenarios:
  • User ID as partition key and game title as sort key for even distribution.
  • Game title as partition key creates potential for uneven distribution.
  • Country as partition key can lead to a hot partition (overutilization of one partition).

DynamoDB Features

• High Availability: Automatic replication across Availability Zones.
• Global Tables: Multi-master, multi-region replication for global applications.
• Transactions: Supports transactions across multiple items and tables.
• Continuous Backup to S3: With 35 days retention; point-in-time restore available.
• Time To Live (TTL): Automatically deletes expired items.
• Item Size Limitation: Maximum of 400 KB per item; larger items can be stored in S3 with a reference in DynamoDB.

Use Cases

• Ideal for high-traffic web applications, e-commerce systems, gaming platforms, and more.

Summary

• DynamoDB offers high performance, scalability, and flexibility for NoSQL database needs.
• Its unique features like schema-less design, global tables, and transaction support make it a versatile choice for various applications.

253. Cassandra and DocumentDB

Introduction

• AWS provides managed services for migrating open-source databases Cassandra and MongoDB to the cloud.

Amazon Managed Cassandra

• Cloud-optimized Cassandra: AWS version designed for easy migration from on-premises deployments.
• Performance: Comparable to DynamoDB; offers single-digit millisecond response time, linear scaling, and a flexible schema.
• Use Cases: Suitable for industrial equipment data collection and scenarios needing numerous columns.
• Key Differences from DynamoDB:
  • Primary Key Structure: Cassandra supports multi-column partition and sort keys, offering more flexibility.
  • Item Size: Cassandra allows up to 2 GB per column (best performance with few MB per column) versus DynamoDB’s 400 KB per item limit.
  • Column Limit: Cassandra supports a virtually unlimited number of columns, whereas DynamoDB’s attributes must fit within its item size limit.
• Advantages: Offers more flexibility in terms of primary key structure and item size.

Amazon DocumentDB

• MongoDB API Compatibility: Designed to be compatible with MongoDB APIs.
• Challenges: There have been reports of compatibility issues with newer MongoDB versions.
• Future Roadmap Uncertainty: Potential incompatibility between future versions of MongoDB and DocumentDB.

Summary

• AWS Managed Cassandra and DocumentDB provide cloud migration options for Cassandra and MongoDB users, respectively.
• Managed Cassandra offers significant flexibility and scalability benefits.
• DocumentDB’s compatibility with MongoDB is a crucial feature, but future compatibility remains uncertain.

254. Amazon ElastiCache - Usage Example, Features

Introduction

• AWS Elasticache is a distributed in-memory data store offering sub-millisecond latency.
• It’s primarily used as a database cache and for capturing frequently changing information.

Use Cases

• Ideal for applications like game leaderboards, user session management, product reviews and ratings, and geospatial applications.
• Provides network isolation and security by deploying in a VPC.

Elasticache Engines

• Offers two engine choices: Memcached and Redis.

Memcached

• Simple key-value store.
• Scales up to 20 nodes and 12 TB of data.
• Provides sub-millisecond read/write operations.
• Suitable for basic caching requirements.

Redis

• Feature-rich, supporting advanced data structures like lists, sorted sets, hash tables, and bit arrays.
• Enables implementation of in-memory queues, automatic maintenance of leaderboards, and geospatial data processing.
• Scales up to 250 nodes and 170 TB of data.
• Provides pub-sub capabilities for building chatrooms, server communication, and tracking social media feeds.
• Supports read replicas across multiple Availability Zones for increased scalability and automatic primary promotion in case of failure.
• Offers backup support to S3 and data export to other regions.
• Includes Lua scripting support.

Integration Example

• Application using DynamoDB for storage integrates Elasticache for real-time aspects like gameplay, leaderboards, and session data.
• Reduces the number of read/write requests to DynamoDB, optimizing resource usage.

Benefits of Elasticache

• High Performance: Ensures sub-millisecond response times, enhancing user experience in real-time applications.
• Reduced Database Load: Decreases the volume of requests to the backend database, enabling efficient scaling with fewer resources.
• Flexibility and Scalability: Offers various data structures and scaling options to fit diverse application needs.

Conclusion

• Elasticache, with its choice of Memcached and Redis, offers significant benefits in terms of performance and database load reduction, making it a versatile solution for high-throughput applications.

255. Amazon Redshift

Introduction

• Amazon Redshift is AWS’s petabyte-scale data warehouse service.
• It utilizes a distributed cluster architecture including a leader node and multiple compute nodes.

Key Features

Distributed Cluster

• Redshift’s architecture allows for scalability by adding more compute nodes to the cluster.

Columnar Storage

• Unlike traditional row-based storage, Redshift uses columnar storage.
• This approach is efficient for analytic queries, which often involve a subset of columns across many rows.
• Each block in columnar storage contains data of a single column, allowing optimized compression based on data type.

Analytics and SQL Support

• Redshift supports SQL with advanced analytics functions.
• Ideal for sophisticated analytics tasks requiring querying across numerous rows but fewer columns.

Redshift Spectrum Integration with S3

• Allows querying data across Redshift tables and files stored in Amazon S3.
• Offers the flexibility to access a large amount of data in file format, enhancing data warehousing capabilities.

Benefits

• Optimized for Analytics: Columnar storage makes it suitable for data warehousing and complex analytical queries.
• Scalable Architecture: Ability to scale by adding more compute nodes.
• Data Compression: Efficient data storage due to compression algorithms tailored for specific data types.
• SQL and Advanced Analytics: Supports a wide range of analytics use cases with familiar SQL syntax.
• Integration with S3: Extends querying capabilities to include unstructured data stored in S3.

Conclusion

• Amazon Redshift offers a highly scalable, columnar storage-based data warehouse solution, making it a powerful tool for businesses needing advanced data analytics capabilities.
• The integration with AWS services like S3 further enhances its utility in diverse data processing scenarios.