Title

AWS re:Invent 2022 - How Yahoo cost optimizes their in-memory workloads with AWS (DAT321)

Summary

Yahoo moved their ad platform to AWS and optimized costs using Amazon ElastiCache.
Maulik Shah from Yahoo and Itai Meos from AWS presented the session.
Yahoo's ad platform generates 320 billion events per day, requiring efficient data processing.
Yahoo initially considered DynamoDB and HBase but settled on ElastiCache due to its performance and cost-effectiveness.
Data tiering in ElastiCache allowed Yahoo to reduce cluster size and costs by 50%.
Yahoo's architecture includes AWS Kinesis, S3, EMR, and Glue, with data querying via Athena or direct S3 access.
Yahoo learned to manage connections efficiently and implement exponential backoff with retries.
Yahoo plans to further optimize costs and migrate on-prem workloads to AWS.
AWS introduced data tiering, allowing storage expansion by moving data from memory to SSDs, saving up to 60% in costs.
Data tiering is transparent to applications and requires no code changes.
AWS also discussed MemoryDB for Redis, a durable in-memory database service.
New ElastiCache features include an improved management console, Memcached 1.6.12, encryption in transit, IPv6 support, Redis log delivery, native JSON support, AWS PrivateLink, Redis 7, and IAM authentication.
ElastiCache is available in all AWS regions and will be included in new regions.

Yahoo's use case demonstrates the scalability and cost-effectiveness of AWS services for handling large-scale data workloads.
Data tiering is a significant innovation that allows for cost savings by intelligently managing data storage between memory and SSDs.
The session highlighted the importance of monitoring and managing connections to maintain performance and reliability.
AWS's commitment to customer feedback is evident in the development of data tiering and other new features.
MemoryDB for Redis offers durability and high performance, expanding the options for customers needing in-memory databases.
The introduction of new features and services like MemoryDB for Redis and data tiering indicates AWS's focus on continuous innovation and addressing diverse customer needs.
The global availability of ElastiCache ensures that customers can rely on consistent performance and features regardless of their geographic location.