EMR Serverless Doubles Worker Sizes for Intensive Workloads
EMR Serverless now offers 32 vCPU/244 GB memory workers, doubling previous limits. This targets compute-heavy Spark and Hive jobs.
Editorial summary and commentary based on the original from AWS What's New. Read the original
EMR Serverless workers just doubled in size. Don't expect a performance miracle, but do expect fewer out-of-memory errors.
What changed
- Worker configurations increased from a maximum of 16 vCPUs and 120 GB memory to 32 vCPUs and 244 GB memory.
- Larger workers are available for Spark and Hive workloads.
- This change applies to all AWS Regions where EMR Serverless is available.
Why it matters
This update directly addresses limitations for memory-intensive workloads on EMR Serverless. The honest version: AWS is acknowledging that previous worker sizes were insufficient for certain Spark and Hive jobs, particularly those experiencing data skew or requiring significant in-memory caching. By doubling the available resources per worker, EMR Serverless can now handle more demanding tasks, potentially reducing out-of-memory failures and improving performance by minimizing data shuffling. Pairs with: AWS Glue for ETL jobs that might otherwise be constrained by EMR Serverless worker limits.
The catch
While larger workers can improve performance, they do not fundamentally change the serverless architecture's overhead. The catch: Users are still subject to EMR Serverless's per-second billing, and larger workers mean higher costs per worker if not carefully managed. The announcement does not specify any new service quotas or limits that might cap the total number of these larger workers a customer can provision, which could be a concern for massive-scale operations. Watch out: Existing Spark and Hive jobs may require tuning to fully benefit from the increased resources; simply switching to larger workers might not yield optimal results.
Ship it
If your Spark or Hive jobs on EMR Serverless are frequently failing with out-of-memory errors or suffering from excessive data shuffling, consider migrating them to the new 32 vCPU/244 GB worker configuration. Test these workloads in a non-production environment first to validate performance gains and cost implications before a full migration.
Bottom line: EMR Serverless now offers larger workers, which should reduce OOM errors and improve performance for memory-intensive Spark/Hive jobs.
— Filed to /aws