Google Cloud has outlined a framework for running Apache Spark workloads on its managed platform, focusing on serverless deployment, cost controls and AI-assisted fault diagnosis.
The guidance explains how users of Managed Service for Apache Spark should choose between managed clusters and serverless infrastructure, and between interactive sessions and batch execution.
Managed clusters, it argues, still suit organisations running steady, round-the-clock pipelines with consistently high utilisation, depending on older Spark 2.x environments, or needing deep control over operating systems, machine shapes and storage layouts.
Serverless, by contrast, is aimed at intermittent and bursty jobs that do not justify permanently running infrastructure. In this model, compute is provisioned when needed and shut down when the work is complete.
Deployment choices
Google Cloud also distinguishes between interactive and batch serverless options. Interactive sessions are intended for development and exploratory analysis, where engineers use notebooks, inspect intermediate results and refine transformations while compute remains available between steps.
Batches are designed for automated production runs. These jobs execute packaged Python, Java or Scala applications from start to finish without manual intervention and are typically scheduled through workflow tools or other orchestration systems.
The two modes are presented as a natural progression from development to production: engineers can test logic in interactive sessions, then move final code into scheduled batch jobs.
Cost and tuning
Much of the guidance focuses on resource tuning in serverless Spark, where costs are tied to Data Compute Units, or DCUs. Default settings can lead to wasted spend or performance problems if workloads are not matched to the right level of memory, processor cores and scaling boundaries.
By default, serverless batches use four cores and 16,000MB of memory. That may not suit memory-heavy jobs processing large uncompressed datasets, which can fail with out-of-memory errors, or processor-heavy jobs where CPUs are saturated while memory remains underused.
Users are advised to adjust driver and executor memory independently and tune the number of cores for each. These settings should be reviewed together, because increasing cores also changes the baseline memory assigned to a workload.
Google Cloud also highlights autoscaling controls as a budget safeguard. Serverless Spark can increase the number of executors when task backlogs build, but poor code or badly designed joins can trigger unnecessary scale-out and higher bills if upper limits are not set.
For that reason, it recommends defining a maximum number of executors through Spark's dynamic allocation settings. A higher ceiling may suit jobs with strict service targets, while a lower one may better fit overnight or lower-priority work where predictable spend matters more than runtime.
Shuffle issues
Shuffle storage is another frequent source of inefficiency. Wide transformations such as joins, groupings and distinct operations redistribute data across the network and write intermediate files to disk, slowing jobs and adding cost if partitioning is poorly configured.
According to Google Cloud, Spark's default of 200 shuffle partitions may be too low for very large datasets. If partitions grow too large for executor memory, data spills to disk instead of being processed in memory.
Its recommendation is to size partitions so each handles roughly 100MB to 200MB of in-memory data, with some trial and error needed to find the right balance for each workload.
AI troubleshooting
Another part of the guidance focuses on Gemini Cloud Assist, which is integrated into the Google Cloud console to diagnose failed Spark jobs. The tool is presented as a way to reduce the need for engineers to manually search driver and executor logs when pipelines fail.
Google Cloud describes a typical failure sequence in a PySpark ETL job reading transaction data from Cloud Storage. In the first example, a job failed because required runtime arguments were missing at submission, and Gemini Cloud Assist identified the issue from the logs and pointed to the relevant lines in the script.
In a second example, the same pipeline failed because a division operation was performed on fields inferred as strings rather than numbers. The assistant identified the faulty transformation and traced the problem to non-numeric values in the source data.
The tool can also suggest revised code to make pipelines more resilient, including schema casting and null handling so invalid records can be skipped instead of causing the whole batch to fail.
The material reflects a wider push by cloud providers to tie infrastructure services more closely to AI-based operational tools. For Spark users, the message is that deployment choice, careful tuning and faster diagnosis are now central to managing large-scale data pipelines.
Google Cloud argues that serverless Spark can reduce operational work, but makes clear that users still need to set explicit resource boundaries and workload parameters to avoid performance bottlenecks and control spending.