BigQuery — Serverless, Dremel & Slots

No clusters to size: how BigQuery turns SQL into massively parallel scans.

0/1 done

Theory

There is no cluster — there's a query and a pool

Snowflake and Databricks ask you to pick a compute size. BigQuery's differentiator is true serverless: you submit SQL, and Google allocates compute from a vast shared pool for the duration of that query. There is nothing to start, size or suspend.

  • Dremel — BigQuery's execution engine breaks a query into a tree of stages and fans it out across thousands of workers that scan Capacitor (columnar) files in Colossus storage in parallel, then aggregate results back up the tree. This is how it scans terabytes in seconds with no cluster in sight.
  • Slots — A slot is BigQuery's unit of compute (a virtual CPU share). A query is granted some number of slots; more slots = more parallelism. You either pay on-demand (billed by bytes scanned) or buy capacity / reservations (a fixed pool of slots for a flat rate, shared across your queries).
  • Separation, taken further — Because compute is fully managed and ephemeral per query, scaling is automatic and invisible; the trade-off is you tune the query and the data layout, not a cluster size.

Use Case Example: An analyst runs an ad-hoc GROUP BY over a 10 TB events table at 2pm. BigQuery spins thousands of Dremel workers, scans the needed columns in parallel, returns in seconds, and releases all that compute the instant the query ends — the team never provisioned, sized, or paused a thing.

Analogy

BigQuery is a ride-hailing app for compute. You don't own a car (cluster) or keep it parked and insured. You request a ride (submit a query), the platform instantly dispatches as many cars as the trip needs (slots / Dremel workers), and you stop paying the moment you arrive. On-demand pricing is paying per trip (bytes scanned); a reservation is a monthly car-share pass (a fixed slot pool) that's cheaper once you ride enough.

Massively parallel, serverless

Click a node to focus its neighbourhood · drag to pan · scroll to zoom

Dremel's execution tree

A query fans out across many leaf workers scanning storage in parallel, then aggregates back up — all from a shared slot pool, no cluster to manage.

Reflect

Serverless removes the sizing decision but doesn't remove cost discipline — it relocates it. On Snowflake you tune the warehouse; on BigQuery on-demand you tune the bytes your query touches. Same lesson, different knob.

  • Does your team prefer 'never size a cluster' or 'control exactly which cluster runs what'? That preference predicts platform fit.
  • If you pay per byte scanned, who on your team knows how many bytes their dashboard touches?

Reading in progress · 0 of 1 activity done