Spotify's RAP Index Could Change How Earth Scientists Query Their Data Lakes

The Data Lake Bottleneck

Every Earth scientist knows the drill. You've got terabytes of satellite imagery, sensor readings, or climate model outputs sitting in a cloud bucket. The data is rich, but pulling out a specific record—say, the soil moisture reading for a particular plot on a particular day—feels like searching for a grain of sand on a beach. Traditional data lakes are built for sweeping scans, not surgical strikes.

Spotify hit this same wall, albeit with playlists instead of precipitation maps. Their solution, announced recently, is called Random Access Parquet, or RAP. It's an external index layer that sits on top of Apache Parquet files, letting you retrieve a single row without scanning thousands of files. For Earth sciences, the implications are immediate and practical.

Why Point Queries Stumble in Data Lakes

Data lakes have become the de facto home for massive analytical datasets. Tools like Trino and BigQuery excel at scanning entire datasets to compute aggregates or train models. But they choke on key-based lookups—the kind you'd use to answer "what was the temperature at this station at 2 PM on July 4th?"

The problem is overhead. Even when cloud storage offers millisecond access to individual objects, the query planner has to figure out which files to read, then traverse metadata, then discover the files. For a point query, that's a lot of work for a single record. Spotify notes that they store petabytes in Bigtable for online serving and exabytes in a Google Cloud Storage data lake. Copying data between those systems at that scale is expensive—both in dollars and in time.

RAP: An Index Layer for Parquet

RAP attacks the problem by adding an external index that maps query keys—like a user ID, or in our case, a sensor ID or geographic coordinate—directly to the Parquet file and the row position within that file. When a query comes in, it first hits the index, then performs a targeted range read from object storage. No more scanning thousands of files.

The index is built as data is written to Apache Iceberg tables. It generates append-only index fragments, leaving the immutable Parquet files untouched. This is a big deal. It means the same dataset can serve both analytical workloads (like running a climate model) and online applications (like a real-time air quality dashboard) without duplicating storage.

What This Means for Earth Science Workflows

Think about a typical Earth science pipeline. You ingest satellite data into a data lake. You run big batch analytics to produce mosaics or time series. But then you also want to serve individual tiles to a web app, or let a researcher query a specific pixel's history. Traditionally, you'd have to copy the data to a separate database for that kind of access.

With RAP, you could skip the copy. The same Parquet files work for both. That's not just a convenience; it's a shift in how we design data systems for scientific research. You could keep all your data in one place, with one schema, and still get low-latency point queries.

Sorting and Layout: The Hidden Hero

Spotify also shared several storage layout tricks that make point queries even faster. One is sorting data by query key—so if you often query by station ID, you sort your files so records for the same station live close together. That reduces the number of files you need to touch. Another is interleaving value columns so that multiple attributes can be read in a single continuous read. For example, if you often query both temperature and humidity together, you can store those columns interleaved so a single read gets both.

These techniques add a little extra size to the files or index, but they can slash the number of storage operations. In some cases, a point query can be satisfied with just a few kilobytes read. That's a huge win for interactive applications.

Secondary Indexes and Flexible Query Dimensions

Primary indexes are great for a single key, but what if you need to query by different dimensions? For Earth sciences, that's common—maybe you want to query by location, by time, or by sensor type. Spotify's RAP supports secondary indexes without rewriting the Parquet files. You can add an index on any column, like a buyer ID or seller ID in their case, or a latitude/longitude pair in ours.

Hash-based indexes handle exact-match queries, while sorted indexes support range queries. These secondary indexes are managed at the serving layer, so you can add new access paths without changing your data pipeline. The underlying Parquet dataset remains the same for analytical scans and interactive point queries.

Z-ordering and Hilbert curves are also mentioned as ways to improve data locality for secondary query dimensions. These are spatial indexing techniques that map multi-dimensional data to a single dimension while preserving proximity. For geospatial data, that's a natural fit.

A Step Toward Unified Data Serving

Spotify's RAP is part of a broader industry push to make data lakes serve more than just analytics. Google Cloud recently described a lakehouse architecture based on Apache Iceberg that aims to reduce data duplication for AI applications. RAP takes a different route: it adds a dedicated external index optimized for point queries while staying compatible with existing Parquet files and Iceberg tables.

The data engineering community is taking notice. Some see RAP as a prime example of extending open data formats to support interactive workloads. Others point out that as cloud object storage gets faster, the bottleneck shifts to query planning and metadata access—and RAP attacks exactly that by precomputing indexes.

Practical Considerations for Earth Science Teams

If you're running a research lab or a data-centric Earth science project, here's what you might take away. First, you don't have to choose between analytical depth and operational speed. With RAP, you can have both on the same dataset. Second, the storage layout techniques are worth studying even if you don't adopt RAP itself. Sorting by key and interleaving columns can improve query performance regardless of your serving layer.

But RAP is not a silver bullet. It adds an index layer that needs to be maintained and queried. It's designed for point queries, not for complex joins or full-text search. And while it works with open formats like Parquet and Iceberg, you'll need to integrate it into your existing stack.

Still, the direction is promising. Data lakes are becoming more than just dumping grounds for analytics. They're evolving into active, serving-grade systems. For Earth sciences, where datasets are massive and queries are often precise, that evolution is welcome.

One thing is clear: the days of copying petabytes just to serve a single record may be numbered. And that's a good thing for anyone who's ever waited on a query to return a single measurement from a mountain of data.