Paging Through a Parquet File in DuckDB: File_row_number or Offset?

You have a large Parquet file and an API that has to return its contents, but responses have a size ceiling, so the caller pages through it. LIMIT/OFFSET is the obvious way to write that and it is the wrong one. I measured file_row_number against it: 2.53x faster on a file with 163 row groups. Speed is the boring half of the answer. OFFSET will also hand back the wrong rows without telling you.
36 points by rustyconover 84 days ago rusty.today 3 comments

Summary

Using a large Parquet file with an API can be tricky due to size limits on responses. The article highlights how using file_row_number instead of OFFSET can significantly improve performance and accuracy when paging through data.

Comments (3 of ) ⌛

Loading more comments... ⌛
Scroll down to load more comments
No comments available.
Type to search HN and engineering blog articles
Press Escape to close  |  ↑↓ to navigate  |  Enter to open