Algorithms on billion-scale graph using 10GB RAM: I love DataFusion
In a previous post, I was sceptical about using Apache DataFusion for graph analytics. However, after some thought and experimentation, I changed my mind. I have reconsidered my approach and now have a working implementation of basic graph algorithms, such as WCC, PageRank and MSSP. For example, I can identify weakly connected components in a graph with two billion edges using just 10 GB of RAM and two cores. This is one of the most widely used graph algorithms and lies behind most entity resolution. The pure CSR of this graph requires over 15 GB of raw memory (or 30 GB if it is treated as undirected, as WCC requires). The hard limit was 10 GB of RAM when using systemd-run -p MemoryMax=10G, and the process was successful. Not Vibe-coded.
Summary
After initial skepticism, the author shares their successful experience implementing key graph algorithms with Apache DataFusion, demonstrating its efficiency in handling large-scale data. With just 10 GB of RAM and two cores, they reveal how to analyze graphs with billions of edges, showcasing practical applications in entity resolution.
Comments (32 of ) ⌛