The Great Blob Rescue: How Pinax and The Graph Saved the Day!

Pull back the curtain and discover how Pinax helped ensure long-term blob storage after EIP-4844.

TL;DR: Thanks to quick work from the Pinax team and a flexible, performant software built by StreamingFast, The Graph saved the blobs from centralized doom.

The Ethereum Improvement Proposal 4844 (EIP-4844), colloquially known as the “blobs” or “proto-danksharding” proposal, enhances Ethereum’s scalability by enabling temporary bulk data storage, a critical component for the network’s future expansion.

However, these blobs are inherently ephemeral, programmed to disappear after 18 days unless preserved.

Peek behind the scenes at how Pinax and The Graph saved the blobs in a testament to web3 collaboration.

Who will save the blobs?

Ethereum’s approach to blob storage was summarized in a bullet point nestled in the FAQs of eip4844.com. Among other methods for storing the full history, the FAQ suggested:

Third-party indexing protocols like The Graph will likely store the full history.

This expectation urged The Graph core developers to a unified goal:

Save the blobs.

Developers of decentralized applications (dApps) need blobs saved for dApp data, and Layer-2 rollups need blobs saved for rollup history. The Graph seemed an obvious enabler for saving these blobs in a decentralized way, aligning with Ethereum’s decentralized ethos.

Pinax rises to the challenge

The core devs working on The Graph discussed several approaches to saving the blobs. Eventually, the teams settled on using Firehose and Substreams technology to store the blobs in Firehose flat files.

Fueled by a team intimately familiar with Firehose and Substreams, Pinax kicked into high gear.

After core devs agreed on an approach, Pinax back-end developer Frederik Schöll adapted StreamingFast’s Firehose RPC poller for use with beacon chains. This foundational work ensured the basic tooling was available to save the blobs.

The quick deployment of the adapted RPC poller marked the team’s initial victory, but the work was far from over.

The ensuing rapid development phase saw enhancements like adding block timestamps and other metadata. In this cross-team environment, Pinax CIO Matthew Darwin facilitated coordination, tested and monitored solutions, orchestrated the work within the team, and provided daily updates on the most recent GitHub releases.

Pinax full-stack developer Yaro Shkvorets made blobs accessible by creating a beacon-compatible API that integrates with a subgraph, ensuring blob access for participants in The Graph Network.

The pesky timestamp

The project encountered some technical hurdles. During the testing phase, the Edge & Node team faced gateway syncing issues, with some nodes millions of blocks behind. The gateway logs provided some context but deepened the mystery, with transport errors and beacon blocks that did not appear on block explorers.

Pinax traced the error to mismatches in the “timestamp” fields, a problem exacerbated by the lack of a “timestamp” field in specifications for the Phase 0 and Altair hard forks.

After troubleshooting and collaboration, the teams proposed and successfully integrated an alternative method to derive timestamps, stabilizing the system and enabling blob streams in Substreams and storage in subgraphs.

You can see some of the results on the Pinax GitHub.

Upgrade executed

The Ethereum network executed the Dencun hard fork in epoch 269,568 on March 13, with the blobs saved from day zero.

This success was not just the result of Pinax and The Graph’s engineering efforts but a testament to the power of collaboration and swift action in the blockchain community. Together, the teams solved a pressing emergent issue without disruptions, ensuring Ethereum’s level-2 infrastructure remained decentralized. These cross-organization efforts exemplify the resilience and adaptability at the heart of the blockchain community.

What innovative uses can you imagine for blobs? Comment below with your most creative and groundbreaking concepts, or share feedback directly with The Graph! Your unique ideas can help shape the future of blobs.

Explore saved blobs or add your own memes at blob.fm/gallery!

💡 This article answers questions like:
- What is Ethereum Improvement Proposal 4844 (EIP-4844)?
- What are blobs (binary large objects) on Ethereum?
- How did Pinax and The Graph save the blobs?
- How does Firehose and Substreams technology store the blobs long-term?