TL;DR: Get all the alpha from Alexandre at StreamingFast as he shares the team’s areas of focus for this quarter: optimizing Substreams onboarding time, increasing the number of Substreams-powered subgraphs, and understanding Substreams user personas.
Opening remarks
Hello everyone, and welcome to episode 168 of Indexer Office Hours!
GRTiQ 179
Catch the GRTiQ Podcast with Goncalo Sa, co-founder and security researcher of ConsenSys Diligence. ConsenSys Diligence is a comprehensive smart contract audit service that helps startups and enterprises launch and maintain their Ethereum blockchain applications.
Repo watch
The latest updates to important repositories
Execution Layer Clients
- Erigon New release:
- v2.60.5 :
- Improvements include enhanced diagnostics, support for hex-encoded Nonce, enforced gas configuration for Polygon chains, and increased RPC throughput, while bug fixes address TxPool growth and logCount parameter management.
- v2.60.5 :
Ana | GraphOps: It looks like there hasn’t been an update on Erigon v3. I’m not sure if Payne has an update on that, as he is testing it.
[Bug] Some subgraphs can’t be synced under Erigon V3 and OP_Erigon #5558
Payne | Stake🦑Squid posted: v3 forked but was fixed in main 🫡 yet to test things, planning for this week
- sfeth/fireeth: New release:
- v2.6.7 :
- Bump Substreams to v1.9.3: fix high CPU usage on tier1 caused by a bad error handling.
- v2.6.7 :
- Avalanche: New release v1.11.10 :
- This version, compatible with v1.11.0, updates the plugin version to 36 and requires all plugins to update for compatibility.
- Key changes include renaming metrics, fixing block building timer recalculations, refactoring metrics registration, adding reward calculator usage examples, and improving error handling.
- Additional updates involve restructuring the fee calculator API, enabling more flexible CI monitoring, and various other enhancements and bug fixes.
Consensus Layer Clients
Information on the different clients
- Nimbus: New release v24.7.0 :
- Nimbus v24.7.0 is a low-urgency release featuring beacon API improvements and bug fixes. Key updates include support for the publishBlindedBlockV2 API endpoint, improved block proposal rewards without pre-aggregated sync contributions, and fixes for SSZ decoding and parameter handling in the beacon API.
- Additionally, this release restores Sepolia bootnodes and ensures Keymanager API fee recipient changes propagate to builder API relays. Users are advised to update at their convenience within the normal two-week cycle.
Protocol watch
The latest updates on important changes to the protocol
Forum Governance
Forum Research
Core dev updates:
- The Guild July 2024 Update
- GraphOps Update July 2024
- Pinax July 2024 Update
- Edge & Node’s July/August Update
- Semiotic July 2024 Update
- Messari July 2024 Update
- StreamingFast July 2024 Update
- Geo July 2024 Update
Matthew | Pinax posted: Always some ALPHA in there!
Contracts Repository
Open discussion
Core dev update with Alexandre Bourget from StreamingFast
Note: Comments have been lightly edited and condensed.
Abel | GraphOps: One way we’re trying to ensure a tighter feedback loop and connection between core developers and indexers is to bring core developers to IOH to speak in detail about some of their updates and projects.
Today, we are very fortunate to have Alex from StreamingFast to tell us more about what the team has been working on and share some alpha.
Alexandre B | StreamingFast: I could start by reviewing the updates from this month and what we’re working on this quarter. We’ve distilled our objectives and key results, and I want to tell you what we’re focusing on right now. Remember, we have a small team of eight engineers, and we need to focus.
Here are our objectives and key results for the quarter.
Optimize Substreams onboarding time
One of the areas the team is focusing on this quarter is to shrink the onboarding time for new Substreams users.
Listen to Alex speak about how they’re planning to simplify the process of getting up and running with Substreams at 9:11 of the recording.
Increase the number of Substreams-powered subgraphs
Similar to the previous goal, they want to simplify the steps to code and release a Substreams-powered subgraph [Timestamp 10:10].
Clarify and understand our Substreams users (personas)
Zero in on the personas to identify who are the users of Substreams and who benefits most from Substreams, both qualitatively and quantitatively.
Listen to Alex speak about their plans to gather this information at 10:47 of the recording.
Methods for quicker onboarding
Alex asked himself what has changed in technology that would make it simpler for people to onboard to Substreams, and he thought of when they set up a Gitpod to create a development environment.
Gitpod is a solution that allows developers to be always ready to code through automated and standardized development environments.
Keep listening to hear Alex tell the story of how Shopify shifted its development environment into containers and cloud-based IDEs.
What’s the plan?
In light of this, how could we simplify the developer experience for Substreams?
The team is working on having a fully contained developer environment in a container running the Substreams GUI and init, the Postgres database, the Graph Node, and whatever other services are necessary. A fully contained dev environment running all the necessary services, all preinstalled, that can be started with one click, so people can see the potential and power.
GitHub Codespaces is a really powerful tool that we can leverage.
GitHub Codespaces gets you up and coding faster with fully configured, secure cloud development environments native to GitHub.
Questions?
Jim | Wavefive: Who’s thinking about the indexer infrastructure-running customer of Substreams and Firehose from the same point of view?
Listen to Alex’s answer at 16:55 on the importance of StreamingFast’s ability to focus on the developer experience this quarter and that GraphOps is working on integrating this into Launchpad. We need to foster the demand for Substreams from developers and get adoption.
From the chat:
Matthew Darwin | Pinax: GraphOps is working on that with Launchpad.
Ana | GraphOps: We are indeed working to add first Firehose and then Substreams support to Launchpad with the amazing support of Pinax.
Abel | GraphOps: Could you share any rough ETA for this support?
Ana | GraphOps: I don’t have an exact ETA, but I can say we are testing Firehose support now and will very soon release a v1 of the Firehose chart—most likely in a few weeks.
Jim | Wavefive: I’m certainly not suggesting that it’s on you and your eight engineers. It’s a collective effort. It’s good to hear GraphOps is working on that.
Alex: We value and will support anyone who wants to do some deployments. The code is open source, and we will share our knowledge. We will help anyone who wants to try. It’s always in the back of our minds how to scale the backend and have the indexers properly serviced; we’re just not focused on that this quarter.
Modifications to Substreams
Listen to Alex talk about the team’s work on:
- Building an interface so you can discover the contents of the Substreams packages on substreams.dev
- Simplifying the creation of Substreams modules
- Rolling out the base blockchain with extended trace data
Storage costs
- Optimizing storage costs and digging into resource uses for a Substreams cluster
- They want to understand what makes sense in terms of pricing
- What’s the cost model when Substreams queries are used?
- They need to gather data to be able to price these things
- Would love to have collaboration from people with different setups
More StreamingFast updates
Alex discusses an opportunity for the development container to provide an improved subgraph development environment.
Alex talks about rolling out a substrate chain soon, creating a Substreams build command that simplifies the build steps, and working on foundational modules.
Matthew Darwin | Pinax posted: Yaro | Pinax has been playing around with Substreams<->subgraph development recently. He would be good to ask for input / beta test.
A world of data services
Mickey | E&N posted: As we add a world of data services, what kind of info would you want to know about those services as they’re being added to the protocol (compute, storage, whatever)?
Marc-André | Ellipfra: Any rough estimate is going to be useful, but best to have a chance to work with it early so we can build expertise before we need to aggressively scale it.
Mickey | E&N: I love this; good idea. We could partner with y’all for a closed beta.
GRT utility
Mickey | E&N posted: If y’all have creative ideas for how we can extend the function of GRT, let us know (like using GRT as a native gas token on an L3, using GRT as a token inside a dApp, that kinda thing). I know this isn’t an indexer-specific topic, but you all are smart, and any ideas are welcome, of course.
Alex: This is a design goal for Geo, the decentralized knowledge graph… GRT would be used for writing, the capacity to post data in this or that space. That would make it a write token, which is very interesting indeed. Yaniv is leading that project.
Marc-André | Ellipfra posted: Graph appchain on L3???
MoonBoi: GRT as a native gas token as an Avalanche layer 1. 🙂
Matthew Darwin | Pinax: MoonBoi I think Yaniv’s idea is GRT as the native gas token for Geo.
Vince | Nodeify: Bundle graph services with open source defi platforms or tools. Discount for using it on queries.
Jim | Wavefive: GRT in every single DeFi, borrow, lend platform you can see is probably the most high impact if you’re looking for utility for the coin.
stake-machine.eth: GRT on Peapods Finance? I just heard about it, no knowledge so far.
Jim | Wavefive: Imagine the things you could do if you could leverage up your GRT within the protocol, and in the future with data services.
Vince | Nodeify: Multichain staking could bring more use as well. Cross-chain query payments.
Firehose: performance metrics and testing
Listen as Alex explains how the direct Firehose into Graph Node integration can achieve a 3X performance increase (320%).
They recently retested this and discovered that Firehose as a backend, just for replacing JSON RPC, is always faster and cheaper.
Alex talks through some options for how to test Firehose.
Jim | Wavefive: Question for other indexers: how are you playing around with Firehose and Substreams? Are you doing that in production?
Matthew Darwin | Pinax posted: From our side, we do testing on the “testnet.”
Matthew elaborated: We actually have more than two indexers, but we have one running on the testnet and one running on the mainnet. So if we want to test anything, like a brand new chain, for example, which we do a lot because we’re doing the testing for the chain integration process, we’re not hooking that up to the mainnet indexer. First we run it on the testnet using a testnet RPC, testnet Firehose, testnet Substreams, and then once it’s all good, we deploy it on mainnet.
If you want to reduce the unintended consequences, then set up a separate Graph Node stack, where you can test some subgraphs. There are subgraphs published for all the new chains on the testnet. That’s work that InfraDAO does.
I personally haven’t set up a single shard because we started with Firehose before we started with Graph Node, so we’re coming at it from the other direction.
Vince | Nodeify: Having a Substreams shard would make the most sense, you won’t get the same volume to test.
Vincent | Data Nexus: It’s more of a per-graph-node deployment than a per-shard deployment when doing a half Firehose setup. We’re currently doing that in order to sync Substreams while still using RPCs for everything else.
Marc-André | Ellipfra: For different reasons, I was going back and forth between RPC and Firehose at some point, and I didn’t encounter any problem, really. So try it, worst case, you rewind a bit, but I doubt it comes to this.
Matthew Darwin | Pinax: Hau | Pinax goes back and forth between Firehose and RPC.
How many subgraphs per Graph Node?
Payne: How many subgraphs do you have assigned per Graph Node?
Jim | Wavefive 🌊: 140 per, roughly.
Marc-André | Ellipfra: 200-250 subgraphs per graph-node.
Payne: I have 230–250 as well.
Vince | Nodeify: Until it screams, then back 10.
Marc-André | Ellipfra: But yes, I have to restart graph-node from time to time.
Questions about Postgres?
Mickey | E&N posted: We have an awesome new Postgres DBA at Edge & Node. Once he’s a little more onboarded, he can help answer indexer questions or do workshops with you guys, whatever is most helpful.
Jim | Wavefive: I would love that, Mickey. I have so many questions about Postgres, and it’s not my area of expertise. I think a lot of people would be the same.
Mickey | E&N: It would be super helpful if we could start collecting specific questions you have so we could do a more targeted workshop. Abel, could you help facilitate that?
Abel: Yes, I would love to.
No Comments