TL;DR: In this recap of IOH #162, you’ll find links to core dev updates and excerpts from a panel celebrating the Sunrise of decentralized data. Learn how Sunrise will impact indexers and the future of The Graph.
Opening remarks
Hello everyone, and welcome to episode 162 of Indexer Office Hours!
GRTiQ 173
Catch the GRTiQ Podcast with Ruallen Chai, CEO and co-founder of IoTeX and MachineFi.
- IoTeX is a decentralized blockchain platform tailored for the Internet of Things (IoT). It aims to enhance the security, privacy, and efficiency of IoT device interactions and data exchange.
- MachineFi, an innovative concept introduced by IoTeX, merges decentralized finance (DeFi) with IoT devices, enabling machines to participate in financial activities and generate revenue.
Repo watch
The latest updates to important repositories
Execution Layer Clients
- sfeth/fireeth: New release v2.6.2 :
- Added substreams back-filler to populate cache for live requests when the blocks become final.
- Fixed: truncate very long details on error messages to prevent them from disappearing when behind a (misbehaving) load-balancer.
- Avalanche: New release v1.11.8 :
- This version is backward compatible to v1.11.0. It is optional but encouraged.
- Redesigned metrics to use labels rather than custom namespaces.
- Arbitrum-nitro New releases:
- v3.0.1 :
- -node.parent-chain.wallet.* is no longer available. Keys for the batch poster and validator must now be specified with -node.batch-poster.parent-chain-wallet.* and -node.staker.parent-chain-wallet.* respectively.
- The staker and batch poster must be run with different keys. Previously, this would produce unexpected errors as they tried to make transactions, but this will now produce a hard error on startup.
- v3.0.2 :
- This release fixes a crash on startup in v3.0.1, along with fixing the consensus-v30 present in Docker for validators, and an issue with the STOP opcode in the flatCallTracer.
- v3.0.1 :
Graph Orchestration Tooling
Join us every other Wednesday at 5 PM UTC for Launchpad Office Hours and get the latest updates on running Launchpad.
The next one is on July 3. Bring all your questions!
Protocol watch
The latest updates on important changes to the protocol
Forum Governance
- Add Support for Inco
- Add Injective Support
- Matthew posted: FYI: Injective is integrated to Graph Node using Substreams (unlike older Cosmos-based chains, which use Firehose). See Substreams for Injective Developers.
Forum Research
Core dev updates:
- GraphOps Update June 2024
- Pinax June 2024 Update
- Edge & Node’s June/July Update
- Semiotic June 2024 Update
- The Guild June 2024 Update
- Messari June 2024 Update
- Geo June 2024 Update
Contracts Repository
- ❄️ Graph Horizon and Subgraph Service ❄️ #944 (open)
- Graph Horizon: tests and internal review fixes #983 (merged)
Open discussion
Block cache options
A point was raised for discussion by Vincent (Data Nexus) and Payne (StakeSquid):
Can we revisit the idea of having the option to drop the call cache sometime soon? It doesn’t save much time and is a waste of expensive NVMe storage that is quickly being filled with new subgraph deployments.
Payne explained: Given that most indexers, if not all, are running their archive nodes right next to their database, there’s probably no need to store the block cache in the database. Essentially you’re doubling up the storage more or less for each chain just to store the blocks locally. This is not ideal as NVMe storage is expensive. We wonder if we should be looking into completely removing it or making it optional for indexers to run alongside their stack, and want to hear everyone’s thoughts on this.
Derek from Data Nexus posted: I’d be curious to see if we can benchmark these two.
Stake Machine posted: +1
Vince from Nodeify posted: Anything to reduce size with minimal performance impact, sign me up.
Vincent from Data Nexus commented: We still probably want to keep it around for the indexers using RPCs because that seems important for indexers running a stack like that.
Sunrise panel
Pranav, Marian, and Marcus joined from Edge & Node for a Sunrise panel discussion.
Note: Some comments have been lightly edited.
How will Sunrise impact large-scale indexing operations?
Pranav: All the queries from the hosted service are moving to The Graph Network, and for anyone running big, hefty indexers, there is good potential for them to make more in query fees. dApps that were leveraging the hosted service are now moving to the network, so any indexer that is large and has enough infrastructure to support big subgraphs will not just get indexing rewards but also a sizable number of the query fees. As more serious and bigger subgraphs with a lot of dApps and a lot of queries come on the network, there’s a good chance we’ll see a substantial increase in the query fees.
What are best practices for optimizing indexing strategies to ensure high performance?
Pranav: With the indexer selection algorithm up until now, most indexers have used particular configurations and they may want to consider making some tweaks as these serious subgraphs with a lot of entities start making more queries. You may have seen on Graph Explorer that we’ve changed the UI so that it shows the number of queries a subgraph is making and not just curation. Based on that, an indexer can make optimizations for curation and start to give more weight to query fees.
Marcus: If you’re able to reach out to any team or group that you know is using subgraphs and share resources on optimizing them, this will help your operations by educating subgraph developers.
- Subgraph Best Practice 1 – Improve Query Speed with Subgraph Pruning
- Subgraph Best Practice 2 – Improve Indexing and Query Responsiveness By Using @derivedFrom
- Subgraph Best Practice 3 – Improve Indexing and Query Performance by Using Immutable Entities and Bytes as IDs
- Subgraph Best Practice 4 – Improve Indexing Speed by Avoiding eth_calls
How is the development team thinking about scalability? What are some of the things the Edge & Node team and core developer teams are doing to ensure scalability?
Pranav: The hosted service was one indexer being run by Edge & Node, and The Graph Network is hundreds of indexers like you, running different subgraphs, so it is much more fault-proof. It becomes more fault-proof when there is more curation on a particular subgraph, and to help with that, we’re running a curation program. If you’re deploying a subgraph on The Graph Network and you’re going to make some queries, then you can contact any business, support, or devrel person at Edge & Node and tell them your subgraph needs curation, so that other indexers can start allocating on that particular subgraph.
We’re also working toward making the upgrade indexer more fault-proof.
The hosted service was the one that was not scalable to infinite. With The Graph Network, I think we’re on the way to scale infinite because the centralized point of failure with the hosted service has been eliminated.
pili from GraphOps: Did you see an increase in curation with the Sunrise?
Pranav: On the Arbitrum network, not a lot of the community is curating on The Graph Network right now. We’re trying to work toward that. Edge & Node is filling the gap and helping subgraphs with curation so more indexers allocate to those subgraphs.
We’re not seeing an increase in curation and people aren’t moving their curation from Ethereum to Arbitrum. If you have curation on Ethereum, please move it to Arbitrum because that’s where more active subgraph querying is happening. There’s a good amount of share of the queries you can get as a curator if you’re curating on a big dApp that’s making a lot of queries.
💡 Learn more about L2 Transfer Tools and how they can streamline the process of moving subgraphs, signal, stake, or delegation to Arbitrum One.
Alexis from Semiotic Labs posted:
Perhaps the “financial” opportunities related to curation should be more widely known outside of The Graph circles, i.e., show predicted APRs for curators based on historical queries, etc.
Any tips for indexers on how they can manage allocations with the increasing number of subgraphs in the network?
Pranav: More and more chains are becoming live on the network, and there is the potential for indexers to become specialists of a particular chain and, based on their knowledge of those dApps and that ecosystem, make more indexing rewards. Indexers can also reach out to Edge & Node and say that this particular subgraph has less curation and may need more because there are more queries happening.
How can we improve the communication and collaboration between technical teams and indexers in the wider community?
Marcus: We need to share what so many battle-tested indexers here have learned over the years getting The Graph Network off the ground. We’re in the Sunrise—we are in the sun, if you will—so it’s been quite a journey. I’m looking forward to this next phase, where the only option moving forward is The Graph Network, and indexers are going to be more visible and the stars of the show.
I’m thinking about how to talk with indexers and discussing with my internal team how to make your skills more visible and what you do more attractive to the world. If any indexers would like to talk about getting this knowledge share going and how to attract more indexers, please reach out to OxMarcus | E&N on Discord. I’d also like to learn more about how you communicate internally.
I’d like to see what we could do to make more people inside and outside the ecosystem aware of and interested in what you do as indexers. I’d like to make your career more attractive and make it easier for people to hop in and get started.
Marc-André from Ellipfra posted:
My wishlist in terms of collaboration: performance problem reports must be shared with indexers so they can fix items on their end. Edge & Node is well-positioned to receive these complaints, and indexers must hear them.
Marcus: I’ve been working primarily with subgraph developers up until now and I’m curious about the indexer flow and improving your experience. If anyone wants to talk further, I’m happy to.
How do you see the role of indexers evolving as The Graph ecosystem continues to grow and mature?
Marcus: Really, The Graph is the indexers. I mean yes, there are so many other participants, but if there’s no indexing, the roles aren’t as potent.
I’d like to see more understanding of what indexers do. We’re looking at ways to improve how people can onboard into a role within The Graph ecosystem, having that pathway curated for them. We need to communicate that The Graph is expanding, and as we push into this network, business opportunities are only going to improve.
We want to add more indexers to the network, make it look attractive, and make it relatively easy for people to get started.
Pranav: In my opinion, The Graph Network is one of the only networks that is sufficiently decentralized to position itself as one of the biggest compute networks. As things go along, and our scope widens with Semiotic’s vision, StreamingFast’s vision, Geo’s vision, and other core devs’ visions, the Graph Network will become a very big compute network and indexers will play an even bigger role in becoming their own siloed compute nodes, eventually powering the big compute house that’s required for different data services around the world.
Shout out to indexers
Marian and the team at Edge & Node are grateful to the many indexers who helped with the transition to the decentralized network by altruistically indexing subgraphs on different networks without any sort of rewards, indexers like Braindexer, Data Nexus, and Ellipfra.
Marian shared how the last two weeks went from the perspective of the Edge & Node business development (BD) team. Listen at 39:13 of the recording to hear about the BD team’s work and how much indexers helped with the transition.
What have indexers noticed in the last 2–4 weeks?
- Have you already seen substantial changes in your operations?
- What changed for you? Did you spot any patterns or face any challenges?
Listen to the responses at 41:02 of the recording.
What’s the next big thing on the horizon that the Edge & Node team wants to focus on?
Matthew posted: More chains?
Abel: It’s definitely more chains.
Hear what else Pranav has to say, including L1 deprecation and the future of curation at 50:30 of the recording.
No Comments