TL;DR: Major indexers, including Pinax and Upgrade Indexer, have completed their TAP migration. Multiple indexers reported issues with Graph Node v0.36.0, including PostgreSQL performance problems and migration difficulties. It was noted that v0.36.0 is not yet ratified for production use. Upcoming sessions will feature Yaniv Tal discussing GRC-20: Knowledge Graph, Ricky on indexer analytics, and Vince covering Kubernetes and Launchpad. The community also discussed plans for a special episode to celebrate The Graph's 4th birthday on December 17.
Opening remarks
Hello everyone, and welcome to episode 186 of Indexer Office Hours!
GRTiQ 197
Catch the GRTiQ Podcast with Rebecca Liao, CEO and Co-founder at Saga. A Stanford economics graduate and Harvard Law alumna, Rebecca’s remarkable journey spans from corporate law to presidential politics, serving as policy advisor for both the Biden and Clinton campaigns.
⬇️ Skip straight to the open discussion ⬇️
Repo watch
The latest updates to important repositories
Execution Layer Clients
- Avalanche: New releases v1.12.0 :
- Date: 2024-12-02 22:26:05 UTC
- This upgrade includes major changes such as introducing dynamic fees on the P-Chain and activating Cancun EIPs on the C-Chain.
- All Etna supporting mainnet nodes must upgrade by 12 PM ET on December 16, 2024, to ensure compatibility and functionality. Failure to upgrade may lead to performance issues.
- Urgency indicator: Yellow
- Urgency reason: Important for maintaining network functionality.
Consensus Layer Clients
Information on the different clients
- Nimbus: New releases v24.11.0 :
- Date: 2024-11-30 02:45:00 UTC
- Nimbus v24.11.0 introduces performance enhancements and fixes for validator client configurations. Notably, it updates Holesky bootnodes and improves syncing performance, while also introducing an experimental sync method.
- Urgency indicator: Green
- Urgency reason: Low-priority update for convenience.
- Lighthouse: New releases v6.0.0 :
- Date: 2024-12-02 05:57:26 UTC
- Lighthouse v6.0.0 is a major release with breaking changes, including a one-way database migration and the removal of deprecated CLI flags.
- Users of Nethermind should avoid upgrading until compatibility is confirmed. New features include hierarchical state diffs and optimizations for beacon node performance, which enhance overall functionality.
- Urgency indicator: Yellow
- Urgency reason: Critical compatibility changes require attention.
Matthew Darwin | Pinax posted: Anyone try the new Lighthouse yet?
Ana: We [GraphOps] haven’t yet. I don’t know if anyone else has?
Launchpad Stack
- New chart versions released with enhanced features and bug fixes:
- graph-node-0.5.8 and graph-network-indexer-0.5.10
- New stable versions of Polygon, Graph, Arbitrum
Issues:
- Launchpad charts issues: View or report issues
- Launchpad namespaces issues: View or report issues
Protocol watch
The latest updates on important changes to the protocol
Forum Governance
- Request for Information about dispute GDR-20
- Request for information about disputes GDR-21
- Both of these disputes have resulted in slashes.
- The arbitration process, investigation, and decisions have become more efficient.
- GRC-20: Knowledge Graph
- Check out new comments and the discussion on The Graph Forum post.
Contracts Repository
- TRST audit fixes for data service contracts #1076 (open)
- TRST audit fixes for DisputeManager contract #1075 (open)
- TRST audit fixes for Subgraph Service contracts #1074 (open)
- TRST audit fixes for HorizonStaking contract #1073 (open)
- TRST audit fixes for Graph Payments contracts #1072 (open)
Project watch
Final reminder to upgrade to TAP
At the time of publication, the deadline to upgrade has passed. All indexers should have upgraded to Timeline Aggregation Protocol (TAP), the new micro-payments system for queries on The Graph.
Open discussion [10:00]
TAP Migration
We’ll continue the discussion on the TAP migration. Ana from GraphOps and Carlos from Semiotic Labs are here to answer any questions that indexers may have.
chris.eth | GraphOps posted: Check your Arbitrum One Indexers’ TAP status on GraphSeer.
[The two big indexers that hadn’t yet upgraded last week have now completed their migration: Pinax and Upgrade Indexer.]
Matthew Darwin | Pinax: We’re upgraded this week.
As is Edge & Node’s Upgrade Indexer:
Ana: Someone from Edge & Node mentioned one interesting bug. They should let us know if they see it repeated or if they can reproduce it. The bug was that the Scalar-related databases weren’t created by default when upgrading to the Rust version.
Refer to The Graph Indexer Office Hours #184 for the migration process and resources.
Storage and Snapshots
Matthew Darwin | Pinax posted: Pinax storage cluster… 1/2
Matthew: I was just counteracting someone’s earlier comment that we’re all using tiny computers, so had to put a server picture.
Matthew: We [Pinax] have a lot of storage. There are two racks beside each other, this is one of them. These are all old Dell R720 servers that are supporting Ceph. All the disks are exposed as S3 object storage in Ceph. Mainly, it’s storing two things: one is Firehose blocks, obviously we don’t need that much space for Firehose blocks, even though Solana is huge. Mostly, it’s backups, or snapshots, as we like to call them in the web3 world. We have a lot of snapshots for all the different chains, so in case things break, we can get a snapshot from a few days ago, restore back machines, and get up and running again. Whenever you guys ask us for snapshots, we do have them already prepared, it’s just a matter of making them available. Indexers, do not hesitate to ask us for a snapshot. This offer is only for indexers.
chris.eth | GraphOps:
Matthew Darwin | Pinax: Solana is 53 TiB of Firehose blocks.
Abel | GraphOps: Final call for TAP-related questions before we move on to the next discussion topic.
Future Suggestions and Feedback Gathering [20:46]
- What topics do you want to see in the future?
- What suggestions do you have to improve IOH?
- How can we make sure we’re meeting indexers’ needs?
Upcoming Topics
Yaniv Tal, founder of Geo, is coming to IOH next week to discuss GRC-20: Knowledge Graph, a new standard for storing and representing knowledge on The Graph. It can be used by any application that wants to create and consume interoperable information across applications.
- Submit your questions for Yaniv to answer next week.
Ricky, data scientist, is returning to talk about indexer analytics.
- Refer to the IOH #179 recap for Ricky’s earlier session.
Vince from Nodeify [and Pinax] is returning to talk about Kubernetes and Launchpad.
- Refer to the IOH #182 recap for Vince’s earlier session on Talos Linux.
Feedback [29:28]
Louis Bellier | Pinax: Marketing proposal to make IOH better: have a calendar or planning done a little earlier to make more promotion possible. Ask each indexer to promote the content on socials. Or create a pipeline for it.
Matthew Darwin | Pinax: Pipeline: So you know what’s coming up in future episodes and can promote on socials.
chris.eth | GraphOps: Shill us your AI workflow tools.
- Louis Bellier | Pinax: I can, but no need for AI for this, a simple workflow or calendar will be a huge update to what we have now.
Jana | Pinax posted an example of a promotion for Builders Office Hours:
Louis Bellier | Pinax: Example of what could be done in advance ^^^
pili | GraphOps: We already have the calendar for internal IOH planning purposes, maybe you can see how to make the confirmed content public, Abel?
Vince | Nodeify: Is there a poll or anything on what subjects people want? Maybe on socials, too?
Abel: We don’t do polls. Based on these kinds of conversations and talking to people, I intuit what people want. I also listen to feedback and ask questions, but a poll could be a good way to understand people’s perspectives and opinions on what they want to hear at future IOHs and to keep people engaged on socials like Discord. That is a good suggestion and I will take that onboard.
The Graph’s 4th Birthday
Marc-André | Ellipfra: [The Graph] 4-year network anniversary is coming up.
Goldberg | StakeSquid: December 17th
pili | GraphOps: We can organize something like a panel or an open Q&A with OG indexers for the 17th.
Vince | Nodeify: Did we ever record the early days of IOH? Those were amazing, lol.
Abel: Yes, every single IOH has been recorded. They’re all on YouTube: #IndexerOfficeHours
NSun | Graphtronauts:
Matthew Darwin | Pinax: I’d love to hear about the early days of indexers. It’s before my time.
chris.eth | GraphOps: Jim | Wavefive 🌊 is the Founder of IOH.
Jim | Wavefive 🌊: Sure, I can [participate].
Abel: I’ll put together a panel with the OGs for December 17. It will be OGs coming back together, talking about the early days of indexing.
Graph Node Release [40:14]
This part of the discussion focused on how the latest Graph Node release, v0.36.0, has not yet been ratified as ready for production. Refer to the Feature Support Matrix for the specific graph node version range that has been approved for production.
Pierre | Chain-Insights.io posted: Hi guys, sorry, arrived late. Did we speak about Graph Node v0.36.0, is it good to go or does it need fixes?
Ana | GraphOps: What issues are you seeing? There was the dashboard change, which I’ve addressed in the chart, but wondering what other issues you’re seeing? On our side, I haven’t seen any errors or issues. However, we are running v0.36.0 on testnet only.
Payne | Stake🦑Squid: I would avoid 0.36 for now. It’s full of bugs.
Pierre | Chain-Insights.io: Ok, just that my PG goes down with too much load, but this is on my side, I guess.
Ana: Apart from PostgreSQL performance issues, what bugs are you seeing, Payne?
Payne: You have to dig into the change log to find some environment variable that nobody explained… still nobody can explain what it does. I couldn’t update properly between 0.35 and 0.36. There are a bunch of issues that need to be fixed. I know they’re working on a new update to fix these bugs. Also, you can’t downgrade to 0.35. You have to run a bunch of down migrations, and I’m not sure what else they affect in the long term.
Pierre | Chain-Insights.io: Yes, I did run the down scripts to go back. Working so far.
Matthew Darwin | Pinax: What bugs, specifically? Firehose Arbitrum thing: [Bug] arbitrum-one not working #5713
Hau | Pinax: Indexer crash looping Store error: store error: column subgraph_deployment.synced does not exist"
Matthew Darwin | Pinax: That was you downgrading to 0.35.
Hau | Pinax: Nov 28 10:25:58.352 ERRO Unable to get a connection for block ingestor, err: no providers available for chain 'fantom', component: EthereumFirehoseBlockIngestor
(this is just 1, actual is multiple chains)
Nov 28 10:25:58.363 ERRO Stream blocks complete unexpectedly, expecting stream to always stream blocks, component: SubstreamsBlockIngestor Nov 28 10:25:58.366 ERRO Unable to get a connection for block ingestor, err: all providers failed for chain 'scroll', component: EthereumFirehoseBlockIngestor
chris.eth | GraphOps: See the Feature Support Matrix. 0.36.0 is not ratified as ready for production. If you are running 0.36.0 in production, you do so at your own risk.
- Payne | Stake🦑Squid: The Council doesn’t ratify versions anymore.
- chris.eth | GraphOps: That’s true, however, versions are still ratified, but by the graph node team.
Vincent | Data Nexus: Any graph-nodes not ratified need to be released as pre-release only…
- chris.eth | GraphOps: I agree that makes sense, but the rules are the rules. Mickey | The Graph | E&N, could you take this request to the graph-node team? Is it possible to mark a pre-release as a release after it’s made?
- Chris elaborated: I think it’s a very reasonable request that releases on GitHub aren’t marked as the latest release until they’re ratified to run in production. For reference, the Feature Support Matrix specifies a graph node version range, which has been approved for production.
Vince | Nodeify: What’s the process of ratified? Does that require indexers to do so at their own risk, feedback needs to happen if no major testing before release does.
- chris.eth | GraphOps: An update to the feature matrix by the graph-node team.
Vince | Nodeify: But I mean how does the team reach that conclusion?
- chris.eth | GraphOps: I’m not sure. I will add this to the follow-up with Mickey.
chris.eth | GraphOps: Abel | GraphOps, I propose a session with the graph-node team to cover off the concerns mentioned in the chat above:
- How does the graph node team reach consensus that a release is to be ratified as ready?
- How can we speed up the process?
- How can we improve the clarity of comms? Too many places that list the version number (e.g. GitHub releases, feature support matrix, networks.md)
I will loop you [Abel] into the message with Mickey | The Graph | E&N.
Additional Topics [44:53]
Abel closed IOH by making sure he understood what people wanted to learn from the topics suggested on the whiteboard.
How to scale Substreams (Matthew added)
- Matthew Darwin | Pinax: I’m happy to talk about it.
How to scale and operate owned infrastructure (Pili added)
- pili | GraphOps: To understand what others are doing.
More core developer updates and participation (Pili added)
- Abel: This has come up before. We kind of have been doing that periodically, but I need to press core devs more. We need to have a regular session dedicated to tightening the connection between core devs and indexers.
- Matthew Darwin | Pinax: Pinax is here every week. Feel free to ask questions. 😉
Substreams query fees
- Josh Kauffman | StreamingFast.io: I didn’t write it, but I think it’s to do with fees that we’ve collected from serving Substreams directly to consumers, and we’ve passed those query fees over the network (so that protocol tax was paid, and so that The Graph product usage is being accounted for and shown to the world).
- pili | GraphOps: For context: StreamingFast Substreams & Firehose revenues for May 2024
- chris.eth | GraphOps: It’s probably worth having a session to discuss in more depth with indexers, Abel | GraphOps.
Indexer resource for potential new indexers
- Josh Kauffman | StreamingFast.io: An interesting episode, but harder to organize: find people who are THINKING about becoming an indexer, find out all of their questions, what they are having trouble getting answered, what’s keeping them from doing it already, etc. Build out a community resource from that. It may be best done async, but it may help with the discussion.
No Comments