TL;DR: In episode 176 of The Graph Indexer Office Hours, key topics included indexers sharing their experiences with switching to the Timeline Allocation Protocol (TAP), troubleshooting issues with the indexer-agent during upgrades, and configuring ingress endpoints for the public indexer URL. Participants also delved into managing subgraph deployments across multiple nodes, utilizing indexing rules, and bringing to light an off-chain subgraph CLI bug. The session covered methods for verifying that indexer services are properly answering queries, finding Substreams-powered subgraphs, and configuring indexer setups. The meeting concluded with a preview of upcoming support for the Fuse network.
Opening remarks
Hello everyone, and welcome to episode 176 of Indexer Office Hours!
GRTiQ 187
Catch the GRTiQ Podcast with Eshan Chordia, Co-Founder at Lumino AI. Lumino AI builds decentralized AI solutions focused on making machine learning models more accessible and affordable. Eshan shares his journey into tech and web3, his time at Protocol Labs working on Filecoin’s cryptoeconomics team, and the origins behind Lumino AI. The podcast also explores the challenges and opportunities within the AI and web3 spaces, Eshan’s vision for decentralization, and why he believes that democratizing access to machine learning models is crucial for the future of tech.
Repo watch
The latest updates to important repositories
Execution Layer Clients
- sfgeth v2.0:
- geth v1.13.12 fh2.4
- Geth New release v1.14.9:
- This maintenance release introduces support for the new multicall spec, enabling the eth_simulateV1 call to sequentially execute multiple eth_calls across blocks, along with optional state and precompile overrides, improved verkle support, and various bug fixes and enhancements across the command line, core, txpool, and networking components. Breaking changes include the removal of the Goerli flag and configuration.
- sfeth/fireeth: New release v2.7.4:
- This release fixed an(other) substreams issue where multiple stores running on the same stage with different initialBlocks will fail to progress.
- Celo: New release v1.8.5:
- This release can be skipped for mainnet.
- The latest release introduces a new command-line flag, –l2migrationblock, which expects a (future) block number. When specified, the node will pause at l2migrationblock – 1 block. This feature is designed to assist chain operators with the migration to the new Cel2 architecture.
- Arbitrum-nitro New releases:
- v3.2.1:
- This release adds support for ArbOS 32 for validators.
- v3.2.0:
- This release includes security fixes with the introduction of ArbOS 32, which will be activated by an emergency upgrade from the Arbitrum Security Council, making this a required update. It adds support for ArbOS 32, custom native tracers, and several improvements related to WASM store management, Redis consumers, and validator functionality. Breaking changes include required version upgrades for security reasons and changes to the Docker entrypoint.
- v3.2.1:
Launchpad Stack
- New chart versions released with enhanced features and bug fixes:
- Erigon
- Nimbus
- Graph-network-indexer
- Firehose-ethereum
- New stable versions of Ethereum, Gnosis, Celo, Monitoring
Issues:
- Launchpad Charts Issues: View or report issues
- Launchpad Namespaces Issues: View or report issues
🔭 Protocol Watch
The latest updates on important changes to the protocol
- Contracts Repository
- Network Subgraphs
- Core network subgraph: Security hotfix for Curator data (v1.1.3)
📋 Project Watch
A detailed update from core developers on
- Indexer Experience Indexer Experience • graphprotocol
- Feedback survey on Project board and WIP: https://mi71abukjtv.typeform.com/to/iAWBrkLx
⛓️ CIP Watch
- Updates to the CIP process + next chains:
🧠 Open Discussion [6:24]
Abel: First I want to acknowledge that there is a new indexer who’s joined us. I thought it might be a good opportunity to ask if they have any questions.
At this point, Abel pulls up a whiteboard in Discord where participants post sticky notes with questions.
Status Update – Switching To TAP [8:32]
Stake-machine.eth asks: How many indexers have already switched over to TAP?
Ana | GraphOps: The experience is definitely a learning curve for a few reasons. First of all, TAP includes the new component tap-agent. There are a few things to be aware of when running tap-agent and the new Rust-based indexer service. In order for those two components to start they both need a config.toml. You can add every single config option as an environment variable, overwriting it, but you still need a config.toml, at least in my testing.
What’s important is that both indexer_service and indexer-tap-agent use the same configuration.
Apart from that, the indexer agent runs as before, except it needs to point to an escrow subgraph.
Then there’s the learning curve of checking whether receipts have been written into the indexer database, making sure that RAVs (request aggregate vouchers) have been created, things like that. There’s definitely a learning curve I would say, but once you get it running it’s quite stable. And the community is very responsive.
Vincent | Data Nexus [12:09]: We have been testing the TAP setup for a while with InfraDAO, and we finally made the switch for mainnet last week. Our experience was pretty smooth.
We did have some issues with the escrow contract not getting topped up so the TAP agent was starting to deny the gateway. But that was fixed pretty quickly.
After that, we had our RAV requests going through, which is the main thing to look for right now to verify that things are working. Unfortunately, there are no indexer service metrics yet, like queries by deployment, for instance. That would probably be our main feature request at this point for the TAP service.
But things are going pretty well. Excited to make this next step in The Graph’s decentralization.
Vincent | Data Nexus posted the TAP Agent Grafana dashboard in chat.
Abel: Ana says that it’s in the immediate pipeline, and it will be added after this week.
Vincent | Data Nexus: We were thinking through where our max willing to lose should be. As the network moves to decentralize, we want to set that at a point that we’re not going to get screwed over. But we’re not really worried about The Edge & Node gateway screwing us over. But if there were a new gateway in the future, we’d probably want to set that quite a bit lower.
So our thought was, it would be cool to have a feature where we can set the max willing to lose per gateway. That way, for more trusted gateways, we could have that set higher. Whereas, if there’s a gateway that we do not trust, we can set that a lot lower. Right now, there’s only one gateway, so that’s not really a big sticking point. Just something that we’d like to see for the future.
Stake-machine.eth shared his experience: Unfortunately, I couldn’t upgrade. I tried to upgrade to the latest index version and had an issue, so downgraded back to v0.20.23. So that’s why I’m asking, because I’m willing to upgrade, but there is a lack of documentation, so I don’t know what to do next.
Issues with indexer-agent [17:30]
Vincent | Data Nexus: You want to close out all of your allocations and just reset everything there. If you’re having a problem with upgrading the indexer agent, the best case for you might be to close out all of your allocations and then completely wipe the agent database. Do this after all your query receipts are put through so you’re not losing out on any queries.
Once all that’s done, just wipe the database and start up a new agent database on the new version. I’m guessing it’s a migration error or something like that, so you should be good at that point starting from scratch, and then you can implement the indexer-service-RS and the agent. Then everything should work at that point. So that’s probably the direction I’d go if I were having an issue like that.
Stake-machine.eth responded in chat:
All allocations were closed, but we didn’t wipe the database. We will try to upgrade to v0.21.3 instead of v0.21.4.
Ingress Endpoints [19:09]
Pierre | Chain-Insights.ai asks on the board: Do I need to create an ingress for the public indexer7600 URL, and to what service do I connect it? I didn’t see anything in the Helm charts.
Ana | GraphOps: For that public indexer flag, yes you do need to create an ingress endpoint, and you should point to the graph network indexer service endpoint. So in your graph-network-indexer-service on the 76000 Port. I’m not 100% sure if it’s absolutely required. I think it is.
Inter-Node Subgraph Deployments [23:21]
Ernani.eth asks on the board: I’m setting up a distributed system where we have multiple indexers to index our subgraphs. But I’m curious, how does one node communicate with other nodes? I mean, if I deploy one subgraph to a node, but another node is responsible for indexing it (configured via deployment rules), how is that subgraph sent to the appropriate node? Does everything happen on Postgres? Is there any TCP-based pub/sub in place?
Vincent | Data Nexus replied in voice chat: So, how to set a graph node for sending all the deployment instructions to the indexer agent.
When you send that deployment instruction, that graph node will go through the deployment rules and assign the subgraph to index, to itself, or potentially to another graph node. Then that goes into the deployment table in the database, and all of the graph nodes are looking for deployments that they’re supposed to index. So when that’s updated, the graph node that was assigned that subgraph will start indexing it.
Indexing Rules [26:32]
Gemma | LunaNova asked in the chat: Do the indexing rules actually work now? They definitely didn’t a while back…
Vincent | Data Nexus responded in voice: We’ve had those rules set up for quite a while, and they weren’t working. The reason was because when you sent the request to a graph node to assign a subgraph, it would go through the deployment rules and assign subgraphs based on the deployment rules. But then the indexer agent, in the next step, would reassign the graph node that’s supposed to be indexing it to a random graph node that was in the list of graph nodes that you give the indexer agent.
I put in a pull request a while ago to remove that second step, so it’s not reassigning to a random graph node anymore. So the deployment rules are actually working at this point. So you can set those up in the graph-node config and then assign programmatically based on things like by chain or by regex and a few other things.
Off-chain Subgraph CLI Bug [28:14]
Ana | GraphOps: There is the option for off-chain subgraphs. I know it works to do it using indexing rules off-chain, but there’s also the CLI and the environment variable. I noticed that the indexer agent doesn’t work with the CLI option. I’m wondering if that’s just a bug that nobody else has come across, or if there’s something else I should know.
Vincent | Data Nexus: That may potentially be a bug. We’ve been using just the indexer agent API exclusively. So there may be some specific issue with the CLI option. From our experience, when we set an off-chain subgraph by the API, it is assigned to the correct graph nodes that we have set by the deployment rules.
Dashboards and Verifying Queries Are Answered [33:00]
John K. asked on the whiteboard: How do I ensure my indexer service answers queries properly? From my end it looks ok, is there a dashboard or something to check?
Jim | Wavefive responded: There’s a basic dashboard that I created a long time ago in the StakeSquid repo and it includes some metrics from the typescript indexer service that tells you if the gateway is querying you or not, for cost models. I often used to use that as a way of knowing whether I’m talking to the gateway or not.
The same dashboard will also show you if you’re getting any queries and if you have any allocations or even if you’re just running an indexer on-chain. If you are on-chain, then at some point, the gateway should try and hit your status endpoint, and you should see metrics on that dashboard.
If you want to look at the raw metrics, you can look at Port 7300/metrics on the service and you’ll see a bunch of stuff there. If you want it to make a bit more sense, then have a look at the indexer query dashboard in the StakeSquid repo under Grafana dashboards folder.
Ana | GraphOps added: You can also tell if you’re looking at the TAP Grafana dashboard. If you’re getting new receipts created, that’s good. New receipts equal you’re getting queries served.
Finding Substreams-Powered Subgraphs [37:07]
John K. asked on the whiteboard: Is there a way to find Substreams-based subgraphs in Graph Explorer?
Jim | Wavefive chimed in: Some of them say Substreams in the name. If you just search Substream on the Explorer you’ll find some. Obviously, it’s not a very specific way of finding them.
Vince | Data Nexus also posted in chat: Network subgraph query for Substreams-Powered subgraphs:
{
subgraphDeploymentManifests(where: {poweredBySubstreams: true}) {
poweredBySubstreams
id
network
schemaIpfsHash
startBlock
}
}
Indexer Setup [44:52]
Pierre | Chain-Insights.ai asked in the chat: How do I configure the subgraph to be indexed? I configured everything, I have my address, everything is running, but how do I index subgraphs? Not my subgraph, but the others, so I can get queries. I didn’t find any documentation yet.
Jim | Wavefive responded: There’s definitely a bunch of indexing documentation. Actions queue, here we go. Let me link this. My advice though, Pierre, would be if you want to have a relatively pleasant experience in figuring out what to what to index, and even to interact with your indexer, indexer-tools written by Vincent from Data Nexus is a really great place to start. You do need to give it access to your indexer-agent API, and it needs to be over an SSL connection. If that’s beyond your ability, I wouldn’t I wouldn’t worry too much about doing that.
There’s actually an IOH where Derek and Vincent go through a demo of indexer tools. That’s where I would start. It’s like a sort of single pane of glass for looking at both your indexer and the network. Choose which chains you’re interested in, seeing where the signal is. You can sort by signal, sort by query fees, sort by allocation, whatever you like. That’s how I operate my indexers day-to-day now, all via that web interface.
Pierre | Chain-Insights.ai followed up: That’s what I want, but I didn’t find where to do that.
Jim | Wavefive: If you just want to do it the basic way, then you would need to use the indexer-cli and talk to your indexer-agent with it using the indexer actions CLI. I’ve linked specifically to that CLI in the documentation. Indexer tools is basically using the same methods to create these cues on your agent and execute them.
Another great thing you can do with indexer tools is, if you just want to test whether or not you can sync something, one of the options in the list is the off-chain syncing manager. If you want to pick any subgraph just to test and see if your indexing is working, you can use that to add one without doing any on-chain activity at all.
Vincent | Data Nexus added: There’s also the status part of the table. If you click on one of the icons there will be a sync and a not sync button. That’ll show up after you connect to your indexer agent. It doesn’t show up when it’s not connected. Once you are connected, on that little status drop-down, there’ll be two different buttons for starting to sync or stopping sync off-chain. And off-chain means you’re just indexing, but you’re not allocating to it with your GRT that you have either self-staked or delegated.
Pierre | Chain-Insights.ai continued with clarification: When I want to do that, can I also do it with this tool? If I want to allocate.
Ana | GraphOps posted a link to the allocations documentation in chat.
Vincent | Data Nexus responded: That’ll be through the allocation wizard. You’ll be able to choose which ones to close, which ones to open. Then the last page will give you the actual CLI options if you just want to do that in the meantime. Then once you’re connected, there’ll just be a few buttons to go through with that automatically, and it’ll use the API to cue those actions.
Closing Remaks [55:15]
Abel closed the show with some updates: A little bit of alpha, we will be talking about Fuse next week. Recently we put out support for Fuse. If you have any questions about what it means to spin up a node in the Fuse network, the team will be joining us to discuss that.
No Comments