TL;DR: Indexers discuss moving to the new payment system (TAP), that they need to wait for the next indexer agent release before moving, and why not everyone should move at the same time. They also talked about the state of indexer documentation and the need to have one place to retrieve up-to-date information that aligns with releases. IOH welcomed two newcomers and heard about their experience with indexing so far.
Opening remarks
Hello everyone, and welcome to episode 177 of Indexer Office Hours!
GRTiQ 188
Catch the GRTiQ Podcast with Riad Wahby, co-founder and CEO at Cubist, a company dedicated to enhancing security and simplifying key management for web3 developers and businesses.
Repo watch
The latest updates to important repositories
Execution Layer Clients
- Erigon: New release v3.0.0-alpha4
- Ana | GraphOps: This release should only be tested on a testnet. I’m not sure if anyone has tested it yet.
Discussion from the chat
Johnathan | Pinax: We are testing BSC Erigon3 and Chapel Erigon 3. Ethereum will be the next. Erigon3 for BSC is ready now. The v1.2.* version will not support after the Bohr hard fork.
Vince | Nodeify: Make sure you rewind those subs, lots of forks last week if you missed them, or your POIs might be in outerspace.
Vincent | Data Nexus: Wait, Erigon2 will be unsupported for the next HF [hard fork]? We just finished the execution step syncing Erigon2 from scratch. 😭
- Geth New releases:
- v1.14.11 :
- A minor release aimed at resolving issues in the CI pipeline to publish new stable and latest Docker images, with changes to the Docker image build process that consolidate amd64 and arm64 tags.
- It also includes updates such as the removal of the totalDifficulty field from RPC, fixes to the simulated backend, and new multi-platform Docker image-building methods.
- v1.14.10 :
- A hotfix release addressing a blob pool regression introduced in v1.14.9, urging users to update promptly, though there’s no immediate risk beyond blob transaction propagation issues.
- It also includes updates like stateless witness production in the engine API and changes to gas cap behavior in simulated calls.
- v1.14.11 :
Consensus Layer Clients
Information on the different clients
- Nimbus: New release v24.9.0 :
- A low-urgency release focused on beacon API improvements and stability fixes, such as resolving DNS hostnames and preventing crashes during UPnP initialization.
- Users are advised to update within the normal two-week cycle.
Launchpad Stack
- New chart versions released with enhanced features and bug fixes:
- Erigon
- Celo
- Arbitrum-Nitro
- Graph-toolbox
- Firehose-Ethereum
- New stable versions of Ethereum, Gnosis, Celo, Monitoring, Arbitrum
Issues:
- Launchpad charts issues: View or report issues
- Launchpad namespaces issues: View or report issues
Protocol watch
The latest updates on important changes to the protocol
Forum Governance
Arbitration
New chain integration requests
Forum Research
Contracts Repository
- fix: update function visibility (OZ N-16) #1051 (merged)
- fix: add security contact to various contract NatSpecs (OZ N-03) #1028 (merged)
- fix: remove redundant getter functions. (OZ N-15) #1050 (closed)
- fix: emit new events in AuthorizedCollector and TokensRescued. (OZ N-14) #1049 (merged)
- fix: remove use of magic numbers in __ProvisionManager_init_unchained function (OZ N-11) #1041 (merged)
- fix: standardize naming conventions throughout the codebase (OZ N-13) #1047 (merged)
- fix: tweaks to thawing pool management #1048 (open)
Open discussion [9:03]
Some comments have been lightly edited and condensed.
Abel mentioned that Fuse Network was unable to make this week’s IOH as planned, so Matthew from Pinax suggested a few topics for discussion, including some about Kubernetes just below.
Kubernetes [10:14]
Matthew introduced a few of the Kubernetes topics for discussion. If you are an indexer and need support to deploy Kubernetes, let us know what you need help with at IOH.
Matthew: The main takeaways from gathering Kubernetes input were the following:
- Infrastructure as code: a lot of the ways you get efficiencies in Kubernetes is to commit all your changes into Git rather than writing on the command line. That might be a different way of operating for some people.
- Vince from Nodeify asked what kind of Kubernetes distributions are people thinking about or wanting to talk about because there are a pile of different options.
- People have different opinions on how things should be done, and it becomes an interesting discussion to understand why people are doing things in certain ways, so as infrastructure operators, we can learn from each other.
- Some are on big cloud providers or deploying infrastructure themselves.
- Different sizes of indexers mean different requirements and setups.
- Kubernetes tools
- How do you do:
- Monitoring
- Workflows
- User management
- Security
- Log management, statistics, alerting
- When you go to deploy a cluster, these kinds of things are important to consider.
- How do you do:
Transitioning to Timeline Aggregation Protocol (TAP) [15:19]
Matthew also mentioned moving to TAP as a possible topic, and indexers chose to talk first about TAP.
TAP is the new micro-payments system for queries on The Graph. It is high-throughput, efficient, and trust-minimized and allows further decentralization of The Graph Network by reducing the need for trust in the gateways.
Initially, there were query metrics missing from the new indexer service, but they have been added now.
Nick | The Graph Foundation posted: I’m curious what indexers are thinking in terms of moving to TAP?
- Vincent | Data Nexus: FYI, metrics are available with the latest main commit. Should have a release soon.
- Jim | Wavefive: With the metrics progress and improvements to escrow, we will swap over.
- Vince | Nodeify: If metrics, me gusta.
- Vincent | Data Nexus: You’ll need to update your query metrics dashboard. I have a working dashboard JSON here: 📁︱indexer-software. Error tracking metrics are still to come though. If you’d like to see distribution of error codes, make sure to react to this GitHub issue to get higher priority: Error metrics information #332.
Matthew asked: What are people waiting for to move to TAP?
- Vince | Nodeify: Metrics for me.
- stake-machine.eth: Docs?
Matthew prompted: What kind of docs do we need? Do we need more time? Anyone need help, advice, more infrastructure, less infrastructure?
- Jim | Wavefive: Specifically metrics parity w/ts agent.
- stake-machine.eth: Examples of ENV [environments] and so on.
Matthew: The reason I’m kind of poking is because the core dev teams would like to stop maintaining the old TypeScript code, so the sooner we can get people to move over to TAP, the better.
Ana | GraphOps: In Semiotic’s presentation from IOH #175, the team mentioned that we should move to TAP when the indexer agent is on the next release, so 0.21.5, and that hasn’t been released yet.
Vincent | Data Nexus: I just finished testing the main branch this morning and gave the thumbs up to Semiotic, so we should have a release soon. Just not in time for IOH today.
Gemma | LunaNova: We usually wait for Payne to test in prod for a while first…
Jim: I believe Semiotic, the team that created this new Rust-based indexer service, was not the team that created the former TypeScript service. So it’s important to make sure we’re not going to miss something with the new service because the metrics were a pretty big miss in terms of understanding indexer user experience. I guess this is a product-manager-type question really, I’m not sure there’ll be anyone here who can answer that.
Ana: Will try to test latest today as well. They’ve [Semiotic] been super responsive to any feedback.
Jim: Yeah, the Semiotic team has been really responsive to feedback. I just want to make sure we try and avoid any pitfalls as we transition away from one team working on this and another team beginning to work on it. If that’s what’s happening here.
Ana: I have suggested to Abel that we do a session where those of us who are running TAP share what we’ve learned, some gotchas, and maybe as part of that, we could also get some understanding of who’s running it, who’s maintaining the code, and the kind of support we’re going to get. Maybe that would be helpful and something that InfraDAO would like to be involved with.
Slimchance posted: I don’t want to speak on their behalf, but I can say that a unified release process has been part of the discussion.
Slimchance elaborated: In that we make sure that this was released the same way as other components in the ecosystem. They get added to the network’s MD file and have a proper release, not just a commit, and follow the example of the existing components.
Slimchance posted: InfraDAO has been working on the testing over the last four months (courtesy of Vince | Nodeify and Vincent | Data Nexus) – they are doing incredible. 🔥
Jim: It would be great to hear from Edge & Node about their experience so far as a gateway operator with the new escrow stuff, and what they’re learning and changing as a result.
Indexer Documentation [28:11]
Vince | Nodeify: How can we keep docs in line with releases? Rather than release notes or Discord chats.
Vince elaborated: For a long time, it’s been the experience for indexers that for new releases or new software, the docs are there but the different pieces are in different places. The indexer community would really benefit from having one place to go to for everything indexer, whether that be The Graph docs or in the repo. We have all these different components and as they start growing for different services, it’s going to be difficult to scale the practice of clicking through seven repos to find out how to run seven different things.
Sometimes documentation isn’t keeping up with releases. When something gets released we’re left to search for it, and a lot of the time you have to ask other indexers or search for a Discord thread.
Make it one place indexers know where to go, and that’s where we go for everything—all the environment variables, how to run it, etc. Instead of having The Graph docs, The Graph indexer repo, The Graph Node repo, this repo, and that repo.
Docs need to follow the release schedule and not be updated later. If you miss IOH one week, you shouldn’t be left wondering why things aren’t working.
Mack posted: Should bounty it out. Whoever does it well, keep them close.
Gemma | LunaNova posted: The docs are always out of date.
Mack posted: It honestly should be part of the code release process.
Gemma | LunaNova posted: I agree that ‘how to set up an indexer’ docs can’t be updated every time there’s a tiny change… but basic things like where new lines are added in config should be kept up to date by the developer, imho.
Mack replied: Yeah, but there can be a list of changesets attached to the guide.
Slimchance: Regarding who’s currently managing TAP, Alexis is not longer working with Semiotic Labs, but Anirudh is the current head of engineering there and Carlos and Gustavo are the ones managing TAP. They are all great engineers; no issues there. I think one of the reasons that we are delaying is that we don’t want everybody switching at once.
On the docs side, indexing docs have been a challenge since we started, and InfraDAO is ready to support with bounties. I think experienced engineers that have been running indexers since the testnet are not that interested in managing and being the head of the docs. It’s a challenge.
Mack: I do really think that the team, the developer, who is doing the PR and constructing the new release should be part of the process. It should be one of the line items to go and update the docs. I know it’s a little more work, but they have all the information about what is changing. So you reduce the feedback loop, reduce the running around to ask questions.
Ana: I have a different opinion in regards to docs. I think that there should be a minimum amount of docs, but then sometimes getting improvement on them and then being written by someone who hasn’t directly worked on the product can help both the product team and the user by identifying the gaps where maybe users don’t understand properly how the product works.
Mack: I can agree with that, but I do think there should be a process if someone from outside is coming to collaborate with the person who actually did the code because then it will go very smoothly. It’s about the process. You don’t want to be the one writing the docs, and having to chase down getting answers about things you need so it just takes longer. But I do agree with you it’s valuable to try and get a kind of middle ground experience and approach.
Jim: I think it takes a very specific type of skills. Ana, I think you’re absolutely right that somebody who is external, someone who is not an indexer but understands a bit about The Graph and is an excellent communicator is usually the type of person that can do really well with that type of work.
There was someone who used to do annotations on stuff, they would ping everyone who was involved in the conversation to summarize their thoughts and then present it on the forums. If we could find someone who could be incentivized to do this with those right skills via a grant or something, if it’s through InfraDAO, that would be a great avenue to take.
Slimchance: Yes, absolutely. We can also empower them to write their own grants. They could write up what is needed; if there are additional docs that need to be written, they can reach out to indexers that know how to do it and set them up with grants. If there’s anyone in the community that you think would fit this role, I’d love to reach out to them and empower them to improve our docs.
Subgraph Radio
Meanwhile, in the chat…
stake-machine.eth: How’s Subgraph Radio doing? I haven’t heard about it lately.
Vincent | Data Nexus: I just had a reason to check it for the first time in a while, and it seems like there’s an issue with Grafana’s GraphQL data source that is keeping me from finding the data I want from the dashboard. Currently trying to get around that. 😅
pili | GraphOps: We will look at it! Thanks for flagging this.
New Indexers on the Mic! [38:41]
A new indexer from last week returned to IOH, along with another new person this week.
Abel invited John to share about how indexing is going for him.
John K.: I set up my indexer just a couple weeks ago, so I did go through the whole documentation. It was a little confusing trying to figure out the difference between the indexer and the indexer-rs (the new Rust version), and trying to figure out what the TAP system even was. So yeah, I definitely would say the docs do need a little bit of work, but I think everything’s going well now.
Jim: What sort of infrastructure are you running, John? Do you have specific objectives for your indexer?
John: It’s a pretty simple setup, I’m on bare metal, Kubernetes, I’m running Talos.
Matthew: Did you use the Launchpad to get started?
John: I followed along The Graph Academy site, I found that to be really good, and then I think it was the GraphOps Kubernetes manifest. I kind of copied mine off of that.
Matthew: All right, so you did. Would it have been helpful to find those Helm charts on Artifact Hub?
John: I didn’t use the Helm charts, I just wrote my own.
Vince: Everyone kind of sees things going the way of Kubernetes with so many services and being able to scale. With people shifting to bare metal providers and wanting to stay on bare metal, some of those providers are less friendly with their networking in terms of bare metal that you’re not hosting yourself, and that makes cloud providers a little more attractive—but also very expensive. I’m wondering if you are hosting your own bare metal or are you using a provider that’s friendly towards your Kubernetes cluster in the networking aspect? Are you using MetalLB to expose your services?
John: I have physical boxes at a location and then (I don’t know if it’s a good setup or not, but it’s working for me), I’m using Cloudflare kind of like as an external ingress.
Vince: Yeah, that makes sense. Some, especially me, like to go with MetalLB to expose services and I’ve tried a few hosted bare metal providers and they haven’t been as friendly towards that kind of thing with their private networks, somehow they’re not as reliable as public IPs.
John: My CNI [container network interface] is Cilium, I’m using the BGP local router and then I guess it comes into Cloudflare.
Vince: Did you write you own Helm charts or raw manifest because you didn’t find anything or didn’t know about Launchpad and the Helm charts that it has right now? Would you find it helpful to have those types of things on Artifact Hub?
John: I had started experimenting with this a long time ago, so I had already written some manifests, so I just updated them from the examples and tried to figure out how all the pieces fit together. But if there was like the pre-made, that would definitely be great.
Mack posted a resource: Setting up a Full Erigon Ethereum Node (might be a bit out of date)
Abel invited Jordan, who is new to the call, to share his experience with indexing.
Jordan: I’m very happy to here, being my first one and all. I come from a background as a developer in the web3 space. I’m very open to learning more about indexing.
Vince posted: I’ll be collaborating with GraphOps on Launchpad charts going forward, updating, etc.
calinah | GraphOps: More docs about navigating the Launchpad stack, just in case: Launchpad Quick Start.
pili | GraphOps: If anyone has questions or needs anything on Launchpad, please shout to me or @calinah | GraphOps [on Discord].
Vince posted: InfraDAO docs, another great resource.
[Unhinged hilarity ensues in the recording… 🤣]
No Comments