TL;DR: The Graph has released Timeline Aggregation Protocol (TAP), a new payment system that minimizes trust and allows for multiple gateways. The release includes a new indexer-service-rs, which replaces the current indexer service and offers improved stability and performance. Indexers are encouraged to migrate to TAP soon, with the old indexer service expected to be deprecated in a few months.
Opening remarks
Hello everyone, and welcome to episode 175 of Indexer Office Hours!
GRTiQ 186
Catch the GRTiQ Podcast with Mark Smargon, co-founder and CEO of Fuse, a leading web3 platform for business and finance. With a deep-rooted background in tech entrepreneurship that began at the age of 14, Mark has been a pioneering force in the crypto space, particularly in building payment solutions that merge the worlds of traditional finance and decentralization.
The Graph deepens support for Solana
The Graph has enhanced its support for Solana, facilitating quicker access to its data. The release includes new tooling, a smoother developer experience, and ready-to-use Substreams-powered subgraphs and foundational modules, all designed to complement Solana’s high-speed ecosystem.
Read The Graph’s blog post, Access Solana Data: Better, Faster, Stronger on The Graph
Chain integration updates
Nick from The Graph Foundation shared that this week, The Graph will make formal announcements about support for Fuse and Blast.
Indexing rewards will be enabled on both of those chains. More information to come!
Repo watch
The latest updates to important repositories
Execution Layer Clients
- Erigon New release v20.60.8 :
- This release schedules the Ahmedabad hardfork for the Polygon mainnet and the Gandhinagar hardfork for Amoy.
- This release is mandatory for all Polygon users.
- sfeth/fireeth: New release v2.7.3 :
- This release fixes several bugs related to cursor handling, store size limits, passthrough mode, and module execution, with a breaking change requiring deletion of specific empty module cache files for consistency.
- Check the release notes to get the specific hash where you need to delete those cache files.
- Arbitrum-nitro New release v3.1.3-rc.2 :
- This release introduces a native tracer for Stylus calls and multiple stability improvements, including fixes for database corruption and invalid state issues in go-ethereum.
- Key user-facing improvements include the ability to force rebuild the local WASM store and support for rebuilding it on initialization.
- Breaking changes include the need to replicate default flags in the entrypoint when overriding it, and validators not using a split validation server must switch to the validator image, which uses a different entrypoint. Additionally, ensure the min basefee is positive for gas estimation and validate the WASM module roots for compatibility.
Launchpad Stack
- New chart versions released with enhanced features and bug fixes:
- Erigon
- Nimbus
- Graph-network-indexer (to support TAP; more on this below in the open discussion)
- Firehose-ethereum (released canary version and still testing; looking for feedback if you have any)
- New stable versions of Ethereum, Gnosis, Celo, Monitoring
Issues:
- Launchpad charts issues: View or report issues
- Launchpad namespaces issues: View or report issues
Protocol watch
The latest updates on important changes to the protocol
Contracts Repository
- chore: add stakeToFeesRatio to SubgraphService initialize #1027 (open)
- feat(horizon): deploy horizon with Hardhat Ignition #1025 (open)
Network Subgraphs
- Core network subgraph: Security hotfix for Curator data (v1.1.3)
Project watch
Auto-grafting is coming to Graph Node
- feature: graft auto sync #5633 (draft)
Open discussion
The release of TAP [11:03]
Gustavo from Semiotic Labs presented on the release of Timeline Aggregation Protocol (TAP).
What is TAP?
- A new payment system
- Trust is minimized
- Allows for multiple gateways (this is huge!)
- Discussed on Core Devs Call #22 [4:47] and Indexer Office Hours #147
- How it works: Timeline Aggregation Protocol (TAP)
New components
New indexer-service-rs
- Replaces the current indexer-service
- More stable
- Higher performance
- Supports only TAP
Rewrite of indexer-service in Rust with TAP payments implementation
New TAP-agent
- Makes sure that balance of receipt will never be more than escrow
- Allows/denies bad gateways
- Aggregates receipts into Receipt Aggregate Vouchers (RAVs)
- More info on TAP agent in Ana’s presentation below
Updated Indexer-agent
- Support to redeem RAVs
New TAP subgraph
Minimum version to run TAP
| Component | Version |
|---|---|
| indexer-service-rs | v1.0.0-rc.6 |
| tap-agent | v1.0.0-rc.6 |
| indexer-agent | #999fe0d (next release: v0.21.5) |
Running the new stack
- Create a config.toml (template)
- Remove old indexer-service
- Deploy new indexer-service-rs
- Deploy tap-agent
- Add a new TAP subgraph flag to indexer-agent
You can use our built Docker images:
If you want to run the stack using Helm charts, there are also updated Helm charts (more info in Ana’s presentation below).
Repositories and issues
- Rewrite of indexer-service in Rust with TAP payments implementation: indexer-rs
- For bugs or feature requests, go here: create issue
- TAP repos:
Documentation
- The Graph’s TAP Migration Guide
- Forum post: Mainnet release of TAP and indexer-service-rs
- GIP-0054: Timeline Aggregation Protocol
Questions [20:49]
Marc-André | Ellipfra posted: Is the new indexer agent going to be able to redeem both legacy and TAP query fees? Or should indexers that are migrating be advised to close all allocations, redeem vouchers, and then upgrade?
Can an indexer roll back to legacy vouchers/receipts if needed?
Alexis | Semiotic Labs: Yes! It will redeem both.
[Before you migrate], it is important to close all your allocations and stop doing business with the gateway because the gateway identifies the version of the indexer service you’re running, so it can detect that you’re supporting the new TAP. If you do a rolling update, it will confuse the gateway. Ideally, you close everything, stop all business, don’t do any new queries, and you update all of the software, then reopen business.
Ana: I have a follow-up question on that. What happens if you’ve done a rolling update and have unclosed allocations, and then you roll back to the previous version of the indexer service? Say, in case you’re testing and your configuration is not where it needs to be. What happens with your allocations?
Alexis: Ideally you would close your allocations first, then roll back. You don’t need to roll back indexer agent, but you’d stop the TAP agent and roll back the indexer service, then reopen business. If you do a rolling update, the gateway won’t be able to communicate properly with the version of the software you’re running and you’ll have failed queries.
Ana: That makes sense. I was more worried about what happens with receipts between the two.
Alexis: The newer indexer agent supports redeeming the old Scalar receipts and the TAP receipts, so that’s good.
Matthew Darwin | Pinax posted: So, recommended to go live on mainnet (arbitrum-one) now?
Alexis | Semiotic Labs: Yes.
Ana | GraphOps: We’ve tested it on arbitrum-one as well.
Matthew Darwin | Pinax: Is there a timeline when everyone should move? (When does old indexer service stop being supported?)
Alexis: We want to deprecate Scalar, and in particular, we really want to deprecate the TypeScript indexer service, so then we’ll also deprecate Scalar. It’s a matter of months, a few months, so you should get going. Ideally, you would first try the new service on your testnet, get comfortable with how it works, and then migrate your mainnet setups.
Semiotic has been secretly running on mainnet with TAP for two months, we fixed some bugs, and for at least a month we’ve been running smoothly, redeeming query fees and serving queries.
Jim | Wavefive posted: How much of a lift is replicating the existing service metrics? They are quite critical for testing; we are talking to the gateway and serving queries.
Gustavo: Currently, we have a lot of metrics inside indexer-rs and TAP agent. Tap agent has better metrics and we’re working to get to same level of metrics for indexer-rs. You can look at query metrics like how many queries per second and average latency. We want to add more granularity related to what are the allocations and what is the subgraph.
Ana | GraphOps posted: It is in the pipeline, I believe for next week.
Marc-André | Ellipfra posted: Is the new indexer service going to reduce query latency? If yes that’s a huge incentive to move quickly!
Gustavo: If you have a lot of load, it’s going to reduce query latency. We tested both indexer-service and indexer-service-rs, and when you get some amount of load, indexer-service-rs keeps up, the P99 stays steady, while the other starts to degrade really fast. So if you have very few queries, if won’t be that different, but if you have some amount of load, it’s going to be better and less resource-intensive. You’re going to use less memory and less CPU.
Alexis: The big news about memory is that it doesn’t have memory leaks. I think we’ve been running for a month in prod, and it’s been using 50 megabytes of memory continuously.
Jim | Wavefive: Maybe I’m missing something because the metrics seemed quite sparse.
Gustavo: The metrics that we have are different in that the same metrics that worked for the old indexer service don’t work for the new one. We are using Autometrics, which basically handles all of that for us. So we probably need to update that.
Derek | Data Nexus posted: Did the hosted service use the TypeScript indexer service? Or is that a new component to the stack with the decentralized network?
Alexis: Ford would be the one to ask for more details. The hosted service is migrating to the Rust version. Edge & Node is very excited about the performance improvements because of course, it has a lot of load. I would think you’re talking about the upgrade indexer. That service is strained quite a bit and the Rust version of the indexer service would really help a lot with the upgrade indexer.
Alexis: The biggest side effect of TAP is that we’re going to be open to allow more gateways run by less trusted parties.
What’s changed in The Graph Network Indexer Chart in Launchpad? [34:48]
Ana from GraphOps presented on what has changed as they have added support to Launchpad charts for TAP (Graph Network Indexer Helm Chart). A chart is a way of templating and packaging Kubernetes resources into releases, so they are easier to manage.
What’s changed?
- Indexer service has been completely replaced.
- TAP agent has been added as a different component.
- The indexer agent has been updated to a newer version.
How TAP works
- A consumer makes a query to the gateway.
- That query is forwarded to the indexer service.
- If the query is served, then a TAP receipt is created in the Scalar TAP receipts table in your indexer database.
The entries appear as in the below screenshot.
- Then you have many more queries.
- Once a certain amount of receipts have been created, there are a bunch of configuration fields that tell you when to create a RAV (receipt aggregate voucher) that will eventually be redeemable on-chain.
- The TAP agent is the new component that takes those receipts from the Scalar TAP receipts database and requests a RAV from the TAP aggregator, which is a component a gateway will have to run.
- You continue operations like that, creating many receipts and many RAVs.
The second valid table in the indexer database is the Scalar TAP RAVs. Below you can see allocation ID, for each allocation you can see some senders and the RAVs that have been created, the value of each allocation and each RAV. In the final and last fields, true or false indicates if an allocation has been closed or not.
For example, the fourth allocation in the table below has not been closed, so the final and last are showing false.
- Once an allocation has been closed, a notification goes to the TAP agent.
- The indexer agent would request an on-chain redemption.
- If that redemption is successful, then that marks the receipt as final, as it has been redeemed.
- You get the final RAV, so marked as final.
- The indexer agent makes the request on-chain.
- It checks against the TAP escrow contract, which the indexer agent now needs to reference.
What has changed for The Graph Network Indexer Launchpad chart? [41:53]
The Graph-Network-Indexer Helm chart now supports TAP.
Configuration changes
Indexer service has been replaced, TAP agent has been created as a new component, and they both rely on a config.toml.
- indexer-service and indexer-tap-agent both rely on a config.toml.
- Currently, the chart will force you to provide a config.toml. If you don’t, you’ll get a very minimal one, but you would probably need to add additional configuration, which you can add yourself or you can use Launchpad namespaces.
- Environment variable overrides: Every option in config.toml can be overridden with environment variables. The format is PREFIX_TOMLHEADING__ (SUBHEADING)___CONFIGOPTION, where the PREFIX depends on the service:
- Use INDEXER_SERVICE for indexer-related options.
- Use TAP_AGENT for TAP-related options.
- However, using config.toml is recommended for two reasons: it forces you to have less repetition and if you need to add two specific deployment IDs with the different environment variables, you could introduce more mistakes.
Chart changes
- indexer-service and indexer-tap-agent both rely on a config.toml.
- indexer-agent still relies mostly on cli arguments, but the cli argument values common to service and tap-agent are computed automatically based on values in indexerDefaults.config.
- It’s important that service and TAP agent run the same version and pass the same config.
- indexer-tap-agent is running as a stateful set to force only one replica at a time.
- Any additional values indexer-agent specific can be passed to indexerAgent.config.
- Support for TAP dashboard added.
- Extensive docs added.
Next steps
- Release new version of graph-network-indexer chart with better docs and more improvements.
- Release stable version of the graph namespace with the new components update.
- We have a canary version you can use right now that we’ve tested and used on Arbitrum Sepolia and Arbitrum One.
Resources
- Launchpad Documentation
- launchpad-charts/charts/graph-network-indexer/README.md
- launchpad-starter
- launchpad-namespaces/graph/
Graph Network Indexer Helm Chart
This is the Launchpad chart if you haven’t seen it before. Previously, there were only indexer agent and indexer service, and now there’s the TAP agent.
It’s important to mention that anything that is passed into indexer defaults-config will be rendered as part of the config.toml. That same config will also be computed automatically for you and for the agent tags.
It’s also important to make sure that anything that’s specific for the indexer is passed under indexer-agent.config.
Watch the recording at 47:14 to see the Grafana dashboard that the chart includes, where you can see total pending fees over balance, unaggregated fees, receipt rate, RAV request rate, and if any senders were blocked. You can find this dashboard in the indexer-rs repo or packaged with the chart.
Big shout out to InfraDAO for testing.
Comments [49:42]
Alexis | Semiotic Labs: We’ve been running the chart in question in prod for a few weeks already! Going real smooth. Much more compact than the ad-hoc YAMLs I had for the TAP components.
Ana: Because we mentioned migration before, it’s worth mentioning that if you’re running namespaces and you update what you point your namespace to instead of stable to canary, or if you’re already using the setup from Launchpad starter, it would automatically migrate everything for you. There might be a few things that you need to add. For example, your indexer address, but not if you’re already set up. If you’re running any of the stack, it would just be redeploying and that would replace your indexer service and add TAP agent.
Jim | Wavefive: For us lowly Docker Compose users, I have everything running and can share the Compose bits if someone else is a lowly, lowly pathetic Docker user like lil’ ol’ me.
I’ve deployed it in my new stack and it works. I can write a gist or a PR for the repo.
Alexis | Semiotic Labs: That’d be super helpful. Helps understanding the stack to see multiple different deployment implementations.
Vince | Nodeify: Yeah, seeing it in Docker really helps people visualize it.
Jim: Yeah, I’ve deployed it in my new stack, so I’ll need to sanitize it, but it works and I’ve got the maximal config setup. So what I’ll do is I’ll try and sanitize those two containers, the TAP agent, and the indexer service, and I’ll either write a gist or a reference guide or something.
Jim | Wavefive later posted to the indexer-software channel: In the interest of others getting indexer-rs running quickly, I’ve put together a roughly sanitized version of the Docker Compose definitions we use and run on testnet: indexer-rs.yml.
Vince | Nodeify: Also, FYI, you can feed Claude a Compose and it will make you a Mermaid diagram.
I got it in one shot. I fed it the entire indexing stack Compose and it showed me exactly how the indexer stack works. The better question is, tell it to make it better. It had some interesting insights, one of them was to introduce Redis for caching and some networking-layer level stuff with load balancing and things like that.
Alexis: Originally, we were thinking of using Redis, but then we thought that using the PostgreSQL message passing was just as good for what we need. Actually, Claude is kind of wrong because Redis is not technically message passing. Unless it was thinking of something else. But we could have used RabbitMQ, for example. PostgreSQL is a very simple and fast enough message broadcast system and it avoids indexers adding yet another component. Responding to Claude I guess, not to Vince, but there you go, you can tell Claude my answer. [😆]
No Comments