Menu

The Graph Indexer Office Hours #185

Events By Dec 02, 2024 No Comments 
TL;DR: During IOH #185's open discussion, the focus was on the Timeline Aggregation Protocol (TAP) migration status, which has reached approximately 82% of queries with two major indexers still pending. A key highlight was the discussion of a zero-downtime upgrade strategy using reverse proxy routing for the Upgrade Indexer, which is critical as it serves as a single point of failure for many subgraphs. The deadline for indexers to migrate to TAP is December 3, 2024.

Opening remarks

Hello everyone, and welcome to episode 185 of Indexer Office Hours!

⬇️ Skip straight to the open discussion ⬇️

Repo watch

The latest updates to important repositories

Graph Stack

  • Graph Node: New releases v0.36.0 :
    • Date: 2024-11-26 15:37:35 UTC
    • Version 0.36.0 introduces notable features such as Substreams support, a new IPFS client, and enhanced error messages, improving overall functionality and debugging processes.
    • Bug fixes addressing performance and stability issues are also included, ensuring smoother operation of the Graph Node infrastructure.
    • Urgency indicator: Yellow
    • Urgency reason: Important updates, but not urgent.
  • Indexer Service & Agent (TS): New release v0.21.9 :
    • Date: 2024-11-25 21:46:27 UTC
    • Release 0.21.9 addresses a bug fix related to reallocating allocations for active processes, ensuring smoother operations in scenarios with over 1000 allocations.
    • Urgency indicator: Yellow
    • Urgency reason: Important fix, but not immediately critical.
  • Subgraph-radio: New release 1.0.7 :
    • Date: 2024-11-21 14:19:00 UTC
    • This release updates the Grafana dashboard and the Graphcast SDK dependency in version 1.0.7. It includes minor fixes and no impactful changes to core functionality.
    • Urgency indicator: Green
    • Urgency reason: Low priority updates, no critical changes.

Launchpad Stack

Launchpad-charts:

  • New chart versions released with enhanced features and bug fixes:
    • graph-node-0.5.6

Launchpad-namespaces:

  • New stable versions of Polygon, Graph, Arbitrum

Issues:

Protocol watch

The latest updates on important changes to the protocol

Forum Governance

Contracts Repository

Open discussion [7:30]

Follow-up Q&A on TAP with Gustavo from Semiotic Labs

Timeline Aggregation Protocol (TAP) is the new micro-payments system for queries on The Graph.

Indexers need to migrate to TAP by December 3, 2024.

For more discussions on TAP, refer to the IOH recaps:

Ana | GraphOps: There was a question that Pierre was asking on Discord, so let’s start with that.

Pierre asked about this error (see below). An indexer service error occurred where the query receipt does not have the minimum value, and there’s the expected value and the received value. I suggested increasing the max receipt value GRT, but that hasn’t seemed to improve it, so I’m wondering, Gustavo, if you have a suggestion for that.

2024-11-23T21:37:14.149104Z ERROR indexer_common::indexer_service::http::indexer_service: An IndexerServiceError occoured., self: Issues with provided receipt: Receipt error: Issue encountered while performing check: Query receipt does not have the minimum value. Expected value: 100000000000000. Received value: 54914160775223.

with

service.tap:
  max_receipt_value_grt: "0.01"

Gustavo | Semiotic Labs: Every time you find a minimum value, it is related to your cost models. So in TAP now, we have the cost models, so we are enforcing the gateways to respect your cost model. So that minimum value is related to your cost model, and there are a few indexers that have their cost model set too high. The gateway also has a budget. It has a target of $40 per million queries, I believe, and if your value is higher than that amount, they’re going to try to send the maximum that they can, and because it’s not enough for your expected value, you’re going to reject a few queries. So I suggest you lower your cost models a bit, probably close to the $40 per million queries.

Max receipt value GRT is something else. We expect TAP to receive small receipts, and if you’re receiving, for example, 10 GRT from a sender, something is not correct. They might be trying to send something that might break your system, and that’s why they’re sending so much GRT, so you will accept it. So that’s why you have that max receipt value GRT, so this is the maximum a receipt can be.

Josh Kauffman | StreamingFast.io posted: From monitoring the migration, are there enough indexers that have moved? Any logs you guys are seeing to see how things have gone?

Gustavo: We are close to 80% of queries, with two big indexers joining us soon. We are ready. But we’re still giving support to anyone who wants to migrate.

I’m going to run the script in the background and can update you on what the percentage of queries is right now.

Current status:

Indexers running36
Total indexers88
Percentage of indexers40.91%
Percentage of queries82.19%
As soon as Pinax and Upgrade Indexer start running TAP:
Indexers running38
Total indexers88
Percentage of indexers43.18%
Percentage of queries96.34%

Marc-André | Ellipfra: Great news. This rollout is going great IMO. Improvements across the board in the software, responsive devs, excellent communication.

Abel | GraphOps: What’s the main challenge that you are facing in the TAP migration? Who is yet to be migrated over?

Mickey | The Graph | E&N: Upgrade indexer hasn’t migrated yet.

Upgrade indexer is the single point of failure on many subgraphs (primarily on chains without rewards), so we can’t entertain any downtime.

Marc-André | Ellipfra: Yeah, the upgrade requires downtime as far as I know.

Gustavo has been helping us scheme and plot a way to avoid downtime. 💓

Matthew Darwin | Pinax: Pinax2 is done, Pinax1 is to do.

Just waiting to make sure pinax2 is stable for a few days before pinax1 is done. (not blocked on anything)

Gemma | LunaNova: We’re hoping to finish infra migration first… it’s a fun race…

Yeah, we’re just moving to shiny new hardware in a new data centre SOON.

Josh Kauffman | StreamingFast.io: We’re still actually not receiving query fees yet on this new version. But I lost my dev for this week, so we’ll fix it next week with Gustavo.

calinah | GraphOps: Send any errors you see as well, so we can address them as they come.

Gustavo | Semiotic Labs: Just send a message, and I’ll happily join a call to help you.

Upgrade to TAP with no downtime [22:07]

Gustavo: Something that I could add here is how we are adjusting to do the upgrade with no downtime for the Upgrade Indexer.

We recommend that indexers close their allocations, upgrade to TAP, and then start using the new software because there’s no easy way to tell the gateway to stop sending you queries while you’re upgrading.

Also, the gateway queries you every minute to check what your version is and updates the receipt type accordingly.

If you follow the recommendation, when you open a new allocation, the gateway queries you, finds the new version, and starts sending the new type of query receipt.

However, the Upgrade Indexer is a single point of failure for many subgraphs and cannot have any downtime.

The main difference for a request from V1 to V2 is that in the new version, the header has a different receipt name: Scaler receipt —> TAP receipt.

What we’re going to do is put a reverse proxy in front of the indexer (e.g., nginx, traefik), so it can route things and you can have a proxy. So you can load balance between multiple indexer services.

You can add a rule inside that to route accordingly to the headers. You have a route rule where if the header contains Scaler receipt, you send to the older indexer, and if the header contains TAP receipt, you send to the new indexer stack.

Make sure both of them are running, and you have different internal hosts for each. Then, you will route accordingly.

Initially, the Upgrade Indexer will route to the old version, but then when they switch to indexer-rs, the next time that the gateway sends a slash version, it will start sending new receipts, but until then you can route to the older stack.

This way, you can have zero downtime and just switch the route in your reverse proxy.

Maybe the bigger indexers could benefit from this strategy, but now that they’ve already migrated, it doesn’t really make sense. But for the Upgrade Indexer, it will result in a smoother transition.

Closing Q&A [31:14]

Gustavo | Semiotic Labs: Other than Pinax and Edge & Node, is everyone here using TAP?

Abel | GraphOps: Payne [StakeSquid] mentioned that he’s working on a bunch of stuff, so he’s not migrated yet, but what about everyone else? Curious to hear from the DappLooker team or InfraDAO, where you are in the TAP migration.

Slimchance [from InfraDAO] posted:

Choubey | DappLooker: Planning to upgrade but have not yet. Just bandwidth from our side.

John K.: Going well… I started on TAP, so didn’t have to migrate.

No questions right now… still scaling up my indexing operation.

Graph Node Upgrade

Matthew Darwin | Pinax: So who already upgraded to latest graph-node?

calinah | GraphOps: Not us.

, , ,
Author

We're a web3 service provider specializing in blockchain indexing operations. Our mission is to enable creators to achieve their true potential with web3 technology. We want to help developers reliably access blockchain data in a consistent format so you can create amazing experiences for your applications.

No Comments

Leave a comment

Your email address will not be published. Required fields are marked *