TL;DR: The open discussion focused on the Firehose chart developed by the GraphOps team for Launchpad, a Kubernetes toolkit for indexers. Carlos from GraphOps presented on Kubernetes basics, Helm charts, and the Firehose-Ethereum chart, demonstrating its deployment flexibility. The team invited indexers to try Launchpad and send any feedback their way.
Opening remarks
Hello everyone, and welcome to episode 178 of Indexer Office Hours!
GRTiQ 189
Catch the GRTiQ Podcast with Hyung-Kyu Choi, also known as “HQ,” co-founder and Chief Operating Officer at DSRV, a well-known blockchain infrastructure provider based in South Korea.
DSRV is known for offering a range of web3 infrastructure solutions, including its validator services, serving over 50 protocols, blockchain RPC service, and development support tools like Welldone Studio.
Repo watch
The latest updates to important repositories
Execution Layer Clients
- Erigon New release v3.0.0-alpha.4 :
- Breaking change release that adds new defaults: -pporf.port=6060 –metrics.port=6061 –diagnostics.addr=localhost:6062, their ports were conflicting in Erigon2.
- Nethermind: New release v1.29.0 :
- This release introduces a breaking change with the replacement of the Discovery.Discv5Enabled flag by a more generic Discovery.DiscoveryVersion flag, defaulting to V4 but allowing for V5 discovery, alongside new heuristic transaction censorship detection, performance improvements in block validation, and memory optimizations for Windows and other platforms.
- Heimdall: New release v1.0.9 :
- This release includes several improvements and bug fixes, such as handling account sequences in the bridge, adding a status endpoint from Tendermint, increasing the max limit for the list span API, and a bug fix for the bridge self-heal feature, along with version updates and contributions from new contributors.
Consensus Layer Clients
- Teku: New release v24.10.0 :
- This release introduces blobs handling and networking improvements, including support for the Ephemery Testnet and engine_getBlobsV1 for faster block import, while renaming key metrics to align with the current version of Prometheus, along with bug fixes such as removing deprecated methods and resolving CVE-2024-7254.
- Notable changes include updated configurations for the Holesky and LUKSO networks, with breaking changes to the naming of certain Prometheus metrics.
Graph Stack
- Indexer Service & Agent (TS): New release v0.21.4-4 :
- This is for the TypeScript version of indexer service and agent.
- Release naming has changed.
- Performance fix release that removes a workaround where we had to query the graph-node for indexingStatus twice to determine a deployment’s state and moves flushOutstanding (receipts) to use READ_COMMITTED isolation level.
Protocol watch
Forum Governance
- Request for information about disputes #GDR-17
- Confirmed that it’s going to be closed, with one dispute resulting in a slash and the other a draw.
- Inspector POI posted: Transaction just went through, thank you.
Contracts Repository
- See the agenda for a list of pull requests.
Open discussion [8:02]
Carlos from the GraphOps team presented a session on the Firehose Launchpad chart they’ve been working on.
Presentation slides: Launchpad: The Kubernetes Toolkit for Indexers
Carlos started with a short intro to address those indexers who are not familiar with Kubernetes and Launchpad charts. He mentioned that the presentation assumes that most of the audience has some experience with Docker containers.
Agenda
Kubernetes [9:39]
- An orchestrator of containers built for large-scale production environments.
- If you’ve been running containers, you probably use Docker Compose.
- Use Kubernetes if you want to launch a lot of containers, play with their configurations, keep them running even if they crash, or scale them to run as backends with a load balancer on top.
- Helps you scale up the usage of containers as a delivery mechanism for workloads.
- Abstracts much of the underlying infrastructure.
- Items like where to store data, how to connect to the outside world or accept connections from the outside world, etc.
- Has a lot of resource types governed by YAML specs.
- Works on reconciliation loops.
- Drives the state of the system by what’s specified. If something crashes, it will endeavor to start it running as it was.
- Is API driven.
- You specify what your workloads are going to run and how they’re going to run.
Watch a quick example of resources at 12:25 showing configuration map, service, part, and deployment.
Helm and Charts [14:37]
- Like a package manager, Helm bundles Kubernetes resources into packages called “charts.”
- Provides lifecycle management (deployment, upgrading).
- Offers go templating for mutating the resources and workload configuration management.
A set of resources represents an application that you want to be running on your Kubernetes cluster. An application will have the container with the workloads and resources that represent storage, config maps, or services. For a given workload deployment, you will have a bunch of resources, and Helm is a tool to bundle those resource types and package them. You can think of Helm as a package manager because it’s mostly responsible for the lifecycle management of deploying the packages, which are called charts.
Another capability is templating the resources that are deploying so you have some flexibility to change them in terms of the shape of the deployment architecture or actually configuring the workload. For example, you could template the arguments that are going to be executed on the container.
Watch a quick example of a Helm chart at 16:14.
Firehose [18:05]
Because indexers know quite a bit about Firehose, Carlos focused on how it’s currently being packaged and delivered, and the challenges in making a chart for Firehose.
- Firehose is made for scalability and follows a micro-service architecture. Defines interfaces and protocols.
- It extracts data from the blockchain by instrumentalizing the chain’s node.
- The component ingesting the node’s data (reader-node) and producing standardized block files is the most chain-specific “connector.” Different binaries: firehose-ethereum for EVM-based chains, firehose-near, firehose-gear, firehose-solana, firehose-starknet, etc.
- Firehose-Ethereum has a lot of common ground with all EVM-based chains like Arbitrum, Polygon, etc.
- The first chart we did targets Firehose-Ethereum, so that will be focused on all EVM-based chains and not any of the others yet.
- This component is also responsible for launching the chain’s node and setting its configuration.
- Releases for different chains bundle different nodes, so there are different images.
If we want the chart to support multiple chains, we’re going to have to run different images, which is a bit of an anti-pattern and complicated given that one of the primary goals of a Helm chart is to do lifecycle management and for that, it should be tied to a specific image.
Goals for Launchpad Chart for Firehose EVM Chains [21:00]
There are a lot of chains and as a starting goal, we’d like to have support for a broad range of them, and we want the support to be very flexible.
- Meant to be used for all chains using the ‘fireeth’ connector and binary.
- Aims to be very flexible in deployment architecture.
- Users can opt in and out of specific components.
Challenges
- Different chains require different container images.
- Workload configuration parameters vary, particularly regarding the chain’s node.
- No one-size-fits-all for everyone or different chains with regards to deployment architecture.
- Firehose architecture involves quite a few components, and there are different ways to deploy them to serve different use cases.
Firehose-Ethereum Chart High-Level Overview [22:49]
- ‘firehoseComponents’ section allows you to specify the deployment architecture (e.g., Firehose-enabled blockchain node, reader, merger, relayer, Firehose gRPC server).
- This is where you have the most flexibility to shape the deployment architecture of the chart.
- ‘firehoseServiceDefaults’ allows you to set defaults per Firehose service running.
- A Firehose service is one of the Firehose components mentioned above.
- ‘firehoseComponentDefaults’ allows you to set defaults inherited by all components.
- Supports templating on the values, which enables configuration-management tied to a source-of-truth parameter.
- Instead of having to repeat parameters, you can actually use templating to point to another parameter or make conditionals.
- Override anything, anywhere and configuration interfaces for common usages and other added value features.
Demo of the Chart Usage
Watch a demo of chart usage at 26:12, starting from initial deployment and going all the way through the process.
Consensus Client Chart as a Dependency [34:40]
Vince | Nodeify: Would it also be beneficial to add a consensus client chart as a dependency, or what would that process look like? Just deploying and pointing it at consensus node service currently I’m assuming.
calinah | GraphOps: That is correct. It’s a bit more complicated to add consensus layer as a dependency at the Helm-chart layer, but we can add it by default at the namespaces layer. I think Carlos has some content on that shortly.
calinah | GraphOps: For example, our Ethereum namespace bundles Erigon, Nimbus, and proxyd by default.
Vince | Nodeify: Yeah, that makes sense! Just wondering for those that are just using the charts abstraction.
calinah | GraphOps: Maybe this kind of workload dependency is something we can explicitly mention in our docs for those users relying only on launchpad-charts.
Carlos added: We do have consensus layer charts, and you can use our set of charts (Launchpad charts) to deploy a consensus layer. Along with this, I could have a values file for a consensus layer chart and I would install it alongside the one on the same namespace. Helm works at the chart level, and when you deploy, the consensus layer needs to be configured with the workflow in an aligned way, which is difficult to do in independent charts. This forces you to add the charts as a dependency and build these Helm charts, which have a lot of dependencies, and that has drawbacks.
With Launchpad, we want to offer a lot of flexibility in terms of being able to configure the charts. We want to keep all of the configuration management inside the scope of that chart. For grouping charts together, for example, if they are functionally dependent or logically connected, we have Launchpad namespaces. The sole purpose of namespaces is to bundle Helm charts together. It’s not released yet, but we will have a namespace for Firehose and it will bring along the consensus layer and focus on doing configuration management across the charts.
Back to the Demo
Watch as the demo continues at 38:09, and Carlos uninstalls the deployment and changes the values a bit to showcase deployment flexibility.
Then Carlos shows how to deploy the same chart but for the Arbitrum chain [44:31].
Why These Choices [47:22]
- Maintaining one chart per chain would not scale at all for a hundred chains, and there is a lot of repetition in the resource templating logic.
- Maximum flexibility and transparency. Flexibility scales poorly with configuration management. This approach decouples Kubernetes resource generation from workload configuration management.
- Suitable for relegating logic parts to a library chart in a practical way, bringing homogeneity and standardization across the charts.
Launchpad Namespaces
More on namespaces, which were mentioned above.
- Built on helmfile, an upper layer that bundles different logically related charts in a single declarative unit.
- Focused on providing inter-chart configuration management.
- Has HA [high availability] architecture deployments in scope.
- Much value added in providing complex stacks in a simple manner, pre-configured to work out-of-the-box.
Launchpad Stack [49:16]
- Taskfiles
- Prepared workflows for common tasks or operations.
- Starter
- Skeleton repo for using the Launchpad stack.
Meanwhile in the chat…
Vince | Nodeify: How opinionated in terms of bucket storage will the chart remain? I know… I know we absolutely should be using S3. But for indexers that want to use local storage.
chris.eth | GraphOps: Minio pod exposing S3 over local storage?
Vince | Nodeify: Not a bad idea!
Coming Next [50:00]
- Expanding Firehose support for Substreams, sinks, and many more chains.
- Standardizing this approach in a library chart and rebasing our existing charts.
Carlos thanked the Pinax team for their support and collaboration.
Your Next Steps to Try Launchpad
- Join us in Discord: #kubernetes-launchpad.
- Get a Kubernetes cluster.
- Clone Launchpad Starter, play with it and enjoy very fast deployment of complex software stacks.
- Reach out for any issues/bugs, and let us hear from you!
Resources
- Launchpad Docs Introduction
- Launchpad Starter
- Launchpad Charts
- Launchpad Namespaces
- Launchpad Taskfiles
chris.eth | GraphOps: Chart covered today: Firehose-Ethereum Helm Chart
Q&A [51:12]
Ana | GraphOps: I’m just wondering if InfraDAO is planning on running Firehose anytime soon and if this would be useful.
Slimchance: Yeah, this is very useful. We will be running it for all non-EVM chains that require Firehose but we will also be running Firehose for all chains in the future, so definitely very helpful.
Vince will be our main contact on that on the non-CIP [chain integration process] chains, the EVM chains. I’ll defer to him on exact timing on that.
Vince: We’re definitely going to be using it in production for the non-EVM chains, and then also using that for Substreams if that becomes its own thing and scales out or whatever else may come, but it’s definitely the way to scale and makes things a lot easier at larger scale, so when we have a lot of chains coming through CIP, this is going to be hugely helpful.
Ana: Awesome, so it sounds like very soon there’ll be a lot more users, not just us. It’ll be good to get it tested.
Vince: Yeah, and I don’t have a timeline on it but fairly soon as we work on charts, Pinax will also be running and testing and I will also be running and testing the other charts as well, specifically the RPC charts and everything else that comes with that.
No Comments