Stream Seamless Antelope Data with Epic New Foundational Substreams Modules

Easily build fast, efficient, and responsive Antelope applications using Substreams.

TL;DR: Foundational modules make Substreams easy. Get streaming Antelope data from Pinax. Then put it in your favorite database using purpose-built Substreams sinks from Pinax.

Pinax has released new foundational Substreams modules for Antelope blockchains like EOS and WAX. These modules allow Antelope developers to stream blockchain event data using a simple syntax. To use Substreams, get your free API key in the Pinax app.

Substreams technology offers a lightweight way to stream data. Developers can use filtered Substreams data feeds in their applications. Alternatively, they can pipe data to a database for later transformation and use.

Pinax foundational modules offer faster indexing by pre-filtering actions based on the user’s query. This accelerates the process of receiving streaming data from Substreams. Foundational modules provide transactions, actions, authorization, and more.

Try it out

To test these new foundational modules, use the Substreams command-line interface (CLI) to connect to the Pinax endpoint. For example, using Substreams CLI, use the Pinax endpoint at eos.substreams.pinax.network and the Substreams package https://spkg.io/pinax-network/antelope-common-v0.4.0.spkg, followed by the desired module and queries. For example:

substreams gui -e eos.substreams.pinax.network:443 <https://spkg.io/pinax-network/antelope-common-v0.4.0.spkg> filtered_actions -s -10000 -p filtered_actions="code:eosio.token && action:transfer" --production-mode

substreams gui -e eos.substreams.pinax.network:443 <https://spkg.io/pinax-network/antelope-common-v0.4.0.spkg> filtered_actions_extra -s -10000 -p filtered_actions_extra="code:eosio.token && action:transfer && (data.to:myaccount || data.from::myaccount)"" --production-mode

Deploy it

To deploy the modules in applications, use the Pinax endpoint with a gRPC client to stream data directly. Pinax has published a JavaScript gRPC streaming example for streaming Antelope blocks in the Substreams-js GitHub repo, as well as an example for Antelope foundational modules:

const token = process.env.SUBSTREAMS_API_KEY;

const baseUrl = "<https://eos.substreams.pinax.network:443>";

// User parameters

const manifest = "<https://spkg.io/pinax-network/antelope-common-v0.4.0.spkg>";

const outputModule = "filtered_transactions";

const startBlockNum = -86400*4*2;

const productionMode = true;

const params = [`filtered_transactions=code:eosio && (action:buyram || action:sellram || action:ramtransfer)`]

Full code example.

Substreams sinks

Instead of setting up a gRPC client in your app, you can direct the stream into a common database engine or a message broker. For example, you can use a Substreams sink to store the streamed gRPC data for later use.

To help developers, Pinax published a Substreams sink example (and template). This example can output streaming blockchain data to many destinations, including Postgres, ClickHouse, RabbitMQ, and Prometheus.

The Substreams foundational modules package includes modules for querying Antelope data by all_transactions, all_actions, filtered_transactions, filtered_actions, and index_actions.

The modules support filtering queries by action, top-level action, and top-level parameters within an action. They can also filter by smart contract account, action receiver account, signing account, and signing permission.

What are foundational Substreams modules?

Essentially, Substreams modules are blueprints for querying filtered blockchain data feeds called Substreams.

Pinax’s new foundational modules let consumers subscribe to a “Substreams” feed. This feed streams actions and transactions from Antelope chains.

Users define the rules of their query, and Substreams filters out all the noise. Substreams provides only the necessary data. Data consumers can access all this without writing a single line of backend code.

What is Substreams?

Substreams is a data streaming technology that filters and indexes blockchain data. Indexing data allows sorting and filtering by fields other than block number, like contract or action name.

A Substreams feed ingests data from a Firehose worker that receives incoming block data from a blockchain node. Substreams will stream this data indexed by block and by whatever other indexes you set.

Developers can index and sink the data into the database of their choice. Substreams can also sink to subgraphs compatible with The Graph decentralized network.

However, Substreams is a read-only streaming service. It provides data streaming, but with no native data stores or transaction submission functions.

How does Substreams work?

A Substreams provider extracts data from blockchain nodes, then stores the blockchain data and makes it available.

To consume this data, a developer creates a Rust program that defines the transformations they want to apply to the blockchain data. Then, the Rust program is packed into a Substreams package.

The Antelope foundational modules are an example of one of these Substreams packages. Because Pinax provides these pre-built packages, Antelope users do not have to use Rust for common queries.

To begin streaming data, the consumer sends the Substreams package to the Substreams provider for execution. The Substreams provider will start streaming data back at high speed.

Data consumers can then use Substreams services to consume the extracted data in multiple formats. (SQL, Webhooks, Stream, Subgraphs, etc.)

Why Substreams?

While this read-only streaming approach requires a database for storage and a wallet component (like Anchor or MetaMask) for blockchain write functions, the modularity of this solution allows Substreams to act as a more scalable sub-component of a larger technology stack.

From the standpoint of a dApp developer, Substreams can serve streaming chain data. Storing chain data using a Substreams sink like ClickHouse can serve most chain query needs.

Pinax built these foundational modules to use common syntax. This syntax allows developers to use familiar workflows to stream and filter using Substreams. You get all the power of Substreams without learning Rust.

Usage

Test, debug, and explore the modules using the Substreams CLI. When you’re ready, you can use a gRPC client to stream the output to a Substreams sink, or directly into your app.

Use the Pinax Substreams endpoint and include your $SUBSTREAMS_API_KEY environment variable. Set the environment variable with the export SUBSTREAMS_API_KEY=”<YOUR-API-KEY>” command. Or, use a file named .env with your environment variables. Specify the Substreams package, module type, and search parameters.

For example:

export SUBSTREAMS_API_KEY="321ac5**************************************64a6"

substreams gui -e eos.substreams.pinax.network:443 <https://spkg.io/pinax-network/antelope-common-v0.4.0.spkg> filtered_actions -s 370000000 -p filtered_actions="code:atomicassets && action:createcol" --production-mode

This example includes the gui option, which lets you see and evaluate the requested output in a nice visual representation. The run option would display the module output in the terminal.

The -e option specifies the endpoint. In this case, we’re using the EOS endpoint eos.substreams.pinax.network:443. To use the WAX endpoints, replace eos with wax.

The example uses the module filtered_actions. Other available modules are listed below.

It uses the -s tag to start at block 370,000,000.

The -p tag specifies the parameters for the filtered_actions module. Possible parameters are listed on GitHub.

In this case, the query applies a filter within the filtered actions module: filtered_actions=”code:atomicassets && action:createcol”. This command will only return the createcol (create collection) actions from the atomicassets smart contract.

The –production-mode option parallelizes processing and caches output on the backend for subsequent uses.

gRPC implementation is similar. This example requires the installation of @substreams/node and @substreams/manifest.

const outputModule = "filtered_transactions";

const startBlockNum = -86400*4*2;

const productionMode = true;

const params = [`filtered_transactions=code:eosio && (action:buyram || action:sellram || action:ramtransfer)`]

In this case, the module parameters result in a stream of RAM-related transactions starting 691,200 blocks ago, or four days ago.

Modules

Previous examples use the filtered_transactions foundational module. There are three types of foundational modules:

• filtered-* – modules with outputs defined by a query. These are the most commonly used modules. When indexed, they will skip large chunks of empty blocks, improving performance. This type includes:
  • filtered_actions – flattened actions in a block, filtered based on the query
  • filtered_transactions – transactions in a block, filtered based on the query

• all-* – modules with outputs that include only certain messages in a block, such as actions and transactions, minimizing bytes billed. These “utility” modules are unlikely to be useful for end users. This type includes:
  • all_transactions – all transactions in a block
  • all_actions – all flattened actions in a block

• index-* – modules that index blocks with related keys. These “utility” modules are unlikely to be useful for end users. This type includes:
  • index_actions – index blocks with relevant action-related keys
  • index_actions_extra – “heavy” block index that also includes action authorization and parameters

• There are also extra modules, which contain extra data and are “utility” modules that are unlikely to be useful for end users.
  • filtered_actions_extra – flattened actions in a block filtered based on the query and index_actions_extra index
  • filtered_transactions_extra – transactions in a block filtered based on the query and index_actions_extra index

More Details

Find expressions and queries, as well as more details, in the GitHub repository.

Discover the power and flexibility of Substreams. Get your free API key in the Pinax app!

💡 This article answers questions like:
- What is Substreams and how does it work?
- Why should developers use Substreams for chain data?
- What are foundational Substreams modules?
- How do foundational Substreams modules help developers stream blockchain data?
- What types of foundational modules are included in this release?
- How can a developer deploy the new Substreams foundational modules? 
- What expressions can a developer use in queries?