Menu

Important Tech Update: Transition From Dfuse By September 25

Pinax, Substreams By Sep 18, 2024 No Comments

Use subgraphs, Substreams, and APIs from Pinax.

Blockchain technology holds promise as a neutral layer of the internet. Web3 is a place where code establishes trust without intermediaries.

Bitcoin, Ethereum, and others have delivered on a piece of this promise. Blockchains store data immutably across countless decentralized nodes. However, direct access to this blockchain data can be complex and resource-intensive. Most blockchains need a data availability layer.

The dfuse stack has provided this layer for users of EOS, WAX, and other Antelope / EOSIO blockchains. Unfortunately, the software is not under active development. Due to many factors, dfuse service will end on September 25.

Many have relied on the dfuse data stack for their chain data. Fortunately, Pinax services provide continued access to high-quality live and historical blockchain data. This guide will walk through the process of making the switch.

This guide covers some key functionalities:

Effective replacements for dfuse components include:

Many functions have multiple replacement options. Choose the component that works best for your use case.

Query with Subgraphs (GraphQL)

Subgraphs are customizable indexes for blockchain data. A subgraph transforms raw blockchain data into structured data. This indexed data is easier to use in decentralized applications. Indexers then store this data, using the subgraph as a data template.

Subgraphs can serve performant queries with enough flexibility for most uses. This format can also provide decentralized data availability through networks like The Graph.

The easiest way to get started with subgraphs is experimenting in Graph Explorer. View EOS and WAX subgraphs in Graph Explorer to generate and fine-tune queries. Pinax has built subgraphs with two weeks of chain history for:

Graph Explorer is a playground for querying subgraphs with GraphQL.

On the left is the GraphQL query. On the right is the response window.

Click the folder icon to explore possible queries and build your subgraph query. Alternatively, you can look at the GraphQL schema and type out your own queries. When formatting your own queries, keep in mind that GraphQL does not support OR statements.

Once you’ve built a query, you can execute the query to test it out. Refine the fields until you get exactly the data you want.

When you have the perfect query, you can find example code for cURL, React, and other frameworks at the bottom. Copy-paste the example into your code and test it out. Make sure to add your API key to replace the {api-key} URL slug.

To see an example, you can run the following query in Graph Explorer:

{
blocks(orderBy: number) {
id
timestamp
number
date
}
}

Which results in this response:

{
"data": {
"blocks": [
{
"date": "2018-06-09",
"id": "0000000267f3e2284b482f3afc2e724be1d6cbc1804532ec62d4e7af47c30693",
"number": "2",
"timestamp": "1528545390"
},
{
"date": "2018-06-09",
"id": "00000003d93442ea55d07be4d515700e2b9737c1f485e8a13ebb3550c1a8bb44",
"number": "3",
"timestamp": "1528545390"
},
... ... ...
... ... ...
"timestamp": "1528545439"
}
]
}

In addition to querying blocks with the blocks subgraph, you can query transactions with the transactions subgraph. Like the blocks subgraph, craft your GraphQL query using Graph Explorer or using GraphQL syntax.

{
actions(
orderDirection: asc
first:20
where:{
  account:"swap.defi"
  }
  ) {
    block{
      number
      timestamp
    }
    transaction {
      id
    }
    account
    name
    jsonData
    receiver
    dbOps {
      code
      tableName
      primaryKey
      newDataJson
    }
  }
}

Paginate this GraphQL statement using globalSequence (the index of all actions). Knowing the last number used in the global sequence, you can include the globalSequence with the suffix *_gt (Greater Than) to exclude all previous actions.

To sort actions in order from oldest to newest, you can sort by globalSequence in ascending order. This feature is only available in Antelope transactions subgraphs version 0.4.2 and later. It only applies to the actions index.

This results in the response:

{

"data": {
"actions": [
{
"account": "swap.defi",
"block": {
"number": "391628232",
"timestamp": "1725036952500000"
},
"dbOps": [],
"jsonData": "{\"fee\":\"0.0732 USN\",\"owner\":\"ijinian21312\",\"pair_id\":9,\"quantity_in\":\"24.4316 USN\",\"quantity_out\":\"50.1259 EOS\",\"reserve0\":\"52059.4393 EOS\",\"reserve1\":\"25322.3449 USN\",\"trade_price\":\"2.05373414402307519\"}",
"name": "swaplog",
"receiver": "swap.defi",
"transaction": {
"id": "0000384a08f2971d667d03f98d4a07391b153e9737da5899e2ed32b5dc3dcc7a"
}
},
... ... ...
... ... ...
{
"account": "swap.defi",
"block": {
"number": "393020833",
"timestamp": "1725733294000000"
},
"dbOps": [],
"jsonData": "{\"fee\":\"4.6963 SVX\",\"owner\":\"trader.mlt\",\"pair_id\":1811,\"quantity_in\":\"1565.4670 SVX\",\"quantity_out\":\"5.55357691 CHEX\",\"reserve0\":\"6869680.5333 SVX\",\"reserve1\":\"24438.27160223 CHEX\",\"trade_price\":\"0.00355110379706754\"}",
"name": "swaplog",
"receiver": "swap.defi",
"transaction": {
"id": "0017d24a9dfd26d30e2c62cdea70ba4f5ce9948c84caf262cfba574b26a4f218"
}
}
]
}

Pinax has extensive experience developing and hosting subgraphs. If your application is in need of a subgraph, or your subgraph is in need of hosting, contact us at Pinax and tell us what you need.

Set Up Your API Key

Subgraphs use The Graph decentralized network, and require an API key from The Graph. All other Pinax services use a Pinax API key.

  • Sign up for a free API key in the Pinax app.
  • Store this key securely as it will be used to authenticate your requests.
  • When using Substreams, include the API key in an environment variables file, usually named .env.
    Add the following text on a new line in the file: SUBSTREAMS_API_KEY=<your-api-key>Or, set the environment variable using the console: export SUBSTREAMS_API_KEY=”<YOUR-API-KEY>”
  • When using the Pinax Antelope API, add your API key to the HTTP GET request header.
    curl --request GET --url 'https://eos.api.pinax.network/v1/tokens/holders?contract=eosio.token&symcode=EOS' --header 'Accept: application/json' --header 'X-Api-Key: PINAX_API_KEY'

Note: When using endpoints for other networks, replace the network name in the endpoint.

Stream with Substreams (gRPC)

To stream block data, use a Substreams gRPC stream. Make sure all dependencies are installed.

Install Dependencies:

  • To stream data with Substreams, you need to install substreams along with the @substreams/node library.
  • You will also likely need to install @substreams/manifest
  • When using the .mjs example, ensure your project is using ESM (ECMAScript Modules). If you are using CommonJS, you may need to transpile the code using tools like Babel.

npm install substreams

npm install @substreams/node

npm install @substreams/manifest

See an example on GitHub that demonstrates how to stream block data using Substreams. User-defined parameters include:

...
...
// User parameters

const manifest = "https://github.com/pinax-network/substreams/releases/download/blocks-v0.1.0/blocks-v0.1.0.spkg";

const outputModule = "map_blocks";

const startBlockNum = -10;

const productionMode = true;
...
...
  • The code linked on GitHub listens to the latest EOS blocks, providing the block ID (hash), number, and timestamp.
  • Make sure to set up proper error handling and restart policies in case the stream fails.

The blocks-v0.1.0.spkg example above is just one example of how Substreams can provide streaming Antelope data. The Foundational Modules package antelope-common-v0.4.0.spkg provides more detailed transaction and action data.

For example, the following script will stream actions related to the swap.defi contract that are either deposit, sellallrex, sellpartrex, sellrex, sellrexlog , buyrex or buyrexlog :

substreams run -e eos.substreams.pinax.network:443 https://spkg.io/pinax-network/antelope-common-v0.4.0.spkg filtered_actions -s -5000 -p filtered_actions="(account:swap.defi || receiver:swap.defi) && (action:deposit || action:sellallrex || action:sellpartrex || action:sellrex || action:sellrexlog || action:buyrex || action:buyrexlog)" --production-mode
  • eos.substreams.pinax.network:443: The Substreams endpoint for EOS.
  • filtered_actions: A foundational module that lets you filter by contract and action.
  • –s: Start block for stream. 5000 will start streaming at block 5000. -5000 will start streaming 5000 blocks before the head block.
  • -p: Query parameters.
  • –production-mode: Writes indexes to the backend for reuse.

You can see the example implementation to see how it’s used in context. Find out more about foundational modules in our recent blog post.

Substreams queries can be adjusted for various contracts and actions (like in the searchTransactionsForward query). This approach gives more control and flexibility compared to dfuse.

Developers can also write their own Substreams modules. These custom modules can apply custom transformations and filtering, to stream already transformed data into an app or database. Dive in deeper by visiting the Substreams documentation.

Substreams Sinks

Substreams sinks are tools that allow you to take the data processed by Substreams and send it somewhere useful—like a database, for example. Think of sinks as the connectors that help you get the data from the blockchain into a place where you can store or analyze it. You can use different Substreams sinks to pipe the streamed data into a database like ClickHouse.

When you run Substreams, it pulls in blockchain data, processes it, and you use a sink to decide where that data ends up. Some common sinks include:

  • Databases (like PostgreSQL or ClickHouse)
  • Files (like CSV or JSON files)
  • Subgraphs

To send the data to a database like ClickHouse, use a tool like substreams-sink-clickhouse, which connects Substreams to ClickHouse. You can watch a video tutorial to see the tool in action.

Learn more about Substreams Sinks on the Pinax blog.

Send Transactions with Transaction Retry (Nodeos RPC)

dfuse Method:

In dfuse, you might have used the /v1/chain/push_transaction endpoint to push transactions to the blockchain.

Pinax Antelope Method:

Many developers have moved from eos-js to WharfKit, as eos-js is no longer actively maintained. You can push transactions with automatic transaction retry using WharfKit. See the WharfKit documentation for more details.

Alternatively, you can use the Antelope RPC endpoints directly to push transactions. The new send_transaction2 API will re-submit the transaction until it is included in a block. It is available on API nodes that have the transaction-retry option enabled.

Warning: full failure traces are now returned by default instead of exceptions. Be careful not to confuse a returned trace as an indication of speculative execution success. Verify receipt and except fields of the returned trace.

Here’s how to do it:

POST https://eos.api.eosnation.io/v1/chain/send_transaction2
Content-Type: application/json
{
"return_failure_trace": true,
"retry_trx": true,
"retry_trx_num_blocks": 10,
"transaction": {
"signatures": [],
"compression": false,
"packed_context_free_data": "string",
"packed_trx": "string"
}
}

Following these steps to send transactions with automatic transaction retry to ensure the transaction is successfully executed. Find details in the official documentation.

Get Head Block Data (Nodeos RPC)

Use get_info to get the current head block.

At any time, you can get the head block number and head block hash using the /chain/get_info endpoint:

  • Note: No authentication is required for Nodeos RPC API requests.

GET https://eos.api.eosnation.io/v1/chain/get_info

The resulting response will look something like this:

{
"server_version": "7cd03d6c",
"chain_id": "aca376f206b8fc25a6ed44dbdc66547c36c6c33e3a119ffbeaef943642f0e906",
"head_block_num": 392571495,
"last_irreversible_block_num": 392571159,
"last_irreversible_block_id": "17662917de5f0cf3a7309bf36120c7a91dbe6114516b94cf65fc88daa990ea30",
"head_block_id": "17662a67543992f24d2a4fdf2570445e3a8953646d3649b1e6d549ab6ad1d897",
"head_block_time": "2024-09-05T03:56:41.500",
"head_block_producer": "bp.defi",
"virtual_block_cpu_limit": 200000,
"virtual_block_net_limit": 1048576000,
"block_cpu_limit": 200000,
"block_net_limit": 1048576,
"server_version_string": "v5.0.2",
"fork_db_head_block_num": 392571495,
"fork_db_head_block_id": "17662a67543992f24d2a4fdf2570445e3a8953646d3649b1e6d549ab6ad1d897",
"server_full_version_string": "v5.0.2-7cd03d6c122b89b5fbbe75d5556810c108d97765-dirty",
"total_cpu_weight": "382735099695247",
"total_net_weight": "95748971443138",
"earliest_available_block_num": 1,
"last_irreversible_block_time": "2024-09-05T03:53:53.500"
}

Find details in the official documentation.

Get Specific Block Data (Nodeos RPC)

Use get_block to get the specified block by number or ID.

POST https://eos.api.eosnation.io/v1/chain/get_block
Content-Type: application/json
{
"block_num_or_id": 123
}

The resulting response will look something like:

{
"timestamp": "2018-06-09T11:57:30.500",
"producer": "eosio",
"confirmed": 0,
"previous": "0000007ac6d22a894d40056f6a1d013a26f55228a38a73cc4514092c831d2b7d",
"transaction_mroot": "0000000000000000000000000000000000000000000000000000000000000000",
"action_mroot": "d5ad70147f03f25f8acae0b373873b5ca2bd2bf8d94e8ccff0d5193a062a6494",
"schedule_version": 0,
"new_producers": null,
"producer_signature": "SIG_K1_KdMcxT5gwyypxRXUo1SJvuPsWC1RAtK1okV5SsDbCFUep5MYZoro6199B1XbhcG7noqCHRRWfW4QANjumhnvyu7XSgx9cS",
"transactions": [],
"id": "0000007b5dd0cd9fb87cf16213c8ffa4e8b9031879377b296d4d833a50a6236d",
"block_num": 123,
"ref_block_prefix": 1659993272
}

  • You can retrieve the block hash (id), number (block_num), and timestamp for a specific block number or ID, similar to the dfuse /v1/chain/get_block endpoint.

Find details in the official documentation.

Get Transaction Details (Pinax Antelope API)

dfuse Method:

In dfuse, you might have used the /v0/transactions/{txId} REST API to retrieve transaction details.

Pinax Antelope Method:

Use the Transactions API to retrieve transaction details. Here’s an example using a simple GET request:

curl --request GET \
--url https://eos.api.pinax.network/v1/transactions/hash/{hash} \
--header 'Accept: application/json' \
--header 'X-Api-Key: 123'

This will give you similar information to dfuse’s /v0/transactions/ API. Try it out and find more details in the Pinax Antelope API documentation.

Get Currency Balance (Nodeos RPC, Pinax Antelope API)

dfuse Method:

In dfuse, /v1/chain/get_currency_balance to monitors the balance of an account.

Nodeos RPC Method:

You can use the built-in get_currency_balance endpoint to get the balance of an account, scoped by a specific token symbol:

POST https://eos.api.eosnation.io/v1/chain/get_currency_balance
Content-Type: application/json
{
"code": "string",
"account": "string",
"symbol": "string"
}

Find details in the official documentation.

Pinax Antelope API Method:

You can achieve the same, but for multiple tokens, with the Antelope Token API. Here’s how:

curl --request GET \
  --url 'https://eos.api.pinax.network/v1/account/balances?account=eosio&contract=eosio' \
  --header 'Accept: application/json' \
  --header 'X-Api-Key: 123'

This will return the currency balance of the specified account, similar to dfuse. It differs from the native get_currency_balance endpoint in that it returns token balances for all tokens the account owns, rather than for a single token.

Only the “account” field is required. By default, the request is scoped to all tokens and contracts. Specify a contract or symcode to filter results by smart contract or token symbol. Explore the API documentation to try it and find out more.

Get Table Scopes (Nodeos RPC)

This native nodeos RPC now includes the /v1/chain/get_table_by_scope endpoint. This takes the contract, and optionally a specific table, as parameters. The endpoint returns the scopes of that contract, optionally filtered by table. For the current state, this replaces the /v0/state/table_scopes query in dfuse.

POST https://eos.api.eosnation.io/v1/chain/get_table_rows
Content-Type: application/json
{
"code": "wombattokens",
"table": "accounts",
"limit": 10,
}

More details and other options are available in the official documentation.

Get Current Table Rows (Nodeos RPC)

This query provides the current state of table rows. It replaces the /v0/state/table/row query in dfuse for current data.

Using Chain API (Retrieving Table Data)

This example retrieves rows from a smart contract table using the get_table_rows endpoint.

Command:

curl -X POST https://wax.api.eosnation.io/v1/chain/get_table_rows 
 '{
"code": "wombattokens",
"table": "accounts",
"scope": "your_scope",
"limit": 10
}'

More details and other options are available in the official documentation.

Transition from dfuse to Pinax by September 25

Pinax offers several powerful options for getting blockchain data.

  • Substreams provides a stream of indexed and filtered blockchain data.
  • Subgraphs provide an easy way to query blockchain data using GraphQL commands. Many subgraphs are supported by decentralized hosting from multiple indexers on The Graph network. Pinax has extensive experience developing and hosting subgraphs, and is happy to build or host subgraphs. Contact Pinax for more information.
  • The native chain API has many built-in features that can be used without any authentication.
  • Pinax’s new Antelope API provides more powerful and extensive information than the native API.

By following this guide, you can seamlessly adapt your applications to use subgraphs, Substreams, and the various APIs offered by Pinax. With the September 25 deadline approaching, we encourage you to make the switch promptly to ensure a smooth transition and continue leveraging reliable blockchain data for your projects. Feel free to reach out on our public Discord server if you’d like more information.

, , , , , , , , ,
Author

Hi, I'm Andrew. After spending the first decade of my adult life as a music entrepreneur, and then stumbling into a blockchain obsession in 2017, I am now a technical writer and product owner with a commitment to facilitating builders and realizing the potential of decentralized money and power. My vision for the future is that all industries will encapsulate the business logic and value streams of their respective products into automated contracts that adequately reward each participant based on the value they create. These contracts are published open-source, and anybody who thinks value should be rewarded differently can publish their own version and compete in the marketplace based on the results of their theory of value and the behavior it incentivizes. Decentralized organizations will compete on which supply chain participants are most rewarded, what governance structure the organization uses, whether to offer voting rights for tokenized customer rewards, and so much more. It all stems from the central theme of blockchain: Money is power. We can now create our own money. We can now create our own power. We must use it wisely.

No Comments

Leave a comment

Your email address will not be published. Required fields are marked *