Last year I curated a series of videos for a partner project to introduce Subsquid and explain its core concepts, starting from the basics, up to some less-known features. With the new year, I decided to rewrite parts of the videos, re-adapt and re-record them to the latest changes in the SDK and create a new series that we decided to call Subsquid Academy.
Introduction
In the previous lesson, we discussed what Subsquid is and why you should be using it.
In this article, we are going to dive deeper into the development flow, taking the necessary steps to index and transform EVM smart contract log information. I am going to show you how to develop a project from start to finish, and I am going to explain every step as I go.
We are going to start with the template repository we introduced in the previous lesson
We are going to start by changing the schema of the database and the final API. This will include using a command line tool to generate TypeScript classes that model entities defined in our schema.
We then need to obtain the contract’s ABI in JSON format, and we’ll be using a new command line tool to generate some boilerplate code for us. This will take care of decoding contract events with the right type, and will create wrappers around contract state calls.
We’ll go over this in more detail when we get to this point.We need to write a little bit of code to map on-chain information to our models and make the necessary transformations to write data in the correct format.
And finally, we are going to launch our project and check the results.
Discussing the different aspects of squid development while working on a concrete example should make it easier to understand. For this reason, I strongly encourage everyone watching to follow along and prepare their development environment, before advancing to the next videos.
You can find all the code relating to this article in this repository:
github.com/RaekwonIII/ethereum-name-service..
EVM log indexing
But let’s start with introducing the project for this lesson.
For this lesson, I decided to index events of the Ethereum Name Service (ENS) smart contract, and track the transfers of its Non-Fungible Tokens.
The ENS tokens and smart contract are available on this page on Etherscan.
The page shows the contract address, which will be useful later on, and there is some information about Transfers at the bottom.
Each Transfer reports the addresses that exchanged the token and some more information at the bottom when clicking on "Click to see More".
Subsquid’s SDK libraries are going to help us decode this data from the smart contract logs, but we can customize all the aspects of this process.
Index ERC-* smart contracts on Ethereum
Let’s access the squid–ethereum-template repository:
github.com/subsquid/squid-evm-template
We are going to fork the repository into our own account, then clone it locally, and then open it with our IDE.
Note: thanks to the Subsquid CLI, you can skip this step and simply initialize a new project with the command:
sqd init
We need to install dependencies, so let’s open a terminal and type
npm i
Schema
Let’s open the schema.graphql file and delete its content. We are going to replace it entirely for today’s project. As we said, we want to index the transfers of ERC-721 tokens.
To do so, we need these entities in our database:
A
Tokenentity, to represent each one of the NFTs in the collectionAn
Owner, which will intuitively represent the accountA
Contractentity, in which we are going to store contract informationA
Transferentity, to capture each transaction
Here is the resulting schema:
type Token @entity {
id: ID!
owner: Owner
transfers: [Transfer!]! @derivedFrom(field: "token")
contract: Contract
}
type Owner @entity {
id: ID!
ownedTokens: [Token!] @derivedFrom(field: "owner")
}
type Contract @entity {
id: ID!
name: String! @index
symbol: String!
totalSupply: BigInt!
tokens: [Token!]! @derivedFrom(field: "contract")
}
type Transfer @entity {
id: ID!
token: Token!
from: Owner
to: Owner
timestamp: BigInt!
block: Int! @index
transactionHash: String!
}
A couple of notes:
The
transfersfield ofTokenentity (same asownedTokens,tokens, ofOwnerandContractentities, respectively) is marked as a derived field, this means it will not exist on the database for this table, but because eachTransferis storing whichTokenwas exchanged, thetransfersfield on this entity can just be derived from itThis is how you can represent what it’s commonly referred to as a One-to-Many relationship
The
timestamp,block,transactionHashfields are completely optional fields, but they have been added to show how to source them when we'll be implementing the mapping code
I am choosing these entities and these fields, because I think they are representative and they can be useful, if I want to analyze transactions, or I want to use them in a frontend application.
We could also be adding things like the event ID that stores the transfer information, or changing things around with the relations between entities.
Here’s a pro-tip: in the
Makefile, we can find shortcuts to some of the useful commands we are using in our project.
So once we are happy with the schema, we need to open up the terminal and type
make codegen
And new models are going to appear in src/model/generated replacing the previous ones.
Next, we are going to deal with the contract’s Application Binary Interface, or ABI.
Evm typegen
Many of you should already be familiar with the ABI of a smart contract. It stands for Application Binary Interface and it is a description of the available functions, the arguments, the generated events, and the types of a smart contract.
It is useful, because once a smart contract is compiled and deployed, how would anyone know how it works? What are its inputs and outputs?
ERC-20 and ERC-721 smart contracts have standard ABIs, so they are not very hard to find on the internet. However, in most cases, we can use Etherscan to obtain a contract’s ABI, by visiting the contract’s page, clicking on the Contract tab, and either copy or export the ABI.
We need to import this into our project, so let’s copy the ABI, go back to our project, and create a file named ens.json and paste the ABI in here.
[
{
"inputs": [
{ "internalType": "contract ENS", "name": "_ens", "type": "address" },
{ "internalType": "bytes32", "name": "_baseNode", "type": "bytes32" }
],
"payable": false,
"stateMutability": "nonpayable",
"type": "constructor"
},
{
"anonymous": false,
"inputs": [
{
"indexed": true,
"internalType": "address",
"name": "owner",
"type": "address"
},
{
"indexed": true,
"internalType": "address",
"name": "approved",
"type": "address"
},
{
"indexed": true,
"internalType": "uint256",
"name": "tokenId",
"type": "uint256"
}
],
"name": "Approval",
"type": "event"
},
{
"anonymous": false,
"inputs": [
{
"indexed": true,
"internalType": "address",
"name": "owner",
"type": "address"
},
{
"indexed": true,
"internalType": "address",
"name": "operator",
"type": "address"
},
{
"indexed": false,
"internalType": "bool",
"name": "approved",
"type": "bool"
}
],
"name": "ApprovalForAll",
"type": "event"
},
{
"anonymous": false,
"inputs": [
{
"indexed": true,
"internalType": "address",
"name": "controller",
"type": "address"
}
],
"name": "ControllerAdded",
"type": "event"
},
{
"anonymous": false,
"inputs": [
{
"indexed": true,
"internalType": "address",
"name": "controller",
"type": "address"
}
],
"name": "ControllerRemoved",
"type": "event"
},
{
"anonymous": false,
"inputs": [
{
"indexed": true,
"internalType": "uint256",
"name": "id",
"type": "uint256"
},
{
"indexed": true,
"internalType": "address",
"name": "owner",
"type": "address"
},
{
"indexed": false,
"internalType": "uint256",
"name": "expires",
"type": "uint256"
}
],
"name": "NameMigrated",
"type": "event"
},
{
"anonymous": false,
"inputs": [
{
"indexed": true,
"internalType": "uint256",
"name": "id",
"type": "uint256"
},
{
"indexed": true,
"internalType": "address",
"name": "owner",
"type": "address"
},
{
"indexed": false,
"internalType": "uint256",
"name": "expires",
"type": "uint256"
}
],
"name": "NameRegistered",
"type": "event"
},
{
"anonymous": false,
"inputs": [
{
"indexed": true,
"internalType": "uint256",
"name": "id",
"type": "uint256"
},
{
"indexed": false,
"internalType": "uint256",
"name": "expires",
"type": "uint256"
}
],
"name": "NameRenewed",
"type": "event"
},
{
"anonymous": false,
"inputs": [
{
"indexed": true,
"internalType": "address",
"name": "previousOwner",
"type": "address"
},
{
"indexed": true,
"internalType": "address",
"name": "newOwner",
"type": "address"
}
],
"name": "OwnershipTransferred",
"type": "event"
},
{
"anonymous": false,
"inputs": [
{
"indexed": true,
"internalType": "address",
"name": "from",
"type": "address"
},
{
"indexed": true,
"internalType": "address",
"name": "to",
"type": "address"
},
{
"indexed": true,
"internalType": "uint256",
"name": "tokenId",
"type": "uint256"
}
],
"name": "Transfer",
"type": "event"
},
{
"constant": true,
"inputs": [],
"name": "GRACE_PERIOD",
"outputs": [{ "internalType": "uint256", "name": "", "type": "uint256" }],
"payable": false,
"stateMutability": "view",
"type": "function"
},
{
"constant": false,
"inputs": [
{ "internalType": "address", "name": "controller", "type": "address" }
],
"name": "addController",
"outputs": [],
"payable": false,
"stateMutability": "nonpayable",
"type": "function"
},
{
"constant": false,
"inputs": [
{ "internalType": "address", "name": "to", "type": "address" },
{ "internalType": "uint256", "name": "tokenId", "type": "uint256" }
],
"name": "approve",
"outputs": [],
"payable": false,
"stateMutability": "nonpayable",
"type": "function"
},
{
"constant": true,
"inputs": [{ "internalType": "uint256", "name": "id", "type": "uint256" }],
"name": "available",
"outputs": [{ "internalType": "bool", "name": "", "type": "bool" }],
"payable": false,
"stateMutability": "view",
"type": "function"
},
{
"constant": true,
"inputs": [
{ "internalType": "address", "name": "owner", "type": "address" }
],
"name": "balanceOf",
"outputs": [{ "internalType": "uint256", "name": "", "type": "uint256" }],
"payable": false,
"stateMutability": "view",
"type": "function"
},
{
"constant": true,
"inputs": [],
"name": "baseNode",
"outputs": [{ "internalType": "bytes32", "name": "", "type": "bytes32" }],
"payable": false,
"stateMutability": "view",
"type": "function"
},
{
"constant": true,
"inputs": [{ "internalType": "address", "name": "", "type": "address" }],
"name": "controllers",
"outputs": [{ "internalType": "bool", "name": "", "type": "bool" }],
"payable": false,
"stateMutability": "view",
"type": "function"
},
{
"constant": true,
"inputs": [],
"name": "ens",
"outputs": [
{ "internalType": "contract ENS", "name": "", "type": "address" }
],
"payable": false,
"stateMutability": "view",
"type": "function"
},
{
"constant": true,
"inputs": [
{ "internalType": "uint256", "name": "tokenId", "type": "uint256" }
],
"name": "getApproved",
"outputs": [{ "internalType": "address", "name": "", "type": "address" }],
"payable": false,
"stateMutability": "view",
"type": "function"
},
{
"constant": true,
"inputs": [
{ "internalType": "address", "name": "owner", "type": "address" },
{ "internalType": "address", "name": "operator", "type": "address" }
],
"name": "isApprovedForAll",
"outputs": [{ "internalType": "bool", "name": "", "type": "bool" }],
"payable": false,
"stateMutability": "view",
"type": "function"
},
{
"constant": true,
"inputs": [],
"name": "isOwner",
"outputs": [{ "internalType": "bool", "name": "", "type": "bool" }],
"payable": false,
"stateMutability": "view",
"type": "function"
},
{
"constant": true,
"inputs": [{ "internalType": "uint256", "name": "id", "type": "uint256" }],
"name": "nameExpires",
"outputs": [{ "internalType": "uint256", "name": "", "type": "uint256" }],
"payable": false,
"stateMutability": "view",
"type": "function"
},
{
"constant": true,
"inputs": [],
"name": "owner",
"outputs": [{ "internalType": "address", "name": "", "type": "address" }],
"payable": false,
"stateMutability": "view",
"type": "function"
},
{
"constant": true,
"inputs": [
{ "internalType": "uint256", "name": "tokenId", "type": "uint256" }
],
"name": "ownerOf",
"outputs": [{ "internalType": "address", "name": "", "type": "address" }],
"payable": false,
"stateMutability": "view",
"type": "function"
},
{
"constant": false,
"inputs": [
{ "internalType": "uint256", "name": "id", "type": "uint256" },
{ "internalType": "address", "name": "owner", "type": "address" }
],
"name": "reclaim",
"outputs": [],
"payable": false,
"stateMutability": "nonpayable",
"type": "function"
},
{
"constant": false,
"inputs": [
{ "internalType": "uint256", "name": "id", "type": "uint256" },
{ "internalType": "address", "name": "owner", "type": "address" },
{ "internalType": "uint256", "name": "duration", "type": "uint256" }
],
"name": "register",
"outputs": [{ "internalType": "uint256", "name": "", "type": "uint256" }],
"payable": false,
"stateMutability": "nonpayable",
"type": "function"
},
{
"constant": false,
"inputs": [
{ "internalType": "uint256", "name": "id", "type": "uint256" },
{ "internalType": "address", "name": "owner", "type": "address" },
{ "internalType": "uint256", "name": "duration", "type": "uint256" }
],
"name": "registerOnly",
"outputs": [{ "internalType": "uint256", "name": "", "type": "uint256" }],
"payable": false,
"stateMutability": "nonpayable",
"type": "function"
},
{
"constant": false,
"inputs": [
{ "internalType": "address", "name": "controller", "type": "address" }
],
"name": "removeController",
"outputs": [],
"payable": false,
"stateMutability": "nonpayable",
"type": "function"
},
{
"constant": false,
"inputs": [
{ "internalType": "uint256", "name": "id", "type": "uint256" },
{ "internalType": "uint256", "name": "duration", "type": "uint256" }
],
"name": "renew",
"outputs": [{ "internalType": "uint256", "name": "", "type": "uint256" }],
"payable": false,
"stateMutability": "nonpayable",
"type": "function"
},
{
"constant": false,
"inputs": [],
"name": "renounceOwnership",
"outputs": [],
"payable": false,
"stateMutability": "nonpayable",
"type": "function"
},
{
"constant": false,
"inputs": [
{ "internalType": "address", "name": "from", "type": "address" },
{ "internalType": "address", "name": "to", "type": "address" },
{ "internalType": "uint256", "name": "tokenId", "type": "uint256" }
],
"name": "safeTransferFrom",
"outputs": [],
"payable": false,
"stateMutability": "nonpayable",
"type": "function"
},
{
"constant": false,
"inputs": [
{ "internalType": "address", "name": "from", "type": "address" },
{ "internalType": "address", "name": "to", "type": "address" },
{ "internalType": "uint256", "name": "tokenId", "type": "uint256" },
{ "internalType": "bytes", "name": "_data", "type": "bytes" }
],
"name": "safeTransferFrom",
"outputs": [],
"payable": false,
"stateMutability": "nonpayable",
"type": "function"
},
{
"constant": false,
"inputs": [
{ "internalType": "address", "name": "to", "type": "address" },
{ "internalType": "bool", "name": "approved", "type": "bool" }
],
"name": "setApprovalForAll",
"outputs": [],
"payable": false,
"stateMutability": "nonpayable",
"type": "function"
},
{
"constant": false,
"inputs": [
{ "internalType": "address", "name": "resolver", "type": "address" }
],
"name": "setResolver",
"outputs": [],
"payable": false,
"stateMutability": "nonpayable",
"type": "function"
},
{
"constant": true,
"inputs": [
{ "internalType": "bytes4", "name": "interfaceID", "type": "bytes4" }
],
"name": "supportsInterface",
"outputs": [{ "internalType": "bool", "name": "", "type": "bool" }],
"payable": false,
"stateMutability": "view",
"type": "function"
},
{
"constant": false,
"inputs": [
{ "internalType": "address", "name": "from", "type": "address" },
{ "internalType": "address", "name": "to", "type": "address" },
{ "internalType": "uint256", "name": "tokenId", "type": "uint256" }
],
"name": "transferFrom",
"outputs": [],
"payable": false,
"stateMutability": "nonpayable",
"type": "function"
},
{
"constant": false,
"inputs": [
{ "internalType": "address", "name": "newOwner", "type": "address" }
],
"name": "transferOwnership",
"outputs": [],
"payable": false,
"stateMutability": "nonpayable",
"type": "function"
}
]
Let’s now head to the terminal window and type
npx squid-evm-typegen –help
This is another command from Subsquid’s SDK, and it generates TypeScript facades for EVM transactions, logs and calls. It needs two arguments: the first argument is the destination folder, the second one is the ABI location.
Let’s go ahead and launch the command
npx squid-evm-typegen src/abi ens.json
It has created 3 files, the interesting one is src/abi/ens.ts.
It defines two objects:
one has
LogEventclasses to decode smart contract Events and access their topicsanother one with
Funcclasses that allow us to decode smart contract functions and access their input
Devs can decide to consult one or the other, but oftentimes, Events can be more useful, because there might be additional information attached to them, at times.
For example, this project will focus on the Transfer Event.
And then, we have a Contract class to directly call smart contract functions that are not payable from our indexer. Often times, the tokenURI() function is used when handling ERC721, because it provides the URL of the token metadata, including the image URL.
Another use case for this class would be to fetch the contract’s metadata (instead of hard-coding it, like I am going to do later on), by calling name() , symbol() and other similar functions.
But in the case of ENS contract and tokens, these functions are not implemented.
Implement mapping logic
Now that we have defined a schema, generated models to interact with the database, generated code to interact with the contract, it's time to map the on-chain data to our database schema and models.
To implement these mappings, we need to work on the file named processor.ts in the src folder.
The processor is already correctly configured to connect to the Ethereum archive, but we need to uncomment the chain paramenter and add the RPC endpoint for direct communication with Ethereum nodes.
This is using an environment variable and I need to open up the .env file and add a value to this variable. I know that Ankr has public and free RPC endpoints, so let's add its URL:
DB_NAME=squid
DB_PORT=23798
GQL_PORT=4350
# JSON-RPC node endpoint, both wss and https endpoints are accepted
RPC_ENDPOINT="https://rpc.ankr.com/eth"
The template tracks transactions, and we are not going to do that, so we need to replace the addTransaction section with a new function call, named addLog. The section where the EvmBatchProcessor class is instantiated should look like this:
import { events } from "./abi/ens";
const contractAddress =
"0x57f1887a8BF19b14fC0dF6Fd9B2acc9Af147eA85".toLowerCase();
const processor = new EvmBatchProcessor()
.setDataSource({
chain: process.env.RPC_ENDPOINT,
archive: "https://eth.archive.subsquid.io",
})
.addLog(contractAddress, {
filter: [
[
events.Transfer.topic,
],
],
data: {
evmLog: {
topics: true,
data: true,
},
transaction: {
hash: true,
},
},
});
We have to specify the contract address, but that can be copied from the Etherscan page.
Then we need to specify what events from that contract we are interested in, using the ABI facade we previously generated. In our case, we want the Transfer event, and this is where the ENS contract facade code we generated comes in handy.
Note: The syntax for event filtering follows the standard from ethers library.
Finally we want to specify what data we want to receive from the Archive. We want the emvLog , with its topics and its data, and then we want the transaction that generated it, with its hash.
Then it’s time to work on the processor execution where processor.run() is executed. We need to start processing the blocks in the data bundle we receive, and the already present double for loop, allows us to iterate through blocks and items in them.
What we need to do is to make sure we are processing an evmLog, by verifying the kind of the item. This is useful for the strong typings provided by TypeScript.
I also add a check for the item address: in this case it’s redundant, but it’s good practice for when you might index events from multiple contracts.
Then, it’s usually necessary to verify the item’s topic is the same as the log we want to index, and we do this by comparing the item’s topic with the topic from the event class in our interface. As before, in this specific scenario, it’s redundant, but it’s once again good practice, for when you’ll be indexing multiple types of events.
Our code should look like this now:
processor.run(new TypeormDatabase(), async (ctx) => {
const ensDataArr: ENSData[] = [];
for (let c of ctx.blocks) {
for (let i of c.items) {
if (i.address === contractAddress && i.kind === "evmLog") {
if (i.evmLog.topics[0] === events.Transfer.topic) {
// what to do with the item now?!
}
}
}
}
});
But we need to extract the event information, and for that, I say we create a new function. We’ll simply call it handleTransfer.
To fully leverage the strong typings of TypeScript, we should be creating an object that will be the return value of these functions, and will contain all the fields we need.
I'm naming it ENSData and add the same fields we defined in our schema.
type ENSData = {
id: string;
from: string;
to: string;
tokenId: bigint;
timestamp: bigint;
block: number;
transactionHash: string;
};
As for the handleTransfer function, we need to pass it a special context, and its type is called LogHandlerContext (imported from the @subsquid/evm-processor library). This is using generics and we need to provide the Store class (imported from the @subsquid/typeorm-store library), and an object, representing the item we are indexing, which we can copy directly from the processor settings at the top. Then we add ENSData as the return type.
Inside the function, we can use the generated contract interface to decode the event. And we can add the right fields and types to the object we want to return. It should look like this now.
function handleTransfer(
ctx: LogHandlerContext<
Store,
{ evmLog: { topics: true; data: true }; transaction: { hash: true } }
>
): ENSData {
const { evmLog, block, transaction } = ctx;
const { from, to, tokenId } = events.Transfer.decode(evmLog);
const ensData: ENSData = {
id: `${transaction.hash}-${evmLog.address}-${tokenId.toBigInt()}-${
evmLog.index
}`,
from,
to,
tokenId: tokenId.toBigInt(),
timestamp: BigInt(block.timestamp),
block: block.height,
transactionHash: transaction.hash,
};
return ensData;
}
Then we need to save all the information we collected in our for loop to the database, and for this, I say we create a new function, we name it saveENSData.
This will be asynchronous, because we’ll interact with the database. We pass it a BlockHandler context, which we can use to interface with the database, and the array of transfer data. Let me show you the code, and I'll comment it below.
async function saveENSData(
ctx: BlockHandlerContext<Store>,
ensDataArr: ENSData[]
) {
const tokensIds: Set<string> = new Set();
const ownersIds: Set<string> = new Set();
for (const ensData of ensDataArr) {
tokensIds.add(ensData.tokenId.toString());
if (ensData.from) ownersIds.add(ensData.from.toLowerCase());
if (ensData.to) ownersIds.add(ensData.to.toLowerCase());
}
const tokens: Map<string, Token> = new Map(
(await ctx.store.findBy(Token, { id: In([...tokensIds]) })).map((token) => [
token.id,
token,
])
);
const owners: Map<string, Owner> = new Map(
(await ctx.store.findBy(Owner, { id: In([...ownersIds]) })).map((owner) => [
owner.id,
owner,
])
);
const transfers: Set<Transfer> = new Set();
for (const ensData of ensDataArr) {
const {
id,
tokenId,
from,
to,
block,
transactionHash,
timestamp,
} = ensData;
let fromOwner = owners.get(from);
if (fromOwner == null) {
fromOwner = new Owner({ id: from.toLowerCase() });
owners.set(fromOwner.id, fromOwner);
}
let toOwner = owners.get(to);
if (toOwner == null) {
toOwner = new Owner({ id: to.toLowerCase() });
owners.set(toOwner.id, toOwner);
}
const tokenIdString = tokenId.toString();
let token = tokens.get(tokenIdString);
if (token == null) {
token = new Token({
id: tokenIdString,
contract: await getOrCreateContractEntity(ctx.store),
});
tokens.set(token.id, token);
}
token.owner = toOwner;
if (toOwner && fromOwner) {
const transfer = new Transfer({
id,
block,
timestamp,
transactionHash,
from: fromOwner,
to: toOwner,
token,
});
transfers.add(transfer);
}
}
await ctx.store.save([...owners.values()]);
await ctx.store.save([...tokens.values()]);
await ctx.store.save([...transfers]);
}
We can start with identifying all the tokenIds and ownerIds we find by iterating through the ENS data and store the ids in the respective sets.
Then, we need to find the tokens that already exist in the database and map the tokenId to the database model, this is so we can avoid verifying for existence on the database every time and use the map instead, which is much faster. Then we are doing the exact same thing for the owners.
Next, there is another for loop over ENSData to process the single transfer, and eventually update or create tokens and token owners (senders and receivers), using the map we previously built.
A small interjection here, because when creating a new Token (if (token == null)) we need to link it to a Contract model on the database. For this, a getOrCreateContractEntity function, with a singleton instance is created outside the function:
let contractEntity: Contract | undefined;
export async function getOrCreateContractEntity(
store: Store
): Promise<Contract> {
if (contractEntity == null) {
contractEntity = await store.get(Contract, contractAddress);
if (contractEntity == null) {
contractEntity = new Contract({
id: contractAddress,
name: "Ethereum Name Service",
symbol: "ENS",
totalSupply: 0n,
});
await store.insert(contractEntity);
}
}
return contractEntity;
}
I want to build a get or create contract entity function, that will manage a singleton instance of the contract model.
We check if the singleton is defined, and if not, we check that a contract with the given address exists, and if not we create it. But when creating it, we need some information, like name, symbol and total supply.
In other instances, this information should be available by interacting with the Contract API from the code we generated with our ABI. As mentioned earlier, this is not the case for ENS, as this contract does not implement functions like name() or totalSupply(). Luckily, we can easily obtain that in our contract page on Etherscan.
Back to our saveENSData function and the newly created Token instance, we add the contract model and add the new instance to the map. The token owner should always be updated, whether it already exists on the database, or we just created it, so this operation is done after the Token creation block.
The next part is about creating a new instance of the Transfer database model, adding all the necessary information, and adding the model itself to the set of Transfer models we created earlier.
At the very end of the function, we save all the sets we have populated. Thanks to the store database interface, we can save in batch, which is more efficient and will increase the indexing performance.
Once the handleTransfer and saveENSData functions are done, we can go back and call our functions to get the data we want. We need to build the LogHandlerContext argument, with the current context, the block, and the item itself.
Because we are getting transfer data for every EVM log we encounter, we need an array to store them and we need to add the single event data to it inside the loop.
The function call to saveENSData, is done outside the double for loop, to save all of our work. The processor.run() function call should look like this now:
processor.run(new TypeormDatabase(), async (ctx) => {
const ensDataArr: ENSData[] = [];
for (let c of ctx.blocks) {
for (let i of c.items) {
if (i.address === contractAddress && i.kind === "evmLog") {
if (i.evmLog.topics[0] === events.Transfer.topic) {
const ensData = handleTransfer({
...ctx,
block: c.header,
...i,
});
ensDataArr.push(ensData);
}
}
}
}
await saveENSData(
{
...ctx,
block: ctx.blocks[ctx.blocks.length - 1].header,
},
ensDataArr
);
});
We have finished implementing the indexing logic, we made sure to extract information from the EVM logs of the smart contract execution that are stored on-chain and we mapped this to our Data models. Then we made sure to save everything on the database and we saw how to do this efficiently.
Once again, if you are unsure about the code we developed so far, or just want to double check, you can verify your work against the code in this repository.
We now need to compile our code, run it, then launch the GraphQL server to inspect the result of our work.
Run and verify
In order to launch the project and confirm our work so far, the first thing we have to do is use a terminal window and run this command:
make up
This will read the docker-compose.yml file in the root folder of the project (we explored it at the end of lesson 1), and create a new Docker container with our database.
Next, we need to delete any files in the db/migrations folder. This is because we have a new schema, so we have to create a new migration file as well. These are files filled with SQL statements to create tables and apply properties to them.
Then I am going to build the project, by running this command in a console window:
make build
This is important because we need our compiled to be up to date, in order to generate the right migration file.
And now, let’s create the migration file, using the automated tool we saw in lesson number one, just using a different parameter.
In our terminal, let’s type
npx squid-typeorm-migration generate
A new file will be created, we can see it under the db/migrations folder.
Let’s now launch the project by running this command in the console:
make process
This will check for migrations that have not been applied yet and will run the compiled JavaScript of our processor.ts file.
You should logs, showing us the status of processing, compared to the total number of blocks on the chain, as well as some velocity metrics. The logs will block the terminal, so open another console window, and launch te command
make serve
To start the GraphQL server that comes with the template. This will be useful to inspect the data we are generating. The logs say it’s listening to port 4350, so you can now open a browser window to the URL: http://localhost:4350/graphql
Here we have a GraphQL playground, where we can type in an arbitrary query in the center, or use the suggestions on the left.
One of the first things we can do is verify how many tokens we have found so far, using the tokensConnection query, which has a totalCount field.
query MyQuery {
tokensConnection(orderBy: id_ASC) {
totalCount
}
}
Now let’s see how many transfers we have found so far, using the transfersConnection query.
query MyQuery {
transfersConnection(orderBy: id_ASC) {
totalCount
}
}
We can also inspect some of these transfers using the transfers query. Since there’s more than ten thousand transfers, I am not going to fetch them all, but I am going to put a limit of ten, and order them by block number, basically asking for the latest.
We can get some information about the tokens exchanged in these transfers, using this nested object notation. And about the contract as well, just to have an idea.
query MyQuery {
transfers(limit: 10, orderBy: block_DESC) {
token {
id
contract {
id
symbol
name
}
}
block
from {
id
}
to {
id
}
}
}
There’s one more thing I want to show you. If we switch the ordering, let’s also only ask for the block number, for simplicity, we can verify the block number where the first transfer of this smart contract was generated.
query MyQuery {
transfers(limit: 10, orderBy: block_ASC) {
block
}
}
We can use this information to further optimize our project. Let’s go back to our IDE and in the section where we configure the processor, we can add a new line and call the setBlockRange function.
const processor = new EvmBatchProcessor()
.setDataSource({
chain: process.env.RPC_ENDPOINT,
archive: "https://eth.archive.subsquid.io",
})
.setBlockRange({ from: 9380427 })
.addLog(contractAddress, {
filter: [
[
events.Transfer.topic,
],
],
data: {
evmLog: {
topics: true,
data: true,
},
transaction: {
hash: true,
},
},
});
As the name intuitively indicates, this will configure the range inside which the processor will be executing. In our case, we want to set a starting point right before the first Transfer, so we collect all the information we need, ignoring the initial portion of the blockchain.
We can stop the processor (by hitting Ctrl+C in the terminal where we launched it), then reset the database with these two commands in succession:
make down
make up
And launch the processor again:
make process
Nothing should actually change, but this was just an additional feature I wanted to show you, and you can use it to further optimize your indexer.
This is the end of this section, and the end of our second lesson as well.
We have learnt how to index execution of EVM smart contracts on Ethereum using Subsquid. Starting with the template repository, customizing it to our specific contract and project, and we also learned how to implement our custom mapping logic.
Stay tuned for the next instalments in this series, the next one will be about hosting our project in Subsquid’s hosting service, called Aquarium.
Subsquid socials: