03 Feb Kick Back & Reap the Harvest

This is the tale of a NEM enthusiast turned Symbol node operator, and details the script he wrote to simplify node reward kickbacks.

Begin at the beginning…

Harvesting NEM in the early days

Harvesting in NEM’s early days would occasionally hit a lucky block with 100+ XEM in transaction fees from some enormous transfer; this rare prize amounted to less than $0.01 USD at the time, and a nice surprise when most blocks came up empty. NEM quickly outgrew its initial fee structure and a substantial reduction in fees directly impacted block rewards. Harvesting NEM was never particularly profitable, it was more about participating than profiting – a sentiment likely shared by early Bitcoin miners. The Bitcoin whitepaper said “Proof-of-work is essentially one-CPU-one-vote”; I always read this as an unwritten dogma of inclusive decentralization. By the time NEM entered the scene, meaningful participation in Bitcoin’s POW was increasingly exclusive and rapidly centralizing – CPUs no longer had much of a vote. The copycat altcoin boom was also in full swing with a long list of rug pulls and hollow promises. Only a handful of projects valued innovation or inclusion over profit, and NEM was among them. Founded on principles of inclusivity and fairness, it attracted a passionate community and a superb team of developers. I didn’t harvest NEM for profit – I harvested NEM because it seemed to capture elements of Satoshi’s vision that were increasingly left behind as Blockchain evolved.

From XEM to XYM

When Symbol came out I expected it to follow in NEM’s footsteps with negligible fee-based block rewards. How wrong I was! I was shocked to see a whopping 134 XYM reward for my first harvested block, and initially dismissed as a fluke. The next few blocks I harvested highlighted my ignorance of Symbol’s tokenomics and the massive trove of XYM slated for block rewards.  I also learned that nodes receive 25% the block rewards from delegated harvesters, meaning a quarter of my harvest was getting skimmed by a node operator. I ran some numbers and found that hosting a node to keep the 25% node reward would more than pay for itself. I also saw that a handful of node operators were offering harvesters a kickback on their node rewards; a quarter of the harvest seemed like a steep price to pay, and kickbacks seemed like a great solution. I imagined a future where node operators light-heartedly competed for harvesters through their contributions to the Symbol ecosphere; engaging users and doing what they could to support the platform – something reminiscent of the support and engagement of Bitcoin mining pool operators in the early days.. Kickbacks fall far short of this vision, but it seemed a solid first step towards connecting with harvesters and building engagement.

Firing up a Node

For the sake of speed and simplicity I chose http://allnodes.me for my node; it’s hard to beat speed and convenience when rewards are at stake! Allnodes was incredibly easy and fast to set up, has impeccable uptime, and was quick to deploy new releases.. I incentivised a few harvesters with the promise of generous kickbacks, but this meant actively monitoring my node’s harvest to manage payouts. At first I used Symbol’s blockchain explorer to figure out which blocks others had harvested on my node. This was time consuming, and left me worrying that I would miss a block or double-pay a kickback.. I quickly realised that any successful node with kickbacks would need a better solution, so I started building one.

Writing a Harvest Monitor with Kickbacks In Mind

When it comes to building on the Symbol platform, the powerful JavaScript-based Command Line Interface (CLI) is an obvious choice.. I ended up choosing a different path due to my lack of Java experience and an interest in exploring Symbol’s REST APIs. NEM’s robust APIs were always a highlighted feature, so I was curious to see if I could make due with Symbol’s API. I like that APIs remove barriers to development and integration; they’re very accessible and don’t require a specific programming language or custom modules.. Many eyes glaze over when terms like ‘API’ come up, but many API endpoints are just ugly websites with awkward text – you don’t have to be a tech whiz to use them! 

I started by playing around with a handful of endpoints in my browser to figure out where I could get the information I was looking for, then began writing a python script to do the legwork. Navigating Symbol’s REST APIs was a bit of a challenge – it took me some time to realise the API documentation site has a lot of click-to-reveal content, so I was working with a lot of missing details. After a few hours of research and experimentation I finally got things working – I was able to track blocks harvested by delegators at the click of a button! The breakthrough was exhilarating, and it wasn’t long before I had a functional script. The vision of node operators as collaborative contributors had stuck with me, so I went out of my way to share and improve on my solution – hoping to inspire other node operators with the spirit of collaboration and provide some sample code to prospective developers. 

A Handy Alternative

Since writing this script I’ve found https://symbol-tools.com/ to be a handy alternative. Their node list tracks delegated harvesters and the last 50 harvested blocks on each node; the inclusion of  harvest date is a nice touch that you won’t find so easily in the official wallet or explorer.

Using the Node Harvest Kickback Tracker script

Source: https://github.com/orisdorch/XYM-NodeHarvestKickbackTracker-API

Prerequisite: Python v3.7.6 or greater (other versions may not be compatible) with the following libraries: requests, base64, datetime

Recommended: An operational API node used by yourself with other harvesters

Configuring and running the script:

1. Download or copy & paste the source code from https://github.com/orisdorch/XYM-NodeHarvestKickbackTracker-API to create a local .py script
2. Edit the user defined variables portion of the script (to define your node, harvesting details, and the local log file location):

### USER DEFINED VARIABLES - you will need to provide these! ###
APINodeURL = 'http://my.node.me:3000'   #Provide the node that should be queried and relevant port (can be your own node or any other API node)
HarvestedBlocksLocalFilePath = 'C:\\Users\\USER\\Desktop\\harvestedblocks.txt'   #Provide the location of the (headerless) csv file that will be used to track blocks harvested by others
myBeneficiaryAddress = 'SAAA244WMCB2JXGNQTQHQOS45TGBFF4V2MJBVOQ'   # Your node's beneficiary address (without hyphens) - this is where node rewards are sent, and is used to identify blocks harvested on your node. note: use hexToAddress() to convert a public key to an address
myHarvesterAddress = 'SAAA244WMCB2JXGNQTQHQOS45TGBFF4V2MJBVOQ'   # Additional harvesting address to be identified as yours (allnodes doesn't allow you to specify the node reward address, so use your primary harvesting address here. Otherwise this might be your node's delegation address - i.e. 'nodePublicKey' in http://my.node/node/info )
myOtherAddress = myHarvesterAddress   #Optional third address to identify as yours. Use myHarvesterAddress if a third address is not required.
kickbackMultiplier = 0.2   # percent to kickback to harvesters (i.e. 0.2 = 20%)

3. Run the script for the first time
4. This will create a new log file, then query the Symbol blockchain to search for blocks harvested by delegates harvesting on your node. A ‘kickback’ amount is calculated and the status is defaulted as unpaid

a previous output file has been found at the target location ... 
retrieving last block from previous output file 
Previous output file found and last block identified as 685121

Collecting paged data from http://xym123.allnodes.me:3000/statements/transaction?type=8515&artifactId=6BED913FA20223F8&targetAddress=SABQ123WMCB2JXGNQTQHQOS45TGBFF4V2MJBVOD&fromHeight=685121&pageSize=100&pageNumber=1 Page 1 is the last page of results, analyzing results…
{'height': '685123', 'harvestAmount': '123163852', 'nodeReward(satoshis)': '43987089', 'kickbackAmount': 'N/A', 'Address': ‘SABQ123WMCB2JXGNQTQHQOS45TGBFF4V2MJBVOD’, 'Date/Time': '2021-11-10 10:28:55', 'Status': 'Harvested by me'}
Collecting data from http://mynode.allnodes.me:3000/blocks/686123
{'height': '686123', 'harvestAmount': '123119500', 'nodeReward(satoshis)': '3971249', 'kickbackAmount': 'N/A', 'Address': ‘SABQ123WMCB2JXGNQTQHQOS45TGBFF4V2MJBVOD’, 'Date/Time': '2021-11-11 09:25:05', 'Status': 'Harvested by me'}
Collecting data from http://mynode.allnodes.me:3000/blocks/687123
{'height': '687123', 'harvestAmount': '123819365', 'nodeReward(satoshis)': ‘4221201', 'kickbackAmount': 'N/A', 'Address': ‘NABQ123WMCB2JXGNQTQHQOS45TGBFF4V2MJBVOD’, 'Date/Time': '2021-11-11 16:36:28', 'Status': ‘Unpaid’}
Collecting data from http://mynode.allnodes.me:3000/blocks/698123
{'height': '698123', 'harvestAmount': '123119500', 'nodeReward(satoshis)': '3971249', 'kickbackAmount': 'N/A', 'Address': ‘SABQ123WMCB2JXGNQTQHQOS45TGBFF4V2MJBVOD’, 'Date/Time': '2021-11-12 13:36:49', 'Status': 'Harvested by me'}
Collecting data from http://mynode.allnodes.me:3000/blocks/691123
{'height': '691123’, 'harvestAmount': '123119500', 'nodeReward(satoshis)': '3971249', 'kickbackAmount': 'N/A', 'Address': ‘NABQ123WMCB2JXGNQTQHQOS45TGBFF4V2MJBVOD’, 'Date/Time': '2021-11-12 22:14:05', 'Status': 'Unpaid’}

2 blocks harvested by others and 3 by you since the last check

5. Details of all harvested blocks are stored in the log file, and blocks harvested by others default to an UNPAID status. Once harvesters are manually paid out, the node operator can update the log file to mark them as paid – changes to payment status will be picked up the next time the script is run. The script does not automatically re-query, as payments are not automated. The intent is for the node operator to run the script periodically when they have a chance to send kickbacks. For example, on a daily, weekly, or monthly basis.

Sample log file contents:
height,123456,harvestAmount,134422724,nodeReward(satoshis),48008115,kickbackAmount,9.601623,Address,SAAA244WMCB2JXGNQTQHQOS45TGBFF4V2MJBVOQ,Date/Time,2021-05-02 13:37:51,Status,PAID,
height,123457,harvestAmount,134422724,nodeReward(satoshis),48008115,kickbackAmount,N/A,Address,SAAB244WMCB2JXGNQTQHQOS45TGBFF4V2MJBVOQ,Date/Time,2021-05-02 13:37:51,Status,Harvested by me, 

6. Once the query is complete, you are prompted with a menu allowing you to re-query the blockchain or use other features such as listing unpaid blocks, listing addresses actively harvesting on your node, displaying the total number of blocks harvested on your node, and listing all blocks harvested by you or others.

Enter: 
   1 to refresh data from blockchain 
   2 to show unpaid block kickbacks 
   3 to show details of all blocks harvested by others 
   4 to show details of all harvested blocks 
   5 for total # of harvested blocks on your node (including yours) 
   6 to see harvesters delegated to your node (incl. node beneficiary address) 
  x to exit 

That’s really all there is to it – a simple tool to identify and track harvested blocks and payments with a few extra features thrown in. Node operators, developers, and harvesters: feel free to reach out with any questions or challenges you might be having with the script (Orisdorch on Discord) – I’m happy to help! My only hope is that this adds value in one way or another to the Symbol community.