17 Jul Getting started with Symbol in Python – part 3 (part 2 😂)

Posted at 09:19h in Developer & Guides, Harvesting by
0 Likes

This is and update on the last @nobu_kyutech example which incorporates some extra features based on my own code and code from @kurikou02‘s qiita post. Thank you!

Introduction

Hi guys, I am back! Just a quick update as I saw this post from @kurikou02 and thought it was cool so I integrated with the code from our last post on using Python to obtain the harvesting history of an account. The updated version will add a timestamp to return the date (and if you want to time) of a harvest. I have also added my own code which will make a plot of cumulative harvest income over time and give some additional stats. So.. let’s get started (again!).

First we need to import some Python packages (this time we need a few more).

import urllib.request
import json
import datetime
import matplotlib.pyplot as plt
import pandas as pd

Next we need to set up our node, account and the transaction type we want to fetch (in our case the harvest fee):

# Set the node URL
NODEURL = "http://xymharvesting.net:3000"

# Set the address of the account that we want to query
ADDRESS = "NANZVLZJTVXAK2772CAETNCFXL7FRVTJNGYAH4Q"

# Set the enum value of the receipt "Harvest_Fee"
ENUM_Harvest_Fee = 8515

Getting block dates

This next section sets up some functions to obtain the time and date that a block was harvested and is taken from @kurikou02‘s article (thank you!):

# Genesis block time
NEMESIS_BLOCK_TIMESTAMP = "2021-03-16 00:06:25"

# Function to get block information from height

def get_blocks( height ):
    req = urllib.request.Request(NODEURL + '/blocks/' + height)
    with urllib.request.urlopen(req) as res:
        return json.load(res)

# Get the block generation date and time
def get_datetime( height ):
    data = get_blocks( height ) 
    # Returns milliseconds since the Nemesis block was generated
    ts = int(data['block']['timestamp'])
    nb_ts = datetime.datetime.strptime(NEMESIS_BLOCK_TIMESTAMP, '%Y-%m-%d %H:%M:%S')
    dt = nb_ts + datetime.timedelta(milliseconds=ts)
#    return datetime.datetime.strftime(dt,'%Y-%m-%d %H:%M:%S')
    return datetime.datetime.strftime(dt,'%Y-%m-%d')

We now get the hexadecimal account address (this is explained in the previous post):

# Look up the 48 character hexadecimal format version of ADDRESS

req = urllib.request.Request(NODEURL + '/accounts/' + ADDRESS)
with urllib.request.urlopen(req) as res:
    accountInfo = json.load(res)
ADDRESS48 = accountInfo['account']['address']

Obtaining *all* blocks harvested

Next we run the code to retrieve our harvest history. Now I have made a few changes to the previous post. I will go through them below:

  1. We have put the code to fetch harvest income into a loop so that it goes through every page of transactions to provide ALL harvesting income for an account (not just the first page)
  2. We have added a call to @kurikou02‘s function to return the time that a block was harvested
  3. We not only print the harvest information out to screen but we store it in a dictionary with the date as a key and the cumulative harvest income as the value. We will use this later to plot these data
  4. We keep a count of the total harvesting income which we print when we have obtained all harvesting events

# Now we can get a list of all statements in descending order (newest first)
# The URL would be http://xymharvesting.net:3000/statements/transaction?targetAddress=NBMWZZZOCUKZXXHKB7R3P3G2ZCO4CSGIXC37USI&order=desc

i = 1
total = 0
output = {}

url = NODEURL + '/statements/transaction'
params = {
    'targetAddress': ADDRESS,
    'order': 'asc',
    'pageNumber' : i,
}

req = urllib.request.Request('{}?{}'.format(url, urllib.parse.urlencode(params)))
with urllib.request.urlopen(req) as res:
    data = json.load(res)
    
    # We need to go through the results and find only those receipt types equal to "Harvest_Fee" which we store as ENUM_Harvest_Fee
    # Then we print the block height and the amount of XYM harvested (we divide by 1,000,000 to return the amount in XYM)
    # This time we have this code in a while loop and it will increment a page counter each time we return a page of results
    # This allows us to obtain ALL occasions on which the account has harvested a block
    
while data['data']:
    for d in data['data']:
        height = d['statement']['height']
        for receipt in d['statement']['receipts']:
            if receipt['type'] == ENUM_Harvest_Fee and receipt['targetAddress'] == ADDRESS48:
                amount = int(receipt['amount'])/1000000
                time = get_datetime(height)
                #Here we print the block, harvest date and amount harvested 
                print("date:{}, height:{}, amount:{}".format(time, height, amount))
                # I keep track of the total amount harvested:
                total = total+amount
                # Next I add the date as a dictionary key if the key is not present with the current total harvest income as the value
                # If the key does exist then I just add the harvest amount to the existing total for that key (that day)
                if time in output.keys():
                    output[time]+=amount
                else:
                    output[time]=total
    i=i+1
    # Here is where we increment the page number to make sure we get ALL harvest events (not just the first page)
    params = {
    'targetAddress': ADDRESS,
    'order': 'asc',
    'pageNumber' : i,
    }
    req = urllib.request.Request('{}?{}'.format(url, urllib.parse.urlencode(params)))
    with urllib.request.urlopen(req) as res:
        data = json.load(res)

# Here I print the total amount that account has harvested

print ("TOTAL:{}".format(total))

We can see from the output that all harvested blocks are printed (this time oldest -> newest) along with the block height and the amount. The total harvest income is printed at the end – in this case the account has harvested a total of 13529.82 XYM at time of writing.

date:2021-03-18, height:2971, amount:7.491807
date:2021-03-18, height:3707, amount:6.704021
date:2021-03-18, height:3928, amount:3.282062
date:2021-03-18, height:4823, amount:11.914786
date:2021-03-19, height:7907, amount:182.95082
date:2021-03-19, height:8237, amount:185.422051
date:2021-05-01, height:131319, amount:48.06607
date:2021-05-02, height:134135, amount:48.096028
date:2021-05-03, height:136550, amount:134.39793
date:2021-05-03, height:136550, amount:47.99926
date:2021-05-05, height:142041, amount:47.99926
date:2021-05-05, height:142098, amount:48.077236
date:2021-05-06, height:144504, amount:134.4736
date:2021-05-06, height:144504, amount:48.026285
date:2021-05-06, height:146394, amount:47.99926
date:2021-05-09, height:153662, amount:134.42992
date:2021-05-09, height:153662, amount:48.010685
date:2021-05-09, height:154795, amount:135.097684
date:2021-05-09, height:154795, amount:48.249172
date:2021-05-10, height:156874, amount:134.52491
date:2021-05-10, height:156874, amount:48.04461
date:2021-05-11, height:160183, amount:47.99926
date:2021-05-11, height:160626, amount:47.99926
date:2021-05-11, height:161096, amount:134.44126
date:2021-05-11, height:161096, amount:48.014735
date:2021-05-12, height:162508, amount:134.39793
date:2021-05-12, height:162508, amount:47.99926
date:2021-05-13, height:164709, amount:48.003685
date:2021-05-13, height:165676, amount:134.39793
date:2021-05-13, height:165676, amount:47.99926
date:2021-05-16, height:174526, amount:128.717976
date:2021-05-16, height:174526, amount:45.970705
date:2021-05-16, height:174676, amount:45.94113
date:2021-05-17, height:176796, amount:128.704638
date:2021-05-17, height:176796, amount:45.965942
date:2021-05-17, height:178270, amount:45.953626
date:2021-05-18, height:179241, amount:45.953536
date:2021-05-18, height:179683, amount:128.676746
date:2021-05-18, height:179683, amount:45.95598
date:2021-05-19, height:182222, amount:128.700557
date:2021-05-19, height:182222, amount:45.964484
date:2021-05-20, height:184329, amount:45.95483
date:2021-05-22, height:190328, amount:128.719928
date:2021-05-22, height:190328, amount:45.971402
date:2021-05-22, height:191886, amount:128.635166
date:2021-05-22, height:191886, amount:45.94113
date:2021-05-23, height:195301, amount:45.950055
date:2021-05-23, height:195598, amount:45.94113
date:2021-05-24, height:195822, amount:45.945755
date:2021-05-25, height:201057, amount:128.635166
date:2021-05-25, height:201057, amount:45.94113
date:2021-05-26, height:202469, amount:128.635166
date:2021-05-26, height:202469, amount:45.94113
date:2021-05-26, height:204360, amount:128.635166
date:2021-05-26, height:204360, amount:45.94113
date:2021-05-28, height:208746, amount:128.635166
date:2021-05-28, height:208746, amount:45.94113
date:2021-05-28, height:208952, amount:128.666246
date:2021-05-28, height:208952, amount:45.95223
date:2021-05-28, height:209710, amount:45.94113
date:2021-05-30, height:215334, amount:45.94113
date:2021-05-31, height:218384, amount:128.647556
date:2021-05-31, height:218384, amount:45.945555
date:2021-06-01, height:219846, amount:45.94113
date:2021-06-01, height:220246, amount:45.946788
date:2021-06-01, height:221035, amount:128.635166
date:2021-06-01, height:221035, amount:45.94113
date:2021-06-02, height:221691, amount:128.635166
date:2021-06-02, height:221691, amount:45.94113
date:2021-06-02, height:223947, amount:128.67305
date:2021-06-02, height:223947, amount:45.95466
date:2021-06-05, height:230870, amount:45.94113
date:2021-06-06, height:235476, amount:128.635166
date:2021-06-06, height:235476, amount:45.94113
date:2021-06-06, height:235520, amount:128.635166
date:2021-06-06, height:235520, amount:45.94113
date:2021-06-07, height:237184, amount:128.635166
date:2021-06-07, height:237184, amount:45.94113
date:2021-06-09, height:243076, amount:128.635166
date:2021-06-09, height:243076, amount:45.94113
date:2021-06-09, height:244345, amount:128.635166
date:2021-06-09, height:244345, amount:45.94113
date:2021-06-10, height:245362, amount:128.635166
date:2021-06-10, height:245362, amount:45.94113
date:2021-06-15, height:261748, amount:45.94113
date:2021-06-17, height:265443, amount:45.94113
date:2021-06-17, height:266727, amount:128.711612
date:2021-06-17, height:266727, amount:45.968432
date:2021-06-17, height:266864, amount:128.692208
date:2021-06-17, height:266864, amount:45.961502
date:2021-06-17, height:267030, amount:45.94113
date:2021-06-18, height:268888, amount:128.635166
date:2021-06-18, height:268888, amount:45.94113
date:2021-06-19, height:271936, amount:128.635166
date:2021-06-19, height:271936, amount:45.94113
date:2021-06-19, height:272218, amount:128.635166
date:2021-06-19, height:272218, amount:45.94113
date:2021-06-20, height:275326, amount:45.94113
date:2021-06-21, height:277240, amount:46.191082
date:2021-06-21, height:278368, amount:128.635166
date:2021-06-21, height:278368, amount:45.94113
date:2021-06-22, height:281708, amount:128.635166
date:2021-06-22, height:281708, amount:45.94113
date:2021-06-24, height:285000, amount:128.679608
date:2021-06-24, height:285000, amount:45.957002
date:2021-06-25, height:288337, amount:128.668139
date:2021-06-25, height:288337, amount:45.952906
date:2021-06-27, height:295139, amount:45.94113
date:2021-06-27, height:295468, amount:128.635166
date:2021-06-27, height:295468, amount:45.94113
date:2021-06-27, height:296431, amount:45.990036
date:2021-06-28, height:297766, amount:128.635166
date:2021-06-28, height:297766, amount:45.94113
date:2021-06-29, height:301550, amount:45.94843
date:2021-07-01, height:306851, amount:45.944586
date:2021-07-02, height:308316, amount:46.191042
date:2021-07-02, height:308951, amount:45.94113
date:2021-07-04, height:314318, amount:128.635166
date:2021-07-04, height:314318, amount:45.94113
date:2021-07-04, height:316381, amount:45.948522
date:2021-07-05, height:317662, amount:128.635166
date:2021-07-05, height:317662, amount:45.94113
date:2021-07-06, height:319818, amount:45.94113
date:2021-07-06, height:321830, amount:128.635166
date:2021-07-06, height:321830, amount:45.94113
date:2021-07-06, height:322136, amount:45.942718
date:2021-07-07, height:322410, amount:46.191042
date:2021-07-07, height:323447, amount:45.946023
date:2021-07-07, height:324092, amount:45.946128
date:2021-07-07, height:324293, amount:128.648151
date:2021-07-07, height:324293, amount:45.945768
date:2021-07-07, height:324743, amount:45.944918
date:2021-07-07, height:324952, amount:45.954943
date:2021-07-08, height:325447, amount:45.94334
date:2021-07-08, height:325981, amount:45.952733
date:2021-07-08, height:327161, amount:46.189686
date:2021-07-08, height:327406, amount:45.94113
date:2021-07-08, height:327662, amount:46.141057
date:2021-07-09, height:328347, amount:45.94113
date:2021-07-09, height:328918, amount:45.95358
date:2021-07-09, height:329026, amount:45.943374
date:2021-07-09, height:330175, amount:45.94113
date:2021-07-10, height:331577, amount:128.647486
date:2021-07-10, height:331577, amount:45.94553
date:2021-07-10, height:331586, amount:129.345237
date:2021-07-10, height:331586, amount:46.194727
date:2021-07-10, height:331706, amount:45.94113
date:2021-07-10, height:333159, amount:45.94113
date:2021-07-10, height:333204, amount:45.942949
date:2021-07-11, height:334226, amount:45.94113
date:2021-07-11, height:334898, amount:128.655903
date:2021-07-11, height:334898, amount:45.948536
date:2021-07-11, height:335226, amount:45.944415
date:2021-07-11, height:335489, amount:45.94553
date:2021-07-12, height:337200, amount:46.441097
date:2021-07-12, height:337703, amount:46.191042
date:2021-07-12, height:337717, amount:45.94113
date:2021-07-12, height:338024, amount:45.947178
date:2021-07-12, height:338368, amount:45.94221
date:2021-07-12, height:338650, amount:45.94113
date:2021-07-12, height:338653, amount:45.94113
date:2021-07-12, height:338706, amount:45.94113
date:2021-07-12, height:338910, amount:45.94113
date:2021-07-12, height:338985, amount:45.94113
date:2021-07-12, height:339311, amount:45.943045
date:2021-07-12, height:339525, amount:45.945555
date:2021-07-13, height:340603, amount:45.96697
date:2021-07-13, height:342059, amount:45.94113
date:2021-07-13, height:342333, amount:45.94113
date:2021-07-14, height:342602, amount:45.94113
date:2021-07-14, height:342606, amount:45.945855
date:2021-07-14, height:343142, amount:45.94113
date:2021-07-14, height:343256, amount:45.952906
date:2021-07-14, height:343599, amount:45.94533
date:2021-07-14, height:344277, amount:128.635166
date:2021-07-14, height:344277, amount:45.94113
date:2021-07-14, height:344837, amount:128.635166
date:2021-07-14, height:344837, amount:45.94113
date:2021-07-14, height:344863, amount:45.94113
date:2021-07-15, height:345754, amount:45.944568
date:2021-07-15, height:345913, amount:45.95653
date:2021-07-15, height:345925, amount:45.952906
date:2021-07-15, height:346251, amount:45.94113
date:2021-07-15, height:346281, amount:45.94113
date:2021-07-15, height:346551, amount:128.635166
date:2021-07-15, height:346551, amount:45.94113
date:2021-07-15, height:346569, amount:45.945955
date:2021-07-15, height:346681, amount:45.94573
date:2021-07-15, height:346904, amount:45.945875
date:2021-07-15, height:347303, amount:45.942874
date:2021-07-15, height:347342, amount:45.945955
date:2021-07-15, height:347650, amount:45.94113
date:2021-07-15, height:348255, amount:45.94113
date:2021-07-16, height:348305, amount:45.94245
date:2021-07-16, height:348502, amount:45.964615
date:2021-07-16, height:350103, amount:45.94113
date:2021-07-16, height:350187, amount:45.94113
date:2021-07-16, height:350252, amount:45.94113
date:2021-07-17, height:351303, amount:45.94113
date:2021-07-17, height:351973, amount:45.94113
TOTAL:13529.817574999983

Plotting cumulative harvesting income

Great! But what about that dictionary we were storing data in? What will we do with this? Well we can plot the harvesting history with matplotlib. We first need to get the data into the right format by:

  1. We read the dictionary into a pandas dataframe
  2. We add two column headings “Date” and “XYM”
  3. We tell pandas that the first column represents a date in the format year/month/day
  4. Next we will fill in any missing dates (e.g. if we don’t harvest every day we add in the days that are not present in the table between the first date and the most recent date
# Finally we can plot the harvest history of the account
# We set up a pandas dataframe and import our dictionary of date : amount pairs

data = pd.DataFrame(output.items())
data.columns =['Date', 'XYM']

# We can add a column heading, index based on date and then add any missing dates where the account did not harvest

data['Date'] =  pd.to_datetime(data['Date'], format='%Y/%m/%d')
data.set_index('Date', inplace=True)
data = data.resample('D').ffill().reset_index()

# The dataframe then looks like this and shows the cumulative total harvest income over time

data

The pandas dataframe will look like this:

	Date	XYM
0	2021-03-18	29.392676
1	2021-03-19	397.765547
2	2021-03-20	397.765547
3	2021-03-21	397.765547
4	2021-03-22	397.765547
...	...	...
117	2021-07-13	11857.463578
118	2021-07-14	12482.283651
119	2021-07-15	13208.204860
120	2021-07-16	13437.935315
121	2021-07-17	13529.817575

Next we can plot these data which will show us the cumulative harvest income over this period. Here we label our axes, give it a title and then show the plot of income over time.

# Finally! We can plot the contents of the pandas dataframe data

fig, ax = plt.subplots(figsize=(20, 10))

ax.plot(data['Date'], data['XYM'], 'go-')
ax.set_title("Harvest history")
plt.xlabel("Date")
plt.ylabel("XYM Harvested")

plt.show()

Once we run this code we get a plot that looks like this:

Plot of cumulative harvest income for an account over time

And that’s it for todays post. We have covered adding the date (or if you prefer to, date and time) to our block information. Looping through all pages of results, dictionaries in Python and finally making a simple plot with matplotlib. That’s actually quite a lot.

Finally, apologies for my code if it isn’t very elegant, as I explained in my first post. I am a Python novice so this was sometimes trial and error and there are probably more elegant ways to do it!

Thanks again for reading, there is a link to the Jupyter Notebook for this exercise here. Why don’t you have a go and see if you can improve the code and add new features! How about adding the average daily, weekly and monthly income of the account for a start?

Tags:
Avatar photo
NineLives
admin@symbolblog.com

I'm a Symbol and NEM enthusiast and run this blog to try to grow awareness of the platform in the English-speaking world. If you have any Symbol news you would like me to report on or you have an article that you would like to publish then please let me know!

No Comments

Sorry, the comment form is closed at this time.