11 Feb Setting up the other kind of Dual-Node (XEM-XYM)

Imagine this: you’re an old school NEMber that was enticed into running a Symbol node for the node rewards, but you overbought your VPS (Virtual Private Server) package and are left pondering what to do with that unused oomph.. Hrumm – the NEM network is looks a little slim these days, maybe you could fire up a NEM node for the heck of it…?

My Contabo VPS L package includes 8 vCPU Cores, 30 GB RAM, and 800 GB on SSD. While I haven’t benchmarked the CPU and SSD performance, a quick search for others benchmark results seems to indicate that you get what you pay for – Contabo VPSs are inexpensive and it seems this is reflected in their performance. Even so, Symbol only seems to use ~2% of the CPU, 6% of the RAM, and <30GB of disk space (not including backups), so I had plenty of unused resources to work with.

Docker for the win

I’m fairly new to Docker, having first come across it when I setup an Unraid home NAS. It didn’t take long for me to fall in love with Docker, so I was rather delighted to see it used by Symbol’s Bootstrap. The learning curve for Linux is fairly steep and at times rather unforgiving; Docker does an amazing job of simplifying Linux software installations by pre-packaging them in tidy and disposable boxes. Good docker images contain everything they need to work- you don’t need to worry about dependencies and whatnot because they’re self-contained and intended to work out of the box.. Not all docker images are simple and there’s a lot more to docker than pre-packaged images, but so far as this post is concerned, Docker is a framework that makes it easy to add functionality. The idea of co-hosting XYM and XEM nodes recently popped into my head and I figured I’d give it a shot. After a couple hours of unsuccessfully fiddling around with an outdated nem-docker github repository, the ever-reliable cryptoBeliever pointed me to an updated version at https://github.com/NemProject/nem-docker.

Getting Started

I connected to my VPS using SSH, downloaded the docker project and opened the newly added folder using:

$ git clone https://github.com/NemProject/nem-docker.git
$ cd nem-docker

A Design Issue…

The nem-docker repository was nearly perfect for my purposes, but there was one problematic detail – the docker script was designed for a fresh server without any extra user accounts, but my server was already running a Symbol node and had a couple accounts on it, I had to make a few tweaks to make this work. You can skip these steps If you haven’t created any user accounts on your server, but I’d still recommend running the following command to confirm that UID (user ID) 1000 hasn’t been used.

$ less /etc/passwd

Working around the expected nem user ID…

This may not be the best approach, as I suspect I could have fixed this much more easily by deleting the two references to UID 1000 rather than working around them.. In this case the repository could be updated to eliminate these steps.

1. Adding the ‘nem’ user account using the next available userid on my server:

$ sudo useradd nem

2. Checking the new account’s UID [in my case it’s 1002 – “nem:x:1002:1002::/home/nem:/bin/sh”].

$ less /etc/passwd

3. Removing the instructions to create the NEM user ID.. I use the vim text editor, but there are plenty of alternatives out there! My server doesn’t have a GUI (it’s command-line only), so I often use filezilla to connect via SFTP when I want to view/edit files – this approach would work just fine, but in this example I’m using vim:

$ vim ~/nem-docker/Dockerfile

Edit line 16 from ‘RUN useradd –uid 1000 nem’ to ‘#RUN useradd –uid 1000 nem’ – the added ‘#’ overrides this step, as we already created the nem user account. Vim is a little tricky – you have to hit ‘i’ for edit privileges, then once you’ve made your changes, hit ESC and enter ‘:w” to save, then ESC again and ‘:qa” to exit.

4. updating the UID in the boot file:

$ vim ~/nem-docker/boot.sh

Here we edit line 38 “chown -R 1000 nem” to “chown -R 1002 nem” –> replace 1002 with whatever UID you had for the nem account. We need this command to run, so can’t comment it out with a # like the last one.

Firing up your new NEM Node

Now we can fire up our node and start synchronizing with the network using:

$ sudo ./boot.sh

I used sudo here because I had permission issues without it.. You may not need this after the first run – especially if you’re logged in as root. Worth noting that you have to be in the nem-docker folder for this to work. Further instructions can be found in the repository’s readme. Stopping your node is as easy as:

$ sudo ./stop.sh

Now that your node is running, you may need to open the node’s ports on your firewall.. If you don’t have a firewall, I’d highly recommend using one to block inbound traffic except on the few ports you need. I use UFW for my firewall, which Docker has a bad habit of ignoring, so my node’s ports were exposed without any extra steps – yours might not be. I confirmed that my NEM node was up and running by pointing my web browser to my_node_url_OR_IPaddress.com:7890/chain/height — this should show the block height (i.e: {“height”:279767}). You should also be able to see your node listed on nemnodes.org/nodes after a 5-10 minutes. CPU use fluctuated while NEM was synching with the network, occasionally exceeding 90%! CPU usage settled into the 10-40% range with spikes up to 50% once fully synched (42 hrs later), NEM also used <7.5GB RAM, 12gb virtual memory, and <6GB of storage. I’d be wary of running NEM and a Symbol mainnet nodes on the same device with any less processing power, as NEM’s CPU use during synchronization could impact Symbol’s performance. If you’re keen to try and already have a Symbol testnet node, it might be a better candidate!

If you’re worried about over-stressing your server, you can keep tabs on your memory and CPU usage using with:

$ top -i

And check your disk space with:

$ df

There are also a bunch of great open source server monitoring tools that bring these sorts of metrics to your browser. Some of the options I’ve found particularly interesting are: Netdata (the easiest option with a plug-and-play docker image, limited functionality but no config needed), Zabbix (see this post on visualizing data from Symbol’s logs), and Prometheus (powerful, extensible, complicated).

Docker Management and Server Security

I may have made this all seem easy and straight-forward, but sorting out the above instructions involved a fair bit of troubleshooting. Portainer’s browser-based Docker overview came in super handy with it’s easy access to my node’s logs.. Portainer is a great example of what I love about Docker – easy-to-install and super handy.. That said, Docker does have a bad habit of overriding firewall rules, which came as a bit of a surprise. I wouldn’t want to expose a tool like Portainer to the internet, so I had to override docker’s firewall rules using:

$ sudo iptables -I DOCKER-USER -p tcp --destination-port 9000 -j DROP

This overrides the docker rule and blocks external connections to Portainer while allowing ‘local’ connections.. In order to connect locally I had to add a port forwarding rule to my SSH connection, allows me to access Portainer ‘locally’ when connected to my VPS by pointing my bowser to 127.0.0.1:9000.. This kind of nuance is a great example of the challenges faced by infrastructure providers – the learning curve can be steep and unforgiving, and one seemingly minor oversight can compromise your entire server. I made the mistake of opening access to my Symbol node’s MongoDB without adding a firewall rule to prevent it from being exposed to the broader internet – it was only a few hours before someone wiped the database and left a ransomware note behind.. That ransom cost me a lot of money — JOKING — lucky for me it’s a blockchain and all I had to do was resync! Running a bare-bones Symbol/NEM node on a VPS is relatively safe, but adding anything more to your server opens you to new attack vectors. Docker is awesome and can quickly add functionality to your node, but it’s not without its risks, and needs to be properly understood to be used safely.. If you add any docker images, at very least check which ports Docker has opened and which you’ve overridden as DOCKER-USER:

$ sudo iptables -S | grep docker

…and which ones you’ve opened/closed:

$ sudo iptables -S | grep user

I would also recommend running a port scan; here’s a free one from the My-Addr Project that will runs a full scan if you specify the port range: 0-65535

Closing Remarks

Hosting a NEM node in addition to your Symbol node isn’t for everyone – it’s more demanding of your server and doesn’t provide any financial benefit, That said, it only takes a few minutes to setup and provides a big contribution to NEM’s resiliency. At the time of writing, Symbol has 1300+ nodes while NEM has <250 nodes. running the latest version. If your server has the extra capacity and you support NEM’s future as a Symbol subchain, why not pitch in by hosting a dual XYM-XEM node? If your server doesn’t have the extra capacity to run a NEM node, it’ll be hard pressed to run a subchain – something worth considering for future-friendly node operators.