18 Nov Symbol blockchain news update – 19/11/21

Well that sucked 🙁 What should have been a celebration turned into a bit of a downer for us and probably a huge panic and a stay up all night trying to diagnose and fix the issue for the devs – sorry guys I really wouldn’t want to have your jobs. So much responsibility, I don’t think I would handle the pressure well 😬 It’s cool though a fix is being worked on and we will go again and next time we will have a bigger party at the end of it! 😁

It seems that probably(?) the Harlock fork did have sufficient support but the chain halted at the fork block. The devs are trying to diagnose what went wrong and release a fix, obviously it is number one priority right now.

Mission aborted. The fork did not pass due to an error. We've sent out the fallback transactions to unstick the network – it's going to take some time for it to catch up.

Post-mortem and next steps coming soon. https://t.co/wmufqTmJ60

— Hatchet (@0x6861746366574) November 18, 2021

The postmortem will come soon and I will cover it in a later post (or update this post in the morning if it comes out tonight). Attempt number two will be coming soon^TM and there is a poll below asking for upgrade timeframe suggestions. At the moment 5 days is the most popular option. I guess because everyone wants to get things fixed and move on. It does throw a spanner in the works though since a) the devs still need to find and fix the bug b) everyone needs to upgrade their nodes again. I don’t know where this leaves Harlock, did we have enough support at the fork height (I think so) and that is why the chain stalled as there was a bug in the Harlock update? I have too many questions here and a total lack of knowledge so I will shut up now so I don’t sound stupid.

We're still triaging why the fork has failed.

A new client update (with new pre-generated transactions) will need to be released to try again (as the transactions are pre-generated).

If we go this route, how long would you want to run the upgrade period for?

— Hatchet (@0x6861746366574) November 18, 2021

It will be interesting to find out what went wrong and what the future steps are so keep your eyes peeled and watch out for tweets by the core devs in the coming hours.

Important update

If you are running a node with a chain height > 3,464,840 then please disable your nodes.

We're currently seeing a fork divergence and need the weaker chain to catch up.

— Hatchet (@0x6861746366574) November 18, 2021

Thanks guys – we appreciate your hard work

Hatchet, Jaguar and Gimre – can I send you guys some beers/coffee/nice food or something? Maybe a some kind of digital voucher, obviously I don’t want your names and postal addresses! I am always amazed by how hard you guys work and feel like I should give something small back. I am sure that the community all feel exactly the same. I can’t overstate how much you’ve done for all of us so I feel really bad that things didn’t go as planned and you have to work even harder to sort things out. I know it is probably a stupid idea but I guess if it is then we still want you to know we are on your side and if we can help in any way just let us know.

That’s it for tonight, I will probably update this article or write something new when I know more. It looks like my son has finally stopped screaming and gone to sleep (I did try to settle him many times – didn’t just let him cry all evening!), I am off to pour a glass of wine and relax now – calm..

And it looks like problem solved – geniuses 🤩🤓

The fork did not pass, we used "fallback" transaction. But right now it's all about saving network.

And seems like we're succeeding and network's gonna be good.#BestForkEver

— gimre 🏴‍☠️ (@NCOSIGIMCITYNRE) November 18, 2021

Update (morning of the 19th November)

Well things are starting to get a bit clearer today, there’s no official postmortem as yet but there are two tweets from Jaguar and Gimre explaining the specifics of what went wrong. It seemed to be a combination of two factors and it sounds like it went pretty close to the wire for the Plan A fix – please read their threads for a full explanation.

In Harlock, as opposed to Cyprus, the treasury reissuance transactions simply reused/abused regular Transfer Transactions. [In Cyprus, we added a new Mosaic Supply Revocation Transaction].

— Jaguar0625 (@Jaguar0625) November 19, 2021

🧵 @Jaguar0625 described issue with multisig transactions, I'm gonna describe the other issue.#BestForkEver #HarlockFork https://t.co/L1scd0nO8j

— gimre 🏴‍☠️ (@NCOSIGIMCITYNRE) November 19, 2021

What went wrong?

From what I can gather from the threads here is a summary of went wrong (edited slightly in the PM due to my lazy summarisation being flagged by Gimre – thank you 😁).

• Reissuance transactions were rejected with a FAILURE_TRANSACTION_NOT_ALLOWED_FOR_MULTISIG error at the fork block.
• Some upgraded nodes with lots of harvesting power were misconfigured meaning fork specific information in the v0.6.98/0.6.9 config files were skipped.
  • This led to the misconfigured nodes failing to stall at the fork and instead they just carried on past block 3,464,800.
• This in turn led to the misconfigured upgraded nodes joining with the non-upgraded nodes to build a stronger combined chain.
• The nodes that did stall were unblocked by a special fallback transaction.
• The combined chain was 120 blocks ahead of the stalled nodes at its worst point.
• Things looked pretty bleak and it was a race against time to fix the problem before block 3,465,160, at which point it would have been unrecoverable and would have needed much more drastic measures (plan B) to fix.
• The community and Trust were awesome and worked to switch off the “bad” nodes and delegate to the “good” nodes in order to reallocate harvesting power to the chain with the fallback transaction.

So it is a huge thank you to the devs and the community members that saved the day. It was a close call.

We can look forward to doing it all again soon! 😁

EDIT: I also missed Hatchet’s tweet (just picked it up). Give this a read too, they are all much more informative than my summary!

Hi! I know you're all eager to understand what happened. We don't have all the details yet, but here is what we do know (I typed this after Jag's tweet, but oh well):

— Hatchet (@0x6861746366574) November 19, 2021