NEM-Symbol Blockchain: bootstrap to shoestring node migration and upgrade

This guide could help your NEM-Symbol blockchain node to migrate from symbol-bootstrap container (discontinued) to symbol shoestring container (python3 module container).

DISCLAIMER

This procedure takes no responsibility on your wallet keys custodial or any losses of XYM.
Funds key is referred hereafter as "Main key" or "Wallet key", it should be stored safely offline.
The procedure described here deletes "Main key" with rm command at section "Removing plain keys for node safety".

PREREQUISITES:

1. GNU/Linux Operating System
2. a running symbol-bootstrap blockchain node

WHAT YOU WILL GET:

a NEM-Symbol blockchain node running on shoestring with latest catapult client version v1.0.3.8 using the same public key, the same node public key, the same remote key, the same vrf key as your existing symbol-bootstrap node.

PYTHON AND SHOESTRING INSTALLATION

Install pyenv and python

   curl -fsSL https://pyenv.run | bash
   echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bashrc
   echo '[[ -d $PYENV_ROOT/bin ]] && export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bashrc
   echo 'eval "$(pyenv init - bash)"' >> ~/.bashrc
   echo 'alias python="$(pyenv which python)"' >> ~/.bashrc
   echo 'alias pip="$(pyenv which pip)"' >> ~/.bashrc
   pyenv install 3.11.12 
   pyenv global 3.11.12
   exec "$SHELL"
   python --version

for more information on this step check here
https://github.com/pyenv/pyenv?tab=readme-ov-file#installation
https://catnotes.xyz/symbol/best-practices/python-new-project

Install Shoestring

   pip install symbol-shoestring
   pip show symbol-shoestring

You should have installed Shoestring Version: 0.2.1
For more information check here
https://qiita.com/ccHarvestasya/items/1c1a60beaf3a80d912c2

FIND YOUR NODE KEYS

NEM-Symbol blockchain client keys

To run your node you need four keys, one for each of these four types:

• Wallet (also known as main or ca.key.pem)
• Node (also known as transport)
• Remote (the remote harvesting key is the one Wallet is linked to)
• VRF (for Verifiable Random Function)

For more information check here
https://catnotes.xyz/symbol/transactions/understanding/links

Find the keys in bootstrap

Change directory to your symbol-boostrap node installation directory.
Your symbol-bootstrap installation directory now on is referred here as . where you find the ./target directory and the ./target/addresses.yml file.
The keys mentioned before are written in addresses.ymlfile, immediately after these respective yml lines

   main:
     privateKey: ...
     publicKey: ...
     address: ...
   transport:
     ...
   remote:
     ...
   vrf:
     ...

Please check with the NEM-Symbol blockchain mainnet explorer at https://symbol.fyi that the ADDRESS of the main key is linked with the remote public key and the vrf public key. Also check the xym balance.
You'll need to decrypt them if they are ENCRYPTED, with this command

symbol-bootstrap decrypt --source target/addresses.yml --destination target/d_addresses.yml

enter the password, now you have decrypted keys in your bootstrap node directory under ./target/d_addresses.yml

Copy the keys to shoestring node directory

Now create a new shoestring node directory (check you don't have it already)
mkdir -p ../shoestring-node
copy the keys there

grep -A1 main d_addresses.yml | sed -n -E 's/[[:space:]]*privateKey:[[:space:]]*//p' > ../shoestring-node/main_k.txt
grep -A1 transport d_addresses.yml | sed -n -E 's/[[:space:]]*privateKey:[[:space:]]*//p' > ../shoestring-node/transport_k.txt
grep -A1 remote d_addresses.yml | sed -n -E 's/[[:space:]]*privateKey:[[:space:]]*//p' > ../shoestring-node/remote_k.txt
grep -A1 vrf d_addresses.yml | sed -n -E 's/[[:space:]]*privateKey:[[:space:]]*//p' > ../shoestring-node/vrf_k.txt

Special case for voting keys

For voting keys we just have dat files. We copy all of them.

   cp ./target/nodes/node/votingkeys/private_key*.dat ../shoestring-node

That's all for the keys, but we need the wallet address too so execute also

grep -A3 main d_addresses.yml | sed -n -E 's/[[:space:]]*address:[[:space:]]*//p' > ../shoestring-node/main_a.txt

NOTE:

The main key should not stay on your online host for security reasons.
Find your way to store offline the main key. Voting nodes have this key being the 3M XYM wallet (could be also a multisig wallet).
The remote key is the key used for node harvesting, it is linked to your main wallet. Your node could not harvest if this key is lost. Backup remote, and VRF accordingly.

CREATE SHOESTRING NODE ON THE SAME KEYS

Change directory to shoestring node created before

   cd ../shoestring-node

Generate and edit init file

   python -m shoestring init mainnet.ini --package mainnet
   vi mainnet.ini

Edit [node] section
Choose features accordingly or leave it there.
Give a value to caCommonName i.e. caCommonName = myhost.com
Give a value to nodeCommonName i.e. nodeCommonName = node

Generate overrides.ini

   printf "[node.localnode]\n" > overrides.ini
   printf "host = "$(hostname) >> overrides.ini
   printf "\nfriendlyName = friendlyname" >> overrides.ini

substitute friendlyname with your node friendly name at wish.

Generate the ca.key.pem

This the certificate contains your wallet key. Voting nodes should have 3M xym balance in it. It is generated from main_k.txt and will be deleted from the filesystem at the end of this article, because it should not stay online for security reasons.

   python -m shoestring pemtool --input ./main_k.txt --output ./ca.key.pem

This command has generated the file ca.key.pem

Generate the node.key.pem

This certificate is the node transport and should be rotated for security reasons but in this article we want a clone of our bootstrap node so we reuse the same transport by generating it from the transport key.

   python -m shoestring pemtool --input ./transport_k.txt --output ./node.key.pem

We update the config file to instruct to use this transport key

   sed -i 's%nodeKey = %& '$PWD'/node.key.pem%' mainnet.ini

You can check that the public key corresponds to your node in https://nemnodes.org/nodes with the value in the last column of your node's row, called node public key, with the output of this command.

openssl pkey -in node.key.pem -pubout -noout -text 

Create shoestring setup

   python -m shoestring setup --package mainnet --ca-key-path ./ca.key.pem --config ./mainnet.ini --overrides ./overrides.ini --directory .

Override remote harvesting key

This setup has generated a new harvesting key so we need to ovverride it using our remote key. We delete the linking transaction and we overwrite the remote key with the one we copied from bootstrap.

   rm linking_transaction.dat
   python -m shoestring pemtool --input remote_k.txt --output ./keys/remote.pem
   cat remote_k.txt | xargs -I{} sed -i -E 's/(harvesterSigningPrivateKey = )(.*$)/\1'{}'/' ./userconfig/resources/config-harvesting.properties

Override VRF key

   python -m shoestring pemtool --input vrf_k.txt --output ./keys/vrf.pem
   cat vrf_k.txt | xargs -I{} sed -i -E 's/(harvesterVrfPrivateKey = )(.*$)/\1'{}'/' ./userconfig/resources/config-harvesting.properties

Update beneficiary address

   cat main_a.txt | xargs -I{} sed -i -E 's/(beneficiaryAddress =)(.*$)/\1 '{}'/' ./userconfig/resources/config-harvesting.properties

Please check that config-harvesting.properties file content has the correct values of your private keys

harvesterSigningPrivateKey = #should have the value of the remote private key
harvesterVrfPrivateKey = #should have the value of the vrf private key
beneficiaryAddress = #should have the address of your wallet (3M for voting nodes)

Override voting keys

  rm -rf ./keys/voting/private_key_tree1.dat
  mv private_key_tree*.dat ./keys/voting/

The setup is completed.
The data directory has not been copied, your node will resync from block 1.
If you would like to start from the last block bootstrap had, you should copy bootstrap data directory. Check NEM-Symbol discord helpdesk for assistance on this shortcut.
Next step should be:
Shut down bootstrap node
Start up shoestring node from the shoestring directory

  docker-compose up -d

eventually clean docker with
prune system -a

REMOVING PLAIN KEYS FILES FOR NODE SAFETY

Remember to delete the keys once the node is functioning well
this means to delete these files after having made an offline backup.
You can backup safely to your system using ssh copy command

   scp

plese check scp manual.
After backing up on cold storage, and having double checked the keys, you could remove the files with pain keys and possibly the old bootstrap directory.

  target/d_addresses.yml
  shoestring_node/main_k.txt
  shoestring_node/transport_k.txt
  shoestring_node/remote_k.txt
  shoestring_node/vrf_k.txt
  shoestring_node/ca.key.pem
  shoestring_node/node.key.pem

CREDITS

Greetings to dusanjp.com and his samdal network that made possible to have sperimental activity