The April 29, 2018 IOTA Snapshot and IRI 18.104.22.168: Behind the Scenes
The full article was originally published by Edward Greve on Medium. Read the full article here.
Snapshots are described briefly in our FAQ, but the IRI team wanted to take this snapshot as an opportunity to explain in more detail exactly how the process works, and how our node operating community can help. For an excellent overview of how to prepare and update for snapshots as an IOTA wallet user, please refer to Ralf’s excellent blog post from the previous snapshot. Specifically, we will cover:
- What happens before, during, and after a snapshot; and
- How you as an IOTA node operator can prepare, validate, and update your nodes.
Two of the most compelling features of the IOTA protocol (among many!) are that (1) the network is totally permissionless, and (2) transactions are totally fee-free. What’s more, beyond the transfer of IOTA tokens, transactions may be used to encode arbitrary data without any token value at all, a fact which many network participants and developers rely on for their apps! The result for a node operator, however, is that the node database grows more-or-less unabated as people continue to transact, develop, test, and grow the network. This is a good thing. In fact, we are counting on it!
Data-only transactions often account for somewhere between 90 and 95% of the database growth rate — this means disk space may quickly run out on smaller nodes. We are actively working on local snapshots so that in the not-too-distant future node operators will be able to more effectively manage their own database size. Meanwhile, we provide an interim solution to help keep the database lean and node operators in business without breaking the bank. This is of course: the snapshot.
The snapshot process consists of 5 steps, with Step 1 occurring approximately 2 weeks prior to the snapshot date (this is why we try to announce snapshots approximately 2 weeks in advance):
- Code freeze. While a snapshot might be as simple as pruning the database and re-starting the Coordinator, we generally take them as an opportunity to introduce a new version of the IOTA node software — IRI. Prior to announcing the release date, all of the latest improvements, patches, and bug fixes that have made it into IRI since the previous snapshot are “frozen” as a release candidate. During the remaining time leading up to the snapshot, the team is at work ‘round-the-clock to test and finalize the release candidate into the next version of IRI.
- Stopping Coo. At the moment the snapshot occurs, the Coordinator stops issuing milestones, and the network reaches consensus on the latest milestone issued. The database state at this point in time will be re-used once (A) the database has been preserved as a new initial state of all addresses with a positive balance, and (B) all of these account balances have been validated by the IOTA Foundation and the community.
- IF validation. Multiple participants from the IOTA Foundation run Snapshot.ixi on different nodes. This tool extracts all addresses with a current balance from the database. Once everyone independently confirms that the contents match, the snapshot itself is signed and ready for community validation.
- Spent addresses. As all IOTA users hopefully know by now, addresses may only be spent from once. To enforce this rule across snapshots, all addresses which have been spent during the current Epoch (time between the previous and current snapshots) are appended to a file called previousEpochsSpentAddresses.txt, which is included in the next database. This ensures that all addresses which have ever been spent from (since the genesis) can be accounted for and prevents address reuse.
- Community validation. Arguably the most important part of this entire process: you, our beloved community node operator! Once the results of all previous steps have been validated and pushed to GitHub by our team, this is your time to shine. We provide the following in the form of a Pull Request (PR), similar to IRI#512:
- the new IRI code
- the next milestone index
- snapshotMainnext.txt (+ signature)
- previousEpochsSpentAddresses.txt (+ signature)
- the snapshot timestamp — the time when the network will be resumed (used to filter old transactions, based on attachment time)
- clear instructions for community members to validate the snapshot
All community members are requested to follow the instructions and validate the snapshot, and we wait 6 hours from the time the PR is pushed to allow enough time for community participation while minimizing overall network downtime. Finally, once everything is confirmed valid, Coo is restarted and begins issuing milestones once again — snapshot complete!
Options for Node Operators
For new node operators — a snapshot is always a good opportunity to get involved and start running your own nodes! iota.partners provides a fairly comprehensive guide to getting set up, and you won’t need any special instructions.
For existing node operators — you have 3 basic options:
- Do nothing. You are free to continue running your existing version of IRI, and as there are no breaking changes in this release, nothing much will change. While the Coordinator is offline, you won’t see any confirmations for about 6 hours. Additionally, you won’t be getting the tag fixes that are shipping with 22.214.171.124, so searching by tag on findTransactions will continue to behave oddly. We strongly advise every node operator to keep IRI up to date to ensure you have the latest performance enhancements and bug fixes.
- Migrate existing database. If you would like to keep your existing database and not lose transaction records due to the snapshot, you can compile IRI from source (branch v126.96.36.199_RC)¹. Then replace your existing iri.jar file with the newly compiled iri-188.8.131.52_RC.jar, and restart your node with the –rescan flag. The –rescan flag will perform a database migration, and you only need to run it once.
- Clean database. If you are running on a smaller VPS and short on disk space, you may want to remove your existing database entirely and start fresh. In this case, you should delete your mainnetdb folder and replace iri.jar with the Mainnet v184.108.40.206jarfile which will be found on the releases page shortly after the snapshot.
¹ Add a command to check out the correct branch when building IRI:
$ git clone https://github.com/iotaledger/iri $ cd iri -> $ git checkout v220.127.116.11_RC $ mvn clean compile $ mvn package
Validating the Snapshot
If you run a node and would like to participate in the validation process, it’s not too difficult. Just make sure you have Node.js installed and follow the instructions here: https://github.com/iotaledger/snapshot-validator.
IF and Community members will be available on Discord in the #snapshots channel to provide support during the snapshot if you would like to participate.
A full set of release notes will be available on the release page when the new version is made available.
This snapshot includes fixes for:
- Tag indexing (#728)
- Running node on testnet with a single flag (#610)
- TCP networking issues (#592)
- ZMQ sending issues (#569)
- Improved milestone solidity tracking (#486)
The April 29, 2018 IOTA Snapshot and IRI 18.104.22.168: Behind the Scenes was originally published in IOTA on Medium, where people are continuing the conversation by highlighting and responding to this story.