Builder's Guide
Search…
Operational Safety
Table of Contents
Overview
This chapter describes the security/safety mechanisms that are implemented in
lnd. We encourage every person that is planning on putting mainnet funds into a Lightning Network channel using lnd to read this guide carefully.
As of this writing, lnd is still in beta and it is considered #reckless to put any life altering amounts of BTC into the network.
That said, we constantly put in a lot of effort to make lnd safer to use and more secure. We will update this documentation with each safety mechanism that we implement.The first part of this document describes the security elements that are used in
lnd and how they work on a high level.
The second part is a list of best practices that has crystallized from bug reports, developer recommendations and experiences from a lot of individuals running mainnet lnd nodes during the last 18 months and counting.aezeed
This is what all the on-chain private keys are derived from.
aezeed is similar to BIP39 as it uses the same word list to encode the seed as a mnemonic phrase. But this is where the similarities end, because aezeed is not compatible with BIP39. The 24 words of aezeed encode a 128 bit entropy (the seed itself), a wallet birthday (days since BTC genesis block) and a version.
This data is encrypted with a password using the AEZ cipher suite (hence the name). Encrypting the content instead of using the password to derive the HD extended root key has the advantage that the password can actually be checked for correctness and can also be changed without affecting any of the derived keys.
A BIP for the aezeed scheme is being written and should be published soon.Important to know:
- As with any bitcoin seed phrase, never reveal this to any person and storethe 24 words (and the password) in a safe place.
- You should never run two different
lndnodes with the same seed! Even ifthey aren't running at the same time. This will lead to strange/unpredictablebehavior or even loss of funds. To migrate anlndnode to a new device,
Wallet password
The wallet password is one of the first things that has to be entered if a new wallet is created using
lnd. It is completely independent from the aezeed cipher seed passphrase (which is optional). The wallet password is used to encrypt the sensitive parts of lnd's databases, currently some parts of wallet.db and macaroons.db. Loss of this password does not necessarily mean loss of funds, as long as the aezeed passphrase is still available. But the node will need to be restored using the SCB restore procedure.TLS
By default the two API connections
lnd offers (gRPC on port 10009 and REST on port 8080) use TLS with a self-signed certificate for transport level security. Specifying the certificate on the client side (for example lncli) is only a protection against man-in-the-middle attacks and does not provide any authentication. In fact, lnd will never even see the certificate that is supplied to lncli with the --tlscertpath argument. lncli only uses that certificate to verify it is talking to the correct gRPC server.
If the key/certificate pair (tls.cert and tls.key in the main lnd data directory) is missing on startup, a new self-signed key/certificate pair is generated. Clients connecting to lnd then have to use the new certificate to verify they are talking to the correct server.Macaroons
Macaroons are used as the main authentication method in
lnd. A macaroon is a cryptographically verifiable token, comparable to a JWT or other form of API access token. In lnd this token consists of a list of permissions (what operations does the user of the token have access to) and a set of restrictions (e.g. token expiration timestamp, IP address restriction). lnd does not keep track of the individual macaroons issued, only the key that was used to create (and later verify) them. That means, individual tokens cannot currently be invalidated, only all of them at once.
See the high-level macaroons documentation or the technical README for more information.Important to know:
- Deleting the
*.macaroonfiles in the<lnd-dir>/data/chain/bitcoin/mainnet/folder will triggerlndto recreate the default macaroons. But this doesNOT invalidate clients that use an old macaroon. To make sure allpreviously generated macaroons are invalidated, themacaroons.dbhas to bedeleted as well as all*.macaroon.
Static Channel Backups (SCBs)
A Static Channel Backup is a piece of data that contains all static information about a channel, like funding transaction, capacity, key derivation paths, remote node public key, remote node last known network addresses and some static settings like CSV timeout and min HTLC setting.
Such a backup can either be obtained as a file containing entries for multiple channels or by calling RPC methods to get individual (or all) channel data. See the section on keeping SCBs safe for more information.
What the SCB does not contain is the current channel balance (or the associated commitment transaction). So how can a channel be restored using SCBs?
That's the important part: A channel cannot be restored using SCBs, but the funds that are in the channel can be claimed. The restore procedure relies on the Data Loss Prevention (DLP) protocol which works by connecting to the remote node and asking them to force close the channel and hand over the needed information to sweep the on-chain funds that belong to the local node.
Because of this, restoring a node from SCB should be seen as an emergency measure as all channels will be closed and on-chain fees incur to the party that opened the channel initially.
To migrate an existing, working node to a new device, SCBs are not the way to do it. See the section about migrating a node on how to do it correctly.
Important to know:
- contained in that file.
- Restoring a node from SCB relies on the remote node of each channel to beonline and respond to the DLP protocol. That's why it's important torecovered using SCBs.
- The SCB data is encrypted with a key from the seed the node was created with.A node can therefore only be restored from SCB if the seed is also known.
Static remote keys
Since version
v0.8.0-beta, lnd supports the option_static_remote_key (also known as "safu commitments"). All new channels will be opened with this option enabled by default, if the other node also supports it.
In essence, this change makes it possible for a node to sweep their channel funds if the remote node force-closes, without any further communication between the nodes. Previous to this change, your node needed to get a random channel secret (called the per_commit_point) from the remote node even if they force-closed the channel, which could make recovery very difficult.Best practices
aezeed storage
When creating a new wallet,
lnd will print out 24 words to write down, which is the wallet's seed (in the aezeed format). That seed is optionally encrypted with a passphrase, also called the cipher seed passphrase.
It is absolutely important to write both the seed and, if set, the password down and store it in a safe place as there is no way of exporting the seed from an lnd wallet. When creating the wallet, after printing the seed to the command line, it is hashed and only the hash (or to be more exact, the BIP32 extended root key) is stored in the wallet.db file.
There is a tool being worked on that can extract the BIP32 extended root key but currently you cannot restore lnd with only this root key.Important to know:
- Setting a password/passphrase for the aezeed is meant to protect it froman attacker that finds the paper/storage device. Writing down the passwordalongside the 24 seed words does not enhance the security in any way.Therefore the password should be stored in a separate place.
File based backups
There is a lot of confusion and also some myths about how to best backup the off-chain funds of an
lnd node. Making a mistake here is also still the single biggest risk of losing off-chain funds, even though we do everything to mitigate those risks.What files can/should I regularly backup?
The single most important file that needs to be backed up whenever it changes is the
<lnddir>/data/chain/bitcoin/mainnet/channel.backup file which holds the Static Channel Backups (SCBs). This file is only updated every time lnd starts, a channel is opened or a channel is closed.Most consumer Lightning wallet apps upload the file to the cloud automatically.
What files should never be backed up to avoid problems?
This is a bit of a trick question, as making the backup is not the problem. Restoring/using an old version of a specific file called
<lnddir>/data/graph/mainnet/channel.db is what is very risky and should never be done!
This requires some explanation:
The way LN channels are currently set up (until eltoo is implemented) is that both parties agree on a current balance. To make sure none of the two peers in a channel ever try to publish an old state of that balance, they both hand over their keys to the other peer that gives them the means to take all funds (not just their agreed upon part) from a channel, if an old state is ever published. Therefore, having an old state of a channel basically means forfeiting the balance to the other party.As payments in
lnd can be made multiple times a second, it's very hard to make a backup of the channel database every time it is updated. And even if it can be technically done, the confidence that a particular state is certainly the most up-to-date can never be very high. That's why the focus should be on making sure the channel database is not corrupted, closing out the zombie channels and keeping your SCBs safe.Keeping Static Channel Backups (SCB) safe
As mentioned in the previous chapter, there is a file where
lnd stores and updates a backup of all channels whenever the node is restarted, a new channel is opened or a channel is closed: <lnddir>/data/chain/bitcoin/mainnet/channel.backupOne straight-forward way of backing that file up is to create a file watcher and react whenever the file is changed. Here is an example script that automatically makes a copy of the file whenever it changes.
Because the backup file is encrypted with a key from the seed the node was created with, it can safely be stored on a cloud storage or any other storage medium. Many consumer focused wallet smartphone apps automatically store a backup file to the cloud, if the phone is set up to allow it.
Keep lnd updated
lnd updatedWith every larger update of
lnd, new security features are added. Users are always encouraged to update their nodes as soon as possible. This also helps the network in general as new safety features that require compatibility among nodes can be used sooner.Zombie channels
Zombie channels are channels that are most likely dead but are still around. This can happen if one of the channel peers has gone offline for good (possibly due to a failure of some sort) and didn't close its channels. The other, still online node doesn't necessarily know that its partner will never come back online.
Funds that are in such channels are at great risk, as is described quite dramatically in this article .
The TL;DR of the article is that if you have funds in a zombie channel and you need to recover your node after a failure, SCBs won't be able to recover those funds. Because SCB restore relies on the remote node cooperating.
That's why it's important to close channels with peers that have been offline for a length of time as a precautionary measure.
Of course this might not be good advice for a routing node operator that wants to support mobile users and route for them. Nodes running on a mobile device tend to be offline for long periods of time. It would be bad for those users if they needed to open a new channel every time they want to use the wallet. Most mobile wallets only open private channels as they do not intend to route payments through them. A routing node operator should therefore take into account if a channel is public or private when thinking about closing it.
Migrating a node to a new device
As mentioned in the chapters aezeed and SCB you should never use the same seed on two different nodes and restoring from SCB is not a migration but an emergency procedure.
What is the correct way to migrate an existing node to a new device? There is an easy way that should work for most people and there's the harder/costlier fallback way to do it.
Option 1: Move the whole data directory to the new device
This option works very well if the new device runs the same operating system on the same architecture. If that is the case, the whole
/home/<user>/.lnd directory in Linux (or $HOME/Library/Application Support/lnd in MacOS, %LOCALAPPDATA%\lnd in Windows) can be moved to the new device and lnd started there. It is important to shut down lnd on the old device before moving the directory!
Not supported/untested is moving the data directory between different operating systems (for example MacOS -> Linux) or different system architectures (for example 32bit -> 64bit or ARM -> amd64). Data corruption or unexpected behavior can be the result. Users switching between operating systems or architectures should always use Option 2!Option 2: Start from scratch
If option 1 does not work or is too risky, the safest course of action is to initialize the existing node again from scratch. Unfortunately this incurs some on-chain fee costs as all channels will need to be closed. Using the same seed means restoring the same network node identity as before. If a new identity should be created, a new seed needs to be created.
Follow these steps to create the same node (with the same seed) from scratch:
1. On the old device, close all channels (
lncli closeallchannels). The command can take up to several minutes depending on the number of channels. Do not interrupt the command! 1. Wait for all channels to be fully closed. If some nodes don't respond to the close request it can be that lnd will go ahead and force close those channels. This means that the local balance will be time locked for up to two weeks (depending on the channel size). Check lncli pendingchannels to see if any channels are still in the process of being force closed. 1. After all channels are fully closed (and lncli pendingchannels lists zero channels), lnd can be shut down on the old device. 1. Start lnd on the new device and create a new wallet with the existing seed that was used on the old device (answer "yes" when asked if an existing seed should be used). 1. Wait for the wallet to rescan the blockchain. This can take up to several hours depending on the age of the seed and the speed of the chain backend. 1. After the chain is fully synced (lncli getinfo shows "synced_to_chain": true) the on-chain funds from the previous device should now be visible on the new device as well and new channels can be opened.What to do after the move
If things don't work as expected on the moved or re-created node, consider this list things that possibly need to be changed to work on a new device:
- In case the new device has a different hostname and TLS connection problemsoccur, delete the
tls.keyandtls.certfiles in the data directory andrestartlndto recreate them. - If an external IP is set (either with
--externalipor--tlsextraip) thesemight need to be changed if the new machine has a different address. Changingthe--tlsextraipsetting also means regenerating the certificate pair. Seepoint 1. - If port
9735(or10009for gRPC) was forwarded on the router, theseforwarded ports need to point to the new device. The same applies to firewallrules. - It might take more than 24 hours for a new IP address to be visible onnetwork explorers.
- If channels show as offline after several hours, try to manually connect tothe remote peer. They might still try to reach
lndon the old address.
Migrating a node from clearnet to Tor
If an
lnd node has already been connected to the internet with an IPv4 or IPv6 (clearnet) address and has any non-private channels, this connection between channels and IP address is known to the network and cannot be deleted.
Starting the same node with the same identity and channels using Tor is trivial to link back to any previously used clearnet IP address and does therefore not provide any privacy benefits.
The following steps are recommended to cut all links between the old clearnet node and the new Tor node: 1. Close all channels on the old node and wait for them to fully close. 1. Send all on-chain funds of the old node through a Coin Join service (like Wasabi or Samurai/Whirlpool) until a sufficiently high anonymity set is reached. 1. Create a new lnd node with a new seed that is only connected to Tor and generate an on-chain address on the new node. 1. Send the mixed/coinjoined coins to the address of the new node. 1. Start opening channels. 1. Check an online network explorer that no IPv4 or IPv6 address is associated with the new node's identity.Prevent data corruption
Many problems while running an
lnd node can be prevented by avoiding data corruption in the channel database (<lnddir>/data/graph/mainnet/channel.db).The following (non-exhaustive) list of things can lead to data corruption:
- A spinning hard drive gets a physical shock.
lnd's main data directory being written on an SD card or USB thumb drive(SD cards and USB thumb drives must be considered unsafe for critical filesthat are written to very often, as the channel DB is).lnd's main data directory being written to a network drive withoutfsyncsupport.- Unclean shutdown of
lnd. - Aborting channel operation commands (see next chapter).
- Not enough disk space for a growing channel DB file.
- Moving
lnd's main data directory between different operating systems/architectures.
To avoid most of these factors, it is recommended to store
lnd's main data directory on an Solid State Drive (SSD) of a reliable manufacturer. An alternative or extension to that is to use a replicated disk setup. Making sure a power failure does not interrupt the node by running a UPS ( uninterruptible power supply) might also make sense depending on the reliability of the local power grid and the amount of funds at stake.Don't interrupt lncli commands
lncli commandsThings can start to take a while to execute if a node has more than 50 to 100 channels. It is extremely important to never interrupt an
lncli command if it is manipulating the channel database, which is true for the following commands:openchannelclosechannelandcloseallchannelsabandonchannelupdatechanpolicyrestorechanbackup
Interrupting any of those commands can lead to an inconsistent state of the channel database and unpredictable behavior. If it is uncertain if a command is really stuck or if the node is still working on it, a look at the log file can help to get an idea.
Regular accounting/monitoring
Regular monitoring of a node and keeping track of the movement of funds can help prevent problems. Tools like
lndmon can assist with these tasks.Pruned bitcoind node
Running
lnd connected to a bitcoind node that is running in prune mode is not supported! lnd needs to verify the funding transaction of every channel in the network and be able to retrieve that information from bitcoind which it cannot deliver when that information is pruned away.In theory pruning away all blocks before the SegWit activation would work as LN channels rely on SegWit. But this has neither been tested nor would it be recommended/supported.
In addition to not running a pruned node, it is recommended to run
bitcoind with the -txindex flag for performance reasons, though this is not strictly required.Multiple
lnd nodes can run off of a single bitcoind instance. There will be connection/thread/performance limits at some number of lnd nodes but in practice running 2 or 3 lnd instances per bitcoind node didn't show any problems.The --noseedbackup flag
--noseedbackup flagThis is a flag that is only used for integration tests and should never be used on mainnet! Turning this flag on means that the 24 word seed will not be shown when creating a wallet. The seed is required to restore a node in case of data corruption and without it all funds (on-chain and off-chain) are being put at risk.
Last modified 10mo ago
Copy link
Contents
Table of Contents
Overview
aezeed
Wallet password
TLS
Macaroons
Static Channel Backups (SCBs)
Static remote keys
Best practices
aezeed storage
File based backups
Keeping Static Channel Backups (SCB) safe
Keep lnd updated
Zombie channels
Migrating a node to a new device
Migrating a node from clearnet to Tor
Prevent data corruption
Don't interrupt lncli commands
Regular accounting/monitoring
Pruned bitcoind node
The --noseedbackup flag