Debugging LND
Overview
LND ships with a few useful features for debugging, such as a built-in profiler and tunable logging levels. If you need to submit a bug report for LND, it may be helpful to capture debug logging and performance data ahead of time.
Debug Logging
Logging is useful for security and operating purposes. LND logs can typically be found on Linux at ~/.lnd/logs/bitcoin/mainnet/lnd.log, on macOS at ~/Library/Application Support/Lnd/logs/bitcoin/mainnet or in their specified location using the--logdir flag at startup.
By default, LND will log 10MB worth of its history, and additionally keep three blocks of logs around, compressed with gzip as lnd.log.<i>.gz in the same directory.
You can adjust the location of your log files as well their maximum size (in MB) and how many historical log files you expect LND to keep, in your lnd.conf file.
logdir=~/.lnd/logs
maxlogfiles=3
maxlogfilesize=10
debuglevel=debug,PEER=info
Additionally, the debuglevel can be overridden and adjusted without requiring a restart using the command lncli debuglevel --level=
The available debug levels are, in order of descending detail: trace, debug, info, warn, error, critical, off
Example usage:
lncli debuglevel –-level=debug
Additionally, the log level can be adjusted for each individual subsystem. A list of the subsystem can also be obtained with the command lncli debuglevel --show
Varying debug levels for multiple subsystems can be chained together with commands.
Example usage:
debuglevel --level=BTCN=trace,LNWL=debug
Subsystems:
LNWL
lnwallet
Lightning Wallet
DISC
discovery
Discovery
NTFN
chainntnfs
Chain Notifications
CHDB
channeldb
Channel database
HSWC
htlcswitch
HTLC Switch
CMGR
connmgr
Connection Manager
BTCN
neutrino
Neutrino
CNCT
contractcourt
Contract Court
UTXN
contractcourt
Contract Court
BRAR
contractcourt
Contract Court
SPHX
sphinx
Sphinx
SWPR
sweep
Sweep Transactions
SGNR
signrpc
Signature RPC
WLKT
walletrpc
Wallet RPC
ARPC
autopilotrpc
Autopilot RPC
INVC
invoices
Invoices
NANN
netann
Network Announcements
WTWR
watchtower
Watchtower
NTFR
chainrpc
Chain RPC
IRPC
invoicesrpc
Invoices RPC
CHNF
channelnotifier
Channel Notifier
CHBU
chanbackup
Channel backup
PROM
monitoring
Monitoring
WTCL
wtclient
Watch Tower Client
PRNF
peernotifier
Peer Notifier
CHFD
chanfunding
Channel Funding
PEER
peer
Peer
CHCL
chancloser
Channel Closer
Capturing pprof data with lnd
lndlnd has a built-in feature which allows you to capture profiling data at runtime using pprof, a profiler for Go. The profiler has negligible performance overhead during normal operations (unless you have explicitly enabled CPU profiling).
To enable this ability, start lnd with the --profile option using a free port or add profile=9736 to your lnd.conf.
Now, with lnd running, you can use the pprof endpoint on port 9736 to collect runtime profiling data. You can fetch this data using curl like so:
CPU Profile
A CPU profile can be used to analyze LND's CPU usage. You can specify the time duration as a query parameter.
Goroutine profile
The goroutine profile is very useful when analyzing deadlocks and lock contention.
Heap profile
The heap profile is useful to analyze memory allocations.
Visualizing the profile dumps
It can be hard to make sense of the profile dumps by just looking at them. The Golang ecosystem provides tools to analyze those profile dumps either via the terminal or by visualizing them. One of the tools is go tool pprof.
Assuming the profile was fetched via curl as in the examples above a nice svg visualization can be generated for the cpu profile like this:
Last updated