Onboarding to Bitcoin Core

The GitHub side of the Bitcoin Core workflow for Contributors consists primarily of:

Generally, issues are used for two purposes:

GitHub provides their own guide on mastering Issues which is worth reading to understand the feature-set available when working with an issue.

PRs are where Contributors can submit their code against the main codebase and solicit feedback on the concept, the approach taken for the implementation, and the actual implementation itself.

PRs and Issues are often linked to/from one another:

Another use-case of Issues is soliciting feedback on ideas that might require significant changes. This helps free the project from having too many PRs open which aren’t ready for review and might waste reviewers' time. In addition, this workflow can also save Contributors their own valuable time, as an idea might be identified as unlikely to be accepted before the contributor spends their time writing the code for it.

Most code changes to bitcoin are proposed directly as PRs — there’s no need to open an Issue for every idea before implementing it unless it may require significant changes. Additionally, other Contributors (and would-be Reviewers) will often agree with the approach of a change, but want to "see the implementation" before they can really pass judgement on it.

GitHub is therefore used to help store and track reviews to PRs in a public way.

Comments (inside Issues, PRs, Projects etc.) are where all (GitHub) users can discuss relevant aspects of the item and have history of those discussions preserved for future reference. Often Contributors having "informal" discussions about changes on e.g. IRC will be advised that they should echo the gist of their conversation as a comment on GitHub, so that the rationale behind changes can be more easily determined in the future.

Jon Atack provides a guide to reviewing a Bitcoin Core PR in his article How To Review Pull Requests in Bitcoin Core.

Gloria Zhao’s review checklist details what a "good" review might look like, along with some examples of what she personally considers good reviews. In addition to this, it details how potential Reviewers can approach a new PR they have chosen to review, along with the sorts of questions they should be asking (and answering) in order to provide a meaningful review. Some examples of the subject areas Gloria covers include the PR’s subject area, motivation, downsides, approach, security and privacy risks, implementation of the idea, performance impact, concurrency footguns, tests and documentation needed.

This section details some of the processes surrounding code contributions to the Bitcoin Core project along with some common pitfalls and tips to try and avoid them.

When considering changing code it can be helpful to try and first understand the rationale behind why it was implemented that way originally. One of the best ways to do this is by using a combination of git tools:

As well as the discussions in various places on the GitHub repo.

When building Bitcoin Core from source, there are some platform-dependant instructions to follow.

To learn how to build for your platform, visit the Bitcoin Core bitcoin/doc directory, and read the file named "build-*.md", where "*" is the name of your platform. For windows this is "build-windows.md", for macOS this is "build-osx.md" and for most linux distributions this is "build-unix.md".

There is also a guide by Jon Atack on how to compile and test Bitcoin Core.

Finally, Blockchain Commons also offer a guide to building from source.

Bitcoin Core uses Doxygen to generate developer documentation automatically from its annotated C++ codebase. Developers can access documentation of the current release of Bitcoin Core online at doxygen.bitcoincore.org, or alternatively can generate documentation for their current git HEAD using make docs (see Generating Documentation for more info).

Three types of test network are available:

These three networks all use coins of zero value, so can be used experimentally.

They primary differences between the networks are as follows:

One of the roles most in-demand from the project is that of code review, and in fact this is also one of the best ways of getting familiarized with the codebase too! Reviewing a few PRs and adding your review comments to the PR on GitHub can be really valuable for you, the PR author and the bitcoin community. This Google Code Health blog post gives some good advice on how to go about code review and getting past "feeling that you’re not as smart as the programmer who wrote the change". If you’re going to ask some questions as part of review, try and keep questions respectful.

There is also a Bitcoin Core PR Review Club held weekly on IRC which provides an ideal entry point into the Bitcoin Core codebase. A PR is selected, questions on the PR are provided beforehand to be discussed on irc.libera.chat #bitcoin-core-pr-reviews IRC room and a host will lead discussion around the changes.

Aside from review, there are 3 main avenues which might lead you to submitting your own PR to the repository:

Choosing a "good first issue" from an area of the codebase that seems interesting to you is often a good approach. This is because these issues have been somewhat implicitly "concept ACKed" by other Contributors as "something that is likely worth someone working on". Don’t confuse this for meaning that if you work on it that it is certain to be merged though.

If you don’t have a bug fix or new feature in mind and you’re struggling to find a good first issue which looks suitable for you, don’t panic. Instead keep reviewing other Contributors' PRs to continue improving your understanding of the process (and the codebase) while you watch the Issue tracker for something which you like the look of.

When you’ve decided what to work on it’s time to take a look at the current behaviour of that part of the code and perhaps more importantly, try to understand why this was originally implemented in this way. This process of codebase "archaeology" will prove invaluable in the future when you are trying to learn about other parts of the codebase on your own.

The Bitcoin Core project has an IRC channel #bitcoin-core-dev available on the Libera.chat network. If you are unfamiliar with IRC there is a short guide on how to use it with a client called Matrix here. IRC clients for all platforms and many terminals are available.

"Lurking" (watching but not talking) in the IRC channel can both be a great way to learn about new developments as well as observe how new technical changes and issues are described and thought about from other developers with an adversarial mindset. Once you are comfortable with the rules of the room and have questions about development then you can join in too!

There are also regular meetings held on #bitcoin-core-dev which are free and open for anyone to attend. Details and timings of the various meetings are found here.

In reality there are no hard rules on choosing a discussion forum, but in practice there are some common conventions which are generally followed:

If discussing something Bitcoin Core-related, there can also be a question of whether it’s best to open an Issue, to first discuss the problem and brainstorm possible solution approaches, or whether you should implement the changes as you see best first, open a PR, and then discuss changes in the PR. Again, there are no hard rules here, but general advice would be that for potentially-controversial subjects, it might be worth opening an Issue first, before (potentially) wasting time implementing a PR fix which is unlikely to be accepted.

Regarding communication timelines it is important to remember that many contributors are unpaid volunteers, and even if they are sponsored or paid directly, nobody owes you their time. That being said, often during back-and-forth communication you might want to nudge somebody for a response and it’s important that you do this in a courteous way. There are again no hard rules here, but it’s often good practice to give somebody on the order of a few days to a week to respond. Remember that people have personal lives and often jobs outside of Bitcoin development.

Bitcoin Core often backports fixes for bugs and soft fork activations into previous software releases.

Generally maintainers will handle backporting for you, unless for some reason the process will be too difficult. If this point is reached a decision will be made on whether the backport is abandoned, or a specific (new) fix is created for the older software version.

Anyone who contributes code to the codebase is labelled a Contributor by GitHub and also by the community. As of Version 23.0 of Bitcoin Core, there are > 850 individual Contributors credited with changes.

Some Contributors are also labelled as Members of the Bitcoin organisation. There are no defined criteria for becoming a Member of the organisation; persons are usually nominated for addition or removal by current Maintainer/Member/Admin on an ad-hoc basis. Members are typically frequent Contributors/Reviewers and have good technical knowledge of the codebase.

Some members also have some additional permissions over Contributors, such as adding/removing tags on issues and Pull Requests (PRs); however, being a Member does not permit you to merge PRs into the project. Members can also be assigned sections of the codebase in which they have specific expertise to be more easily requested for review as Suggested Reviewers by PR authors.

Some organisation Members are also project Maintainers. The number of maintainers is arbitrary and is subject to change as people join and leave the project, but has historically been less than 10. PRs can only be merged into the main project by Maintainers. While this might give the illusion that Maintainers are in control of the project, the Maintainers' role dictates that they should not be unilaterally deciding which PRs get merged and which don’t. Instead, they should be determining the mergability of changes based primarily on the reviews and discussions of other Contributors on the GitHub PR.

Working on that basis, the Maintainers' role becomes largely janitorial. They are simply executing the desires of the community review process, a community which is made up of a decentralized and diverse group of Contributors.

It is possible for a "rogue PR" to be submitted by a Contributor; we rely on systematic and thorough peer review to catch these. There has been discussion on the mailing list about purposefully submitting malicious PRs to test the robustness of this review process.

In the event that a Maintainer goes rogue and starts merging controversial code, or conversely, not merging changes that are desired by the community at large, then there are two possible avenues of recourse:

In the case that GitHub itself becomes the rogue entity, there have been numerous discussions about how to move away from GitHub, which have been summarized on the devwiki here. This summary came in part from discussions on this GitHub issue.

Bitcoin Core issue #22665 described how BIP125 was not being strictly adhered to by Bitcoin Core. This raised discussion amongst developers about whether the code (i.e. "the implementation") or the BIP itself should act as the specification, with most developers expressing that they felt that "the code was the spec" and any BIP generated was merely a design document to aid with re-implementation by others, and should be corrected if necessary.

For consensus-critical code most Bitcoin Core Developers consider "the code is the spec" to be the ultimate source of truth, which is one of the reasons that recommending running other full node implementations can be so difficult. A knock-on effect of this was that there were calls for review on BIP2 itself, with respect to how BIPs should be updated/amended. Newly-appointed BIP maintainer Karl-Johan Alm (a.k.a. kallewoof) posted his thoughts on this to the bitcoin-dev mailing list.

In summary a BIP represents a design document which should assist others in implementing a specific feature in a compatible way. These features are optional to usage of Bitcoin, and therefore implementation of BIPs are not required to use Bitcoin, only to remain compatible. Simply being assigned a BIP does not mean that an idea is endorsed as being a "good" idea, only that it is fully-specified in a way that others could use to re-implement. Many ideas are assigned a BIP and then never implemented or used on the wider network.

The following diagram gives a brief overview of how the tests are structured within the source directory.

Bitcoin Core’s test coverage reports can be found here.

In order to debug a multi-threaded application like bitcoind using gdb you will need to enable following child processes. Below is shown the contents of a file threads.brk which can be sourced into gdb using source threads.brk, before you start debugging bitcoind. The file also loads break points where new threads are spawned.

With data from the raw block* and rev* files, various LevelDB indexes can be built. These indexes enable fast lookup of data without having to rescan the entire block chain on disk.

Some of these databases are mandatory and some of them are optional and can be enabled using run-time configuration flags.

The consensus model in the codebase can be thought of as a database of the current state of the blockchain. When a new block is learned about it is processed and the consensus code must determine which block is the current best. Consensus can be thought of as a function of available information — it’s output is simply a deterministic function of its input.

There are a simple set of rules for determining the best block:

The result of these rules is a tree-like structure from genesis to the current day, building on only valid blocks.

Whilst this is easy-enough to reason about in theory, the implementation doesn’t work exactly like that. It must consider state, do I go forward or backwards for example.

Pieter Wuille disclosed the possibility of a consensus failure via usage of OpenSSL. The issue was that the OpenSSL signature verification was accepting multiple signature serialization formats (for the same signature) as valid. This effectively meant that a transactions' ID (txid) could be changed, because the signature contributes to the txid hash.

There were a few cases to consider:

In the length descriptor case there is a higher risk of causing a consensus-related chainsplit. The sender can create a normal-length valid signature, but which uses a 5 byte length descriptor meaning that it might not be accepted by OpenSSL on all platforms.

In the second case, signature tweaking or padding, there is a lesser risk of causing a consensus-related chainsplit. However the ability of third parties to tamper with valid transactions may open up off-chain attacks related to Bitcoin services or layers (e.g. Lightning) in the event that they are relying on txids to track transactions.

It is interesting to consider the order of the steps taken to fix this potential vulnerability:

We can see the resulting flag in the script verification enum:

Having OpenSSL as a consensus-critical dependency to the project was ultimately fixed in PR#6954 which switched to using the in-house libsecp256k1 library (as a subtree) for signature verification.

Historically Bitcoin Core used Berkeley DB (BDB) for transaction and block indices. In 2013 a migration to LevelDB for these indices was included with Bitcoin Core v0.8. What developers at the time could not foresee was that nodes that were still using BDB, all pre 0.8 nodes, were silently consensus-bound by a relatively obscure BDB-specific database lock counter.

Bitcoin Core was interpreting a failure to grab the required number of locks as equivalent to block validation failing. This caused some BDB-using nodes to mark blocks created by LevelDB-using nodes as invalid and caused a consensus-level chain split. BIP 50 provides further explanation on this incident.

BDB has caused other potentially-dangerous behaviour in the past. Developer Greg Maxwell describes in a Q&A how even the same versions of BDB running on the same system exhibited non-deterministic behaviour which might have been able to initiate chain re-orgs.

This Bitcoin Core disclosure details a potential inflation bug.

It originated from trying to speed up transaction validation in main.cpp#CheckTransaction() which is now consensus/tx_check.cpp#CheckTransaction(), something which would in theory help speed up IBD (and less noticeably singular/block transaction validation). The result in Bitcoin Core versions 0.15.x → 0.16.2 was that a coin that was created in a previous block, could be spent twice in the same block by a miner, without the block being rejected by other Bitcoin Core nodes (of the aforementioned versions).

Whilst this bug originates from validation, it can certainly be described as a breach of consensus parameters. In addition, nodes of version 0.14.x ⇐ node_version >= 0.16.3 would reject inflation blocks, ultimately resulting in a chain split provided that miners existed using both inflation-resistant and inflation-permitting clients.

SegWit was the first attempt to go beyond simply repurposing OP_NOPs for upgrades. The idea was that the scriptPubKey/redeemScript would consist of a 1 byte push opcode (0-16) followed by a data push between 2 and 40 bytes. The value of the first push would represent the version number, and the second push the witness program. If the conditions to interpret this as a SegWit script were matched, then this would be followed by a witness, whose data varied on whether this was a P2WPKH or P2WSH witness program.

Legacy nodes, who would not have the witness data, would interpret this output as anyonecanspend and so would be happy to validate it, whereas upgraded nodes could validate it using the additional witness against the new rules. To revert to the statue analogy this gave us the ability to work with a new area of the marble which was entirely untouched.

The addition of a versioning scheme to SegWit was a relatively late addition which stemmed from noticing that, due to the CLEANSTACK policy rule which required exactly 1 true element to remain on the stack after execution, SegWit outputs would be of the form OP_N + DATA. With SegWit we wanted a compact way of creating a new output which didn’t have any consensus rules associated with it, yet had lots of freedom, was ideally already non-standard, and was permitted by CLEANSTACK.

The solution was to use two pushes: according to old nodes there are two elements, which was non-standard. The first push must be at least one byte, so we can use one of the OP_N opcodes, which we then interpret as the SegWit version. The second is the data we have to push.

Whilst this immediately gave us new upgrade paths via SegWit versions Taproot (SegWit version 1) went a step further and declared new opcodes inside of SegWit, also evaluated as anyonecanspend by nodes that don’t support SegWit, giving us yet more soft fork upgradability. These opcodes could in theory be used for anything, for example if there was ever a need to have a new consensus rule on 64 bit numbers we could use one of these opcodes.

AcceptToMemoryPool() (ATMP) is where the checks on single transactions occur before they enter the mempool.

You can see the calls to the various *Checks() functions in the call graph, and the order in which they are run.

Let’s take a look inside AcceptToMemoryPool()'s inner function AcceptSingleTransaction() which handles running the checks:

Once AcceptSingleTransaction has acquired the cs_main and m_pool.cs locks it initializes a Workspace struct — a storage area for (validation status) state which can be shared by the different validation checks. Caching this state avoids performing the same computations multiple times and is important for performance. It will pass this workspace, along with the struct of ATMPArgs it received as argument, to the checks.

The Workspace is initialized with a pointer to the transaction (as a CTransactionRef) and holds some additional information related to intermediate state.

We can look at the ATMPArgs struct to see what other information our mempool wants to know about in addition to transaction information.

If all the checks pass and this was not a test_accept submission then we will MemPoolAccept::Finalize the transaction, adding it to the mempool, before trimming the mempool size and updating any affected RBF transactions as required.

TODO: This section should start from AcceptPackage() and flow through from there, including AcceptMultipleTransactions() as a sub-section.

It’s possible to consider multiple transactions for validation together, via AcceptMultipleTransactions() found in src/net_processing.cpp. It’s currently only available from tests (test/tx_package_tests.cpp) and the testmempoolaccept RPC (via ProcessNewPackage()), but the intention is for it to be available to packages received from the P2P network in the future.

This validation flow has been created for usage with Package Mempool Accept, which glozow has written up in a gist (archive).

The flow here is similar to AcceptSingleTransaction() in that we start by grabbing cs_main before initializing validation state and workspaces, however this time we use PackageValidationState and a vector of workspaces, std::vector<Workspace>. Each transaction therefore has it’s own workspace but all transactions in the package share a single validation state. This aligns with the goal of either accepting or rejecting the entire package as a single entity.

Next come two for loops over the vector of workspaces (i.e. transactions). The first performs the PreChecks(), but this time also freeing up coins to be spent by other transactions in this package. This would not usually be possible (nor make sense) within an AcceptTransaction() flow, but within a package we want to be able to validate transactions who use as inputs, other transactions not yet added to our mempool:

If the PreChecks do not fail, we call m_viewmempool.PackageAddTransaction() passing in the workspace. This adds the transaction to a map in our Mempool called std::unordered_map<COutPoint, Coin, SaltedOutpointHasher> m_temp_added;, which is essentially a temporary cache somewhere in-between being validated and being fully added to the mempool.

TODO: Fix after adding section on AcceptPackage

After this first loop we perform PackageMempoolChecks() which first asserts that transactions are not already in the mempool, before checking the "PackageLimits".

The code comments for PreChecks give a clear description of what the PreChecks are for:

The PreChecks function is very long but is worth examining to understand better which checks are undertaken as part of this first stage.

During PreChecks the m_rbf bool will have been set to true if it is determined that this transaction would have to replace an existing transaction from our mempool. If this bool is set, then ReplacementChecks will be run. These checks are designed to check that BIP125 RBF rules are being adhered to.

Following ReplacementChecks we initialise a PrecomputedTransactionData struct in the Workspace which will hold expensive-to-compute data that we might want to use again in subsequent validation steps.

Next we call PolicyScriptChecks() passing in the same ATMPArgs and Workspace that we used with PreChecks. This is going to check the transaction against our individual node’s policies.

PolicyScriptChecks() starts with initialisation of the transaction into a CTransaction, before beginning to check the input scripts against the script flags.

If the script type is SegWit an additional round of checking is performed, this time including the CLEANSTACK rule. The call(s) flag cacheSigStore as true, and cacheFullScriptStore as false, which means that matched signatures will be persisted in the cache, but matched full scripts will be removed.

If the PolicyScriptChecks return true we will move on to consensus script checks, again passing in the same ATMPArgs, Workspace and now PrecomputedTransactionData that we used previously with PolicyScriptChecks.

The main check in here is CheckInputsFromMempoolAndCache() which is going to compare all the transaction inputs to our mempool, checking that they have not already been marked as spent. If the coin is not already spent, we reference it from either the UTXO set or our mempool, and finally submit it through CheckInputScripts() once more, this time caching both the signatures and the full scripts.

PackageMempoolChecks are designed to "Enforce package mempool ancestor/descendant limits (distinct from individual ancestor/descendant limits done in PreChecks)". They take a vector of CTransactionRefs and a PackageValidationState.

Again we take two locks before checking that the transactions are not in the mempool. Any transactions which are part of the package and were in the mempool will have already been removed by MemPoolAccept::AcceptPackage().

Finally we check the package limits, which consists of checking the {ancestor|descendant} {count|size}. This check is unique to packages because we can now add descendants whose ancestors would not otherwise qualify for entry into our mempool with their low effective fee rate.

Provided that consensus script checks pass and this was not a test ATMP call, we will call Finalize() on the transaction. This will remove any conflicting (lower fee) transactions from the mempool before adding this one, finishing by trimming the mempool to the configured size (default: static const unsigned int DEFAULT_MAX_MEMPOOL_SIZE = 300; MB). In the event that this transaction got trimmed, we ensure that we return a TxValidationResult::TX_MEMPOOL_POLICY, "mempool full" result.

Transactions learned about from blocks:

This means that we can validate these transactions based only on our copy of the UTXO set and the data contained within the block itself. We call ProcessBlock() when processing new blocks received from the P2P network (in net_processing.cpp) from net message types: NetMsgType::CMPCTBLOCK, NetMsgType::BLOCKTXN and NetMsgType::BLOCK.

The general flow of ProcessBlock() is that will call CheckBlock(), AcceptBlock() and then ActivateBestChain(). A block which has passed successfully through CheckBlock() and AcceptBlock() has not passed full consensus validation.

CheckBlock() does some cheap, context-independent structural validity checks, along with (re-)checking the proof of work in the header, however these checks just determine that the block is "valid-enough" to proceed to AcceptBlock().

Once the checks have been completed, the block.fChecked value is set to true. This will enable any subsequent calls to this function with this block to be skipped.

AcceptBlock() is used to persist the block to disk so that we can (validate it and) add it to our chain immediately, use it later, or discard it later. AcceptBlock() makes a second call to CheckBlock() but because block.fChecked was set to true on the first pass this second check will be skipped.

It also now runs some contextual checks such as checking the block time, transaction lock times (transaction are "finalized") and witness commitments are either non-existent or valid (link). After this the block will be serialized to disk.

Once the block has been written to disk by AcceptBlock() full validation of the block and its transactions begins via CChainState::ActivateBestChain() and its inner call to ActivateBestChainStep().

Wallets are stored on disk as databases, either using Berkeley Database (BDB) or sqlite format.

The wallet is stored on disk as a Key Value store.

These are some of the records which help us regenerate a descriptor wallet (populating a DescriptorScriptPubKeyMan (DSPKM)) from the database:

For Legacy wallets (populating a LegacyScriptPubKeyMan (LSPKM)) we use the records with *.KEY & SCRIPT.

Wallet metadata may include a tipLocator — the most recent tip — and a wallet version which is used in database upgrades.

To load the wallet we read the database by iterating the records and loading them to CWallet, using ReadKeyValue() to deserialize.

You can see where all the other DB records are deserialized to by examining the ReadKeyValue() function.

The various *ScriptPubkeyMan objects are all owned by the CWallet instance eventually, however LegacyScriptPubKeyMan is both created and owned by CWallet, whereas DescriptorScriptPubKeyMan is created externally to CWallet and only after loading exists in the CWallet context.

Note that TxSpends is not tracked in the wallet database (and loaded at startup), but instead is rebuilt from scratch because it’s fast to do so and we must reload every transaction anyway, so it’s not much more work to regenerate TxSpends at the same time.

There are a number of Key classes in the wallet code and keeping track of their functions can be confusing at times due to naming similarities. Below are listed some of these classes along with some primary functions.

There is encryption in the wallet code, but it is found within both CWallet and *ScriptPubKeyMan so is not yet well encapsulated.

When you unlock an encrypted wallet you can set a timeout. When the timeout expires secret data is deleted from memory, and the wallet "re-locked".

When we learn about a new block the BlockConnected signal is fired after successful validation. This prompts the wallet to iterate all inputs and outputs, calling IsMine() on all of them. As part of the check, we loop over the wallet’s scriptPubkeyMans to check if any of the scripts belong to us.

If a script does belong to us, it will be inserted into mapWallet along with some metadata related to the time. mapWallet contains all the transactions the wallet is interested in, including received and sent transactions.

When we load a wallet into memory, we iterate all TxSpends. TxSpends stores wallet transactions which were already spent and confirmed.

Therefore, when the wallet needs to select coins to spend, it can select from the coins:

mapWallet - TxSpends - notMine

For balance calculation we iterate mapWallet and add values to a Balance struct.

We do some caching during iteration so that we avoid re-calculating the same values for multiple transactions.

Calculating the above requires using TxSpends and IsMine.

When a new transaction involving the wallet takes place, really what happens is that it’s marked as DIRTY, which deletes the cached entry for the parent transaction. This means that the next time GetBalance() is called, debit is recalculated correctly. This Bitcoin Core PR review club goes into more detail on coins being marked as DIRTY and FRESH in the cache.

TxSpends is calculated by looking at the outpoints in the transaction itself.

COutputs are ephemeral — we create them, perform another operation with them and discard them. They are stored in availableCoins which is recreated when calling functions such as GetAvailableBalance(), ListCoins() and CreateTransactionInternal().

In a spending transaction all inputs have their corresponding OutPoints, and we map these to spending transactions in TxSpends.

If a transaction is our own we check for validity with testMempoolAccept before submitting to the P2P network.

For DSPKM running IsMine() is really simple: descriptors generate a list of ScriptPubKeys, and, if the SPK we are interested in is in the list, then it’s ours.

IsMine returns an enum. This is used as a return value, a filter and set of flags simultaneously. There is more background on the general IsMine semantics in the v0.21.0 release notes.

LSPKM can have watch-only and spendable flags set at the same time, but DSPKM is either or, because descriptor wallets do not allow mixtures of spendable and watch-only keys in the same SPKM. Because Legacy wallets are all key-based, we will need to see if a script could have been generated by one of our keys; what type of script it is; and if we have a (private) key for it.

For Legacy watch-only wallets we simply check "do we have this script stored as a script?" (where CScripts in the database are our watch-only scripts)". If we don’t have a CKey for a script but it exists in mapScripts then it’s implicitly watch-only.

A problem with this current method of IsMine for legacy wallets is that it’s tough to figure out what your wallet considers "Mine" — it’s probably a finite set, but maybe not…​

Another consideration is that the LSPKM IsMine includes P2PK outputs — which don’t have addresses! This un-enumerability can be an issue in migration of Legacy to Descriptor wallets.

There is also the possibility that someone can mutate address to different address type and you will still see it as IsMine. E.g. mutate P2PK into P2PKH address and wallet will still detect.

With descriptors we only look for scripts explicitly. With descriptor wallets IsMine might not recognise script hashes from scripts, because it was not told to watch for them and consider them as belonging to it.

We use the IsMine filters in many places, primarily to distinguish between spendable and watch-only:

See the section on how wallets determine whether transactions belong to them using the IsMine enum for more in-depth information.

Conflict tracking is related to changing the state as the mempool tells us about conflicting transactions.

mapTxSpends is a multimap which permits having the same COutPoint mapping to two transactions. (i.e. two transactions spending the same input) This is how we can tell if things are conflicted: look up an outpoint and check to see how many transactions are there, if > 1 then we know that there was a conflict.

If there is a conflict we can look up the wallet transaction and see what state it’s in, and we can be sure about whether it is currently or previously conflicted.

Conflict tracking is particularly relevant for coin selection…​

See Bitcoin Optech for more information on coin selection. There is a section digging deeper into the coin selection code found below. To select inputs to a transaction our primary considerations are privacy and fees.

The below sections form an overview of creating a transaction via CreateTransactionInternal().

Once the coins have been selected they are returned back to CreateTransactionInternal(), which will create the final transaction.

Right now when we determine the change output, we don’t use what selectionResult says the change output should be. What we actually do is make the tx with in? outputs and set the change amount to be the sum inputs-outputs, so the change amount includes the transaction fee. To get the correct change amount we now calculate the size of this after signing, we use dummysigner to add a dummy signature (74 0’s and the correct script), and now we can calculate the correct fee. We reduce that fee from the change output amount, and if this now goes below some threshold? (the "cost of change" thing from BnB) or if it is dust we drop the change output and add it’s value to the fee.

So now we have an unsigned tx which we need to sign.

We pass the tx to CWallet::SignTransaction() which will call IsMine() on each input to figure out which ScriptPubKeyMan (spkman) owns that input, then ask the spkman to fetch its SigningProviders to provide the signer which can sign the transaction, and return that to us.

With PSBTs we have the fillPSBT() method in CWallet which calls *ScriptPubKeyMan::fillPSBT(). We do this because we can add previous UTXOs due to transaction tracking; the SPKM adds the scripts and key derivation paths and will then optionally sign.

In order to facilitate code separation, distinct interfaces between the node and the wallet have been created:

Wallet(s) to be loaded as part of program startup can be specified by passing -wallet= or -walletdir= arguments to bitcoind/bitcoin-qt. If the wallet has been compiled in but no -wallet*= arguments have been passed, then the default wallet directory ($datadir/wallets) will be checked as per GetWalletDir():

Wallets can also be loaded after program startup via the loadwallet RPC.

Wallet verification refers to verification of the -wallet arguments as well as the underlying wallet database(s) on disk.

Wallets loaded via program arguments are first verified as part of AppInitMain() which first verifies wallet database integrity by calling VerifyWallets() via the WalletClientImpl override of client→verify().

VerifyWallets() takes an interfaces::Chain object as argument, which is currently used to send init and error messages (about wallet verification) back to the GUI. VerifyWallets() starts by checking that the walletdir supplied by argument, or default of "", is valid. Next it loops through all wallets it finds in the walletdir and adds them to an std::set called wallet_paths, first de-duplicating them by tracking their absolute paths, and then checking that the WalletDatabase for each wallet exists (or is otherwise constructed successfully) and can be verified.

If this check passes for all wallets, then VerifyWallets() is complete and will return true to calling function AppInitMain, otherwise false will be returned. If VerifyWallets() fails and returns false (due to a corrupted wallet database, but notably not due to an incorrect wallet path), the main program process AppInit() will be immediately interrupted and shutdown.

"Startup" wallet(s) are loaded when client→load() is called on each node.chain_client as part of init.cpp.

The call to load() on the wallet chain_clients has again been overridden, this time by WalletClientImpl's LoadWallets() method. This function works similarly to VerifyWallets(), first creating the WalletDatabase (memory) object for each wallet, although this time skipping the verify step, before creating a CWallet object from the database and adding it to the global list of wallets, the vector vpwallets, by calling AddWallet().

init.cpp is run on a single thread. This means that calls to wallet code block further initialisation of the node.

The interfaces::Chain object taken as argument by LoadWallets() is used to pass back any error messages, exactly as it was in VerifyWallets(). More information on AddWallet() can be found in src/wallet.cpp.

The wallet is finally ready when (all) chain_clients have been started in init.cpp which calls the overridden client→start() method from the WalletClientImpl class, resulting in src/wallet/load.cpp#StartWallets() being called.

This calls the GetWallets() function which returns a vector of pointers to the interfaces for all loaded CWallet objects, called vpwallets. As part of startup PostInitProcess() is called on each wallet which, after grabbing the main wallet lock cs_wallet, synchronises the wallet and mempool by adding wallet transactions not yet in a block to our mempool, and updating the wallet with any relevant transactions from the mempool.

Also, as part of StartWallets, flushwallet might be scheduled (if configured by argument) scheduling wallet transactions to be re-broadcast every second, although this interval is delayed upstream with a random timer.

All wallets loaded into the program are "flushed" (to disk) before shutdown. As part of init.cpp#Shutdown() the flush() method is called on each member of node.chain_clients in sequence. WalletClientImpl again overrides this method to call wallet/load.cpp#FlushWallets() which makes sure all wallet changes have been successfully flushed to the wallet database.

Finally the stop() method is called on each member of node.chain_clients which is overridden by StopWallets(), flushing again and this time calling close() on the database file.

In order to not block the rest of the program during wallet operations, each CWallet has its own recursive mutex cs_wallet:

Most wallet operations whether reading or writing data require the use of the lock so that atomicity can be guaranteed. Some examples of wallet operations requiring the lock include:

In addition to these higher level functions, most of CWallet's private member functions also require a hold on cs_wallet.

If we take a look at the loadwallet RPC we can see similarities to WalletClientImpl's LoadWallets() function.

However this time the function will check the WalletContext to check that we have a wallet context (in this case a reference to a chain interface) loaded. Next it will call wallet.cpp#LoadWallet which starts by grabbing g_loading_wallet_mutex and adding the wallet to g_loading_wallet_set, before calling LoadWalletInternal which adds the wallet to vpwallets and sets up various event notifications.

Further operation of the wallet RPCs are detailed in their man pages, but one thing to take note of is that whilst loadwallet() (and unloadwallet()) both take a wallet_name argument, the other wallet RPCs do not. Therefore in order to control a specific wallet from an instance of bitcoin{d|-qt} that has multiple wallets loaded, bitcoin-cli must be called with the -rpcwallet argument, to specify the wallet which the action should be performed against, e.g. bitcoin-cli --rpcwallet=your_wallet_name getbalance

The CWallet constructor takes a pointer to the chain interface for the wallet, a wallet name and a pointer to the underlying WalletDatabase:

The constructor is not called directly, but instead from the public function CWallet::Create(), which is itself called from CreateWallet(), LoadWallets() (or TestLoadWallet()). In addition to the arguments required by the constructor, CWallet::Create() also has a wallet_flags argument. Wallet flags are represented as a single unit64_t bit field which encode certain wallet properties:

See src/wallet/walletutil.h for additional information on the meanings of the wallet flags.

CWallet::Create() will first attempt to create the CWallet object and load it, returning if any errors are encountered.

If CWallet::Create is creating a new wallet — on its 'first run' — the wallet version and wallet flags will be set, before either LegacyScriptPubKeyMan or DescriptorScriptPubKeyMan's are setup, depending on whether the WALLET_FLAG_DESCRIPTORS flag was set on the wallet.

Following successful creation, various program arguments are checked and applied to the wallet. These include options such as -addresstype, -changetype, -mintxfee and -maxtxfee amongst others. It is at this stage that warnings for unusual or unsafe values of these arguments are generated to be returned to the user.

After the wallet is fully initialized and setup, its keypool will be topped up before the wallet is locked and registered with the Validation interface, which will handle callback notifications generated during the (optional) upcoming chain rescan. The rescan is smart in detecting the wallet "birthday" using metadata stored in the SPKM and won’t scan blocks produced before this date.

Finally, the walletinterface is setup for the wallet before the WalletInstance is returned to the caller.