Here's an overview of all the articles in the series:
In the present article, we will continue the series and add an important wallet manager without which your platform would probably not be complete: The Bitcoin Wallet.
Before we start looking more into how the bitcoin chain works, I want to focus a bit on security.
We've touched on this in the previous articles as well, but recently I've had the opportunity to analyze some of the 'exchange' scripts (written in PHP) some companies made available for purchasing and I think that a review of what we consider a secure wallet would help.
The security of your wallet comes down to where you store the private keys for any generated address.
Most wallets I've seen use a very simple and naive approach to interacting with the blockchain. Every address used is generated on the bitcoin node using getnewaddress and sometimes the wallet is not even protected with a password.
This approach forces you to depend on a single bitcoin node which stores all your keys. In case it fails or is hacked, you lose all your funds.
Instead, a better approach would be to use the bitcoin node only to interact with the bitcoin network (send signed transactions or get block/tx information) and store your private keys in a secure location, one that could be replicated in case of a node failure and easily secured in case of a breach.
This means you need to generate the private/public keys for each address without using the RPC calls provided by the node and store the private key in a secure location.
If you want to go a step further you could create multi-signature wallets where some keys are owned by the user and some by the wallet ensuring that your system can never have access to the funds without the user's permission.
The database is not a secure location for secret information.
It can contain the transaction or historical information or user profile information, but private keys, credentials to your bitcoin nodes or any other similar data should be be encrypted.
Take note: a base64 encoded string is not the same as an encrypted string!
When we talk about security we can not be limited to the code we write, we also need to assess the security risk of our team and the security of our deployment procedures.
To carry out this assessment, it's best if you start by considering these questions:
If every developer in your team has access to your production servers and the changes they do are not tracked through code and git commits then the weakest part in your security could be your own team.
There are a few examples in the media and in our experience where this has been the case:
A production-ready application has at least 3 different environments - one of which is the production one -, uses CI/CD to automatically deploy your platform based on changes in the code, and keeps all credentials secure and encrypted in the cloud.
If you have little or no technical knowledge, make sure you get a trustworthy technical person (or hire a development team) in your team before buying a turnkey solution.
A blockchain project - like any financial project - shouldn't be attempted by everyone and just because people have some experience writing code doesn't mean they have the know-how, processes or standards needed to write blockchain software which deals with people's money.
Coming back to our Bitcoin wallet, let's first see the difference between how the Ethereum and Bitcoin blockchains work.
This is because for Ethereum every transaction happens between two addresses.
For Bitcoin this is not true.
With bitcoin, you can receive coins from multiple addresses and send those coins to multiple destinations.
This is because bitcoin does not have the concept of a balance like Ethereum does. Instead, it uses something called an Unspent Transaction Output or UTXO for short. Simply put, this means that:
This decision architecture is very powerful in how you can use funds from different addresses and split them between different accounts.
It also imposes some more complex requirements in building a raw transaction. We would need to keep track of every deposit that we receive and spend them as inputs whenever we want to make a withdrawal.
Also, since we don't want to spend all the funds we receive, we need to send part of a withdrawal back to an address we own.
For Bitcoin, the fees represent the difference between the sum of all inputs and the sum of all outputs.
Similarly to the Ethereum wallet, we need to support the following actions for our Bitcoin wallet:
We can build the wallet in NodeJs and make use of the bitcoin-core and bitcoin-lib packages.
First thing we need to do is generate a bitcoin deposit address.
The most basic address type is normally the public/private key address. In order to spend funds from this address, we only need to sign the input transaction with the private key of the address. However, using such addresses requires that we pay more fees.
An alternative would be to also generate a segwit address which means that in order to use any deposit transaction we would need the associated script of the address.
const bitcoin = require('bitcoinjs-lib')
const network = bitcoin.networks.bitcoin
// generate basic bitcoin address
function () {
const keyPair = bitcoin.ECPair.makeRandom({
network: network
})
const publicKey = bitcoin.payments.p2pkh({
network: network,
pubkey: keyPair.publicKey
}).address
return
}Now that we have our address, let's see how we sign a raw transaction.
Sending a signed transaction is very simple as the bitcoin node already provides an RPC method for this.
const client = require("./connection")
module.exports = async function (raw_transaction, _opts) {
return await client.sendRawTransaction(raw_transaction)
}The client is just an RPC connection to our bitcoin node:
const Client = require('bitcoin-core');
const config = require('../../config')
const client = new Client({
host: config.bitcoin_rpc_host,
port: config.bitcoin_rpc_port,
username: config.bitcoin_rpc_user,
password: config.bitcoin_rpc_pass,
});
module.exports = client
The next step is signing the actual transaction we wish to send.
If you hold all coins in a wallet on the bitcoin node, then this step is very easy do to, even if it's secure.
In this article we will however look at how to sign a raw transaction with private keys stored in another location (Hashicorp Vault) from a list of unspent transaction outputs.
The exact place where this data is taken from can depend on your application, but the implementation details below should be similar in any bitcoin wallet.
// a few dependencies
const bitcoin = require('bitcoinjs-lib')
/**
* Send bitcoin from a user owned address to any other address.
* @param { from, to, amount } transaction
* @param { private_key } opts
*/
module.exports = async function (transaction, opts) {
const from_address = transaction.from
const to_address = transaction.to
const amount = transaction.amount * 100000000 // convert to satoshi
const byteFee = 40
let sum = 0
let txfee = 17600 // 40 satoshi per byte for 2 inputs and 2 outputs
let required = amount + txfee
let utxos = await get_utxos(from_address)
let inputs = []
// add the list of utxos as inputs for the transaction one at a time
// until the requirements are met and the sum exceeds the amount we need
// to send plus the transaction fee
while (sum < required) {
if (!utxos.length) {
throw new Error("Insufficient funds to process transaction")
}
const utxo = utxos.pop()
inputs.push(utxo)
sum = sum + utxo.amount * 100000000 // from btc to satoshi
txfee = (181 * inputs.length + 2 * 24 + 10) * byteFee
required = amount + txfee
}
let key = load_from_wif(opts.private_key)
let tx = new bitcoin.TransactionBuilder()
// add transaction inputs
inputs.map (input => tx.addInput(input.txid, input.index) // the txid hash and the vout index of the deposit
// add outputs
tx.addOutput(to_address, amount) // to the destination address
tx.addOutput(from_address, sum - amount - txfee) // send change back to the address except for the transaction fees
// sign each transaction
inputs.map((input, index) => tx.sign(index, key.keyPair))
// build the transaction
const rawtx = tx.build()
const txid = rawtx.getId()
const txhash = rawtx.toHex()
// return the transaction id, the raw hash and the list of inputs used
return {txid, txhash, inputs}
}
async function get_utxos(address) {
// return the list of unspent transaction for the given address
// you can either keep track of them by storing them in the database
// or you can use an external system to retrieve them
// @todo Implementation details
return []
}
function load_from_wif(privateKey) {
return bitcoin.ECPair.fromWIF(privateKey)
}One of the first things we notice in the code above is that in order to create a transaction we need to load a list of unspent transaction outputs that we could use as inputs for the new transactions.
Loading this information from a database can be implemented in the get_utxos() function. How we search for these outputs will be discussed in the next section. For now, just know that they are available for us when we create the transaction.
The second thing to notice here is that in order to send a transaction we need to also calculate a fee that we will pay to the miners in order to include our transaction in the next block. Calculating this the right way is important if you want the transaction to be processed quickly.
For the end user it's important how much that fee represents in bitcoins. For the miner the important part is how much is the price in satoshi per byte, therefore we calculate the total fee we will have to pay for the transaction based on how many inputs we have and in how many deposits we split those inputs.
In this case, we split the transaction in two outputs:
to_address`;from_address`.Further checks could be added here in order to ensure that we don't pay too much. They are based on the current transaction fee per byte taken from the current state of the network and the math used to calculate this transaction fee can be found online here or here.
Next up, we calculate which UTXOs we need to include as inputs in the transaction to cover the amount we need to send + fees. Then we add them all as inputs, we add our outputs and then we sign each input used with the KeyPair associated with the from_address.
The only step left after that is building the transaction and returning the results which we will use as input for the sent signed transaction discussed earlier.
The last part we need to tackle is how we search for and save this list of unspent transactions outputs that we used to create a raw transaction.
In order to do this, we first need to look at each block in the chain and get the list of transactions from it. This can be easily done using the client.getBlockHash(block_number) and the client.getBlock(hash, 2) methods of the bitcoin-core package.
The data returned is already represented as a JSON object within the list of transactions, along with inputs and outputs for each object.
const client = require("./connection")
// get all data of the block
async function process_block(height) {
const hash = await client.getBlockHash(height)
const block = await client.getBlock(hash, 2)
return await process_transactions(block.tx)
}
// process all transactions in the block
async function process_transactions(txs) {
for (let i in txs) {
for (let index in txs[i].vouts) {
await process_tx_vout(txs[i], txs[i].vouts[index])
}
}
}
// process vout
async function process_tx_vout(tx, vout) {
const address = vout.scriptPubKey.addresses[0]
// check if the address is of any interest to us (generated by us)
if (!(await is_deposit_addr(address))) {
return;
}
// if it is watched then we save the information of the vout in the database
await save_utxo(tx.txid, vout.n, address, vout.value, vout)
}
// @todo save the data to the database
async function save_utxo(txid, index, address, amount, vout) {
// save the vout into the database
// ...
}
The code above goes through every transaction in the block and in case it finds a transaction that affects the balance of one of our deposit addreses, it saves that unspent trasaction output into the database.
We can call the process_block(height) function for every block and sync our database with the blockchain. This could be done for every address on the chain if we were to build a blockchain explorer or only for the addresses that we are interested in.
Using a very simple approach and a good understanding of how each blockchain implementation works, you can set the building foundation for any blockchain application you found or used online.
The code in this article explains how to build a bitcoin wallet from the ground up without going into some implementation details like saving or loading data from the database. More information of the dependencies we use in our components can be found in the first article of the series: Building a crypto trading platform: Ethereum Wallet
Using a similar approach, you can build similar wallets for all major bitcoin-based altcoins like NEO, Litecoin, Dash, etc.