Use Bitcoin regtest

[oblak-be]

Dear maintainers/contributors

Bitcoinlib version 0.6.13

Goal:

Use Bitcoin regtest for platform code testing having the availability to provide test users with credit and run our platform tests.

Problem:

When opening an existing wallet, the wallet.scan() function fails with error:
bitcoinlib.services.services.ServiceError: No successful response from any serviceprovider: ['bitcoind']

Steps taken:

• add to providers.json:

"bitcoind": {
    "provider": "bitcoind",
    "network": "regtest",
    "client_class": "BitcoindClient()",
    "provider_coin_id": "",
    "url": "http://<user>:<pass>@bitcoin:8332",
    "api_key": "",
    "priority": 10,
    "denominator": 100000000,
    "network_overrides": null
},

• Change prefix_bech32 in network.jon for regtest:

  "regtest":
  {
    "description": "Bitcoin regtest",
    "currency_name": "regtest",
    "currency_name_plural": "bitcoins",
    "currency_symbol": "r฿",
    "currency_code": "rBTC",
    "prefix_address": "00",
    "prefix_address_p2sh": "05",
    "prefix_bech32": "bcrt",
    "prefix_wif": "80",
    "prefixes_wif": [
      ["0488B21E", "xpub", "public",  false, "legacy",      "p2pkh"],
      ["0488B21E", "xpub", "public",  true,  "legacy",      "p2sh"],
      ["0488ADE4", "xprv", "private", false, "legacy",      "p2pkh"],
      ["0488ADE4", "xprv", "private", true,  "legacy",      "p2sh"],
      ["049D7CB2", "ypub", "public",  false, "p2sh-segwit", "p2sh_p2wpkh"],
      ["0295B43F", "Ypub", "public",  true,  "p2sh-segwit", "p2sh_p2wsh"],
      ["049D7878", "yprv", "private", false, "p2sh-segwit", "p2sh_p2wpkh"],
      ["0295B005", "Yprv", "private", true,  "p2sh-segwit", "p2sh_p2wsh"],
      ["04B24746", "zpub", "public",  false, "segwit",      "p2wpkh"],
      ["02AA7ED3", "Zpub", "public",  true,  "segwit",      "p2wsh"],
      ["04B2430C", "zprv", "private", false, "segwit",      "p2wpkh"],
      ["02AA7A99", "Zprv", "private", true,  "segwit",      "p2wsh"]
    ],
    "bip44_cointype": 0,
    "denominator": 0.00000001,
    "dust_amount": 1000,
    "fee_default": null,
    "fee_min": 1000,
    "fee_max": 1000000,
    "priority": 0
  },

• have a functinoing bitcoin-core node running in regtest mode on url from providers.json

• create some wallets using bitcoinlib

• mine some coins on bitcoin node and send some to a bitcoinlib wallet using bitcoin-cli on bitcoin node.

• bitcoinlib commands:

>>> wallet = bitcoinlib.wallets.wallet_create_or_open("test-user")

>>> bitcoinlib.services.bitcoind.BitcoindClient().network
<Network: bitcoin>

>>> wallet.network
<Network: regtest>  

>>> bitcoinlib.services.bitcoind.BitcoindClient().gettransactions(wallet_address)
[
  <Transaction(id=8b7657..., inputs=1, outputs=2, status=confirmed, network=bitcoin)>,   
  <Transaction(id=0b281a...
]

>>> wallet.scan()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/data/.local/lib/python3.8/site-packages/bitcoinlib/wallets.py", line 1849, in scan
    self.transactions_update_confirmations()
  File "/data/.local/lib/python3.8/site-packages/bitcoinlib/wallets.py", line 3021, in transactions_update_confirmations
    srv = Service(network=network, providers=self.providers, cache_uri=self.db_cache_uri)
  File "/data/.local/lib/python3.8/site-packages/bitcoinlib/services/services.py", line 145, in __init__
    self._blockcount = self.blockcount()
  File "/data/.local/lib/python3.8/site-packages/bitcoinlib/services/services.py", line 495, in blockcount
    new_count = self._provider_execute('blockcount')
  File "/data/.local/lib/python3.8/site-packages/bitcoinlib/services/services.py", line 224, in _provider_execute
    raise ServiceError("No successful response from any serviceprovider: %s" % list(self.providers.keys()))
bitcoinlib.services.services.ServiceError: No successful response from any serviceprovider: ['bitcoind']

>>> # set network to bitcoin for test
>>> wallet.network = bitcoinlib.networks.Network("bitcoin")
>>> w.scan()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/data/.local/lib/python3.8/site-packages/bitcoinlib/wallets.py", line 1877, in scan
    if self.scan_key(key):
  File "/data/.local/lib/python3.8/site-packages/bitcoinlib/wallets.py", line 1800, in scan_key
    n_new = self.transactions_update(key_id=key.key_id)
  File "/data/.local/lib/python3.8/site-packages/bitcoinlib/wallets.py", line 3102, in transactions_update
    srv = Service(network=network, providers=self.providers, cache_uri=self.db_cache_uri)
  File "/data/.local/lib/python3.8/site-packages/bitcoinlib/services/services.py", line 145, in __init__
    self._blockcount = self.blockcount()
  File "/data/.local/lib/python3.8/site-packages/bitcoinlib/services/services.py", line 495, in blockcount
    new_count = self._provider_execute('blockcount')
  File "/data/.local/lib/python3.8/site-packages/bitcoinlib/services/services.py", line 224, in _provider_execute
    raise ServiceError("No successful response from any serviceprovider: %s" % list(self.providers.keys()))
bitcoinlib.services.services.ServiceError: No successful response from any serviceprovider: ['bitcoind']
>>> # scan() hangs now, but throws similar error

Maybe I'm doing something wrong or have missed a configuration step?

Any help appreciated and thanks for the great project!

[mccwdev]

It seems related to this: c6efbde
fixed with this pull request: #352

[n1rna]

I'm not sure if the mentioned PR is addressing all the issues mentioned in the issue. I'm also facing the issue with the wrong chain param for the regtest network. The prefix_bech32 value must be bcrt @mccwdev

https://github.com/bitcoin/bitcoin/blob/master/src/kernel/chainparams.cpp#L522

[oblak-be]

After some more testing and digging i found out that

When wallet.scan() is called in our case, BitcoindClient.gettransactions(wallet_address, '', 20) is called and fails

error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/data/.local/lib/python3.8/site-packages/bitcoinlib/services/bitcoind.py", line 223, in gettransactions
    raise ClientError("Address %s not found in bitcoind wallet, use 'importpubkey' or 'importaddress' to add "
bitcoinlib.services.baseclient.ClientError: Address bcrt1qn... not found in bitcoind wallet, use 'importpubkey' or 'importaddress' to add address to wallet.

When changing res['iswatchonly'] = True
wallet.scan() works and the balance is updated.

[n1rna]

My workaround for now is pointing to a custom data directory as follows:

[locations]
# Location of BitcoinLib data, configuration and log files. Relative paths will be based in installation directory
data_dir=./data

In the BCL_CONFIG_FILE and then fixing the networks.json file for my own use case.

P.S. You need to replicate all the required files in the data directory: https://github.com/1200wd/bitcoinlib/tree/master/bitcoinlib/data