Cardano-node: [FR] - support password protected byron keys created with cardano-cli.

[steakandbake]

Internal/Exernal
External

Summary
Unable to transfer funds from a password protected byron address created with cardano-cli.

Steps to reproduce
Steps to reproduce the behavior:

1) Installed cardano-cli from branch release/1.15.x

2) Created skey with password using this command
cardano-cli keygen --byron-formats --secret byron.skey

3) Built public key address with
cardano-cli signing-key-address --secret byron.skey --byron-formats --mainnet

4) Sent mainnet funds to generated public key

5) Synhronized a mainnet node. Confirmed on mainnet tip with
cardano-cli get-tip --mainnet

6) Built transaction
cardano-cli issue-utxo-expenditure --mainnet --byron-formats --tx outtx --wallet-key byron.skey --txin '("211f670a2bbc002fd69da9a408b5641aaa43c32934c5a3aafae23d755ec5a0d1",12)' --txout '("Ae2tdPwwrwYwgzWqEhEmbMdiyxZhcvGvykKYY16FWTLH5mEZkU7z9DaYaxE",999800000)'

7) Submit-tx
cardano-cli submit-tx --mainnet --tx outtx

Expected behavior
Password is prompted for and then transaction is broadcasted and confirmed.

System info (please complete the following information):

• OS: Ubuntu
• Version 20.04
• Node version 1.17

Additional context
Attempting to convert the byron address to shelley address also does not work.

cardano-cli shelley key convert-byron-key --byron-payment-key-type --byron-signing-key-file byron.skey --out-file converted.skey
cardano-cli to-verification --byron-formats --secret byron.skey --to converted.vkey
cardano-cli shelley address build --payment-verification-key-file converted.vkey --mainnet
"Error in $: Failed reading: not a valid json value" ## to-verification does not output a json, but rather a base64

In addition, the --help documentation is incorrect on the issue-utxo-expenditure for --txout parameter. It requires '
cardano-cli issue-utxo-expenditure --txout '("ADDR", LOVELACE)'

Thanks for your assistance.

[intricate]

Attempting to convert the byron address to shelley address also does not work.

Note that you should use the cardano-cli shelley key verification-key command to derive a verification key from a signing key in the new format.

[steakandbake]

Thanks you intricate.
I was able to build the same mainnet address even after converting to shelley.

cardano-cli shelley key verification-key --signing-key-file converted.skey --verification-key-file converted.vkey
cardano-cli shelley address build --payment-verification-key-file converted.vkey --mainnet > converted.addr

Will attempt spending from this converted Shelley key pair on MC4 1.18 too.

[steakandbake]

Everything seems to work with this newly converted to shelley address until cardano-cli shelley transaction submit

Error while submitting tx: ApplyTxError [LedgerFailure (UtxowFailure (UtxoFailure (ExpiredUTxO (SlotNo {unSlotNo = 1000}) (SlotNo {unSlotNo = 131121})))),LedgerFailure (UtxowFailure (MissingVKeyWitnessesUTXOW (WitHashes (fromList [KeyHash "69161c4b2dfcdea38e98c36db214e59e7547abf0861e3541cfe6c1ca"])))),LedgerFailure (UtxowFailure (InvalidWitnessesUTXOW [VKey (VerKeyEd25519DSIGN (PublicKey "\196\214\173\230\197\230\134\129\143x\238\243\144z)\STXv@8!\252]>\196\227\198\206\SYN\\G\198\ESC"))]))]

[intricate]

@steakandbake: You've outlined an ExpiredUTxO error. It seems that you've attempted to submit a transaction with a TTL of 1000, but the current slot is 131121.

When constructing a transaction, you need to specify a TTL in the future. Please see the documentation related to the TTL.

[steakandbake]

@intricate My mistake, offset of the tip + 1000 did not work.

Constructed the txn again and on submit, I'm getting

Error while submitting tx: ApplyTxError [LedgerFailure (UtxowFailure (MissingVKeyWitnessesUTXOW (WitHashes (fromList [KeyHash "69161c4b2dfcdea38e98c36db214e59e7547abf0861e3541cfe6c1ca"])))),LedgerFailure (UtxowFailure (InvalidWitnessesUTXOW [VKey (VerKeyEd25519DSIGN (PublicKey "\196\214\173\230\197\230\134\129\143x\238\243\144z)\STXv@8!\252]>\196\227\198\206\SYN\\G\198\ESC"))]))]

Thanks

[intricate]

The error is no longer related to the TTL; it says MissingVKeyWitnessesUTXOW and InvalidWitnessesUTXOW. Are you sure that you're providing the required witnesses when signing your transaction?

[steakandbake]

The error is no longer related to the TTL; it says MissingVKeyWitnessesUTXOW and InvalidWitnessesUTXOW. Are you sure that you're providing the required witnesses when signing your transaction?

I think the required witnesses is correct. In this transaction, I attempt to register my staking address.

cardano-cli shelley stake-address registration-certificate \
    --staking-verification-key-file stake.vkey \
    --out-file stake.cert

cardano-cli shelley transaction build-raw \
      --tx-in 211f670a2bbc002fd69da9a408b5641aaa43c32934c5a3aafae23d755ec5a0d1#12 \
      --tx-out $(cat converted.addr)+0 \
      --ttl 173960 \
      --fee 0 \
      --out-file txn.tmp \
      --certificate stake.cert

cardano-cli shelley transaction calculate-min-fee \
      --tx-body-file txn.tmp \
      --tx-in-count 1 \
      --tx-out-count 1 \
      --testnet-magic 42 \
      --witness-count 2 \
      --byron-witness-count 0 \
      --protocol-params-file params.json

cardano-cli shelley transaction build-raw \
      --tx-in 211f670a2bbc002fd69da9a408b5641aaa43c32934c5a3aafae23d755ec5a0d1#12 \
      --tx-out $(cat converted.addr)[phone] \
      --ttl 173960 \
      --fee 178437 \
      --certificate-file stake.cert \
      --out-file txn.raw


cardano-cli shelley transaction sign \
      --tx-body-file txn.raw \
      --signing-key-file converted.skey \
      --signing-key-file stake.skey \
      --testnet-magic 42 \
      --out-file tx.signed

cardano-cli shelley transaction submit \
      --tx-file tx.signed \
      --testnet-magic 42 \

Since the original address was a byron format address, I tried building the txn with byron-witness-count too. Same error when submitting.

      --witness-count 1 \
      --byron-witness-count 1 \

Thanks @intricate

[steakandbake]

I believe I'm getting closer.

In the cardano-cli shelley transaction sign step, signed with --mainnet

Error while submitting tx: ApplyTxError [LedgerFailure (UtxowFailure (InvalidWitnessesUTXOW [VKey (VerKeyEd25519DSIGN (PublicKey "\196\214\173\230\197\230\134\129\143x\238\243\144z)\STXv@8!\252]>\196\227\198\206\SYN\\G\198\ESC"))]))]

Error no longer complains for MissingVKeyWitnessesUTXOW

Thanks @SebastienGllmt for the idea

[intricate]

Hmm, the InvalidWitnessesUTXOW error implies that signature verification is failing for some of the provided witnesses. Does this occur on cardano-node and cardano-cli 1.18.0?

[steakandbake]

Yes

[steakandbake]

After importing this formerly byron key, presently converted to a shelley key into CNTools, a famous tool for pool operators to manage pool and wallet operations, the same error occurs regarding InvalidWitnessesUTXOWwhile attempting a simple send transaction. I'm shocked such a core CLI command is exhibiting this issue and that there appears to be no other incidents. Please advise.

[intricate]

Now that I'm taking another look at the Byron-specific code, it seems that we actually lack support for password-protected signing keys in cardano-cli. In fact, I think the only command that supports them is cardano-cli keygen (which is now cardano-cli byron key keygen in more recent revisions of cardano-cli).

Does everything function as expected if you use a non-password-protected signing key?

[steakandbake]

Does everything function as expected for you if you use a non-password-protected signing key?

Yes, non-password works A-OK!

I think you've found it! I suspected it was the password. At no time during the transfer is a password asked for, only at creation with cardano-cli keygen

[intricate]

Apologies, I should have noticed that sooner. I'm checking up on how we're going to deal with these Byron-style password-protected signing keys.

[dcoutts]

We will need to extend cardano-cli shelley key convert-byron-key with a --password flag for reading Byron keys that use passwords.

[steakandbake]

Success. Thank you VERY much @Jimbo4350 @intricate @dcoutts