Pwndbg: utf8 decoding error

[carstein]

Description

Context is not correctly displayed due to utf8 decoding error.

Traceback (most recent call last):
File "/home/carstein/Tools/pwndbg/pwndbg/commands/__init__.py", line 109, in __call__
return self.function(args, *kwargs)
File "/home/carstein/Tools/pwndbg/pwndbg/commands/__init__.py", line 200, in _OnlyWhenRunning
return function(a, *kw)
File "/home/carstein/Tools/pwndbg/pwndbg/commands/context.py", line 86, in context
result.extend(func())
File "/home/carstein/Tools/pwndbg/pwndbg/commands/context.py", line 98, in context_regs
return [pwndbg.ui.banner("registers")] + get_regs()
File "/home/carstein/Tools/pwndbg/pwndbg/ui.py", line 52, in banner
banner = ljust_colored(banner, width, str(config.banner_separator))
File "/home/carstein/Tools/pwndbg/pwndbg/color/__init__.py", line 77, in ljust_colored
return x + ((remaining // len(char) + 1) * char)[:remaining]
UnicodeDecodeError: 'utf8' codec can't decode byte 0xe2 in position 105: unexpected end of data

Steps to reproduce

1. Load binary into gdb
2. break main
3. run

My setup

pwndbg version:
Gdb: GNU gdb (GDB) 8.1
Python: 2.7.13 (default, Nov 24 2017, 17:33:09) [GCC 6.3.0 20170516]
Pwndbg: 1.0.0 build: 869e832
Capstone: 4.0.1024
Unicorn: 1.0.1

[anthraxx]

Python2 is a bitch, you will need to set proper utf8 LC types, there is a hint somewhere what needs to be done

[disconnect3d]

@anthraxx you probably meant this:
https://github.com/pwndbg/pwndbg/blob/6a1fdb282ed34fd7c6579b16ea111a8a11de226d/gdbinit.py#L23-L34

@carstein can u check if that's the issue?

[carstein]

Hey, if you are talking about launching it via:
$ LC_ALL=en_US.UTF-8 PYTHONIOENCODING=UTF-8 /opt/bin/gdb
tried, does not help

[anthraxx]

Are you sure it's enabled in locals and locals were regenerated afterwards? Try exporting the env vars instead, before calling gdb.
Also try setting LC_CTYPE via export.
Try playing around in your environment if Python2 can't use utf8 then it's én environmental issue.

[carstein]

Checked locales and reganeration before
python is able to use utf8 (tested)

>>> print '\xc4\x85'
ą

I've started looking into the code more closely and it is interesting in a sense, that pwndbg is able to construct half of the banner for registers
but fails on the second part (after [ REGISTERS]) in colored/__init__.py
52: banner = ljust_colored(banner, width, str(config.banner_separator))

So, rjust_colored() worked fine, but for some reason ljust_colored() failed. That is bizzare.

[anthraxx]

Nice info, not so much if my theory is correct. Let me test some things and great a PR, if my theory fails we are back at investigating 😄

[disconnect3d]

@anthraxx that's an issue on our side, investigating too :P

[anthraxx]

It's a possibly half consumed sequence of bytes, already on it 😉

[anthraxx]

So Python2 configuration values do not guarantee the type, therefor returning a raw string with Unicode sequence will result in being splitted wrongly in between the sequence instead of logically per character like python3 or Python2 with correct decoded unicode type.
I will test my work around and submit a temporary PR until the Parameter configuration type thingies in #404 land in the tree, they should address such issues and make a workaround obsolete.