Yowsup: Encoding Error

Created on 14 Jun 2016  ·  10Comments  ·  Source: tgalal/yowsup

Trying to reistablish connection with WhatsApp contacts now that whatsapp-purple is abandoned, I tried yowsup once more. Unfortunately, it crashes shortly after logon. Sometimes, a few messages are actually received prior to the UnicodeEncodeError. Revision is todays pip version installed by pip install yowsup2 --upgrade.

$ yowsup-cli demos -l REDACTED -y
yowsup-cli  v2.0.15
yowsup      v2.5.0

Copyright (c) 2012-2016 Tarek Galal
http://www.openwhatsapp.org

This software is provided free of charge. Copying and redistribution is
encouraged.

If you appreciate this software and you would like to support future
development please consider donating:
http://openwhatsapp.org/yowsup/donate


Yowsup Cli client
==================
Type /help for available commands

[offline]:/L
Auth: Logged in!
[connected]:WARNING:axolotl.sessionbuilder:We've already setup a session for this V3 message, letting bundled message fall through...
WARNING:yowsup.layers.axolotl.layer_receive:Received a message that we've previously decrypted, goint to send the delivery receipt myself
WARNING:axolotl.sessionbuilder:We've already setup a session for this V3 message, letting bundled message fall through...
Traceback (most recent call last):
  File "/usr/local/bin/yowsup-cli", line 368, in <module>
    if not parser.process():
  File "/usr/local/bin/yowsup-cli", line 268, in process
    self.startCmdline()
  File "/usr/local/bin/yowsup-cli", line 297, in startCmdline
    stack.start()
  File "/usr/local/lib/python2.7/dist-packages/yowsup/demos/cli/stack.py", line 26, in start
    self.stack.loop(timeout = 0.5, discrete = 0.5)
  File "/usr/local/lib/python2.7/dist-packages/yowsup/stacks/yowstack.py", line 188, in loop
    asyncore.loop(*args, **kwargs)
  File "/usr/lib/python2.7/asyncore.py", line 216, in loop
    poll_fun(timeout, map)
  File "/usr/lib/python2.7/asyncore.py", line 156, in poll
    read(obj)
  File "/usr/lib/python2.7/asyncore.py", line 87, in read
    obj.handle_error()
  File "/usr/lib/python2.7/asyncore.py", line 83, in read
    obj.handle_read_event()
  File "/usr/lib/python2.7/asyncore.py", line 449, in handle_read_event
    self.handle_read()
  File "/usr/local/lib/python2.7/dist-packages/yowsup/layers/network/layer.py", line 102, in handle_read
    self.receive(data)
  File "/usr/local/lib/python2.7/dist-packages/yowsup/layers/network/layer.py", line 110, in receive
    self.toUpper(data)
  File "/usr/local/lib/python2.7/dist-packages/yowsup/layers/__init__.py", line 76, in toUpper
    self.__upper.receive(data)
  File "/usr/local/lib/python2.7/dist-packages/yowsup/layers/stanzaregulator/layer.py", line 29, in receive
    self.processReceived()
  File "/usr/local/lib/python2.7/dist-packages/yowsup/layers/stanzaregulator/layer.py", line 52, in processReceived
    self.processReceived()
  File "/usr/local/lib/python2.7/dist-packages/yowsup/layers/stanzaregulator/layer.py", line 49, in processReceived
    self.toUpper(oneMessageData)
  File "/usr/local/lib/python2.7/dist-packages/yowsup/layers/__init__.py", line 76, in toUpper
    self.__upper.receive(data)
  File "/usr/local/lib/python2.7/dist-packages/yowsup/layers/auth/layer_crypt.py", line 65, in receive
    self.toUpper(payload)
  File "/usr/local/lib/python2.7/dist-packages/yowsup/layers/__init__.py", line 76, in toUpper
    self.__upper.receive(data)
  File "/usr/local/lib/python2.7/dist-packages/yowsup/layers/coder/layer.py", line 35, in receive
    self.toUpper(node)
  File "/usr/local/lib/python2.7/dist-packages/yowsup/layers/__init__.py", line 76, in toUpper
    self.__upper.receive(data)
  File "/usr/local/lib/python2.7/dist-packages/yowsup/layers/logger/layer.py", line 14, in receive
    self.toUpper(data)
  File "/usr/local/lib/python2.7/dist-packages/yowsup/layers/__init__.py", line 76, in toUpper
    self.__upper.receive(data)
  File "/usr/local/lib/python2.7/dist-packages/yowsup/layers/axolotl/layer_control.py", line 44, in receive
    self.toUpper(protocolTreeNode)
  File "/usr/local/lib/python2.7/dist-packages/yowsup/layers/__init__.py", line 76, in toUpper
    self.__upper.receive(data)
  File "/usr/local/lib/python2.7/dist-packages/yowsup/layers/__init__.py", line 189, in receive
    s.receive(data)
  File "/usr/local/lib/python2.7/dist-packages/yowsup/layers/axolotl/layer_receive.py", line 41, in receive
    self.onMessage(protocolTreeNode)
  File "/usr/local/lib/python2.7/dist-packages/yowsup/layers/axolotl/layer_receive.py", line 74, in onMessage
    self.handleEncMessage(protocolTreeNode)
  File "/usr/local/lib/python2.7/dist-packages/yowsup/layers/axolotl/layer_receive.py", line 86, in handleEncMessage
    self.handlePreKeyWhisperMessage(node)
  File "/usr/local/lib/python2.7/dist-packages/yowsup/layers/axolotl/layer_receive.py", line 129, in handlePreKeyWhisperMessage
    self.parseAndHandleMessageProto(pkMessageProtocolEntity, plaintext[:-padding])
  File "/usr/local/lib/python2.7/dist-packages/yowsup/layers/axolotl/layer_receive.py", line 188, in parseAndHandleMessageProto
    self.handleConversationMessage(node, m.conversation)
  File "/usr/local/lib/python2.7/dist-packages/yowsup/layers/axolotl/layer_receive.py", line 217, in handleConversationMessage
    self.toUpper(messageNode)
  File "/usr/local/lib/python2.7/dist-packages/yowsup/layers/__init__.py", line 76, in toUpper
    self.__upper.receive(data)
  File "/usr/local/lib/python2.7/dist-packages/yowsup/layers/__init__.py", line 189, in receive
    s.receive(data)
  File "/usr/local/lib/python2.7/dist-packages/yowsup/layers/__init__.py", line 125, in receive
    recv(node)
  File "/usr/local/lib/python2.7/dist-packages/yowsup/layers/protocol_messages/layer.py", line 21, in recvMessageStanza
    self.toUpper(entity) 
  File "/usr/local/lib/python2.7/dist-packages/yowsup/layers/__init__.py", line 76, in toUpper
    self.__upper.receive(data)
  File "/usr/local/lib/python2.7/dist-packages/yowsup/layers/interface/interface.py", line 80, in receive
    self.entity_callbacks[entityType](entity)
  File "/usr/local/lib/python2.7/dist-packages/yowsup/demos/cli/layer.py", line 512, in onMessage
    MESSAGE_ID = message.getId()
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-1: ordinal not in range(128)

Most helpful comment

@hoehermann
I have a solution for you.

change line 511 of yowsup/demos/cli/layer.py file by:

MESSAGE = messageOut.encode('latin-1').decode() if sys.version_info >= (3, 0) else messageOut.encode('utf-8'),

if you want to see the nicknames or aliases in the "cli". add here the nickname and change this line by:

FROM = self.jidToAlias(sender),

All 10 comments

this seems to happen for voice messages. It tries to decode it as ascii I think

I doubt it's just for voice messages. Here is another one:

$ yowsup-cli demos -l REDACTED -y
yowsup-cli  v2.0.15
yowsup      v2.5.0

Copyright (c) 2012-2016 Tarek Galal
http://www.openwhatsapp.org

This software is provided free of charge. Copying and redistribution is
encouraged.

If you appreciate this software and you would like to support future
development please consider donating:
http://openwhatsapp.org/yowsup/donate


Yowsup Cli client
==================
Type /help for available commands

[offline]:/L
Auth: Logged in!
[connected]:WARNING:axolotl.sessionbuilder:We've already setup a session for this V3 message, letting bundled message fall through...
WARNING:yowsup.layers.axolotl.layer_receive:Received a message that we've previously decrypted, goint to send the delivery receipt myself
WARNING:axolotl.sessionbuilder:We've already setup a session for this V3 message, letting bundled message fall through...
[[email protected](03-06-2016 19:57)]:[B11319BA20EF7EF5251B9B0525E642]   Flunky find ich auch gut
Message B11319BA20EF7EF5251B9B0525E642: Sent delivered receipt and Read
[connected]:
WARNING:axolotl.sessionbuilder:We've already setup a session for this V3 message, letting bundled message fall through...

[[email protected](03-06-2016 19:57)]:[C1DDCE04FBE553BFDC31FEC0F1D2DE]   kein Problem
Message C1DDCE04FBE553BFDC31FEC0F1D2DE: Sent delivered receipt and Read
[connected]:
WARNING:axolotl.sessionbuilder:We've already setup a session for this V3 message, letting bundled message fall through...

[[email protected](03-06-2016 19:57)]:[A7A96B879A8FE32B981DAB264F514E]   Top
Message A7A96B879A8FE32B981DAB264F514E: Sent delivered receipt and Read
[connected]:
WARNING:axolotl.sessionbuilder:We've already setup a session for this V3 message, letting bundled message fall through...

[[email protected](03-06-2016 19:57)]:[0F69D97DCBED20090BF0069CFC8E0B]   Wir maxbe und uns auf den Weg zum Fischers Park
Message 0F69D97DCBED20090BF0069CFC8E0B: Sent delivered receipt and Read
[connected]:
WARNING:axolotl.sessionbuilder:We've already setup a session for this V3 message, letting bundled message fall through...

[[email protected](03-06-2016 19:57)]:[0AE69C5EF9D3192A4455D657233693]   Geschmacklich aber himmelweiter Untefschied
Message 0AE69C5EF9D3192A4455D657233693: Sent delivered receipt and Read
[connected]:
WARNING:axolotl.sessionbuilder:We've already setup a session for this V3 message, letting bundled message fall through...
Traceback (most recent call last):
  File "/usr/local/bin/yowsup-cli", line 368, in <module>
    if not parser.process():
  File "/usr/local/bin/yowsup-cli", line 268, in process
    self.startCmdline()
  File "/usr/local/bin/yowsup-cli", line 297, in startCmdline
    stack.start()
  File "/usr/local/lib/python2.7/dist-packages/yowsup/demos/cli/stack.py", line 26, in start
    self.stack.loop(timeout = 0.5, discrete = 0.5)
  File "/usr/local/lib/python2.7/dist-packages/yowsup/stacks/yowstack.py", line 188, in loop
    asyncore.loop(*args, **kwargs)
  File "/usr/lib/python2.7/asyncore.py", line 216, in loop
    poll_fun(timeout, map)
  File "/usr/lib/python2.7/asyncore.py", line 156, in poll
    read(obj)
  File "/usr/lib/python2.7/asyncore.py", line 87, in read
    obj.handle_error()
  File "/usr/lib/python2.7/asyncore.py", line 83, in read
    obj.handle_read_event()
  File "/usr/lib/python2.7/asyncore.py", line 449, in handle_read_event
    self.handle_read()
  File "/usr/local/lib/python2.7/dist-packages/yowsup/layers/network/layer.py", line 102, in handle_read
    self.receive(data)
  File "/usr/local/lib/python2.7/dist-packages/yowsup/layers/network/layer.py", line 110, in receive
    self.toUpper(data)
  File "/usr/local/lib/python2.7/dist-packages/yowsup/layers/__init__.py", line 76, in toUpper
    self.__upper.receive(data)
  File "/usr/local/lib/python2.7/dist-packages/yowsup/layers/stanzaregulator/layer.py", line 29, in receive
    self.processReceived()
  File "/usr/local/lib/python2.7/dist-packages/yowsup/layers/stanzaregulator/layer.py", line 52, in processReceived
    self.processReceived()
  File "/usr/local/lib/python2.7/dist-packages/yowsup/layers/stanzaregulator/layer.py", line 49, in processReceived
    self.toUpper(oneMessageData)
  File "/usr/local/lib/python2.7/dist-packages/yowsup/layers/__init__.py", line 76, in toUpper
    self.__upper.receive(data)
  File "/usr/local/lib/python2.7/dist-packages/yowsup/layers/auth/layer_crypt.py", line 65, in receive
    self.toUpper(payload)
  File "/usr/local/lib/python2.7/dist-packages/yowsup/layers/__init__.py", line 76, in toUpper
    self.__upper.receive(data)
  File "/usr/local/lib/python2.7/dist-packages/yowsup/layers/coder/layer.py", line 35, in receive
    self.toUpper(node)
  File "/usr/local/lib/python2.7/dist-packages/yowsup/layers/__init__.py", line 76, in toUpper
    self.__upper.receive(data)
  File "/usr/local/lib/python2.7/dist-packages/yowsup/layers/logger/layer.py", line 14, in receive
    self.toUpper(data)
  File "/usr/local/lib/python2.7/dist-packages/yowsup/layers/__init__.py", line 76, in toUpper
    self.__upper.receive(data)
  File "/usr/local/lib/python2.7/dist-packages/yowsup/layers/axolotl/layer_control.py", line 44, in receive
    self.toUpper(protocolTreeNode)
  File "/usr/local/lib/python2.7/dist-packages/yowsup/layers/__init__.py", line 76, in toUpper
    self.__upper.receive(data)
  File "/usr/local/lib/python2.7/dist-packages/yowsup/layers/__init__.py", line 189, in receive
    s.receive(data)
  File "/usr/local/lib/python2.7/dist-packages/yowsup/layers/axolotl/layer_receive.py", line 41, in receive
    self.onMessage(protocolTreeNode)
  File "/usr/local/lib/python2.7/dist-packages/yowsup/layers/axolotl/layer_receive.py", line 74, in onMessage
    self.handleEncMessage(protocolTreeNode)
  File "/usr/local/lib/python2.7/dist-packages/yowsup/layers/axolotl/layer_receive.py", line 86, in handleEncMessage
    self.handlePreKeyWhisperMessage(node)
  File "/usr/local/lib/python2.7/dist-packages/yowsup/layers/axolotl/layer_receive.py", line 129, in handlePreKeyWhisperMessage
    self.parseAndHandleMessageProto(pkMessageProtocolEntity, plaintext[:-padding])
  File "/usr/local/lib/python2.7/dist-packages/yowsup/layers/axolotl/layer_receive.py", line 188, in parseAndHandleMessageProto
    self.handleConversationMessage(node, m.conversation)
  File "/usr/local/lib/python2.7/dist-packages/yowsup/layers/axolotl/layer_receive.py", line 217, in handleConversationMessage
    self.toUpper(messageNode)
  File "/usr/local/lib/python2.7/dist-packages/yowsup/layers/__init__.py", line 76, in toUpper
    self.__upper.receive(data)
  File "/usr/local/lib/python2.7/dist-packages/yowsup/layers/__init__.py", line 189, in receive
    s.receive(data)
  File "/usr/local/lib/python2.7/dist-packages/yowsup/layers/__init__.py", line 125, in receive
    recv(node)
  File "/usr/local/lib/python2.7/dist-packages/yowsup/layers/protocol_messages/layer.py", line 21, in recvMessageStanza
    self.toUpper(entity) 
  File "/usr/local/lib/python2.7/dist-packages/yowsup/layers/__init__.py", line 76, in toUpper
    self.__upper.receive(data)
  File "/usr/local/lib/python2.7/dist-packages/yowsup/layers/interface/interface.py", line 80, in receive
    self.entity_callbacks[entityType](entity)
  File "/usr/local/lib/python2.7/dist-packages/yowsup/demos/cli/layer.py", line 512, in onMessage
    MESSAGE_ID = message.getId()
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe4' in position 58: ordinal not in range(128)

As you can see, messages are received. But things fall apart as soon someone sends me a message containing a German umlaut (ä in this example). I assume, the error occurrs when writing messages to the terminal. Python might try to encode with ASCII although my terminal uses UTF-8.

@hoehermann are you sure if encrypted media in groups is supported by master code?

@jlguardi No, I am unaware aof the support of encrypted media in group chats. As you state in https://github.com/jlguardi/yowsup/issues/87#issuecomment-224777859 , there is none.
By looking at the trace, I cannot relate this problem to encrypted media in a group chat. If this is the case, I still would like yowsup to ignore the message and print a warning rather than crashing alltogether. Problem persists in f510d44 .

I am currently investigating this issue. For future reference, receiving an image in an encrypted group chat triggers an exception being handled and printed as

DUMP:
<binary data>
<list of char>
<list of int>
Traceback (most recent call last):
  File "./yowsup-cli", line 368, in <module>
    if not parser.process():
  File "./yowsup-cli", line 268, in process
    self.startCmdline()
  File "./yowsup-cli", line 297, in startCmdline
    stack.start()
  File "/media/data/Eigene/Programmieren/Kompilieren/yowsup_tgalal/yowsup/demos/cli/stack.py", line 26, in start
    self.stack.loop(timeout = 0.5, discrete = 0.5)
  File "/media/data/Eigene/Programmieren/Kompilieren/yowsup_tgalal/yowsup/stacks/yowstack.py", line 188, in loop
    asyncore.loop(*args, **kwargs)
  File "/usr/lib/python2.7/asyncore.py", line 216, in loop
    poll_fun(timeout, map)
  File "/usr/lib/python2.7/asyncore.py", line 156, in poll
    read(obj)
  File "/usr/lib/python2.7/asyncore.py", line 87, in read
    obj.handle_error()
  File "/usr/lib/python2.7/asyncore.py", line 83, in read
    obj.handle_read_event()
  File "/usr/lib/python2.7/asyncore.py", line 449, in handle_read_event
    self.handle_read()
  File "/media/data/Eigene/Programmieren/Kompilieren/yowsup_tgalal/yowsup/layers/network/layer.py", line 102, in handle_read
    self.receive(data)
  File "/media/data/Eigene/Programmieren/Kompilieren/yowsup_tgalal/yowsup/layers/network/layer.py", line 110, in receive
    self.toUpper(data)
  File "/media/data/Eigene/Programmieren/Kompilieren/yowsup_tgalal/yowsup/layers/__init__.py", line 76, in toUpper
    self.__upper.receive(data)
  File "/media/data/Eigene/Programmieren/Kompilieren/yowsup_tgalal/yowsup/layers/stanzaregulator/layer.py", line 29, in receive
    self.processReceived()
  File "/media/data/Eigene/Programmieren/Kompilieren/yowsup_tgalal/yowsup/layers/stanzaregulator/layer.py", line 49, in processReceived
    self.toUpper(oneMessageData)
  File "/media/data/Eigene/Programmieren/Kompilieren/yowsup_tgalal/yowsup/layers/__init__.py", line 76, in toUpper
    self.__upper.receive(data)
  File "/media/data/Eigene/Programmieren/Kompilieren/yowsup_tgalal/yowsup/layers/auth/layer_crypt.py", line 65, in receive
    self.toUpper(payload)
  File "/media/data/Eigene/Programmieren/Kompilieren/yowsup_tgalal/yowsup/layers/__init__.py", line 76, in toUpper
    self.__upper.receive(data)
  File "/media/data/Eigene/Programmieren/Kompilieren/yowsup_tgalal/yowsup/layers/coder/layer.py", line 35, in receive
    self.toUpper(node)
  File "/media/data/Eigene/Programmieren/Kompilieren/yowsup_tgalal/yowsup/layers/__init__.py", line 76, in toUpper
    self.__upper.receive(data)
  File "/media/data/Eigene/Programmieren/Kompilieren/yowsup_tgalal/yowsup/layers/logger/layer.py", line 14, in receive
    self.toUpper(data)
  File "/media/data/Eigene/Programmieren/Kompilieren/yowsup_tgalal/yowsup/layers/__init__.py", line 76, in toUpper
    self.__upper.receive(data)
  File "/media/data/Eigene/Programmieren/Kompilieren/yowsup_tgalal/yowsup/layers/axolotl/layer_control.py", line 44, in receive
    self.toUpper(protocolTreeNode)
  File "/media/data/Eigene/Programmieren/Kompilieren/yowsup_tgalal/yowsup/layers/__init__.py", line 76, in toUpper
    self.__upper.receive(data)
  File "/media/data/Eigene/Programmieren/Kompilieren/yowsup_tgalal/yowsup/layers/__init__.py", line 189, in receive
    s.receive(data)
  File "/media/data/Eigene/Programmieren/Kompilieren/yowsup_tgalal/yowsup/layers/axolotl/layer_receive.py", line 41, in receive
    self.onMessage(protocolTreeNode)
  File "/media/data/Eigene/Programmieren/Kompilieren/yowsup_tgalal/yowsup/layers/axolotl/layer_receive.py", line 74, in onMessage
    self.handleEncMessage(protocolTreeNode)
  File "/media/data/Eigene/Programmieren/Kompilieren/yowsup_tgalal/yowsup/layers/axolotl/layer_receive.py", line 86, in handleEncMessage
    self.handlePreKeyWhisperMessage(node)
  File "/media/data/Eigene/Programmieren/Kompilieren/yowsup_tgalal/yowsup/layers/axolotl/layer_receive.py", line 129, in handlePreKeyWhisperMessage
    self.parseAndHandleMessageProto(pkMessageProtocolEntity, plaintext[:-padding])
  File "/media/data/Eigene/Programmieren/Kompilieren/yowsup_tgalal/yowsup/layers/axolotl/layer_receive.py", line 171, in parseAndHandleMessageProto
    m.ParseFromString(serializedData)
  File "/usr/local/lib/python2.7/dist-packages/google/protobuf/message.py", line 185, in ParseFromString
    self.MergeFromString(serialized)
  File "/usr/local/lib/python2.7/dist-packages/google/protobuf/internal/python_message.py", line 1080, in MergeFromString
    if self._InternalParse(serialized, 0, length) != length:
  File "/usr/local/lib/python2.7/dist-packages/google/protobuf/internal/python_message.py", line 1116, in InternalParse
    pos = field_decoder(buffer, new_pos, end, self, field_dict)
  File "/usr/local/lib/python2.7/dist-packages/google/protobuf/internal/decoder.py", line 633, in DecodeField
    if value._InternalParse(buffer, pos, new_pos) != new_pos:
  File "/usr/local/lib/python2.7/dist-packages/google/protobuf/internal/python_message.py", line 1116, in InternalParse
    pos = field_decoder(buffer, new_pos, end, self, field_dict)
  File "/usr/local/lib/python2.7/dist-packages/google/protobuf/internal/decoder.py", line 502, in DecodeField
    field_dict[key] = _ConvertToUnicode(buffer[pos:new_pos])
  File "/usr/local/lib/python2.7/dist-packages/google/protobuf/internal/decoder.py", line 469, in _ConvertToUnicode
    return local_unicode(byte_str, 'utf-8')
UnicodeDecodeError: 'utf8' codec can't decode byte 0xff in position 0: 'utf8' codec can't decode byte 0xff in position 0: invalid start byte in field: com.whatsapp.proto.UrlMessage.jpeg_thumbnail

This differs from my examples above.

I am now on f510d44 using python 2.7. I helped myself by editing yowsup/demos/cli/layer.py, surrounding the flaky call to self.__class__.MESSAGE_FORMAT.format with a try-except block. A less than ideal solution, but at least I can follow conversations now. Exceptions occur when receiving emoji in messeges like

Hot Town, Summer in da City 🍻🍻☀☀☀☀

I really needed those. :[

@hoehermann
I have a solution for you.

change line 511 of yowsup/demos/cli/layer.py file by:

MESSAGE = messageOut.encode('latin-1').decode() if sys.version_info >= (3, 0) else messageOut.encode('utf-8'),

if you want to see the nicknames or aliases in the "cli". add here the nickname and change this line by:

FROM = self.jidToAlias(sender),

I did not try the jidToAlias hint, but your proposal for layer.py is looking very good. I want to test it some more, but it looks stable.

god bless your soul! adding the .encode worked for me. been struggling the whole afternoon

MESSAGE = messageOut.encode('latin-1').decode() if sys.version_info >= (3, 0) else messageOut.encode('utf-8')

For me it worked changing line 511 as follows:
MESSAGE = messageOut.encode('utf-8').decode() if sys.version_info >= (3, 0) else messageOut
But it still crash when receiving files (image, video, PDF, etc)
I'm using Python version 3.6

Was this page helpful?
0 / 5 - 0 ratings

Related issues

thatskriptkid picture thatskriptkid  ·  3Comments

darode picture darode  ·  4Comments

alexvong1995 picture alexvong1995  ·  4Comments

bzero picture bzero  ·  3Comments

MauricioAHM picture MauricioAHM  ·  5Comments