Python: when use exec api tail xxx.log error "'utf-8' codec can't decode bytes"

[laixiaohuai]

When I'am exec pod use tail -200f xxx.log.The following error is often reported

Exception in thread Thread-426:
Traceback (most recent call last):
File "/data/pyenv/versions/3.7.0/lib/python3.7/threading.py", line 917, in _bootstrap_inner
self.run()
File "/data/SeaOPS/Base/api/views_file/kubernetes.py", line 591, in run
requests[fileno] = self.chan.read_stdout()
File "/data/pyenv/versions/3.7.0/lib/python3.7/site-packages/kubernetes/stream/ws_client.py", line 122, in read_stdout
return self.read_channel(STDOUT_CHANNEL, timeout=timeout)
File "/data/pyenv/versions/3.7.0/lib/python3.7/site-packages/kubernetes/stream/ws_client.py", line 86, in read_channel
ret = self.peek_channel(channel, timeout)
File "/data/pyenv/versions/3.7.0/lib/python3.7/site-packages/kubernetes/stream/ws_client.py", line 78, in peek_channel
self.update(timeout=timeout)
File "/data/pyenv/versions/3.7.0/lib/python3.7/site-packages/kubernetes/stream/ws_client.py", line 182, in update
data = data.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 4095-4096: unexpected end of data

The utf-8 decode error is currently seen on the source side

Here's some of my code
..........
while self.chan.is_open():
events = self.epoll.poll()
# print("events")
if not self.chan.is_open():
logger.info('container stream connect is closed')
self.websocket.send(bytes("container stream connect is closed", 'utf-8'))
self.websocket.close()
for fileno, event in events:
# print("adsfadsaf:", fileno, event)
if event & select.EPOLLIN:
requests[fileno] = self.chan.read_stdout()
# try:
# requests[fileno] = self.chan.read_stdout()
# except Exception as ee:
# logger.info(str(ee))
self.websocket.send(bytes(requests[fileno], 'utf-8'))
elif event & select.EPOLLHUP:
# print('------------EPOLLHUP-----------')
# 注销对此socket连接的关注
self.epoll.unregister(fileno)
# 关闭socket连
self.websocket.close()
............

Can anyone help me see why that is?Is it a Bug or something else?

[leavest]

I get the same error when execute vim command.
my code sample

from kubernetes import client, stream
self.core_api = client.CoreV1Api()
self.stream = stream.stream(
            self.core_api.connect_get_namespaced_pod_exec,
            name=name,
            namespace=namespace,
            container=container,
            command=command,
            stdin=True,
            stdout=True,
            stderr=True,
            tty=True,
            _preload_content=False
        )

error message:

Exception in thread Thread-180:
Traceback (most recent call last):
  File "/Users/zuili/project/apps/cloud/tools/terminal.py", line 38, in run
    if self.stream.peek_stderr():
  File "/Users/zuili/.pyenv/versions/env_opsoa/lib/python3.5/site-packages/kubernetes/stream/ws_client.py", line 130, in peek_stderr
    return self.peek_channel(STDERR_CHANNEL, timeout=timeout)
  File "/Users/zuili/.pyenv/versions/env_opsoa/lib/python3.5/site-packages/kubernetes/stream/ws_client.py", line 78, in peek_channel
    self.update(timeout=timeout)
  File "/Users/zuili/.pyenv/versions/env_opsoa/lib/python3.5/site-packages/kubernetes/stream/ws_client.py", line 180, in update
    data = data.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xbd in position 30: invalid start byte

I print the data in "kubernetes/stream/ws_client.py", line 180, and the last one raise UnicodeDecodeError

b'\x01' <class 'bytes'>
b'\x01' <class 'bytes'>
b'\x01root@test-f54787bd8-v9n26:/# ' <class 'bytes'>
b'\x01v' <class 'bytes'>
b'\x01i' <class 'bytes'>
b'\x01m' <class 'bytes'>
b'\x01\r\n' <class 'bytes'>
b'\x01\x1b[?1000h\x1b[?1049h\x1b[?1h\x1b=\x1b[2;1H\xbd\x1b[6n\x1b[2;1H  \x1b[1;1H' <class 'bytes'>

version
kubernetes==8.0.0
python==3.5.5
os==10.13.6

[leavest]

你解决问题了吗？我这样修改，我的问题解决了。仅限python3的，python2.7依然报错。

# kubernetes/stream/ws_client.py  line 182
- data = data.decode("utf-8")

+ import chardet
+ encode = chardet.detect(data).get('encoding')
+ data = data.decode(encode or "utf-8")

[laixiaohuai]

你解决问题了吗？我这样修改，我的问题解决了。仅限python3的，python2.7依然报错。

# kubernetes/stream/ws_client.py  line 82
- data = data.decode("utf-8")

+ import chardet
+ encode = chardet.detect(data).get('encoding')
+ data = data.decode(encode or "utf-8")

你是修改源码了是吧

[leavest]

你解决问题了吗？我这样修改，我的问题解决了。仅限python3的，python2.7依然报错。

# kubernetes/stream/ws_client.py  line 82
- data = data.decode("utf-8")

+ import chardet
+ encode = chardet.detect(data).get('encoding')
+ data = data.decode(encode or "utf-8")

你是修改源码了是吧

是的

[laixiaohuai]

你解决问题了吗？我这样修改，我的问题解决了。仅限python3的，python2.7依然报错。

# kubernetes/stream/ws_client.py  line 82
- data = data.decode("utf-8")

+ import chardet
+ encode = chardet.detect(data).get('encoding')
+ data = data.decode(encode or "utf-8")

你是修改源码了是吧

是的

好的，我也试试，测看看结果

[laixiaohuai]

你解决问题了吗？我这样修改，我的问题解决了。仅限python3的，python2.7依然报错。

# kubernetes/stream/ws_client.py  line 82
- data = data.decode("utf-8")

+ import chardet
+ encode = chardet.detect(data).get('encoding')
+ data = data.decode(encode or "utf-8")

你是修改源码了是吧

是的

好的，我也试试，测看看结果

2019-01-16 11:23:02,712 - ERROR - kubernetes.py[line:713] - get - 'NoneType' object has no attribute 'fileno' Exception in thread Thread-430: Traceback (most recent call last): File "/data/pyenv/versions/3.7.0/lib/python3.7/threading.py", line 917, in _bootstrap_inner self.run() File "/data/SeaOPS/Base/api/views_file/kubernetes.py", line 587, in run self.chan.update(timeout=300) File "/data/pyenv/versions/3.7.0/lib/python3.7/site-packages/kubernetes/stream/ws_client.py", line 181, in update data = data.decode(encode or "utf-8") UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe5 in position 4096: unexpected end of data
我还会报错，纳闷

[leavest]

我还会报错，纳闷

你使用哪个 websocket 的包。
试一下换个 websocket 的包吧
我用 channels 、Flask-Sockets 都可以，用 dwebsocket 又有另外的问题

[laixiaohuai]

我还会报错，纳闷

你使用哪个 websocket 的包。
试一下换个 websocket 的包吧
我用 channels 、Flask-Sockets 都可以，用 dwebsocket 又有另外的问题

我用的就是dwebsocket

[sadlar]

麻烦请教，django如果使用channels怎么实现web terminal呢

[gvalentinivmw]

I ran into the same issue for the port-forward api. The proposed fix worked for me. Any chance to get it merged?

Giorgio

[aaronjwood]

The above patch also worked on my end with Python 3.7

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.