Mbed-os: Fail to update cmsis pack index.json

[ccli8]

Description

OS:
Windows 10/64-bit

mbed-os:
41ecde9c30 Merge pull request #9371 from 0xc0170/test_travis_master

python
3.7.2

mbed-cli
1.8.3

Following the doc, I run the below command under mbed-os directory to update cmsis pack index.json:

mbed export -i uvision --update-packs

It fails with:

Exception in thread Thread-4:
Traceback (most recent call last):
  File "c:\program files\python37\lib\urllib\request.py", line 1317, in do_open
    encode_chunked=req.has_header('Transfer-encoding'))
  File "c:\program files\python37\lib\http\client.py", line 1229, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "c:\program files\python37\lib\http\client.py", line 1275, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "c:\program files\python37\lib\http\client.py", line 1224, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "c:\program files\python37\lib\http\client.py", line 1016, in _send_output
    self.send(msg)
  File "c:\program files\python37\lib\http\client.py", line 956, in send
    self.connect()
  File "c:\program files\python37\lib\http\client.py", line 928, in connect
    (self.host,self.port), self.timeout, self.source_address)
  File "c:\program files\python37\lib\socket.py", line 707, in create_connection
    for res in getaddrinfo(host, port, 0, SOCK_STREAM):
  File "c:\program files\python37\lib\socket.py", line 748, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno 11001] getaddrinfo failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\CCLi8\IoT\mbed\mbed-examples\nu-psa\mbed-os\tools\arm_pack_manager\__init__.py", line 108, in cache_file
    fd.write(urlopen(url).read())
  File "c:\program files\python37\lib\urllib\request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "c:\program files\python37\lib\urllib\request.py", line 525, in open
    response = self._open(req, data)
  File "c:\program files\python37\lib\urllib\request.py", line 543, in _open
    '_open', req)
  File "c:\program files\python37\lib\urllib\request.py", line 503, in _call_chain
    result = func(*args)
  File "c:\program files\python37\lib\urllib\request.py", line 1345, in http_open
    return self.do_open(http.client.HTTPConnection, req)
  File "c:\program files\python37\lib\urllib\request.py", line 1319, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 11001] getaddrinfo failed>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\program files\python37\lib\threading.py", line 917, in _bootstrap_inner
    self.run()
  File "C:\CCLi8\IoT\mbed\mbed-examples\nu-psa\mbed-os\tools\arm_pack_manager\__init__.py", line 63, in run
    self.func(url)
  File "C:\CCLi8\IoT\mbed\mbed-examples\nu-psa\mbed-os\tools\arm_pack_manager\__init__.py", line 133, in cache_pdsc_and_pack
    self.cache_file(url)
  File "C:\CCLi8\IoT\mbed\mbed-examples\nu-psa\mbed-os\tools\arm_pack_manager\__init__.py", line 110, in cache_file
    stderr.write(e.reason)
TypeError: write() argument must be str, not gaierror

Issue request type

[ ] Question
[ ] Enhancement
[x] Bug

[ciarmcom]

Internal Jira reference: https://jira.arm.com/browse/MBOCUSTRIA-455

[bridadan]

I'm getting different errors, and my process doesn't seem to finish:

mbed export -i uvision --update-packs
[mbed] WARNING: If you're using Python 3 with Mbed OS 5.8 and earlier versions, Python errors will occur when compiling, testing and exporting
---
[mbed] Working path "C:\Users\bridan01\onedrive_arm\Documents\dev\mbed-os" (program)
Not FoundNot FoundException in thread Thread-12:
Traceback (most recent call last):
  File "C:\Users\bridan01\AppData\Local\Programs\Python\Python36-32\lib\threading.py", line 916, in _bootstrap_inner
    self.run()
  File "C:\Users\bridan01\onedrive_arm\Documents\dev\mbed-os\tools\arm_pack_manager\__init__.py", line 63, in run
    self.func(url)
  File "C:\Users\bridan01\onedrive_arm\Documents\dev\mbed-os\tools\arm_pack_manager\__init__.py", line 135, in cache_pdsc_and_pack
    self.cache_file(self.pdsc_to_pack(url))
  File "C:\Users\bridan01\onedrive_arm\Documents\dev\mbed-os\tools\arm_pack_manager\__init__.py", line 124, in pdsc_to_pack
    content = self.pdsc_from_cache(url)
  File "C:\Users\bridan01\onedrive_arm\Documents\dev\mbed-os\tools\arm_pack_manager\__init__.py", line 458, in pdsc_from_cache
    return BeautifulSoup(fd, "html.parser")
  File "C:\Users\bridan01\AppData\Local\Programs\Python\Python36-32\lib\site-packages\bs4\__init__.py", line 245, in __init__
    markup = markup.read()
  File "C:\Users\bridan01\AppData\Local\Programs\Python\Python36-32\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 163140: character maps to <undefined>

Forbidden[ ERROR ] https://toshiba.semicon-storage.com/content/dam/toshiba-ss/shared/docs/product/micro/device-family-Pack/Toshiba.TXZ4-M4G_DFP.pdsc does not appear to be a conforming .pdsc file
Forbidden[ ERROR ] https://toshiba.semicon-storage.com/content/dam/toshiba-ss/shared/docs/product/micro/device-family-Pack/Toshiba.TXZ4-M4K1_DFP.pdsc does not appear to be a conforming .pdsc file
Forbidden[ ERROR ] https://toshiba.semicon-storage.com/content/dam/toshiba-ss/shared/docs/product/micro/device-family-Pack/Toshiba.TX03-M344_DFP.pdsc does not appear to be a conforming .pdsc file
Forbidden[ ERROR ] https://toshiba.semicon-storage.com/content/dam/toshiba-ss/shared/docs/product/bluetooth-publishing-program/Toshiba.TC35678_ROM002.pdsc does not appear to be a conforming .pdsc file
Forbidden[ ERROR ] https://toshiba.semicon-storage.com/content/dam/toshiba-ss/shared/docs/product/bluetooth-publishing-program/Toshiba.TC3567C_ROM002.pdsc does not appear to be a conforming .pdsc file
Forbidden[ ERROR ] http://software-dl.ti.com/msp430/msp430_public_sw/mcu/msp430/msp432cmsis/latest/exports/TexasInstruments.MSP432P4xx_DFP.pdsc does not appear to be a conforming .pdsc file
Forbidden[ ERROR ] http://software-dl.ti.com/msp430/msp430_public_sw/mcu/msp430/msp432cmsis/latest/exports/TexasInstruments.MSP432E4_DFP.pdsc does not appear to be a conforming .pdsc file
Exception in thread Thread-13:
Traceback (most recent call last):
  File "C:\Users\bridan01\AppData\Local\Programs\Python\Python36-32\lib\threading.py", line 916, in _bootstrap_inner
    self.run()
  File "C:\Users\bridan01\onedrive_arm\Documents\dev\mbed-os\tools\arm_pack_manager\__init__.py", line 63, in run
    self.func(url)
  File "C:\Users\bridan01\onedrive_arm\Documents\dev\mbed-os\tools\arm_pack_manager\__init__.py", line 135, in cache_pdsc_and_pack
    self.cache_file(self.pdsc_to_pack(url))
  File "C:\Users\bridan01\onedrive_arm\Documents\dev\mbed-os\tools\arm_pack_manager\__init__.py", line 124, in pdsc_to_pack
    content = self.pdsc_from_cache(url)
  File "C:\Users\bridan01\onedrive_arm\Documents\dev\mbed-os\tools\arm_pack_manager\__init__.py", line 458, in pdsc_from_cache
    return BeautifulSoup(fd, "html.parser")
  File "C:\Users\bridan01\AppData\Local\Programs\Python\Python36-32\lib\site-packages\bs4\__init__.py", line 245, in __init__
    markup = markup.read()
  File "C:\Users\bridan01\AppData\Local\Programs\Python\Python36-32\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 31656: character maps to <undefined>

Exception in thread Thread-1:
Traceback (most recent call last):
  File "C:\Users\bridan01\AppData\Local\Programs\Python\Python36-32\lib\threading.py", line 916, in _bootstrap_inner
    self.run()
  File "C:\Users\bridan01\onedrive_arm\Documents\dev\mbed-os\tools\arm_pack_manager\__init__.py", line 63, in run
    self.func(url)
  File "C:\Users\bridan01\onedrive_arm\Documents\dev\mbed-os\tools\arm_pack_manager\__init__.py", line 133, in cache_pdsc_and_pack
    self.cache_file(url)
  File "C:\Users\bridan01\onedrive_arm\Documents\dev\mbed-os\tools\arm_pack_manager\__init__.py", line 102, in cache_file
    makedirs(dirname(dest))
  File "C:\Users\bridan01\AppData\Local\Programs\Python\Python36-32\lib\os.py", line 210, in makedirs
    makedirs(head, mode, exist_ok)
  File "C:\Users\bridan01\AppData\Local\Programs\Python\Python36-32\lib\os.py", line 210, in makedirs
    makedirs(head, mode, exist_ok)
  File "C:\Users\bridan01\AppData\Local\Programs\Python\Python36-32\lib\os.py", line 210, in makedirs
    makedirs(head, mode, exist_ok)
  File "C:\Users\bridan01\AppData\Local\Programs\Python\Python36-32\lib\os.py", line 220, in makedirs
    mkdir(name, mode)
NotADirectoryError: [WinError 267] The directory name is invalid: 'C:\\Users\\bridan01\\AppData\\Local\\Temp\\fex.autochips.com:90'

Caching Files 982/986

It would then stay stuck at 982. This was using the commit mentioned in the original post. I also tried the latest master and I had the same issue. Any ideas @theotherjimmy ?

[theotherjimmy]

Yeah, use the code from https://github.com/ARMmbed/mbed-os/pull/8757 The arm_pack_manager code is pretty unmaintained and deprecated.

[bridadan]

I ran the code @theotherjimmy mentioned, it seemed to help it complete. I think there's still a few changes coming in for #8757, but once its merged I think this issue will be resolved.

[ccli8]

The issue is also related to my network environment. After clarifying it and changing mbed-os to 120e43ed14 Merge pull request #9928 from kfnta/secure_release, I can generate index.json now.