Hydra: UnicodeDecodeError: 'charmap' codec can't decode byte 0x8d in position 1: character maps to <undefined>

[arisliang]

🐛 Bug

Description

Codec error when yaml has some chinese characters

Checklist

• [x] I checked on the latest version of Hydra
• [x] I created a minimal repro

To reproduce

* Minimal Code/Config snippet to reproduce *
main.py

import hydra
import omegaconf


@hydra.main(config_path='.', config_name='config')
def print_hi(cfg: omegaconf.DictConfig) -> None:
    print(omegaconf.OmegaConf.to_yaml(cfg))


if __name__ == '__main__':
    print_hi()

config.yaml

单: 1

* Stack trace/error message *

Traceback (most recent call last):
  File "C:\Users\user\anaconda3\envs\poolobs\lib\site-packages\hydra\_internal\utils.py", line 198, in run_and_report
    return func()
  File "C:\Users\user\anaconda3\envs\poolobs\lib\site-packages\hydra\_internal\utils.py", line 350, in <lambda>
    overrides=args.overrides,
  File "C:\Users\user\anaconda3\envs\poolobs\lib\site-packages\hydra\_internal\hydra.py", line 104, in run
    run_mode=RunMode.RUN,
  File "C:\Users\user\anaconda3\envs\poolobs\lib\site-packages\hydra\_internal\hydra.py", line 512, in compose_config
    from_shell=from_shell,
  File "C:\Users\user\anaconda3\envs\poolobs\lib\site-packages\hydra\_internal\config_loader_impl.py", line 156, in load_configuration
    from_shell=from_shell,
  File "C:\Users\user\anaconda3\envs\poolobs\lib\site-packages\hydra\_internal\config_loader_impl.py", line 262, in _load_configuration
    run_mode=run_mode,
  File "C:\Users\user\anaconda3\envs\poolobs\lib\site-packages\hydra\_internal\config_loader_impl.py", line 805, in _merge_defaults_into_config
    hydra_cfg = merge_defaults_list_into_config(hydra_cfg, user_list)
  File "C:\Users\user\anaconda3\envs\poolobs\lib\site-packages\hydra\_internal\config_loader_impl.py", line 793, in merge_defaults_list_into_config
    package_override=default1.package,
  File "C:\Users\user\anaconda3\envs\poolobs\lib\site-packages\hydra\_internal\config_loader_impl.py", line 698, in _merge_config
    package_override=package_override,
  File "C:\Users\user\anaconda3\envs\poolobs\lib\site-packages\hydra\_internal\config_loader_impl.py", line 592, in _load_config_impl
    package_override=package_override,
  File "C:\Users\user\anaconda3\envs\poolobs\lib\site-packages\hydra\_internal\config_repository.py", line 48, in load_config
    package_override=package_override,
  File "C:\Users\user\anaconda3\envs\poolobs\lib\site-packages\hydra\_internal\core_plugins\file_config_source.py", line 33, in load_config
    header_text = f.read(512)
  File "C:\Users\user\anaconda3\envs\poolobs\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 117: character maps to <undefined>

Expected Behavior

Run without error, and print:

单: 1

System information

• Hydra Version : 1.0.3
• Python version : 3.7.7
• Virtual environment type and version : conda 4.8.5
• Operating system : Windows 7

[omry]

Thanks for reporting.
If you want to try to fix this, I think changing line

with open(full_path,") as f:

to

with open(full_path, encoding="utf-8") as f:

here might solve this.

(To be accepted, such PR also needs a test).

[arisliang]

Where is the test procedure? Any link?

[omry]

Check the developer guide.

[shagunsodhani]

@arisliang Feel free to tag me if you have any questions or want reviews.

[arisliang]

Any rough target date for v1.0.4?

[shagunsodhani]

I do not know :( The milestone tracker does not show an end date.

[omry]

There is no target date for 1.0.4, but the issues opened against it will need to be addressed before it can be released.

[jieru-hu]

@arisliang
could you provide more info on your os, hydra version etc?

I could not reproduce this on Hydra 1.0.3 on my MAC

$ pip freeze | grep hydra
hydra-core==1.0.3

config.yaml

单: 1

my_app.py

@hydra.main(config_path='.', config_name='config')
def print_hi(cfg: DictConfig) -> None:
    print(OmegaConf.to_yaml(cfg))


if __name__ == '__main__':
    print_hi()```

$ python /Users/jieru/workspace/hydra-fork/hydra/examples/tutorials/basic/your_first_hydra_app/2_config_file/my_app.py

单: 1

```

Pls paste your output from pip freeze

[jieru-hu]

nvm, i was able to reproduce this on windows :)

[jieru-hu]

fixed in #1124, going to cherry-pick to 1.0 branch.

[jieru-hu]

fixed on 1.0_branch, closing now.