Iocage: Corruption with listing/starting ALL jails

[notjoe]

I am using iocage as provided in FreeNAS 11.2-Beta3

root@zeus:~ # iocage --version
Version 1.0 ALPHA 1
root@zeus:~ #

I can manually start all the jails if I start them one by one. If I attempt to run "iocage start ALL" I get the following output.

root@zeus:~ # iocage start ALL
Traceback (most recent call last):
  File "/usr/local/bin/iocage", line 10, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/iocage_cli/start.py", line 54, in cli
    ioc.IOCage(jail=jail, rc=rc).start()
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/iocage.py", line 1584, in start
    self.__jail_order__("start")
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/iocage.py", line 183, in __jail_order__
    conf = ioc_json.IOCJson(path).json_load()
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/ioc_json.py", line 238, in json_load
    conf = json.load(conf)
  File "/usr/local/lib/python3.6/json/__init__.py", line 299, in load
    parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
  File "/usr/local/lib/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/lib/python3.6/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
root@zeus:~ #

The same happens if I try to do an iocage list

root@zeus:~ # iocage list
Traceback (most recent call last):
  File "/usr/local/bin/iocage", line 10, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/iocage_cli/list.py", line 81, in cli
    dataset_type, header, _long, _sort, plugin=plugins, quick=quick)
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/iocage.py", line 1143, in list
    quick=quick
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/ioc_list.py", line 75, in list_datasets
    _all = self.list_all(ds)
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/ioc_list.py", line 153, in list_all
    conf = iocage_lib.ioc_json.IOCJson(mountpoint).json_load()
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/ioc_json.py", line 238, in json_load
    conf = json.load(conf)
  File "/usr/local/lib/python3.6/json/__init__.py", line 299, in load
    parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
  File "/usr/local/lib/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/lib/python3.6/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
root@zeus:~ #

[skarekrow]

Hmm one of your jails must be causing some great unhappiness. It looks like a malformed JSON

Hold off until RC1 and then come back here pasting your iocage debug output.

It will save the output in the same location as the jails are mounted in the /debug dataset. So /mnt/POOL/iocage/debug for instance.

Make sure to sanitize anything you don’t want public like your IP address when you paste it here.

It should help us narrow down which jail is having these specific issues.

Did iocage start ALL work previously for you with these same jails? If you can, tell me the history of them. What FreeNAS version they were created with?

• Brandon

[gronke]

@notjoe here's a Python snippet you can execute on your command-line to determine which jail(s) have invalid config.json files. You can copy this snippet to your shell:

python3.6 -c 'import json
import os
jails_dir = "/iocage/jails"
for jail in os.listdir(jails_dir):
  config_file = f"{jails_dir}/{jail}/config.json"
  try:
    with open(config_file) as f:
      json.load(f)
  except:
    print(f"Invalid Jail Config: {jail}")'

It walks over all jails (in the /iocage/jails directory) and attempts to read the JSON data. Please modify the path in case your jails are stored somewhere else.

[notjoe]

python3.6 -c 'import json import os jails_dir = "/iocage/jails" for jail in os.listdir(jails_dir): config_file = f"{jails_dir}/{jail}/config.json" try: with open(config_file) as f: json.load(f) except: print(f"Invalid Jail Config: {jail}")'

Worked perfectly and my problem is now fixed. thanks!