Recommenders: [BUG] in integration test of wikidata
Description
============================= test session starts ==============================
platform linux -- Python 3.6.8, pytest-5.0.1, py-1.8.0, pluggy-0.12.0
rootdir: /data/home/recocat/cicd/7/s
collected 29 items / 12 deselected / 17 selected
tests/integration/test_criteo.py . [ 5%]
tests/integration/test_movielens.py ......... [ 58%]
tests/integration/test_notebooks_python.py ......F [100%]
=================================== FAILURES ===================================
__________________________ test_wikidata_integration ___________________________
self = Index(['name', 'value', 'type', 'filename'], dtype='object')
key = 'lenght_result', method = None, tolerance = None
@Appender(_index_shared_docs['get_loc'])
def get_loc(self, key, method=None, tolerance=None):
if method is None:
if tolerance is not None:
raise ValueError('tolerance argument only valid if using pad, '
'backfill or nearest lookups')
try:
> return self._engine.get_loc(key)
/anaconda/envs/nightly_reco_base/lib/python3.6/site-packages/pandas/core/indexes/base.py:2657:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> ???
pandas/_libs/index.pyx:108:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> ???
pandas/_libs/index.pyx:132:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> ???
pandas/_libs/hashtable_class_helper.pxi:1601:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> ???
E KeyError: 'lenght_result'
pandas/_libs/hashtable_class_helper.pxi:1608: KeyError
During handling of the above exception, another exception occurred:
notebooks = {'als_deep_dive': '/data/home/recocat/cicd/7/s/notebooks/02_model/als_deep_dive.ipynb', 'als_pyspark': '/data/home/rec...baseline_deep_dive.ipynb', 'data_split': '/data/home/recocat/cicd/7/s/notebooks/01_prepare_data/data_split.ipynb', ...}
tmp = '/tmp/pytest-of-recocat/pytest-1082/tmpwyijg271'
@pytest.mark.integration
def test_wikidata_integration(notebooks, tmp):
notebook_path = notebooks["wikidata_KG"]
MOVIELENS_SAMPLE_SIZE = 5
pm.execute_notebook(notebook_path, OUTPUT_NOTEBOOK, kernel_name=KERNEL_NAME,
parameters=dict(MOVIELENS_DATA_SIZE='100k',
MOVIELENS_SAMPLE=True,
MOVIELENS_SAMPLE_SIZE=MOVIELENS_SAMPLE_SIZE))
> result = pm.read_notebook(OUTPUT_NOTEBOOK).dataframe["lenght_result"]
@almudenasanz please check when you are available
In which platform does it happen?
How do we replicate the issue?
Expected behavior (i.e. solution)
Other Comments
bug
miguelgfierro
All 10 comments
hey @almudenasanz, there is another error in the same test:
tests/integration/test_notebooks_python.py ......F [100%]
=================================== FAILURES ===================================
__________________________ test_wikidata_integration ___________________________
notebooks = {'als_deep_dive': '/data/home/recocat/cicd/7/s/notebooks/02_model/als_deep_dive.ipynb', 'als_pyspark': '/data/home/rec...baseline_deep_dive.ipynb', 'data_split': '/data/home/recocat/cicd/7/s/notebooks/01_prepare_data/data_split.ipynb', ...}
tmp = '/tmp/pytest-of-recocat/pytest-1104/tmpyx54eslt'
@pytest.mark.integration
def test_wikidata_integration(notebooks, tmp):
notebook_path = notebooks["wikidata_KG"]
MOVIELENS_SAMPLE_SIZE = 5
pm.execute_notebook(notebook_path, OUTPUT_NOTEBOOK, kernel_name=KERNEL_NAME,
parameters=dict(MOVIELENS_DATA_SIZE='100k',
MOVIELENS_SAMPLE=True,
MOVIELENS_SAMPLE_SIZE=MOVIELENS_SAMPLE_SIZE))
results = pm.read_notebook(OUTPUT_NOTEBOOK).dataframe.set_index("name")["value"]
> assert results["lenght_result"] == MOVIELENS_SAMPLE_SIZE
E assert 184 == 5
Totally normal, I was not exporting the right variable for the assert. Just did a PR
almudenasanz
on 31 Jul 2019
😄1
hey @almudenasanz
tests/integration/test_criteo.py . [ 5%]
tests/integration/test_movielens.py ......... [ 58%]
tests/integration/test_notebooks_python.py ......F [100%]
=================================== FAILURES ===================================
__________________________ test_wikidata_integration ___________________________
notebooks = {'als_deep_dive': '/data/home/recocat/cicd/7/s/notebooks/02_model/als_deep_dive.ipynb', 'als_pyspark': '/data/home/rec...baseline_deep_dive.ipynb', 'data_split': '/data/home/recocat/cicd/7/s/notebooks/01_prepare_data/data_split.ipynb', ...}
tmp = '/tmp/pytest-of-recocat/pytest-1122/tmpx00synt0'
@pytest.mark.integration
def test_wikidata_integration(notebooks, tmp):
notebook_path = notebooks["wikidata_KG"]
MOVIELENS_SAMPLE_SIZE = 5
pm.execute_notebook(notebook_path, OUTPUT_NOTEBOOK, kernel_name=KERNEL_NAME,
parameters=dict(MOVIELENS_DATA_SIZE='100k',
MOVIELENS_SAMPLE=True,
> MOVIELENS_SAMPLE_SIZE=MOVIELENS_SAMPLE_SIZE))
tests/integration/test_notebooks_python.py:173:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/anaconda/envs/nightly_reco_base/lib/python3.6/site-packages/papermill/execute.py:94: in execute_notebook
raise_for_execution_errors(nb, output_path)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
nb = {'cells': [{'cell_type': 'code', 'metadata': {'inputHidden': True, 'hide_input': True}, 'execution_count': None, 'sour...nd_time': '2019-07-31T12:48:30.058293', 'duration': 15.144893, 'exception': True}}, 'nbformat': 4, 'nbformat_minor': 2}
output_path = 'output.ipynb'
def raise_for_execution_errors(nb, output_path):
"""Assigned parameters into the appropriate place in the input notebook
Parameters
----------
nb : NotebookNode
Executable notebook object
output_path : str
Path to write executed notebook
"""
error = None
for cell in nb.cells:
if cell.get("outputs") is None:
continue
for output in cell.outputs:
if output.output_type == "error":
error = PapermillExecutionError(
exec_count=cell.execution_count,
source=cell.source,
ename=output.ename,
evalue=output.evalue,
traceback=output.traceback,
)
break
if error:
# Write notebook back out with the Error Message at the top of the Notebook.
error_msg = ERROR_MESSAGE_TEMPLATE % str(error.exec_count)
error_msg_cell = nbformat.v4.new_code_cell(
source="%%html\n" + error_msg,
outputs=[
nbformat.v4.new_output(output_type="display_data", data={"text/html": error_msg})
],
metadata={"inputHidden": True, "hide_input": True},
)
nb.cells = [error_msg_cell] + nb.cells
write_ipynb(nb, output_path)
> raise error
E papermill.exceptions.PapermillExecutionError:
E ---------------------------------------------------------------------------
E Exception encountered at "In [19]":
E ---------------------------------------------------------------------------
E ImportError Traceback (most recent call last)
E /anaconda/envs/nightly_reco_base/lib/python3.6/site-packages/tqdm/_tqdm.py in pandas(tclass, *targs, **tkwargs)
E 612 # pandas>=0.23.0
E --> 613 from pandas.core.groupby.groupby import DataFrameGroupBy, \
E 614 SeriesGroupBy, GroupBy, PanelGroupBy
E
E ImportError: cannot import name 'DataFrameGroupBy'
E
E During handling of the above exception, another exception occurred:
E
E ImportError Traceback (most recent call last)
E <ipython-input-19-6ccb9974139b> in <module>
E ----> 1 tqdm().pandas(desc="Number of movies completed")
E 2 result = pd.concat(list(movies.progress_apply(lambda x: wikidata_KG_from_movielens(x), axis=1)))
E
E /anaconda/envs/nightly_reco_base/lib/python3.6/site-packages/tqdm/_tqdm.py in pandas(tclass, *targs, **tkwargs)
E 614 SeriesGroupBy, GroupBy, PanelGroupBy
E 615 except ImportError:
E --> 616 from pandas.core.groupby import DataFrameGroupBy, \
E 617 SeriesGroupBy, GroupBy, PanelGroupBy
E 618
E
E ImportError: cannot import name 'PanelGroupBy'
/anaconda/envs/nightly_reco_base/lib/python3.6/site-packages/papermill/execute.py:241: PapermillExecutionError
can you take a look?
miguelgfierro
on 1 Aug 2019
Seems related to this tqdm issue: https://github.com/tqdm/tqdm/issues/780
almudenasanz
on 1 Aug 2019
I will avoid using tqdm in the notebook until the issue is resolved
almudenasanz
on 1 Aug 2019
@almudenasanz :-)
tests/integration/test_notebooks_python.py ......F [100%]
=================================== FAILURES ===================================
__________________________ test_wikidata_integration ___________________________
notebooks = {'als_deep_dive': '/data/home/recocat/cicd/7/s/notebooks/02_model/als_deep_dive.ipynb', 'als_pyspark': '/data/home/rec...baseline_deep_dive.ipynb', 'data_split': '/data/home/recocat/cicd/7/s/notebooks/01_prepare_data/data_split.ipynb', ...}
tmp = '/tmp/pytest-of-recocat/pytest-1197/tmpaqmy5olp'
@pytest.mark.integration
def test_wikidata_integration(notebooks, tmp):
notebook_path = notebooks["wikidata_KG"]
MOVIELENS_SAMPLE_SIZE = 5
pm.execute_notebook(notebook_path, OUTPUT_NOTEBOOK, kernel_name=KERNEL_NAME,
parameters=dict(MOVIELENS_DATA_SIZE='100k',
MOVIELENS_SAMPLE=True,
MOVIELENS_SAMPLE_SIZE=MOVIELENS_SAMPLE_SIZE))
results = pm.read_notebook(OUTPUT_NOTEBOOK).dataframe.set_index("name")["value"]
> assert results["lenght_result"] == MOVIELENS_SAMPLE_SIZE
E assert 4 == 5
I cannot reproduce this, very time I run the notebook locally I get the 5 results. I will do a commit changing the movie titles, to see if it could be related
almudenasanz
on 6 Aug 2019
👍1
hey @almudenasanz
there is an error now in the unit test:
tests/unit/test_notebooks_python.py .......F. [100%]
=================================== FAILURES ===================================
______________________________ test_wikidata_runs ______________________________
notebooks = {'als_deep_dive': '/data/home/recocat/cicd/3/s/notebooks/02_model/als_deep_dive.ipynb', 'als_pyspark': '/data/home/rec...baseline_deep_dive.ipynb', 'data_split': '/data/home/recocat/cicd/3/s/notebooks/01_prepare_data/data_split.ipynb', ...}
tmp = '/tmp/pytest-of-recocat/pytest-1232/tmppef49a1w'
@pytest.mark.notebooks
def test_wikidata_runs(notebooks, tmp):
notebook_path = notebooks["wikidata_KG"]
MOVIELENS_SAMPLE_SIZE = 5
pm.execute_notebook(notebook_path, OUTPUT_NOTEBOOK, kernel_name=KERNEL_NAME,
parameters=dict(MOVIELENS_DATA_SIZE='100k',
MOVIELENS_SAMPLE=True,
> MOVIELENS_SAMPLE_SIZE=MOVIELENS_SAMPLE_SIZE))
tests/unit/test_notebooks_python.py:76:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/anaconda/envs/reco_base/lib/python3.6/site-packages/papermill/execute.py:94: in execute_notebook
raise_for_execution_errors(nb, output_path)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
nb = {'cells': [{'cell_type': 'code', 'metadata': {'inputHidden': True, 'hide_input': True}, 'execution_count': None, 'sour...end_time': '2019-08-07T10:28:38.307981', 'duration': 8.438732, 'exception': True}}, 'nbformat': 4, 'nbformat_minor': 2}
output_path = 'output.ipynb'
def raise_for_execution_errors(nb, output_path):
"""Assigned parameters into the appropriate place in the input notebook
Parameters
----------
nb : NotebookNode
Executable notebook object
output_path : str
Path to write executed notebook
"""
error = None
for cell in nb.cells:
if cell.get("outputs") is None:
continue
for output in cell.outputs:
if output.output_type == "error":
error = PapermillExecutionError(
exec_count=cell.execution_count,
source=cell.source,
ename=output.ename,
evalue=output.evalue,
traceback=output.traceback,
)
break
if error:
# Write notebook back out with the Error Message at the top of the Notebook.
error_msg = ERROR_MESSAGE_TEMPLATE % str(error.exec_count)
error_msg_cell = nbformat.v4.new_code_cell(
source="%%html\n" + error_msg,
outputs=[
nbformat.v4.new_output(output_type="display_data", data={"text/html": error_msg})
],
metadata={"inputHidden": True, "hide_input": True},
)
nb.cells = [error_msg_cell] + nb.cells
write_ipynb(nb, output_path)
> raise error
E papermill.exceptions.PapermillExecutionError:
E ---------------------------------------------------------------------------
E Exception encountered at "In [13]":
E ---------------------------------------------------------------------------
E JSONDecodeError Traceback (most recent call last)
E <timed exec> in <module>
E
E <ipython-input-12-009ef2d271da> in wikidata_descriptions_from_list(names)
E 5 if entity_id != "entityNotFound":
E 6 json_links = query_entity_links(entity_id)
E ----> 7 entity_description = query_entity_description(entity_id)
E 8 related_entities,related_names = read_linked_entities(json_links)
E 9 d = pd.DataFrame({"name": n,
E
E /data/home/recocat/cicd/3/s/reco_utils/dataset/wikidata.py in query_entity_description(entityID)
E 152 description = "descriptionNotFound"
E 153
E --> 154 description = r.json().get("results", {}).get("bindings", [{}])[0].get("o",{}).get("value", "descriptionNotFound")
E 155 return description
E
E /anaconda/envs/reco_base/lib/python3.6/site-packages/requests/models.py in json(self, **kwargs)
E 895 # used.
E 896 pass
E --> 897 return complexjson.loads(self.text, **kwargs)
E 898
E 899 @property
E
E /anaconda/envs/reco_base/lib/python3.6/site-packages/simplejson/__init__.py in loads(s, encoding, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, use_decimal, **kw)
E 516 parse_constant is None and object_pairs_hook is None
E 517 and not use_decimal and not kw):
E --> 518 return _default_decoder.decode(s)
E 519 if cls is None:
E 520 cls = JSONDecoder
E
E /anaconda/envs/reco_base/lib/python3.6/site-packages/simplejson/decoder.py in decode(self, s, _w, _PY3)
E 368 if _PY3 and isinstance(s, bytes):
E 369 s = str(s, self.encoding)
E --> 370 obj, end = self.raw_decode(s)
E 371 end = _w(s, end).end()
E 372 if end != len(s):
E
E /anaconda/envs/reco_base/lib/python3.6/site-packages/simplejson/decoder.py in raw_decode(self, s, idx, _w, _PY3)
E 398 elif ord0 == 0xef and s[idx:idx + 3] == '\xef\xbb\xbf':
E 399 idx += 3
E --> 400 return self.scan_once(s, idx=_w(s, idx).end())
E
E JSONDecodeError: Expecting value: line 1 column 1 (char 0)
/anaconda/envs/reco_base/lib/python3.6/site-packages/papermill/execute.py:241: PapermillExecutionError
----------------------------- Captured stderr call -----------------------------
do you know what might be happening?
miguelgfierro
on 7 Aug 2019
Seems like the request for the description has failed, but I was not able to reproduce it locally. I added in the function a try/except for that case.
The last issue and this one seem like issues with the requests, like we are getting less responses than expected. But I'm not able to reproduce them locally
almudenasanz
on 7 Aug 2019
@pytest.mark.integration
def test_wikidata_integration(notebooks, tmp):
notebook_path = notebooks["wikidata_KG"]
MOVIELENS_SAMPLE_SIZE = 5
pm.execute_notebook(notebook_path, OUTPUT_NOTEBOOK, kernel_name=KERNEL_NAME,
parameters=dict(MOVIELENS_DATA_SIZE='100k',
MOVIELENS_SAMPLE=True,
MOVIELENS_SAMPLE_SIZE=MOVIELENS_SAMPLE_SIZE))
results = pm.read_notebook(OUTPUT_NOTEBOOK).dataframe.set_index("name")["value"]
> assert results["lenght_result"] == MOVIELENS_SAMPLE_SIZE
E assert 4 == 5
it looks we are hitting this error again. Could it be that there is a query that is not giving back any result? maybe you could debug to see whether we are always returning these movies:
L.A. Confidential (1997) 70
Kolya (1996) 64
Jackie Brown (1997) 51
Legends of the Fall (1994) 45
Heavyweights (1994) 42
I'll be away for 2 weeks, if you need anything, please contact @gramhagen
miguelgfierro
on 8 Aug 2019
Was this page helpful?
0 / 5 - 0 ratings
Related issues
msalvaris
路
6Comments
ngcferreira
路
4Comments
miguelgfierro
路
4Comments
miguelgfierro
路
3Comments
miguelgfierro
路
5Comments