Turicreate: Turicreate in Docker not able to invoke lambda workers

[datahack007]

unable to load configuration from /opt/app-root/lib/python3.6/site-packages/turicreate/_scripts/_pylambda_worker.py
unable to load configuration from /opt/app-root/lib/python3.6/site-packages/turicreate/_scripts/_pylambda_worker.py
unable to load configuration from /opt/app-root/lib/python3.6/site-packages/turicreate/_scripts/_pylambda_worker.py
unable to load configuration from /opt/app-root/lib/python3.6/site-packages/turicreate/_scripts/_pylambda_worker.py
unable to load configuration from /opt/app-root/lib/python3.6/site-packages/turicreate/_scripts/_pylambda_worker.py
unable to load configuration from /opt/app-root/lib/python3.6/site-packages/turicreate/_scripts/_pylambda_worker.py
unable to load configuration from /opt/app-root/lib/python3.6/site-packages/turicreate/_scripts/_pylambda_worker.py
unable to load configuration from /opt/app-root/lib/python3.6/site-packages/turicreate/_scripts/_pylambda_worker.py
2020-10-01 21:54:36 - 140465864115968 - ERROR - recom_svc:130 - Traceback (most recent call last):
File "/opt/app-root/lib/python3.6/site-packages/turicreate/data_structures/sarray.py", line 1920, in apply
return SArray(_proxy=self.__proxy__.transform(fn, dtype, skip_na, seed))
File "turicreate/_cython/cy_sarray.pyx", line 130, in turicreate._cython.cy_sarray.UnitySArrayProxy.transform
File "turicreate/_cython/cy_sarray.pyx", line 141, in turicreate._cython.cy_sarray.UnitySArrayProxy.transform
RuntimeError: Cannot evaluate lambda. No Lambda workers have been successfully started.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "recom_svc.py", line 122, in postitem
recomendations_information = itemrecomendation(python_dict_item)
File "./recom_proc.py", line 60, in itemrecomendation
model = tc.recommender.item_content_recommender.create(tc.SFrame(c_d), item_id = "StockCode")
File "/opt/app-root/lib/python3.6/site-packages/turicreate/toolkits/recommender/item_content_recommender.py", line 180, in create
item_data[c] = item_data[c].apply(lambda s: {s : 1})
File "/opt/app-root/lib/python3.6/site-packages/turicreate/data_structures/sarray.py", line 1920, in apply
return SArray(_proxy=self.__proxy__.transform(fn, dtype, skip_na, seed))
File "/opt/app-root/lib/python3.6/site-packages/turicreate/_cython/context.py", line 49, in __exit__
raise exc_type(exc_value)
RuntimeError: Cannot evaluate lambda. No Lambda workers have been successfully started.

[TobyRoseman]

Please let us know a little more information about your setup:
1 - What operating system are you using for your docker image?
2 - What version of Python are you using?
3 - What version of TuriCreate are you using?

In addition, please run the following code:

import turicreate as tc
tc._sys_util.test_pylambda_worker()

Please copy and paste the complete output into this issue. Also please attach the zip file that gets generated.

[datahack007]

heyy,

# set base image (host OS)
FROM python:3.6.9 

# set the working directory in the container
WORKDIR /opt/Recomsystem

# copy the dependencies file to the working directory
COPY requirements.txt .

# Creating Application Source Code Directory
RUN mkdir -p /opt/Recomsystem
RUN mkdir -p /opt/applogs

# install dependencies
RUN pip3 install -r requirements.txt
RUN pip3 install --upgrade pip
RUN pip3 install --no-cache-dir -r requirements.txt
RUN pip3 install pymongo[srv]

# Copying src code to Container
COPY ./ /opt/Recomsystem
COPY startup /opt/Recomsystem
RUN chmod 755 /opt/Recomsystem/startup

# copy the content of the local src directory to the working directory
ENV PATH=/root/.local:$PATH
ENV PYTHONPATH=.:/usr/local/lib/python36.zip:/usr/local/lib/python3.6:/usr/local/lib/python3.6/lib-dynload:/usr/local/lib/python3.6/site-packages:/usr/local/lib/python3.6/site-packages/turicreate/_scripts:/opt/Recomsystem
ENV PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
# Exposing Ports
EXPOSE 8000

# command to run on container start
CMD [ "/opt/Recomsystem/startup" ]

[datahack007]

This is my requirments.txt

numpy>=1.18.1
flask>=1.1.1
requests>=2.18.4
jsonschema>=3.0.0a4
uwsgi>=2.0.18
flask-swagger-ui>=3.25.0
pymongo>=3.10.1
pandas>=0.25.3
turicreate==5.6

[datahack007]

import turicreate as tc
tc._sys_util.test_pylambda_worker()

heyy @TobyRoseman ,
I have tried adding this to my .py file but still its giving me the same error

unable to load configuration from /usr/local/lib/python3.6/site-packages/turicreate/_scripts/_pylambda_worker.py
unable to load configuration from /usr/local/lib/python3.6/site-packages/turicreate/_scripts/_pylambda_worker.py
unable to load configuration from /usr/local/lib/python3.6/site-packages/turicreate/_scripts/_pylambda_worker.py
unable to load configuration from /usr/local/lib/python3.6/site-packages/turicreate/_scripts/_pylambda_worker.py
unable to load configuration from /usr/local/lib/python3.6/site-packages/turicreate/_scripts/_pylambda_worker.py
unable to load configuration from /usr/local/lib/python3.6/site-packages/turicreate/_scripts/_pylambda_worker.py
unable to load configuration from /usr/local/lib/python3.6/site-packages/turicreate/_scripts/_pylambda_worker.py
unable to load configuration from /usr/local/lib/python3.6/site-packages/turicreate/_scripts/_pylambda_worker.py
2020-10-03 00:27:52 - 139764833097472 - ERROR - recom_svc:130 - Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/turicreate/data_structures/sarray.py", line 1920, in apply
return SArray(_proxy=self.__proxy__.transform(fn, dtype, skip_na, seed))
File "turicreate/_cython/cy_sarray.pyx", line 130, in turicreate._cython.cy_sarray.UnitySArrayProxy.transform
File "turicreate/_cython/cy_sarray.pyx", line 141, in turicreate._cython.cy_sarray.UnitySArrayProxy.transform
RuntimeError: Cannot evaluate lambda. No Lambda workers have been successfully started.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "recom_svc.py", line 122, in postitem
recomendations_information = itemrecomendation(python_dict_item)
File "./recom_proc.py", line 61, in itemrecomendation
model = tc.recommender.item_content_recommender.create(tc.SFrame(c_d), item_id = "StockCode")
File "/usr/local/lib/python3.6/site-packages/turicreate/toolkits/recommender/item_content_recommender.py", line 180, in create
item_data[c] = item_data[c].apply(lambda s: {s : 1})
File "/usr/local/lib/python3.6/site-packages/turicreate/data_structures/sarray.py", line 1920, in apply
return SArray(_proxy=self.__proxy__.transform(fn, dtype, skip_na, seed))
File "/usr/local/lib/python3.6/site-packages/turicreate/_cython/context.py", line 49, in __exit__
raise exc_type(exc_value)
RuntimeError: Cannot evaluate lambda. No Lambda workers have been successfully started.

[pid: 6|app: 0|req: 1/2] 172.17.0.1 () {40 vars in 536 bytes} [Sat Oct 3 00:27:33 2020] POST /postitem/v1 => generated 1576 bytes in 19688 msecs (HTTP/1.1 400) 2 headers in 82 bytes (1 switches on core 1)

[TobyRoseman]

The version of TuriCreate that you are using is quite old. Please try updating to the latest version of TuriCreate.

The code I shared was not intended to fix the issue but to help diagnose the problem.

If you are still having the problem after updating to the latest version of TuriCreate, please run the following code, in a standalone script:

import turicreate as tc
tc._sys_util.test_pylambda_worker()

Please copy and paste the complete output into this issue. Also please attach the zip file that gets generated.

[datahack007]

heyy @TobyRoseman
I have updated my TC version to 6.4.1

my New requirments.txt file
numpy>=1.18.1
flask>=1.1.1
requests>=2.18.4
jsonschema>=3.0.0a4
uwsgi>=2.0.18
flask-swagger-ui>=3.25.0
pymongo>=3.10.1
pandas>=0.25.3
turicreate>=6.4.1

but still i am getting the same error
unable to load configuration from /usr/local/lib/python3.6/site-packages/turicreate/_scripts/_pylambda_worker.py
unable to load configuration from /usr/local/lib/python3.6/site-packages/turicreate/_scripts/_pylambda_worker.py
unable to load configuration from /usr/local/lib/python3.6/site-packages/turicreate/_scripts/_pylambda_worker.py
unable to load configuration from /usr/local/lib/python3.6/site-packages/turicreate/_scripts/_pylambda_worker.py
unable to load configuration from /usr/local/lib/python3.6/site-packages/turicreate/_scripts/_pylambda_worker.py
unable to load configuration from /usr/local/lib/python3.6/site-packages/turicreate/_scripts/_pylambda_worker.py
unable to load configuration from /usr/local/lib/python3.6/site-packages/turicreate/_scripts/_pylambda_worker.py
unable to load configuration from /usr/local/lib/python3.6/site-packages/turicreate/_scripts/_pylambda_worker.py
2020-10-03 01:56:32 - 140546258036480 - ERROR - recom_svc:130 - Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/turicreate/data_structures/sarray.py", line 1947, in apply
return SArray(_proxy=self.__proxy__.transform(fn, dtype, skip_na, seed))
File "cy_sarray.pyx", line 130, in turicreate._cython.cy_sarray.UnitySArrayProxy.transform
File "cy_sarray.pyx", line 141, in turicreate._cython.cy_sarray.UnitySArrayProxy.transform
RuntimeError: Cannot evaluate lambda. No Lambda workers have been successfully started.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "recom_svc.py", line 122, in postitem
recomendations_information = itemrecomendation(python_dict_item)
File "./recom_proc.py", line 60, in itemrecomendation
model = tc.recommender.item_content_recommender.create(tc.SFrame(c_d), item_id = "StockCode")
File "/usr/local/lib/python3.6/site-packages/turicreate/toolkits/recommender/item_content_recommender.py", line 200, in create
item_data[c] = item_data[c].apply(lambda s: {s: 1})
File "/usr/local/lib/python3.6/site-packages/turicreate/data_structures/sarray.py", line 1947, in apply
return SArray(_proxy=self.__proxy__.transform(fn, dtype, skip_na, seed))
File "/usr/local/lib/python3.6/site-packages/turicreate/_cython/context.py", line 50, in __exit__
raise exc_type(exc_value)
RuntimeError: Cannot evaluate lambda. No Lambda workers have been successfully started.

[pid: 7|app: 0|req: 2/2] 172.17.0.1 () {40 vars in 536 bytes} [Sat Oct 3 01:56:12 2020] POST /postitem/v1 => generated 1537 bytes in 20305 msecs (HTTP/1.1 400) 2 headers in 82 bytes (1 switches on core 1)

issue_tc_pylw.zip

I have also attached a file zip file I have generated

Thanks,

[TobyRoseman]

Unfortunately, there is a bug in tc._sys_util.test_pylambda_worker() which is preventing us from getting complete results. (I've fixed this bug in #3349).

The information that is present in the zip file looks good.

Please trying running this standalone code and let me know the results:

import turicreate as tc
tc.config.set_runtime_config('TURI_CACHE_FILE_LOCATIONS', '/tmp/')
tc.SArray(range(200)).apply(lambda x: x+1)

[datahack007]

Heyy @TobyRoseman ,

import turicreate as tc
tc.config.set_runtime_config('TURI_CACHE_FILE_LOCATIONS', '/tmp/')
tc.SArray(range(200)).apply(lambda x: x+1)

These are the results observed when I ran the above code

Precision-3540:/home/tarun/Desktop/recomendationsystemecomm/recomsystem# docker exec -it -u 0 76521830835a /bin/bash
root@76521830835a:/opt/Recomsystem# python3
Python 3.6.9 (default, Nov 23 2019, 06:41:34)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

import turicreate as tc
tc.config.set_runtime_config('TURI_CACHE_FILE_LOCATIONS', '/tmp/')
tc.SArray(range(200)).apply(lambda x: x+1)
dtype: int
Rows: 200
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, ... ]

[TobyRoseman]

Great, looks like that worked fine. Try adding tc.config.set_runtime_config('TURI_CACHE_FILE_LOCATIONS', '/tmp/') to your script right after you import turicreate. Let me know if that works.