Tm1py: ProcessService - execute_process_with_return does not pass parameters to the process called

[EricSteingrabe]

Describe the bug
When trying to run an unbound process via execute_process_with_return, the function does not pass the parameters to the underlying process.

To Reproduce
Any process with parameters can be used to replicate this issue. You will experience that the process will run with the parameters it was saved with.

Expected behavior
The expected behaviour of the function would be to pass the parameters to the process that is called.

Version
TM1py 1.6.2
TM1 Server Version: 2.0.9.4

[jrobinsonAG]

Hi @EricSteingrabe could you share a code snippet to better understand exactly how you are using 'execute_process_with_return'?

for example

status = tm1.processes.execute_with_return(process_name='Cub_Load_GL_TB_Master',
                                           pParameter1='P1Value',
                                           pParameter2='P2Value')
logger.info(f'TB Master complete with status: {status[1]}')

[rkvinoth]

with execute_proces_with_return method, you have to explicitly create a Process object and add all the required parameters to it.

1. Is your usecase to create a new Process in python and execute it?
   or
2. Execute an existing TI process in your instance?

For the former you use this as an example:

process = Process(name="Test_Process")
process.add_parameter(name="P0", prompt="logoutput message", value="Here you go")
process.prolog_procedure = "LOGOUTPUT('WARN', P0 ) ;"
success, status, _ = tm1.processes.execute_process_with_return(process=process)
print("Process {}".format(status))

For the latter, follow @jrobinsonAG's example.

[EricSteingrabe]

Ok, I give more detail about my issue. We are loading a csv file via python and create the TI process using python as part of the execution but prior to the script I'm posting below. The TI process named is "_tmp_Upload" and it has 1 parameter called pFile which contains the filepath of the file to be loaded into the system.

We currently run the below function and this is executing successfully:

def create_run_upload_ti(connection, cube_name, file, data_type):

    ti_name = '_tmp_Upload'
    run_ti_params = {'pFile': file}

    create_upload_ti(ti_name=ti_name, cube_name=cube_name, connection=connection, data_type=data_type)
    ret_val_upload = connection.processes.execute_with_return(ti_name, **run_ti_params)

    return ret_val_upload

Please note that the create_upload_ti function simply creates a TI process in the TM1 server. We had some locking issues within TM1 with the creation and deletion of the process and therefore wanted to change this to use an unbound process. Therefore I changed the script to this:

def create_run_upload_ti(connection, cube_name, file, data_type):

    ti_name = '_tmp_Upload'
    run_ti_params = {'pFile': file}

    py_ti = create_upload_process(ti_name=ti_name, cube_name=cube_name, connection=connection, data_type=data_type)
    ret_val_upload = connection.processes.execute_process_with_return(process=py_ti, **run_ti_params)

    return ret_val_upload

Running the above scripts gives me the following error message "Error: Prolog procedure line (0): Unable to open data source "d:\data directory\"." This is the path to the data directory of the TM1 server and you receive this error message when you don't give a path in the parameter and it assumes the data directory.

Subsequently I checked the ProcessServices in tm1py and I think the function processes.execute_with_return passes parameters over to the process and the function processes.execute_process_with_return does not. Specifically I think this code is missing in the processes.execute_process_with_return function.

        parameters = dict()
        if kwargs:
            parameters = {"Parameters": []}
            for parameter_name, parameter_value in kwargs.items():
                parameters["Parameters"].append({"Name": parameter_name, "Value": parameter_value})

Below is the whole code from the ProcessServer.py as a reference:

  def execute_process_with_return(self, process: Process, **kwargs) -> Tuple[bool, str, str]:
        """
        Run unbound TI code directly
        :param process: a TI Process Object
        :return: success (boolean), status (String), error_log_file (String)
        """
        url = "/api/v1/ExecuteProcessWithReturn?$expand=*"

        payload = json.loads("{\"Process\":" + process.body + "}")

        response = self._rest.POST(
            url=url,
            data=json.dumps(payload, ensure_ascii=False),
            **kwargs)

        execution_summary = response.json()

        success = execution_summary["ProcessExecuteStatusCode"] == "CompletedSuccessfully"
        status = execution_summary["ProcessExecuteStatusCode"]
        error_log_file = None if execution_summary["ErrorLogFile"] is None else execution_summary["ErrorLogFile"][
            "Filename"]
        return success, status, error_log_file

    def execute_with_return(self, process_name: str, timeout: float = None, **kwargs) -> Tuple[bool, str, str]:
        """ Ask TM1 Server to execute a process.
        pass process parameters as keyword arguments to this function. E.g:

        self.tm1.processes.execute_with_return(
            process_name="Bedrock.Server.Wait",
            pWaitSec=2)

        :param process_name: name of the TI process
        :param timeout: Number of seconds that the client will wait to receive the first byte.
        :param kwargs: dictionary of process parameters and values
        :return: success (boolean), status (String), error_log_file (String)
        """
        url = format_url("/api/v1/Processes('{}')/tm1.ExecuteWithReturn?$expand=*", process_name)
        parameters = dict()
        if kwargs:
            parameters = {"Parameters": []}
            for parameter_name, parameter_value in kwargs.items():
                parameters["Parameters"].append({"Name": parameter_name, "Value": parameter_value})

        response = self._rest.POST(
            url=url,
            data=json.dumps(parameters, ensure_ascii=False),
            timeout=timeout,
            **kwargs)
        execution_summary = response.json()
        success = execution_summary["ProcessExecuteStatusCode"] == "CompletedSuccessfully"
        status = execution_summary["ProcessExecuteStatusCode"]
        error_log_file = None if execution_summary["ErrorLogFile"] is None else execution_summary["ErrorLogFile"][
            "Filename"]
        return success, status, error_log_file

Again this is my suspicion that this is missing.

[rkvinoth]

As I mentioned earlier in my comment, you have to add the parameter to the Process object that you are creating.

Try this:

def create_run_upload_ti(connection, cube_name, file, data_type):
    ti_name = '_tmp_Upload'

    py_ti = create_upload_process(ti_name=ti_name, cube_name=cube_name, connection=connection, data_type=data_type)
    py_ti.add.add_parameter(name='pFile', prompt='Path to the csv file', value=file)
    ret_val_upload = connection.processes.execute_process_with_return(process=py_ti)

    return ret_val_upload

[MariusWirtz]

@rkvinoth, @EricSteingrabe
I think you have to call the remove_parameter first before you can add it. I assume the pFileparameter already exists.
Maybe try this:

from TM1py import Process

def create_run_upload_ti(connection, cube_name, file, data_type):
    ti_name = '_tmp_Upload'

    py_ti: Process = create_upload_process(ti_name=ti_name, cube_name=cube_name, connection=connection, data_type=data_type)
    py_ti.remove_parameter(name='pFile')
    py_ti.add_parameter(name='pFile', prompt='Path to the csv file', value=file)
    ret_val_upload = connection.processes.execute_process_with_return(process=py_ti)

    return ret_val_upload

[MariusWirtz]

But considering your use case @EricSteingrabe I think it's a fair point.
For your situation, it would be convenient to execute unbound processes with parameters.

We should look into supporting parameters on unbound processes

[rkvinoth]

Agreed, that would be nice for parallel processing as well.

[EricSteingrabe]

@rkvinoth @MariusWirtz - I've tested both of your solutions and can confirm that @MariusWirtz solution works. You'll need to remove the parameter first (as it exists in the process definition) and then add it again.

Thanks all!

[MariusWirtz]

Closed with #588