Galaxy: `Unable to finish job`: Augustus

[remimarenco]

Hi,

I am running a workflow containing Augustus tool, which outputs a gff3 file. The workflow is available here.

The job state says Error and the output is Unable to finish job.

I posted the traceback below.

I was not having this issue until I switch to a PostgreSQL database (but I had others due to SQLite).

I have tried to remove the post-process of the tool in the workflow (renaming), and just run the tool. All end up in having the Unable to finish job.

Traceback (most recent call last):
  File "/home/svc_galaxy/new/galaxy/lib/galaxy/jobs/runners/local.py", line 128, in queue_job
    job_wrapper.finish( stdout, stderr, exit_code )
  File "/home/svc_galaxy/new/galaxy/lib/galaxy/jobs/__init__.py", line 1321, in finish
    ActionBox.execute(self.app, self.sa_session, pja.post_job_action, job)
  File "/home/svc_galaxy/new/galaxy/lib/galaxy/jobs/actions/post.py", line 561, in execute
    ActionBox.actions[pja.action_type].execute(app, sa_session, pja, job, replacement_dict)
  File "/home/svc_galaxy/new/galaxy/lib/galaxy/jobs/actions/post.py", line 407, in execute
    sa_session.flush()
  File "/home/svc_galaxy/new/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/orm/scoping.py", line 150, in do
    return getattr(self.registry(), name)(*args, **kwargs)
  File "/home/svc_galaxy/new/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2004, in flush
    self._flush(objects)
  File "/home/svc_galaxy/new/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2122, in _flush
    transaction.rollback(_capture_exception=True)
  File "/home/svc_galaxy/new/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/home/svc_galaxy/new/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2086, in _flush
    flush_context.execute()
  File "/home/svc_galaxy/new/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 373, in execute
    rec.execute(self)
  File "/home/svc_galaxy/new/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 532, in execute
    uow
  File "/home/svc_galaxy/new/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 170, in save_obj
    mapper, table, update)
  File "/home/svc_galaxy/new/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 692, in _emit_update_statements
    execute(statement, multiparams)
  File "/home/svc_galaxy/new/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 914, in execute
    return meth(self, multiparams, params)
  File "/home/svc_galaxy/new/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/home/svc_galaxy/new/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/home/svc_galaxy/new/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
    context)
  File "/home/svc_galaxy/new/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1344, in _handle_dbapi_exception
    util.reraise(*exc_info)
  File "/home/svc_galaxy/new/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
    context)
  File "/home/svc_galaxy/new/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute
    cursor.execute(statement, parameters)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xf6' in position 197: ordinal not in range(128)

Hope somebody can help,

Remi

[mvdbeek]

Hi @remimarenco,

the workflow link you've shared probably only works for you, you would need to post the URL from the sharing menu.
That said, u'\xf6' is ö, if you have control over this server can you print stdout and stderr from lib/galaxy/jobs/runners/local.py and see if they contain ö for the failing job?
If they do, can you try this patch?

[remimarenco]

Thanks Marius, will try that now.

[dannon]

Extra info: Looking through the trace, since this is the initial flush I'd guess it's probably unrelated to the actual PJAs you have within the workflow. If @mvdbeek's fix doesn't work, let me know and I can dig more.

[remimarenco]

Thanks Dannon, will let you know. PJA is the acronym for Post Job Action, right?

Augustus is the slowest tool I have, so it is still running...will let you know soon.

[dannon]

@remimarenco Yep, exactly, PostJobAction, which is all of the dataset cleanup, hide, rename, etc., actions that you can assign within a workflow.

[remimarenco]

I have ran the tool and got this stacktrace:

Traceback (most recent call last):
  File "/home/svc_galaxy/new/galaxy/lib/galaxy/jobs/runners/local.py", line 131, in queue_job
    job_wrapper.finish( stdout, stderr, exit_code )
  File "/home/svc_galaxy/new/galaxy/lib/galaxy/jobs/__init__.py", line 1324, in finish
    self.sa_session.flush()
  File "/home/svc_galaxy/new/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/orm/scoping.py", line 150, in do
    return getattr(self.registry(), name)(*args, **kwargs)
  File "/home/svc_galaxy/new/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2004, in flush
    self._flush(objects)
  File "/home/svc_galaxy/new/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2122, in _flush
    transaction.rollback(_capture_exception=True)
  File "/home/svc_galaxy/new/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/home/svc_galaxy/new/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2086, in _flush
    flush_context.execute()
  File "/home/svc_galaxy/new/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 373, in execute
    rec.execute(self)
  File "/home/svc_galaxy/new/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 532, in execute
    uow
  File "/home/svc_galaxy/new/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 170, in save_obj
    mapper, table, update)
  File "/home/svc_galaxy/new/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 692, in _emit_update_statements
    execute(statement, multiparams)
  File "/home/svc_galaxy/new/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 914, in execute
    return meth(self, multiparams, params)
  File "/home/svc_galaxy/new/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/home/svc_galaxy/new/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/home/svc_galaxy/new/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
    context)
  File "/home/svc_galaxy/new/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1344, in _handle_dbapi_exception
    util.reraise(*exc_info)
  File "/home/svc_galaxy/new/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
    context)
  File "/home/svc_galaxy/new/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute
    cursor.execute(statement, parameters)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xf6' in position 197: ordinal not in range(128)

Do you need something else from it?

[mvdbeek]

@remimarenco
Is that with the patch, or are you trying to print stdout and stderr?
If this is with trying to print stdout and stderr, those should be just before the stacktrace.
If this is with the patch then my first guess (problem with special characters in stdout/stderr) is not the right one, so maybe @dannon can chime in.

[remimarenco]

This is with the patch :)

[remimarenco]

stdout and stderr are empty through the UI. I can try to print them manually in the code (so I guess I need to add the print stdout and print stderr to the patch)

[mvdbeek]

Hmm, yes, I guess it's worth a shot. It would be interesting to know where that ö is coming from. Does this also happen with the test-data that comes with the tool?

[dannon]

Any way you can search for where the likely ö is coming from? (edit: heh, looks like we had the same next step planned!)

[remimarenco]

Running the tool with the prints. Waiting for the output.

I can try, although it did not happen on SQLite. Not sure why it happens now.

[remimarenco]

# Oliver Keller, Stefanie König and Lizzy Gerischer.

Fourth comment line.

Should I activate something in Postgre to support unicode?

[mvdbeek]

When you say it works on sqlite, is that with the same dataset ?
How does your db connection string look like?
Is that postgresql://galaxy:galaxy@localhost:5432/galaxy?client_encoding=utf8 ?
(copy'n paste from bjoerns docker container galaxy:galaxy / localhost of course replaced with your credentials ...)
The important part I is the ?client_encoding=utf8
What about the LC_ALL and LANG environmental variables?

[remimarenco]

Thanks for helping so quickly.

• Yes same dataset
• database_connection = postgresql:///new_galaxy_test?user=svc_galaxy
• Can't see them atm, will activate in a few seconds on this server

[mvdbeek]

yeah, try appending ?client_encoding=utf8.

[remimarenco]

Running the tool and asking to print right after:

        stdout = unicodify(stdout)
        stderr = unicodify(stderr)

Leads to kind of the same traceback (I guess it makes sense):

Traceback (most recent call last):
  File "/home/svc_galaxy/new/galaxy/lib/galaxy/jobs/runners/__init__.py", line 104, in run_next
    method(arg)
  File "/home/svc_galaxy/new/galaxy/lib/galaxy/jobs/runners/local.py", line 130, in queue_job
    print stdout
  File "/home/svc_galaxy/new/galaxy/lib/galaxy/util/pastescript/serve.py", line 889, in write
    fileobj.write(text)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xf6' in position 197: ordinal not in range(128)

Will restart the server with all the changes

[remimarenco]

Well...I added ?client_encoding=utf8 at the end of the postgre connection string, and stderr / stdout...it did not change anything :/.

[remimarenco]

Weird thing...I can't see LC_ALL and LANG environmental variables on Augustus job, but I can on others....

LC_ALL (runtime environment variable)   C
LANG (runtime environment variable) en_US.UTF-8

[remimarenco]

So I fixed the issue thanks to the client_encoding part.

Now I have a DataError: (psycopg2.DataError) value too long for type character varying(255) error.

Do I understand well that it is trying to save in the db, via parameters: info the whole output file?

Traceback:

Traceback (most recent call last):
  File "/home/svc_galaxy/new/galaxy/lib/galaxy/jobs/runners/local.py", line 131, in queue_job
    job_wrapper.finish( stdout, stderr, exit_code )
  File "/home/svc_galaxy/new/galaxy/lib/galaxy/jobs/__init__.py", line 1324, in finish
    self.sa_session.flush()
  File "/home/svc_galaxy/new/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/orm/scoping.py", line 150, in do
    return getattr(self.registry(), name)(*args, **kwargs)
  File "/home/svc_galaxy/new/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2004, in flush
    self._flush(objects)
  File "/home/svc_galaxy/new/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2122, in _flush
    transaction.rollback(_capture_exception=True)
  File "/home/svc_galaxy/new/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/home/svc_galaxy/new/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2086, in _flush
    flush_context.execute()
  File "/home/svc_galaxy/new/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 373, in execute
    rec.execute(self)
  File "/home/svc_galaxy/new/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 532, in execute
    uow
  File "/home/svc_galaxy/new/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 170, in save_obj
    mapper, table, update)
  File "/home/svc_galaxy/new/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 692, in _emit_update_statements
    execute(statement, multiparams)
  File "/home/svc_galaxy/new/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 914, in execute
    return meth(self, multiparams, params)
  File "/home/svc_galaxy/new/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/home/svc_galaxy/new/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/home/svc_galaxy/new/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
    context)
  File "/home/svc_galaxy/new/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1341, in _handle_dbapi_exception
    exc_info
  File "/home/svc_galaxy/new/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb)
  File "/home/svc_galaxy/new/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
    context)
  File "/home/svc_galaxy/new/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute
    cursor.execute(statement, parameters)
DataError: (psycopg2.DataError) value too long for type character varying(255)
 [SQL: 'UPDATE history_dataset_association SET update_time=%(update_time)s, info=%(info)s, blurb=%(blurb)s, peek=%(peek)s, tool_version=%(tool_version)s, metadata=%(_metadata)s WHERE history_dataset_association.id = %(history_dataset_association_id)s'] [parameters: {'info': u'\n##gff-version 3\n# This output was generated with AUGUSTUS (version 3.1.0).\n# AUGUSTUS is a gene prediction tool written by Mario Stanke ([email protected]),\n# Oliver Keller, Stefanie K\xf6nig and Lizzy Gerischer.\n# Please cite: Mario Stanke, Ma', 'peek': u'##gff-version 3\n\n\n\n\n', 'update_time': datetime.datetime(2016, 6, 22, 23, 48, 57, 938243), 'history_dataset_association_id': 660, 'tool_version': '', '_metadata': <psycopg2.extensions.Binary object at 0x7f72c421e710>, 'blurb': '10 lines, 21 comments'}]

[remimarenco]

Should I modify the Wiki to tell that PostgreSQL also needs UTF-8?

https://wiki.galaxyproject.org/Admin/Config/Performance/ProductionServer#Switching_to_a_database_server

[remimarenco]

Ok fixed all this story, thanks to @dannon help in the end.

So my database was encoded in SQL_ASCII by default:

svc_galaxy galaxy% psql new_galaxy_test -c 'SHOW SERVER_ENCODING'
 server_encoding
-----------------
 SQL_ASCII
(1 row)

It seems like the createdb is taking a specific template in the server of my university, asking to encode in ASCII.

I dumped and dropped the db. Recreated it with the right encoding and template and good to go!

Thanks @mvdbeek for the encoding path and @dannon for showing me that the DataError: (psycopg2.DataError) value too long for type character varying(255) could be related to encoding.

Now everything is fixed!

Do you think we should warn the user the database is not in UTF8? I know this is hard to ask because this is related to tools output...really don't know about that!

[mvdbeek]

Maybe same issue as in to #714 ?
ping @dannon

[nsoranzo]

I think this can be closed, feel free to reopen otherwise.

[dannon]

Yep, this particular issue was resolved and it was a problem with the installation, not something we fixed in Galaxy.