Ghidra: python script bug

[niedabao1]

Describe the bug
Running python script in the program will report an error.

To Reproduce
Steps to reproduce the behavior:

1. Use ghidra open a file
2. Click on ' Display Script Manager'
3. Just run a python script.
4. error" Cannot create PyString with non-byte value"

Expected behavior
no have java error

Attachments
Error running script: AskScriptPy.py
java.lang.IllegalArgumentException: Cannot create PyString with non-byte value
java.lang.IllegalArgumentException: Cannot create PyString with non-byte value
at org.python.core.PyString.(PyString.java:57)
at org.python.core.PyString.(PyString.java:70)
at org.python.core.PyString.(PyString.java:74)
at org.python.core.Py.newString(Py.java:643)
at ghidra.python.GhidraPythonInterpreter.(GhidraPythonInterpreter.java:103)
at ghidra.python.GhidraPythonInterpreter.get(GhidraPythonInterpreter.java:89)
at ghidra.python.PythonScript.runInNewEnvironment(PythonScript.java:110)
at ghidra.python.PythonScript.run(PythonScript.java:48)
at ghidra.app.script.GhidraScript.executeNormal(GhidraScript.java:365)
at ghidra.app.script.GhidraScript.doExecute(GhidraScript.java:220)
at ghidra.app.script.GhidraScript.execute(GhidraScript.java:198)
at ghidra.app.plugin.core.script.RunScriptTask.run(RunScriptTask.java:57)
at ghidra.util.task.Task.monitoredRun(Task.java:126)
at ghidra.util.task.TaskRunner.lambda$startTaskThread$1(TaskRunner.java:94)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)

Environment (please complete the following information):
Ghidra Version: 9.1.2
Java Home: D:\XXOO\ghidra\java11\jdk-11.0.5
JVM Version: AdoptOpenJDK 11.0.5
OS: Windows 10 10.0 amd64

Additional context
chinese/japanese language windows 10

[ryanmkurtz]

Duplicate of #1602. You are probably selecting a path with non-ascii characters in it.

[astrelsky]

@ryanmkurtz out of curiosity is the problem with being unable to handle non-ascii characters is the path a ghidra problem or a jython problem?

[ryanmkurtz]

I believe it's happening because while Java can handle unicode under the hood in its String datatype, Jython, which is based on Python 2.7, cannot. I see similar issues being raised over at the Jython GitHub page, with @Jeff5 referencing a possible fix in Jython 2.7.2.

[niedabao1]

@astrelsky i delete every non-ascii charaters in the path ,then i restart my computer, but it's still doesn't work :-(

[niedabao1]

so,can next version replace ghidra's jython to 2.7.2?

[ryanmkurtz]

I put in #1786. Hopefully it will be accepted for the 9.2 release.

When you run AskScriptPy.py, where does the exception occur? Is it as soon as you select a file from the "Choose a file" dialog?

[ryanmkurtz]

Also, do you get the error if you run the python_basics.py script from Ghidra?

[astrelsky]

You should be able to replace the jython 2.7.1 jar with the 2.7.2 jar in the ghidra installation.

[ryanmkurtz]

You should be able to replace the jython 2.7.1 jar with the 2.7.2 jar in the ghidra installation.

There's more to it than that for Jython. The Lib directory from the jar gets extracted during the build process and put into the data directory, in a directory named after the version of jython.

[astrelsky]

You should be able to replace the jython 2.7.1 jar with the 2.7.2 jar in the ghidra installation.

There's more to it than that for Jython. The Lib directory from the jar gets extracted during the build process and put into the data directory, in a directory named after the version of jython.

Oh I see. I'll look later and see if i can put together instructions to manually update it then. It may be helpful for users encountering this on 9.1.2.

[ryanmkurtz]

That triggered a memory of me writing the instructions for another issue:

1. Download jython-standalone-2.7.2.jar
2. Copy it to your Ghidra/Features/Python/lib directory
3. Delete jython-standalone-2.7.1.jar from that directory
4. Rename jython-standalone-2.7.2.jar to jython-standalone-2.7.1.jar in that directory
5. Extract/unzip your new jython-standalone-2.7.1.jar file
6. In the extracted directory, there will be a subdirectory named Lib. Copy that subdirectory to Ghidra/Features/Python/data/jython-2.7.1/Lib (delete the old Lib dir first).

[niedabao1]

@ryanmkurtz yes,when i Run the python script loaded in the "script manager" ,the error will hapeen,and show the information about this error.i will try replace the jython by myselve,thanks you and @astrelsky

[niedabao1]

@ryanmkurtz i use you way to replace the jyphon-file ,but it's dosen't work :-(
i record a video (it's 42s,gif too big)about what the error happen,hope this work for you.

[niedabao1]

maybe jython developer did not fix this error 😆

[jeff5]

Maybe not.

When I'd fixed all the errors like that in string content, then all the errors related to the path, I still found that a non-ascii user name or host name would cause problems. (It's to do with temp directories and the expansion of relative paths to absolute.) There may be no ascii characters in your path, but that doesn't mean we won't find you some from elsewhere :( sorry.

Console/default encoding can be a pain too. I have my PC localised to Chinese just to give Jython a bit more stress in tests.

It should be much better in 2.7.2, but this is puzzling me: https://bugs.jython.org/issue2890 . I wonder now why I didn't make the message tell you what it choked on.

The idea of installing a 2.7.2 and calling it 2.7.1 to get it executed seems a tad fragile to me. Maybe it could be a configuration item in Ghidra permitting experimental up/down grades to a user?

[niedabao1]

@jeff5 thanks you test,so this problem can't be solved right now,i should learning some java to write plugin🤔

[zagruzkaaa]

Is there a solution to this problem?

[dragonmacher]

Is there a solution to this problem?

We just updated the master branch Jython to 2.7.2. It is not clear to me yet if this fixes the issue. You can try running from the master branch or wait until Ghidra 9.2 is released to test.

[zagruzkaaa]

Is there a solution to this problem?

We just updated the master branch Jython to 2.7.2. It is not clear to me yet if this fixes the issue. You can try running from the master branch or wait until Ghidra 9.2 is released to test.

Jython 2.7.2 doesn't solve problems

[dragonmacher]

@zagruzkaaa > Jython 2.7.2 doesn't solve problems

O.K. At this point, if we can recreate the issue, we may be able to create a patch. It appears from the code that the script name is find, but the path to the script must contain unicode (or non-ascii) characters.

Will you provide us with the stack trace and the full path to the script that is shown in the stack trace?

[zagruzkaaa]

@zagruzkaaa > Jython 2.7.2 doesn't solve problems

O.K. At this point, if we can recreate the issue, we may be able to create a patch. It appears from the code that the script name is find, but the path to the script must contain unicode (or non-ascii) characters.

Will you provide us with the stack trace and the full path to the script that is shown in the stack trace?

This is all that is displayed:

Error running script: ghidra.py java.lang.IllegalArgumentException: Cannot create PyString with non-byte value java.lang.IllegalArgumentException: Cannot create PyString with non-byte value at org.python.core.PyString.<init>(PyString.java:57) at org.python.core.PyString.<init>(PyString.java:70) at org.python.core.PyString.<init>(PyString.java:74) at org.python.core.Py.newString(Py.java:643) at ghidra.python.GhidraPythonInterpreter.<init>(GhidraPythonInterpreter.java:103) at ghidra.python.GhidraPythonInterpreter.get(GhidraPythonInterpreter.java:89) at ghidra.python.PythonScript.runInNewEnvironment(PythonScript.java:110) at ghidra.python.PythonScript.run(PythonScript.java:48) at ghidra.app.script.GhidraScript.executeNormal(GhidraScript.java:365) at ghidra.app.script.GhidraScript.doExecute(GhidraScript.java:220) at ghidra.app.script.GhidraScript.execute(GhidraScript.java:198) at ghidra.app.plugin.core.script.RunScriptTask.run(RunScriptTask.java:57) at ghidra.util.task.Task.monitoredRun(Task.java:126) at ghidra.util.task.TaskRunner.lambda$startTaskThread$1(TaskRunner.java:94) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) at java.base/java.lang.Thread.run(Thread.java:832)

The path to the file is as follows: D:\ghidra\sripts\ghidra.py
Username on windows contains unicode

[zagruzkaaa]

@zagruzkaaa > Jython 2.7.2 doesn't solve problems

O.K. At this point, if we can recreate the issue, we may be able to create a patch. It appears from the code that the script name is find, but the path to the script must contain unicode (or non-ascii) characters.

Will you provide us with the stack trace and the full path to the script that is shown in the stack trace?

If I just want to open python with Window -> Python i'm getting the same error

[jpleasu]

our calls to Py.newString will fail on non-Ascii java strings. We should be using newStringOrUnicode instead.
I've tested with a non-Ascii scripting directory name and the Python console - I see the same error, and it's fixed by this change.