Wpscan: Another "invalid byte sequence UTF-8"

[MLubinski]

Running the brute forcer i get another utf-8 error.

ruby wpscan.rb --url www.domain.com -w passwords.lst -U admin

[ERROR] invalid byte sequence in UTF-8
Trace :
/opt/wpscan/lib/common/models/wp_user/brute_forcable.rb:136:in strip' /opt/wpscan/lib/common/models/wp_user/brute_forcable.rb:136:inblock in passwords_from_wordlist'
/opt/wpscan/lib/common/models/wp_user/brute_forcable.rb:133:in each' /opt/wpscan/lib/common/models/wp_user/brute_forcable.rb:133:inpasswords_from_wordlist'
/opt/wpscan/lib/common/collections/wp_users/brute_forcable.rb:16:in brute_force' wpscan.rb:299:inmain'
wpscan.rb:327:in `

'

[ethicalhack3r]

Hi,

Ruby 1.9.x this time? or still only on 2.0?

[MLubinski]

I just tried it with 2.0.0-p195 and 1.9.2-p320 with the same effect. What is the required format for the password list?

[erwanlr]

What version of WPScan are you using ? (looks like the last one)

Could you tell me the output of file --mime passwords.lst (where passwords.lst is your password file :p

[MLubinski]

version v2.1rcf7d905
rockyou.lst: UTF-8 Unicode C program text

[erwanlr]

I was able to reproduce on the BT5, will investigate

[erwanlr]

What a noob, I know what the problem :

The rockyou file is defined as using UTF-8 encoding but some chars are not UTF-8, you can see them by running :

cat /pentest/passwords/wordlists/rockyou.txt | grep -n -P "[\x80-\xFF]"

So, it's a file encoding issue

[FireFart]

On current version of kali:

root@kali:/etc/john# gunzip /usr/share/wordlists/rockyou.txt.gz
root@kali:/usr/share/wordlists# ls -alh
total 134M
drwxr-xr-x   2 root root 4.0K May 17 21:53 .
drwxr-xr-x 428 root root  16K May  1 19:02 ..
-rw-r--r--   1 root root 134M Mar  3 14:09 rockyou.txt
root@kali:/usr/share/wordlists# file rockyou.txt 
rockyou.txt: C++ source, UTF-8 Unicode text
root@kali:/usr/share/wordlists# uname -a
Linux kali 3.7-trunk-686-pae #1 SMP Debian 3.7.2-0+kali6 i686 GNU/Linux
root@kali:/usr/share/wordlists# grep -n -P "[\x80-\xFF]" rockyou.txt | wc -l
14534
root@kali:/usr/share/wordlists# grep -n -P "[\x80-\xFF]" rockyou.txt | head
3902:contraseña
8613:muñeca
17685:cariño
20424:toño
21919:pequeña
23194:!"£$%^
31311:teextraño
35206:muñequita
35933:niña
36145:españa

[erwanlr]

More explanation : http://stackoverflow.com/questions/10046469/rails-heroku-and-invalid-byte-sequence-in-utf-8-error?answertab=active#tab-top

Solution : convert the rockyou to a real UTF-8 :

iconv -f ISO-8859-1 -t UTF-8 /pentest/passwords/wordlists/rockyou.txt > rockyou_utf8.txt

Tip to see if the encoding of a file matches the encoding of the content :
Just do a vi file_to_open and look at the bottom

"rockyou_utf8.txt" 14344391L, 140056870C <- No problem
"/pentest/passwords/wordlists/rockyou.txt" [converted] 14344391L, 140056870C <- troubles !

Edit: another encoding testing

vi -c 'let $enc = &fileencoding | execute "!echo Encoding:  $enc" | q' rockyou_utf8.txt 

Encoding: utf-8

vi -c 'let $enc = &fileencoding | execute "!echo Encoding:  $enc" | q' /pentest/passwords/wordlists/rockyou.txt 

Encoding: latin1

This command could be created as an alias to check the encoding ;) (See http://vim.wikia.com/wiki/Bash_file_encoding_alias)

[FireFart]

What about using:
http://ruby-doc.org/core-2.0/String.html#method-i-encode

and replace invalid characters?

:invalid
If the value is :replace, encode replaces invalid byte sequences
in str with the replacement character. The default is to raise
the Encoding::InvalidByteSequenceError exception

[ethicalhack3r]

I think using encode is worth a try, I'll have a go at it when I get time.