Twint: "url" issue on Lookup

[marcoferre]

• [x] Python version is 3.6;
• [x] Updated Twint with pip3 install --user --upgrade -e git+https://github.com/twintproject/twint.git@origin/master#egg=twint;
• [x] I have searched the issues and there are no duplicates of this issue/question/request.

Command Ran

for t in tweet_list:
  b = twint.Config()
  b.Username = t['username']
  b.Store_object = True

  twint.run.Lookup(b)

  user = twint.output.users_list[-1]
  t.update(user.__dict__)

  print(t)
  time.sleep(1)

It runs 3 times than this exception

CRITICAL:root:twint.get:User:'url'
ERROR:root:twint.run:Twint:Lookup:Unexpected exception occurred.
Traceback (most recent call last):
  File "/content/src/twint/twint/run.py", line 307, in Lookup
    await get.User(self.config.Username, self.config, db.Conn(self.config.Database))
  File "/content/src/twint/twint/get.py", line 228, in User
    await Users(j_r, config, conn)
  File "/content/src/twint/twint/output.py", line 177, in Users
    user = User(u)
  File "/content/src/twint/twint/user.py", line 31, in User
    _usr.url = ur['data']['user']['legacy']['url']
KeyError: 'url'

---

Environment Details

Google Colab, Win

[ytaijp]

same here.

success with

c.Username = "twitter"
twint.run.Lookup(c)

failed:
c.Username = "jack"
twint.run.Lookup(c)

[aabid0193]

same issue here

[seonake]

It is not 'url' problem... it is 'data', but may be related.

CRITICAL:root:twint.get:User:'data'
ERROR:root:twint.run:Twint:Lookup:Unexpected exception occurred.
Traceback (most recent call last):
File "/root/.local/lib/python3.6/site-packages/twint/run.py", line 307, in Lookup
await get.User(self.config.Username, self.config, db.Conn(self.config.Database))
File "/root/.local/lib/python3.6/site-packages/twint/get.py", line 228, in User
await Users(j_r, config, conn)
File "/root/.local/lib/python3.6/site-packages/twint/output.py", line 177, in Users
user = User(u)
File "/root/.local/lib/python3.6/site-packages/twint/user.py", line 21, in User
if 'data' not in ur and 'user' not in ur['data']:
KeyError: 'data'

---

KeyError Traceback (most recent call last)
in ()
31 c.Username = cuenta
32
---> 33 twint.run.Lookup(c)
34 df = twint.storage.panda.User_df
35

11 frames
/root/.local/lib/python3.6/site-packages/twint/user.py in User(ur)
19 def User(ur):
20 logme.debug(__name__ + ':User')
---> 21 if 'data' not in ur and 'user' not in ur['data']:
22 msg = 'malformed json! cannot be parsed to get user data'
23 logme.fatal(msg)

KeyError: 'data'

[MrNullPoint]

I think it because json parse error, sometimes when we query a user such as @jack, twitter return something like .... user is suspend... (a json can be parsed), so I changed twint/user.py some code, add try...except..., after reinstall twint, this problem is solved.
some code here:

    try:
        _usr.name = ur['data']['user']['legacy']['name']
    except:
        _usr.name = ''
    try:
        _usr.username = ur['data']['user']['legacy']['screen_name']
    except:
        _usr.username = ''
    try:
        _usr.bio = ur['data']['user']['legacy']['description']
    except:
        _usr.bio = ''
    try:
        _usr.location = ur['data']['user']['legacy']['location']
    except:
        _usr.location = ''
    try: 
        _usr.url = ur['data']['user']['legacy']['url']
    except:
        _usr.url = ''

[himanshudabas]

@MrNullPoint this has already been fixed here.
Also try except won't be the desired solution for this, because if something breaks, we should be able to diagnose that issue. try except would simply suppress the issue, which would be much worse, as the final scraped dataset in an insonsistent state.

[micaelamaria]

@himanshudabas -- I keep getting the error when I run twint -u USERNAME --user-full (i musing last kubuntu)

CRITICAL:root:twint.get:User:'url'
ERROR:root:twint.run:Twint:Lookup:Unexpected exception occurred.
Traceback (most recent call last):
File "/home/mik/src/twint/twint/run.py", line 307, in Lookup
await get.User(self.config.Username, self.config, db.Conn(self.config.Database))
File "/home/mik/src/twint/twint/get.py", line 228, in User
await Users(j_r, config, conn)
File "/home/mik/src/twint/twint/output.py", line 177, in Users
user = User(u)
File "/home/mik/src/twint/twint/user.py", line 31, in User
_usr.url = ur['data']['user']['legacy']['url']
KeyError: 'url'
Traceback (most recent call last):
File "/home/mik/.local/bin/twint", line 11, in
load_entry_point('twint', 'console_scripts', 'twint')()
File "/home/mik/src/twint/twint/cli.py", line 339, in run_as_command
main()
File "/home/mik/src/twint/twint/cli.py", line 326, in main
run.Lookup(c)
File "/home/mik/src/twint/twint/run.py", line 386, in Lookup
run(config)
File "/home/mik/src/twint/twint/run.py", line 329, in run
get_event_loop().run_until_complete(Twint(config).main(callback))
File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "/home/mik/src/twint/twint/run.py", line 235, in main
await task
File "/home/mik/src/twint/twint/run.py", line 270, in run
await self.Lookup()
File "/home/mik/src/twint/twint/run.py", line 307, in Lookup
await get.User(self.config.Username, self.config, db.Conn(self.config.Database))
File "/home/mik/src/twint/twint/get.py", line 228, in User
await Users(j_r, config, conn)
File "/home/mik/src/twint/twint/output.py", line 177, in Users
user = User(u)
File "/home/mik/src/twint/twint/user.py", line 31, in User
_usr.url = ur['data']['user']['legacy']['url']
KeyError: 'url'

[himanshudabas]

@micaelamaria
My patch hasn't been merged to the master yet.
If you need to use twint urgently, you can install directly from my branch.
Although I must warn you that there will be some other issues in this branch.
If you do experience some other issue in the above branch, try installing from this branch. This branch is still a work in progress, but it'll be much more stable.

[micaelamaria]

@himanshudabas - how do I install the package from your branch using the command line? I tried infinite options, and none seems to be working :(

[himanshudabas]

@micaelamaria

Try this :

pip3 install --user --upgrade git+https://github.com/himanshudabas/twint.git@origin/twint-fixes#egg=twint