Twint: [QUESTION] sending tweets to Elasticsearch with only python script

[StefanoSetti]

I've checked the following:

• [x] Python version is 3.5 or higher.
• [X] Using the latest version of Twint.
• [X] I have searched the issues and there are no duplicates of this issue/question/request.

I'm trying to send tweets from an python script directly to Elasticsearch because i don't need kebana for what i am doing, but using config.Elasticsearch = "localhost:9200" brings to me this error:

day = weekday(strftime("%A", localtime(Tweet.datetime)))
NameError: name 'weekday' is not defined

Python script

import twint

# Set-up TWINW config
config = twint.Config()
config.Username      = "revenantRED"
config.Since         = "2018-06-10"
config.Store_json    = True
config.Elasticsearch = "localhost:9200"

# Start searching
twint.run.Profile(config)

what is wrong?

Environment Details

Using Fedora 28, executing scripts from terminal, ES is in local

[haccer]

Hi @StefanoSetti

I believe this is a bug, I'm going to look into this right now

[pielco11]

Hmmm also strange to me, fixed removing the function and using the same array. Give it a try pip3 install --upgrade -e git+https://github.com/twintproject/twint.git@origin/dev#egg=twint

EDIT: forgot a git+

[StefanoSetti]

Hmmm also strange to me, fixed removing the function and using the same array. Give it a try pip3
install --upgrade -e https://github.com/twintproject/twint.git@origin/dev#egg=twint

@pielco11 Is this a work-around? Because i need to export my project in to containers and i would like to have a stable solution to this problem

[pielco11]

@StefanoSetti Yes, that's a workaround, there are two way: call a function (Pythonic way) and don't; since the issue is calling the function I just moved the code inside the called function into the "mother" function. Others parts of the code (twint/Pandas.py) are calling that function and everything works great, so it's quite strange.

Now everything seems to work as expected, no changes for the client side

[StefanoSetti]

@pielco11 so, this command pip3 install --upgrade -e https://github.com/twintproject/twint.git@origin/dev#egg=twint is something that i want to do in my container set-up right?

[pielco11]

Yes, now everything in aligned in the master branch so just run:
pip3 install --upgrade -e git+https://github.com/twintproject/twint.git@origin/master#egg=twint

When @haccer will upgrade the package you'll have just to do pip3 install --upgrade twint, but nothing changes

[StefanoSetti]

@pielco11 OK, thanks for the help

grazie!

[pielco11]

No problem :+1: , in case do not hesitate to ask

[StefanoSetti]

I will, thanks a lot.
That's my first time working with Elasticsearch and twitter, so it will be an interesrting travel.

[StefanoSetti]

I just did what you say but a new error occurred: ModuleNotFoundError: No module named 'MySQLdb'

[pielco11]

pip install mysqlclient

It should be part of the origin/mysql branch, but when one operates with merge and PR this happens, sorry for the inconvenience

[pielco11]

Deleted the files that were giving you errors

[StefanoSetti]

ok

[StefanoSetti]

I got a big problem, i still can't manage to install pip install mysqlclient the error returned is:

Command "/usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-45_4tg8r/mysqlclient/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-b2x9tv5e-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-45_4tg8r/mysqlclient/

and now nothing works with twint, i return the same error as before: ModuleNotFoundError: No module named 'MySQLdb'

[pielco11]

Ok I removed something around, should work

[StefanoSetti]

so now i do: pip3 install --upgrade twint and everything should work, right?

[pielco11]

Nope, again upgrade from git.... I tried a couple of things and indexing into elasticsearch and I'm not having errors, pip install --upgrade -e git+https://github.com/twintproject/twint.git@origin/master#egg=twint

MySQLdb will not be imported

[StefanoSetti]

now it works!
Thank you @pielco11

[pielco11]

Thank you too and my apologies for the inconvenience

(closing this since everything is working right)