Python-telegram-bot: schedule_removal, AttributeError: 'bool' object has no attribute 'set'

[Poolitzer]

I get the following error quite consistently with my bot which has quite some different jobs with long(er) names and payloads. As you can see, it uses the persistent jobs snippet, which I suspect to be the problem. I was not able to reproduce it with a more simple dummy bot, and I am not deep enough in the code to find the problem by myself, someone else might?

File "/home/user/gifsupportbot/venv/lib/python3.7/site-packages/telegram/ext/conversationhandler.py", line 330, in handle_update
    timeout_job.schedule_removal()
  File "/home/user/gifsupportbot/venv/lib/python3.7/site-packages/telegram/ext/jobqueue.py", line 408, in schedule_removal
    self._remove.set()
AttributeError: 'bool' object has no attribute 'set'

I use my own forked version. It doesnt affect anything with the jobqueue though.

[Eldinnie]

That snippet does not aquire the lock on the queue. It is quite possible conversationhandler is trying to remove a job that is in the process of being saved.

[Poolitzer]

hmm, it should then probably, so people can use it safely :)

[schcriher]

I was just writing about this, I mention it here (I hope that's right).

The 'save_jobs' function should take the mutex block before iterating over all jobs:

def save_jobs(jq):
    with jq._queue.mutex:
        if jq:
            job_tuples = jq._queue.queue
        else:
            job_tuples = []

        with open(JOBS_PICKLE, 'wb') as fp:
            for next_t, job in job_tuples:
                # rest of the code

I didn't know whether to edit the wiki or not, so I mention it here.

[schcriher]

In addition, in the load_jobs function, the job._job_queue is not restored, that was done by the JobQueue.put function that no longer exists (The JobQueue._put function does not).

• JobQueue.put deprecated: v6.0.0
• JobQueue.put deleted: v10.0.0

[Bibo-Joshi]

@schcriher would you like to update the code snippet accordingly?

[schcriher]

@Bibo-Joshi sure, but what I said in december is not the complete solution, trying to generate a way to reproduce the error with certainty, I found that by deactivating a job manually, the initial error persists. I guess the right solution would be to have a Lock on the job.

On the other hand I found another error (this is easily solved) the job save_jobs_job can be accumulated between restarts (it is saved each time).

If I find a possible solution I edit the wiki with it, if not I will at least put this second error.

---

~_Erratum: job._job_queue is restored, but in Job.job_queue.setter (that was previously done by the JobQueue._put function)_~ (It must be restored manually)

[schcriher]

@Bibo-Joshi I did a proof of concept that works very well

Files: https://gist.github.com/schcriher/e6f04c43c59625bef42613f98882af7a

• jobs_orig.py as it is in the code snippet

• jobs_fix.py with the modifications (both of the problem of this thread and the second problem I mentioned)

• test.sh bash script that runs three times each of the above (emulating several consecutive starts)

• trace.log the output of the previous script, where the occurrence of both problems is observed, not happening in the proposed solution.

If you give me the okey I'll add it to the wiki.

[Bibo-Joshi]

Wow, thanks for your effort! I will have to look into it in detail, which might take a few days.

@Poolitzer are you okay with closing this issue, if @schcriher s solution is added to the wiki?

[Poolitzer]

Sure, go ahead, looks good to me

[Bibo-Joshi]

Had some unexpected waiting time at work, so I had a look. Didn't run it myself but looks good for putting it in the wiki. Please also add a note, that Jobs are a easily scared species and the snippet may still fail in special cases and also that it may not be up to date with master (we have some PRs in the 12.5 milestone that add changes to Jobs …) ;)

[schcriher]

The PoC has an intentional sleep placed on it, to make the error more probable (but they're only for the test).

It would be great if Job is safe for threads (and serializable) in the 12.5 milestone (I will be looking into this).

I already edited the wiki with the new snippet :+1:

[Bibo-Joshi]

Thanks!
Sure, feel free to pr ahead :)