Telethon: All dialogs

[thamma]

Hey,

I'm trying to obtain all chats from a users history using the GetDialogsRequest.
The request provides a limit parameter which is server-sided capped at 100 (which was also discussed in another issue).

Thus, to obtain all chat entities, I'd like to move a sliding window using some offset. The https://core.telegram.org/method/messages.getDialogs method however differs in arguments from GetDialogsRequest (self, offset_date, offset_id, offset_peer, limit, exclude_pinned=None).

I can as mentioned, obtain the first 100 chats, but even if i provide a valid entity es offset_peer (or any positive integer for offset_id, I do not move the sliding window. Yet I don't see another way of providing the proper offset.

Am I using these methods wrongly? Can you provide a minimal working example of getDialogs using an offset?

Thanks for your time

[h-sameri]

Am I using these methods wrongly? Can you provide a minimal working example of getDialogs using an offset?

i dont think so but the only parameter that works is offset_date.
this issue is already mentioned in wiki: https://github.com/LonamiWebs/Telethon/wiki/Retrieving-all-dialogs

from telethon.tl.functions.messages import GetDialogsRequest
from telethon.tl.types import InputPeerEmpty

dialogs = []
users = []
chats = []

last_date = None
chunk_size = 20
while True:
    result = client(GetDialogsRequest(
                 offset_date=last_date,
                 offset_id=0,
                 offset_peer=InputPeerEmpty(),
                 limit=chunk_size
             ))
    dialogs.extend(result.dialogs)
    users.extend(result.users)
    chats.extend(result.chats)
    last_date = min(msg.date for msg in result.messages)
    if not result.dialogs:
        break

[Lonami]

Thanks @h-sameri that was the right answer :)

[ghost]

I have several issues with the code posted above. Sometimes, I get an error on last_date = min... because len(result.messages) == 0. Turns out the dialog types are DialogsSlice. The only usable property is count, and that doesn't provide any data to increase the offset to avoid getting the same thing returned again. The second issue is that I'm getting a FloodWaitError but it completely freezes execution, and doesn't provide any info until I Ctrl-C to exit.

Traceback (most recent call last):
  File "/var/www/html/Telethon/telethon/telegram_bare_client.py", line 583, in _invoke
    raise next(x.rpc_error for x in requests if x.rpc_error)
telethon.errors.rpc_error_list.FloodWaitError: A wait of 29 seconds is required

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "myfile.py", line 82, in <module>
    limit=chunk_size
  File "/var/www/html/Telethon/telethon/telegram_bare_client.py", line 486, in __call__
    result = self._invoke(call_receive, request, ordered=ordered)
  File "/var/www/html/Telethon/telethon/telegram_bare_client.py", line 612, in _invoke
    sleep(e.seconds)

So it appears your code isn't able to sleep as it's supposed to :/

[Lonami]

I don't have many dialogs but the sleep should be fine. How many dialogs fo you have? I can't see min() date in the current Telethon's implementation, it uses [-1].

[ghost]

I probably have about 100

[Lonami]

That should be a single request, there's no need to sleep. Are you not using client.get_dialogs()?

[ghost]

client.iter_dialogs

[Lonami]

@RickyFisher98 yeah that should work.

[ghost]

I don't understand how things work for other people but not me when I'm using the exact same code.

I'm using Python 3.5.3

[Lonami]

Same, I would love reproducible bugs. 100 or less dialogs means no slice. A slice means there will always be another non-empty one if it returns the limit…

[ghost]

I have a lot of dialogs but I don't know how many. I have no way to tell. This is the code I'm using:

from telethon.tl.functions.messages import GetDialogsRequest
from telethon.tl.types import InputPeerEmpty

last_date = None;
chunk_size = 20

while True:
        result = client(GetDialogsRequest
        (
                offset_date=last_date,
                offset_id=0,
                offset_peer=InputPeerEmpty(),
                limit=chunk_size
        ))

        if (len(result.messages) == 0): # because min complains on DialogsSlice objects
                print('zero messages')
                print(result.stringify())
                continue

        last_date = min(msg.date for msg in result.messages)

        for chat in result.chats:
                if hasattr(chat, 'entity'):
                        print(chat.entity.title if hasattr(chat.entity, 'title') else '(no title)');
                else:
                        print(str(chat.id) + ' (no entity)')

        if not result.dialogs:
                break

This produces lots of xxxxxxxx (no entity) and

zero messages
DialogsSlice(
        messages=[
        ],
        dialogs=[
        ],
        count=251,
        users=[
        ],
        chats=[
        ]
)

[Lonami]

I have no way to tell.

You just posted count=251 though? That should be 3 requests…

[ghost]

I have no idea what a DialogsSlice is. With the chunk size set to 20, it's just spamming
zero messages DialogsSlice( messages=[ ], dialogs=[ ], count=251, users=[ ], chats=[ ] )

until the end of time because last_date isn't incrementing, because there are no messages in DialogsSlices. The same with chunk size 100.

[Lonami]

But client.iter_dialogs() fails? What's your code using this method?

[ghost]

def get_all_chats(type = None, limit = None, offset_date = None):
        global client
        array_chats = []

        for dialog in client.iter_dialogs(limit, offset_date):
                if (not type is None) and (not isinstance(dialog.entity, type)):
                        continue

                array_chats.append(dialog)

        return array_chats

get_all_chats()

This just hangs forever, no output whatsoever.

[Lonami]

What does DEBUG logging say?

[ghost]

If I Ctrl C while it's hanging:

Traceback (most recent call last):
  File "start.py", line 70, in <module>
    get_all_chats(None, None, None)
  File "start.py", line 53, in get_all_chats
    for dialog in client.iter_dialogs(limit, offset_date):
  File "/var/www/html/Telethon/telethon/telegram_client.py", line 629, in iter_dialogs
    r = self(req)
  File "/var/www/html/Telethon/telethon/telegram_bare_client.py", line 486, in __call__
    result = self._invoke(call_receive, request, ordered=ordered)
  File "/var/www/html/Telethon/telethon/telegram_bare_client.py", line 543, in _invoke
    self._sender.connection.get_timeout()
  File "/usr/lib/python3.5/threading.py", line 549, in wait
    signaled = self._cond.wait(timeout)
  File "/usr/lib/python3.5/threading.py", line 297, in wait
    gotit = waiter.acquire(True, timeout)
KeyboardInterrupt

[Lonami]

Version?

[ghost]

What of?

[Lonami]

Well you're having troubles with the library so I'd like to know if you're using an up-to-date version of this library ^^'

pip show telethon should do.

[ghost]

pip show telethon : -bash: pip: command not found
pip3 show telethon hangs for about 10 seconds, and outputs nothing

[Lonami]

…how did you install the library then if pip can't find it?

[ghost]

No clue. I found version.py:

0.19.1.6

[Lonami]

You must have installed the library somehow! But v0.19.1.6 should definitely be working…

[ghost]

Have you tried my exact code with an account of yours?

[Lonami]

Definitely. client.iter_dialogs() works flawlessly.

[ghost]

I mean have you tried the code I posted above which I wrote myself?

[Lonami]

No but you ultimately use the method that does work for me, so I assume it works too.

[Lonami]

Can you move this to @TelethonChat? I'm out of ideas and can't test now.

[ghost]

If I limit it to 100, it works. With no limit, it hangs forever.

[Kyle2142]

for reference, code in this comment works on my side, with nothing more than aesthetic changes