Mailkit: imap.fetch() always returns "The IMAP server replied to the 'FETCH' command with a 'NO' response: FETCH Server error while fetching messages" for one particular message in a mailbox on imap.mail.yahoo.com

[mcoyle012]

This is my first week using MailKit. I've got the latest released version installed.

I was looking at some sample code here https://components.xamarin.com/gettingstarted/mailkit for my initial console application. I'm aiming this at a mailbox on imap.mail.yahoo.com.

I found that I could not use the (0, -1, ...) invocation of fetch because I always got an ImapCommandException (below). So I began hunting that down. I began a binary search to find the failing index. Turns out it was 9972.

I found that:

• retrying after catching the exception didn't do any good--persistent failure, and
• if f I changed which MessageSummaryItems I was fetching, it would no longer throw the exception.

In binary #1 I get this exception every single time on any fetch that includes message 9972. I can retry after catching the exception and it never succeeds.

HResult=-2146233088
Message=The IMAP server replied to the 'FETCH' command with a 'NO' response: FETCH Server error while fetching messages
Source=MailKit
StackTrace:
at MailKit.Net.Imap.ImapFolder.Fetch(Int32 min, Int32 max, MessageSummaryItems items, CancellationToken cancellationToken)
at ConsoleApplication2.Program.Main(String[] args) in C:\Users\ExtSvc-IntDevDocuments\Visual Studio 2015\Projects\ConsoleApplication2\ConsoleApplication2\Program.cs:line 63
at System.AppDomain._nExecuteAssembly(RuntimeAssembly assembly, String[] args)
at System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence assemblySecurity, String[] args)
at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart()
InnerException:

In binary #2, I never get the exception thrown on message 9972.

The only difference between binary #1 and binary #2 is this:

binary #1: always fails
foreach (var summary in inbox.Fetch(i, j, MessageSummaryItems.Full | MessageSummaryItems.UniqueId))

Total messages: 23983
Recent messages: 0
Fetching 9971..9971
("text" "plain" ("charset" "iso-8859-1") NIL NIL "quoted-printable" 5245 NIL NIL NIL NIL 152): {0}
Fetching 9972..9972
Exception thrown: 'MailKit.Net.Imap.ImapCommandException' in MailKit.dll
MailKit.Net.Imap.ImapCommandException: The IMAP server replied to the 'FETCH' command with a 'NO' response: FETCH Server error while fetching messages
at MailKit.Net.Imap.ImapFolder.Fetch(Int32 min, Int32 max, MessageSummaryItems items, CancellationToken cancellationToken)
at ConsoleApplication2.Program.Main(String[] args) in C:\Users\ExtSvc-IntDev\documents\visual studio 2015\Projects\ConsoleApplication2\ConsoleApplication2\Program.cs:line 67: ImapCommandException: {0}
Fetching 9972..9972
Exception thrown: 'MailKit.Net.Imap.ImapCommandException' in MailKit.dll
MailKit.Net.Imap.ImapCommandException: The IMAP server replied to the 'FETCH' command with a 'NO' response: FETCH Server error while fetching messages
at MailKit.Net.Imap.ImapFolder.Fetch(Int32 min, Int32 max, MessageSummaryItems items, CancellationToken cancellationToken)
at ConsoleApplication2.Program.Main(String[] args) in C:\Users\ExtSvc-IntDev\documents\visual studio 2015\Projects\ConsoleApplication2\ConsoleApplication2\Program.cs:line 67: ImapCommandException: {0}
Fetching 9972..9972
Exception thrown: 'MailKit.Net.Imap.ImapCommandException' in MailKit.dll
MailKit.Net.Imap.ImapCommandException: The IMAP server replied to the 'FETCH' command with a 'NO' response: FETCH Server error while fetching messages
at MailKit.Net.Imap.ImapFolder.Fetch(Int32 min, Int32 max, MessageSummaryItems items, CancellationToken cancellationToken)
at ConsoleApplication2.Program.Main(String[] args) in C:\Users\ExtSvc-IntDev\documents\visual studio 2015\Projects\ConsoleApplication2\ConsoleApplication2\Program.cs:line 67: ImapCommandException: {0}
Fetching 9972..9972
Exception thrown: 'MailKit.Net.Imap.ImapCommandException' in MailKit.dll
MailKit.Net.Imap.ImapCommandException: The IMAP server replied to the 'FETCH' command with a 'NO' response: FETCH Server error while fetching messages
at MailKit.Net.Imap.ImapFolder.Fetch(Int32 min, Int32 max, MessageSummaryItems items, CancellationToken cancellationToken)
at ConsoleApplication2.Program.Main(String[] args) in C:\Users\ExtSvc-IntDev\documents\visual studio 2015\Projects\ConsoleApplication2\ConsoleApplication2\Program.cs:line 67: ImapCommandException: {0}

binary #2, always succeeds
foreach (var summary in inbox.Fetch(i, j, MessageSummaryItems.Fast))

Total messages: 23983
Recent messages: 0
Fetching 9971..9971
Fetching 9972..9972
Fetching 9973..9973
Fetching 9974..9974
Fetching 9975..9975
Fetching 9976..9976
Fetching 9977..9977
Fetching 9978..9978

[jstedfast]

Can you get a protocol log (see the FAQ to learn how to get one)? That'll be more useful than stack traces.

Thanks.

[mcoyle012]

You bet.  Give me 10 mins or so ...thanks!

[mcoyle012]

Here's the log, unscrubbed.  Please keep the email addresses contained in the log confidential and off the web (light or dark ;-).  Thanks in advance!

Log generated by

                foreach (var summary in inbox.Fetch(9970, 9980, MessageSummaryItems.Full | MessageSummaryItems.UniqueId))
                {
                    var text = summary.Body as BodyPartText;
                    if (text == null)
                    {
                        var multipart = summary.Body as BodyPartMultipart;
                        if (multipart != null)
                            text = multipart.BodyParts.OfType().FirstOrDefault();
                    }
                    if (text == null)
                        continue;
                    // this will download _just_ the text part
                    var part = inbox.GetBodyPart(summary.UniqueId, text);
                    System.Diagnostics.Debug.WriteLine("{0}", text.ToString()
                        );

Connected to imaps://imap.mail.yahoo.com:993/
S: * OK [CAPABILITY IMAP4rev1 ID MOVE NAMESPACE X-ID-ACLID UIDPLUS LITERAL+ CHILDREN XAPPLEPUSHSERVICE XYMHIGHESTMODSEQ AUTH=PLAIN AUTH=LOGIN AUTH=XOAUTH2 AUTH=XYMCOOKIE AUTH=XYMECOOKIE AUTH=XYMCOOKIEB64 AUTH=XYMPKI] IMAP4rev1 imapgate-1.8.1_01.20166
C: A00000000 AUTHENTICATE PLAIN
S: +
C: XXX
S: A00000000 OK AUTHENTICATE completed - Mailbox size in bytes is 7150780663
C: A00000001 CAPABILITY
S: * CAPABILITY IMAP4rev1 ID MOVE NAMESPACE X-ID-ACLID UIDPLUS LITERAL+ XAPPLEPUSHSERVICE XYMHIGHESTMODSEQ
S: A00000001 OK CAPABILITY completed
C: A00000002 NAMESPACE
S: * NAMESPACE (("" "/")) NIL NIL
S: A00000002 OK NAMESPACE completed
C: A00000003 LIST "" "INBOX"
S: * LIST (\Inbox \HasNoChildren) "/" "Inbox"
S: A00000003 OK LIST completed
C: A00000004 EXAMINE Inbox
S: * 23992 EXISTS
S: * 0 RECENT
S: * OK [UIDVALIDITY 1] UIDs valid
S: * OK [UIDNEXT 333878] Predicted next UID
S: * FLAGS (\Answered \Deleted \Draft \Flagged \Seen $Forwarded $Junk $NotJunk)
S: * OK [PERMANENTFLAGS ()] No permanent flags permitted
S: * OK [HIGHESTMODSEQ 102784]
S: A00000004 OK [READ-ONLY] EXAMINE completed; now in selected state
C: A00000005 FETCH 9971:9981 (UID FLAGS INTERNALDATE RFC822.SIZE ENVELOPE BODY)
[snip]
S: A00000005 NO [UNAVAILABLE] FETCH Server error while fetching messages

[mcoyle012]

And here it is with a single request using inbox.Fetch (9972, 9972 ...)

Connected to imaps://imap.mail.yahoo.com:993/
S: * OK [CAPABILITY IMAP4rev1 ID MOVE NAMESPACE X-ID-ACLID UIDPLUS LITERAL+ CHILDREN XAPPLEPUSHSERVICE XYMHIGHESTMODSEQ AUTH=PLAIN AUTH=LOGIN AUTH=XOAUTH2 AUTH=XYMCOOKIE AUTH=XYMECOOKIE AUTH=XYMCOOKIEB64 AUTH=XYMPKI] IMAP4rev1 imapgate-1.8.1_01.20166
C: A00000000 AUTHENTICATE PLAIN
S: +
C: XXX
S: A00000000 OK AUTHENTICATE completed - Mailbox size in bytes is 7150835943
C: A00000001 CAPABILITY
S: * CAPABILITY IMAP4rev1 ID MOVE NAMESPACE X-ID-ACLID UIDPLUS LITERAL+ XAPPLEPUSHSERVICE XYMHIGHESTMODSEQ
S: A00000001 OK CAPABILITY completed
C: A00000002 NAMESPACE
S: * NAMESPACE (("" "/")) NIL NIL
S: A00000002 OK NAMESPACE completed
C: A00000003 LIST "" "INBOX"
S: * LIST (\Inbox \HasNoChildren) "/" "Inbox"
S: A00000003 OK LIST completed
C: A00000004 EXAMINE Inbox
S: * 23993 EXISTS
S: * 0 RECENT
S: * OK [UIDVALIDITY 1] UIDs valid
S: * OK [UIDNEXT 333879] Predicted next UID
S: * FLAGS (\Answered \Deleted \Draft \Flagged \Seen $Forwarded $Junk $NotJunk)
S: * OK [PERMANENTFLAGS ()] No permanent flags permitted
S: * OK [HIGHESTMODSEQ 102788]
S: A00000004 OK [READ-ONLY] EXAMINE completed; now in selected state
C: A00000005 FETCH 9973 (UID FLAGS INTERNALDATE RFC822.SIZE ENVELOPE BODY)
S: A00000005 NO [UNAVAILABLE] FETCH Server error while fetching messages

[jstedfast]

Hey, just an FYI, but when you reply via email, it doesn't go to me - it goes to the issuer tracker which posts stuff publicly.

I've scrubbed the logs, but keep that in mind in the future.

I should probably add a note about that to the FAQ.

Looking at the logs, I'm not sure what can be done about the "NO" response within MailKit itself. It appears to be a bug in the IMAP server.

That said, for this _particular_ issue, you might be able to work around it by subscribing to the MessageSummaryFetched event which gets emitted whenever the full requested data for a message is fetched. So you could do something like this:

void OnMessageSummaryFetched (object sender, MessageSummaryFetchedEventArgs e)
{
     items.Add (e.Message);
}

try {
    inbox.MessageSummaryFetched += OnMessageSummaryFetched;
    inbox.Fetch (9970, 9980, MessageSummaryItems.Full | MessageSummaryItems.UniqueId);
} catch (ImapCommandException) {
} finally {
    inbox.MessageSummaryFetched -= OnMessageSummaryFetched;
}

Hope that helps.

[samysilva]

I discovered the problem.

The EXPUNGE method does not work correctly on YAHOO servers.
It does not reset the sequential message counter (ID).

To solve the problem, just always work using UID

var query = SearchQuery.All;
foreach (UniqueId uid in inbox.Search(query))
{
   try
   {
      var message = inbox.GetMessage(uid);
      ProcessMailMessage(message);
      inbox.AddFlags(uid, MessageFlags.Deleted, false);
   }
   catch (ImapCommandException ex)
   {
      inbox.AddFlags(uid, MessageFlags.Deleted, false);
      Global.Logger.WriteEntry("Error in GetMessage(" + uid.Id + ") " +
         "and was marked as deleted: " + ex.Message);
   }
   catch (Exception ex)
   {
      throw ex;
   }
}

Additionally, I added a try-catch in the "GetMessage" method. Because; Something eventually occurs in the UIDs that jump large ranges, leaving the previous ranges inaccessible.

Example:

Assuming the "inbox.Search (query)" return: 9001,9002,9003,1234501,1234502,1234503

The first 3 UIDs always return errors, being necessary to delete, and only work with the other UIDs.

Example of how to simulate problems in Yahoo
You can verify that there is no "ID 1"

openssl s_client -connect imap.mail.yahoo.com:993 -crlf
? LOGIN [email protected] PASSWORD
? LIST "" "*"
? SELECT INBOX
? SEARCH ALL
? FETCH 1 BODY[TEXT]

Example of how to get around problems in Yahoo

openssl s_client -connect imap.mail.yahoo.com:993 -crlf
? LOGIN [email protected] PASSWORD
? LIST "" "*"
? SELECT INBOX
? UID SEARCH ALL
? UID FETCH 101178 BODY[TEXT]
? LOGOUT

Message Sequence Numbers vs UID

Each message in IMAP has two numbers: it’s message sequence number, and it’s unique identifier. The unique identifier is pretty self-explanatory, with a few caveats following, and the message sequence number is the relative position from the oldest message in the folder. If messages are deleted, sequence numbers are reordered to fill any gaps. As can be imagined this a source of a lot mistakes because if you’re looping through a list of message sequence numbers ascendingly, deleting messages as you go, you’ll end up deleting the wrong messages.

[jstedfast]

As can be imagined this a source of a lot mistakes because if you’re looping through a list of message sequence numbers ascendingly, deleting messages as you go, you’ll end up deleting the wrong messages.

Indeed! I always recommend using UIDs if at all possible, even when not using Yahoo! IMAP. It's really just the best way to do things for any IMAP server and reduces the likelyhood of problems with both the IMAP server and your own code (we all make mistakes and so UIDs are a great way to make sure that we are referring to the message we mean to refer to).