Mailkit: SearchQuery.DeliveredAfter(DateTime.Now.AddHours(-1)) is not working properly.

[anmolk171]

Hi @jstedfast ,

• What were you trying to do?
  I am trying to retrieve only last hour mail in gmail(as my organization is using gmail) but the catch is I received mail from different time zone like UK, USA and other as well and I am in India( GMT +5:30)

• What happened?
  This query return the unusual result like it is giving last three days mails abruptly.

• What did you expect to happen?
  It should return only last one hour mail

• Step-by-step reproduction instructions and/or a simple test case.
  Here is the code which I am using.
  var query = SearchQuery.DeliveredAfter(DateTime.Now.AddHours(-1));

          foreach (var uid in inbox.Search(query))
          {
              var message = inbox.GetMessage(uid);
  
              Console.WriteLine("[match] {0}: {1} {2}", uid, message.Subject,message.Date);
          }
  

If you got an exception, please include the exception Message and StackTrace.

---

Result is like this as I run this on 03-Feb-2019 around 3:00 PM (GMT +5:30) but I am getting mails of 02 Feb as well:
Total messages: 3903
Recent messages: 0
[match] 36794: RE: Database and network maintenance this weekend 02-02-2019 14:47:56 -05:00
[match] 36795: DEV - Status of Task: Polaris to SecMaster Complete Extract - FAILED 02-02-2019 12:11:31 -08:00
[match] 36796: RE: Database and network maintenance this weekend 02-02-2019 20:11:38 +00:00
[match] 36797: RE: (Case 501384)Re: Polaris BDC Wells Interest Rate 02-02-2019 20:14:04 +00:00
[match] 36798: RE: Database and network maintenance this weekend 02-02-2019 13:13:37 -07:00
[match] 36799: *EOD Refresh Success - PROD 02-02-2019 15:16:50 -05:00
[match] 36800: RE: Database and network maintenance this weekend 02-02-2019 15:31:00 -05:00
[match] 36801: Deadlock Report: Detected on :NYPRDPOL-SQL2 02-02-2019 15:31:51 -05:00
[match] 36802: Deadlock Report: Detected on :NYPRDPOL-SQL2 02-02-2019 15:32:10 -05:00
[match] 36803: Deadlock Report: Detected on :NYPRDPOL-SQL2 02-02-2019 15:32:30 -05:00
[match] 36804: Deadlock Report: Detected on :NYPRDPOL-SQL2 02-02-2019 15:32:57 -05:00
[match] 36805: Deadlock Report: Detected on :NYPRDPOL-SQL2 02-02-2019 15:33:22 -05:00
[match] 36806: Deadlock Report: Detected on :NYPRDPOL-SQL2 02-02-2019 15:33:28 -05:00
[match] 36807: RE: Database and network maintenance this weekend 02-02-2019 15:33:41 -05:00
[match] 36808: Deadlock Report: Detected on :NYPRDPOL-SQL2 02-02-2019 15:33:47 -05:00
[match] 36809: Deadlock Report: Detected on :NYPRDPOL-SQL2 02-02-2019 15:34:35 -05:00
[match] 36810: ALERT: SQL Server Integration Services 12.0 Service is Down on NYPRDPOL-SQL2 02-02-2019 15:40:14 -05:00
[match] 36811: RE: Database and network maintenance this weekend 02-02-2019 15:40:49 -05:00
[match] 36812: ALERT: SQL Server Agent (MSSQLSERVER) Service is Down on NYPRDPOL-SQL2 02-02-2019 15:41:50 -05:00
[match] 36813: ALERT: SQL Server (MSSQLSERVER) Service is Down on NYPRDPOL-SQL2 02-02-2019 15:41:50 -05:00
[match] 36814: Accessing Relationship Manager 02-02-2019 20:57:15 +00:00
[match] 36815: PROD : 0 Procs modified today. 02-02-2019 16:00:02 -05:00
[match] 36816: Success: BBG Delta Feed Refresh 02-02-2019 16:31:07 -05:00
[match] 36817: RE: Database and network maintenance this weekend 02-02-2019 16:33:57 -05:00
[match] 36818: RE: Database and network maintenance this weekend 02-02-2019 21:39:32 +00:00
[match] 36819: RE: Accessing Relationship Manager 02-02-2019 21:46:56 +00:00
[match] 36820: RE: Database and network maintenance this weekend 02-02-2019 16:53:41 -05:00
[match] 36821: RESET: SQL Server Integration Services 12.0 Service is Up on NYPRDPOL-SQL2 02-02-2019 16:56:17 -05:00
[match] 36822: RESET: SQL Server (MSSQLSERVER) Service is Up on NYPRDPOL-SQL2 02-02-2019 16:56:58 -05:00
[match] 36823: RESET: SQL Server Agent (MSSQLSERVER) Service is Up on NYPRDPOL-SQL2 02-02-2019 16:56:58 -05:00
[match] 36824: RE: Database and network maintenance this weekend 02-02-2019 16:58:02 -05:00
[match] 36825: PROD: Transaction Load Failed 02-02-2019 22:01:16 +00:00
[match] 36826: PROD: Cash Balance Load Passed 02-02-2019 22:01:29 +00:00
[match] 36827: PROD:Position Load Passed 02-02-2019 22:02:11 +00:00
[match] 36828: PROD: Position Loader Failed 02-02-2019 22:02:28 +00:00
[match] 36829: RE: Database and network maintenance this weekend 02-02-2019 22:25:45 +00:00
[match] 36830: RE: Accessing Relationship Manager 02-02-2019 22:27:19 +00:00
[match] 36831: Re: (Case 501384)Re: Polaris BDC Wells Interest Rate 02-02-2019 15:43:08 -07:00
[match] 36832: Re: (Case 501384)Re: Polaris BDC Wells Interest Rate 02-02-2019 15:44:00 -07:00
[match] 36833: RE: Database and network maintenance this weekend 02-02-2019 15:47:20 -07:00
[match] 36834: Succeeded : [PROD] No. of Stuck Jobs Found - 0 02-02-2019 19:01:26 -05:00
[match] 36835: MIO(PROD) - IVP Security Master Task Status on 02/02/2019 at 19:05 02-02-2019 19:05:08 -05:00
[match] 36836: RE: Database and network maintenance this weekend 03-02-2019 00:17:03 +00:00
[match] 36837: NMC Status WSO_PositionExtract 02-02-2019 19:23:48 -05:00
[match] 36838: Re: Accessing Relationship Manager 03-02-2019 01:04:40 +00:00
[match] 36839: MIO(PROD) - IVP Security Master Task Status on 02/02/2019 at 20:30 02-02-2019 20:30:09 -05:00
[match] 36840: MIO(PROD) - IVP Security Master Task Status on 02/02/2019 at 20:50 02-02-2019 20:50:08 -05:00
[match] 36841: Succeeded : [PROD] No. of Stuck Jobs Found - 0 02-02-2019 22:01:29 -05:00
[match] 36842: DEV - Status of Task: Polaris to SecMaster Complete Extract - FAILED 02-02-2019 19:43:21 -08:00
[match] 36843: WaterFall (PROD) - IVP Security Master Task Status on 02/02/2019 at 22:50 02-02-2019 22:50:14 -05:00
[match] 36844: *EOD Refresh Success - PROD 02-02-2019 22:56:03 -05:00
[match] 36846: MIO(PROD) - IVP Security Master Task Status on 02/02/2019 at 23:30 02-02-2019 23:30:10 -05:00
[match] 36847: EOD Reporting Batch - Success 02-02-2019 23:32:11 -05:00
[match] 36849: 3 new notifications since 7:17 AM (Sunday, Feb 3) 02-02-2019 23:47:18 -05:00
[match] 36850: WaterFall (PROD) - IVP Security Master Task Status on 02/02/2019 at 23:50 02-02-2019 23:50:16 -05:00
[match] 36851: PROD : 0 Procs modified today. 02-02-2019 23:55:05 -05:00
[match] 36852: PROD : 0 Procs modified today. 03-02-2019 00:00:00 -05:00
[match] 36853: Has Position Extract - Success - PROD 03-02-2019 00:20:41 -05:00
[match] 36854: Polaris SecMaster Has Position count and recon 03-02-2019 00:40:27 -05:00
[match] 36855: Security count of Has position TRUE success - PROD 03-02-2019 00:40:36 -05:00
[match] 36856: Succeeded : [PROD] No. of Stuck Jobs Found - 0 03-02-2019 01:01:33 -05:00
[match] 36857: Success - Recon - VPM Position Extract for PB - MonthEnd 03-02-2019 01:05:15 -05:00
[match] 36858: Release notes and test cases for new Add In 03-02-2019 11:38:07 +05:30
[match] 36859: WaterFall (PROD) - IVP Security Master Task Status on 02/03/2019 at 01:30 03-02-2019 01:30:19 -05:00
[match] 36860: [PROD] VPM Hourly Refresh Successful 03-02-2019 01:33:18 -05:00
[match] 36861: FHCNYIVPAPP01: Cash Balance Load Passed Successfully 02-02-2019 22:35:07 -08:00
[match] 36862: Succeeded : [PROD]Polaris-VPM Prime Broker Recon 03-02-2019 01:42:09 -05:00
[match] 36863: Succeeded : [PROD]Polaris-VPM Strategy Recon 03-02-2019 01:42:09 -05:00
[match] 36864: Succeeded : [PROD]Polaris-VPM Fund Recon 03-02-2019 01:42:08 -05:00
[match] 36865: Succeeded : [PROD]Polaris-VPM Rate Recon 03-02-2019 01:42:18 -05:00
[match] 36866: Succeeded : [PROD]Polaris-VPM Security Symbol Recon 03-02-2019 01:42:45 -05:00
[match] 36867: Failed : [Prod]Polaris-VPM SecMaster Data Recon 03-02-2019 01:43:01 -05:00
[match] 36868: Succeeded :[Prod]Polaris-VPM Nav Recon 03-02-2019 01:47:01 -05:00
[match] 36869: Succeeded [Prod]Polaris-VPM Position Denorm Recon Hourly 03-02-2019 01:50:10 -05:00
[match] 36870: PROD: Risk Measures Staging Archival Success 03-02-2019 01:50:24 -05:00
[match] 36871: Succeeded [Prod]Polaris-VPM PnL Denorm Recon Hourly 03-02-2019 01:59:43 -05:00
[match] 36872: [PROD] VPM Recon Task successful 03-02-2019 01:59:45 -05:00
[match] 36873: Succeeded :[PROD] Polaris - VPM Transaction Recon 03-02-2019 02:00:30 -05:00
[match] 36874: Succeeded : [PROD]Period PNL Recon 03-02-2019 02:03:07 -05:00
[match] 36875: [PROD] Load Hourly VPM Data Start Of Day Succeeded. 03-02-2019 02:03:21 -05:00
[match] 36876: Success 02-02-2019 23:52:03 -08:00

---

I have sent you the logs at [email protected] because it contain some sensitive information with subject "Logs for SearchQuery.DeliveredAfter"

[jstedfast]

The DeliveredAfter search query only makes use of the date and not the time (this is a limitation of the IMAP protocol).

What you need is https://tools.ietf.org/html/rfc5032, but unfortunately, GMail does not support that extension.

[anmolk171]

Can I use POP to do that.

[jstedfast]

POP doesn't support any type of search feature.

[lukalule]

@jstedfast Hey!
What practice then would you then recommend for pulling pulling large amounts of e-mails?
I'm just getting into this mail thing and I'm torn between Pop3 and IMAP, as Pop3 on paper looks to better suit my needs but it doesnt have filter capabilities. Then again, IMAP has some search capabilities but I can't very well make use of them as they, for example, don't support filtering by Time part of the DateTime.
What I'm trying to do is connect to Exchange server 2013 on premise and pull ALL (I don't need all mails, rather just ones I hadn't already pulled) the company's e-mails every minute or so, and store it in temporary database as we're switching to new systems. Doesn't seem very efficient pulling all the mails every minute. Maybe the Date part of DateTime in IMAP filter would suffice for just getting that day's worth of e-mails?
And what would be the best practice going about checking and comparing UIDs? Compare the newly pulled ones against ALL the old ones as in this answer of yours?
Cheers

[jstedfast]

If you keep track of the most recent UID that you have processed, you can query for messages with a UID newer than that.

UIDs always increase, so newer messages always have higher integer values.

So all you really need to keep track of locally is the highest UID you've processed and the UIDVALIDITY value (folder.UidValidity) so that you can make sure that you are looking at the same folder (if the user deletes the folder and recreates a folder with the same name, the UIDVALIDITY will be different).

[lukalule]

@jstedfast Hey, thanks for the reply. Been a couple days since I could get back to this.

Ok that part is clear. But what when/if UIVALIDITY changes? Should I just drop my database (I'm storing mails in SQL Server) and and re-populate it with updated email objects with new UID/UIDVALIDITY values, from scratch (the beginning of mailbox)?
I can't find it now but it was somewhere in the IMAP documentation, I think I'd read that if UIDVALIDITY changes, the client SHOULD or MUST (can't remember which keyword was it) discard its cache of UIDs and re-query the server.

• Does that mean that all the data I had stored in the database thus far (until UIDVALIDITY changed) would basically be useless?
• And then, would the way to go about it be deleting the all records from the database, re-querying the server and storing all the new e-mails and their respective UIDs to database anew?

Maybe I'm overthinking this since I'd read that UIDVALIDITY rarely changes, but still I'd have to be prepared for that.
Referencing e-mails via message-ID header isn't recommended I guess? As it's not guaranteed to be there, per IMAP spec.

Sorry as this is not a MailKit issue per se, rather a design one. But I'm using all the resources I can find, as the info about this is rather obscure...

[jstedfast]

But what when/if UIVALIDITY changes? Should I just drop my database (I'm storing mails in SQL Server) and and re-populate it with updated email objects with new UID/UIDVALIDITY values, from scratch (the beginning of mailbox)?

Yes, exactly. If and when the UIDVALIDITY changes, you need to treat it as if it is an entirely new folder that you've never seen before.

I think I'd read that if UIDVALIDITY changes, the client SHOULD or MUST (can't remember which keyword was it) discard its cache of UIDs and re-query the server.

That is correct.

Does that mean that all the data I had stored in the database thus far (until UIDVALIDITY changed) would basically be useless?

Yes.

And then, would the way to go about it be deleting the all records from the database, re-querying the server and storing all the new e-mails and their respective UIDs to database anew?

Correct.

Maybe I'm overthinking this since I'd read that UIDVALIDITY rarely changes, but still I'd have to be prepared for that.
Referencing e-mails via message-ID header isn't recommended I guess? As it's not guaranteed to be there, per IMAP spec.

Generally speaking, it's "rare" - but it also depends on how the end-user behaves. There are 2 scenarios where the UIDVALIDITY is likely to change.

1. Server admins have to restore from a backup or something else goes completely borked on the server. (rare... hopefully)
2. User deletes a folder and recreates it. (rare for most users, but more likely)

FWIW, it's possible for users to delete even the INBOX folder. In this case, however, the server will immediately recreate it.

Consider this possibility as well: if the user has configured server-side filtering via the Sieve protocol or some other way, if the user then deletes a folder that the filters are configured to deliver mail to, then the filter might recreate the folder.

It's also possible that a user deletes a folder and then decides to recreate it.