Suitecrm: after upgrading 7.8.13 to 7.8.22 getting Invalid File Reference from attachments in inbound

[bmwtourer]

might be related to #6418

Issue

inbound email do not store attachments properly, they log in database but do not store in filesystem or upload directory.
running IIs should not be a security issue because if I write an email with attachment the outgoing attachment is recorded properly in the notes table and attachment is found in upload directory.

Expected Behavior

store attachments in upload directory as the notes record shows, currently the received attachment appear to be lost.

Actual Behavior

notes table is updated but no file present

Possible Fix

Steps to Reproduce

send and email to an inbound address that creates cases and try to look at the attachment in crm...
Invalid File Reference

high priority

Context

we can no longer process orders received by inbound emails

Your Environment

SuiteCRM Version used: 7.8.22
not related to browser :
MSsql server
IIs php 7.1

[horus68]

Please edit your subject to remove the "urgent" word, no need for those kind of alerts in here!
Same goes for "after upgrading 7.8.13 to 7.8.22 " that should be only in the details.

For later reference, there is a post in the forum by you here:
https://suitecrm.com/suitecrm/forum/installation-upgrade-help/20689-urgent-after-upgrading-7-8-13-to-7-8-22-getting-invalid-file-reference-from-attachements-in-inbound-emails#72936
(There is no confirmation by others yet)

Note: if your issue is related to an open issue its better to close this one and post in the other issue so it can get more relevance for solutions.

[bmwtourer]

(sorry ... urgent thing)
the problem is not solved but I revert to the 7.8.13 code manually, since the problem was discovered 48 hours after the upgrade and data was added to the system I could not revert to backups (site and sql data). so I manually copied the code (not the upload directory) over the 7.8.22 to go back to 7.8.13
I ran "Quick Repair and Rebuild" and the system appears to be working normally as 7.8.13 again.
I am really surprise this problem has not surfaced before because it is important and 7.8.22 is close to month old...
I tried to find solution but it does not seem to be obvious and the inboundemail.php code has been completely revised.

[pgorod]

Does the URL for the attachment seem to be formed correctly?

Can you try disabling anything that might be interfering at the moment the email is imported (like Anti-virus)?

Can you check both suitecrm.log and the PHP Web server log at the time when the file fails to write to upload folder?

[weedek]

I have this problem after upgrade from 7.8.8 to 7.8.23. Attchment from inbound emails are existing in databasem but not saved to upload directory.
any error I noticed in log files is
Mon Nov 19 18:18:05 2018 [12200][1][INFO] Query:UPDATE job_queue SETassigned_user_id='1',name='Check Inbound Mailboxes',date_modified='2018-11-19 17:18:01',scheduler_id='7aad0a9f -f7fa-46b5-bb94-59de6d97245f',execute_time='2018-11-19 17:18:00',status='done',resolution='success',message='Warning [2]: stat(): stat failed for upload/1 fb23951-1ae8-7325-cb88-5bf2f00bf3f2 w /srv/suitecrm/include/upload_file.php w linii 831\nWarning [2]: stat(): stat failed for upload/1fb23951-1ae8-7325-cb88 -5bf2f00bf3f2 w /srv/suitecrm/include/upload_file.php w linii 831\nWarning [2]: stat(): stat failed for upload/1fb23951-1ae8-7325-cb88-5bf2f00bf3f2 w /srv/s uitecrm/include/upload_file.php w linii 831\nWarning [2]: fopen(upload/1fb23951-1ae8-7325-cb88-5bf2f00bf3f2): failed to open stream: No such file or directo ry w /srv/suitecrm/include/upload_file.php w linii 785\nWarning [2]: copy(upload://1fb23951-1ae8-7325-cb88-5bf2f00bf3f2): failed to open stream: \"UploadStr eam::stream_open\" call failed w /srv/suitecrm/modules/InboundEmail/AOPInboundEmail.php w linii 133\nWarning [2]: stat(): stat failed for upload/1fb23951-1a e8-7325-cb88-5bf2f00bf3f2 w /srv/suitecrm/include/upload_file.php w linii 831\nWarning [2]: fopen(upload/1fb23951-1ae8-7325-cb88-5bf2f00bf3f2): failed to op en stream: No such file or directory w /srv/suitecrm/include/upload_file.php w linii 785\nWarning [2]: copy(upload://1fb23951-1ae8-7325-cb88-5bf2f00bf3f2): failed to open stream: \"UploadStream::stream_open\" call failed w /srv/suitecrm/modules/AOP_Case_Updates/CaseUpdatesHook.php w linii 277\n',target='funct ion::pollMonitoredInboxesAOP',data=NULL,requeue=0,retry_count=NULL,failure_count=NULL,job_delay=0,client='CRON2e00c1e6d741c587e17624588796b8c0:12200', percent_complete=NULL WHERE job_queue.id = 'b30d6e1a-de81-2519-4824-5bf2f0898d84' AND deleted=0

[evgu]

We have also detected the same problem after 7.8.20->7.8.24 upgrade. The following is observed:

• Inbound imails arrive correctly in CRM personal inbox.
• Emails can be viewed and email text displays correctly. When new email via CRM emails, the folloing Error messages are observed in crm-log file. (CRM log level is set to Warn).
  Thu Jan 17 10:21:32 2019 [4952][skip][WARN] InboundEmail::connectMailserver: Request ssl is not defined Thu Jan 17 10:21:32 2019 [4952][skip][WARN] Saving error level. Try to remove the error_reporting() function from your code. Thu Jan 17 10:21:32 2019 [4952][skip][ERROR] Pop error level. Try to remove the error_reporting() function from your code. Thu Jan 17 10:21:32 2019 [4952][skip][WARN] InboundEmail::handleEncodedFilename: unable to resolve encoding Thu Jan 17 10:21:32 2019 [4952][skip][WARN] InboundEmail::saveAttachmentBinaries: file not found: cache/modules/Emails/skip-c8b7-60f5-skip-56331d12d8ca/attachments/26aee1a7432306df975db12e7b6848d60 Thu Jan 17 10:21:32 2019 [4952][skip][WARN] InboundEmail::saveAttachmentBinaries: file not found: cache/modules/Emails/skip-c8b7-60f5-skip-56331d12d8ca/attachments/26aee1a7432306df975db12e7b6848d61 Thu Jan 17 10:21:32 2019 [4952][skip][WARN] InboundEmail::handleEncodedFilename: unable to resolve encoding Thu Jan 17 10:21:32 2019 [4952][skip][WARN] InboundEmail::saveAttachmentBinaries: file not found: cache/modules/Emails/skip-c8b7-60f5-skip-56331d12d8ca/attachments/26aee1a7432306df975db12e7b6848d62 Thu Jan 17 10:21:32 2019 [4952][skip][WARN] InboundEmail::saveAttachmentBinaries: file not found: cache/modules/Emails/skip-c8b7-60f5-skip-56331d12d8ca/attachments/26aee1a7432306df975db12e7b6848d63 Thu Jan 17 10:21:32 2019 [4952][skip][WARN] InboundEmail::handleEncodedFilename: unable to resolve encoding Thu Jan 17 10:21:33 2019 [4952][skip][WARN] InboundEmail::saveAttachmentBinaries: file not found: cache/modules/Emails/skip-c8b7-60f5-skip-56331d12d8ca/attachments/26aee1a7432306df975db12e7b6848d64 Thu Jan 17 10:21:33 2019 [4952][skip][WARN] InboundEmail::saveAttachmentBinaries: file not found: cache/modules/Emails/skip-c8b7-60f5-skip-56331d12d8ca/attachments/26aee1a7432306df975db12e7b6848d65 Thu Jan 17 10:21:33 2019 [4952][skip][WARN] InboundEmail::handleEncodedFilename: unable to resolve encoding Thu Jan 17 10:21:33 2019 [4952][skip][WARN] InboundEmail::saveAttachmentBinaries: file not found: cache/modules/Emails/skip-c8b7-60f5-skip-56331d12d8ca/attachments/26aee1a7432306df975db12e7b6848d66 Thu Jan 17 10:21:33 2019 [4952][skip][WARN] InboundEmail::saveAttachmentBinaries: file not found: cache/modules/Emails/skip-c8b7-60f5-skip-56331d12d8ca/attachments/26aee1a7432306df975db12e7b6848d67 Thu Jan 17 10:21:33 2019 [4952][skip][WARN] InboundEmail::connectMailserver: Request ssl is not defined Thu Jan 17 10:21:33 2019 [4952][skip][WARN] Saving error level. Try to remove the error_reporting() function from your code. Thu Jan 17 10:21:33 2019 [4952][skip][ERROR] Pop error level. Try to remove the error_reporting() function from your code.
• When clicking on email attachment (from within the CRM emails) a new blank page opens displaying text "Invalid File Reference". Nothing is logged in the crm-log file.
• When attaching email to a CRM record (e.g. Importing email to account (from CRM emails)), email record attaches to the account. The following is logged to crm error log file:
  Thu Jan 17 10:24:45 2019 [4919][skip][WARN] InboundEmail::connectMailserver: Request ssl is not defined Thu Jan 17 10:24:45 2019 [4919][skip][WARN] Saving error level. Try to remove the error_reporting() function from your code. Thu Jan 17 10:24:46 2019 [4919][skip][ERROR] Pop error level. Try to remove the error_reporting() function from your code. Thu Jan 17 10:24:46 2019 [4919][skip][WARN] InboundEmail::handleEncodedFilename: unable to resolve encoding Thu Jan 17 10:24:46 2019 [4919][skip][WARN] InboundEmail::saveAttachmentBinaries: file not found: upload://28f4baa9-8aee-d6fe-083a-ae4057620475 Thu Jan 17 10:24:46 2019 [4919][skip][WARN] InboundEmail::saveAttachmentBinaries: file not found: upload://3a56c0ab-d421-c3f6-40c1-5c405eab77dd Thu Jan 17 10:24:46 2019 [4919][skip][WARN] InboundEmail::handleEncodedFilename: unable to resolve encoding Thu Jan 17 10:24:46 2019 [4919][skip][WARN] InboundEmail::saveAttachmentBinaries: file not found: upload://42c9cfcb-a1ee-663f-5b50-5c40571af6df Thu Jan 17 10:24:46 2019 [4919][skip][WARN] InboundEmail::saveAttachmentBinaries: file not found: upload://56ec8e78-8ee6-3e67-7a00-5c4057aa5b1b Thu Jan 17 10:24:46 2019 [4919][skip][WARN] InboundEmail::handleEncodedFilename: unable to resolve encoding Thu Jan 17 10:24:46 2019 [4919][skip][WARN] InboundEmail::saveAttachmentBinaries: file not found: upload://5f82a62b-eds2-89ce-c25c-5c40574e33c0 Thu Jan 17 10:24:46 2019 [4919][skip][WARN] InboundEmail::saveAttachmentBinaries: file not found: upload://682b90fe-3120-26bb-9d43-5c40574915ab Thu Jan 17 10:24:46 2019 [4919][skip][WARN] InboundEmail::handleEncodedFilename: unable to resolve encoding Thu Jan 17 10:24:46 2019 [4919][skip][WARN] InboundEmail::saveAttachmentBinaries: file not found: upload://70c6f86d-732e-c213-1427-5c4057ae9876 Thu Jan 17 10:24:46 2019 [4919][skip][WARN] InboundEmail::saveAttachmentBinaries: file not found: upload://791ac730-889e-124f-7d13-5c4057ea5e51 Thu Jan 17 10:24:46 2019 [4919][skip][DEPRECATED] Formatting correction: Emails->date_sent had formatting automatically corrected. This will be removed in the future, please upgrade your external code Thu Jan 17 10:24:46 2019 [4919][skip][WARN] CaseUpdatesHook: saveEmailUpdate: Not a create case or wrong parent type

We have a CRM instance available where we could assist with testing a bug-fix. This bug is of somewhat high priority to us as it is impacting client data integrity for received emails.

SuiteCRM 7.8.24
MySQLd 5.6.42
PHP 7.0.33
CentOS Linux 7.6.1810

@Dillon-Brown, do you have an indication when this defect could tiaged for development? Thanks in advance!!

[fcorluka]

I have this problem on 7.8.23, just installed fresh 7.8.25 (latest LTS), tried Inbound Mails and the problem is here.

If this is not urgent, then I don't know what would be.
@pgorod For your question about URL, the URL is fine. File is the one that is missing.
Funny side, when email is sent OUT of CRM with attachement everything is fine, I can access file and download it.

Not working URL:
http://localhost/7825/index.php?entryPoint=download&id=81ef07bd-8ebe-80ab-d329-5c52feefe742&type=Notes

Working URL:
http://localhost/7825/index.php?entryPoint=download&id=3553b949-7113-ad51-8869-5c5302d0015d&type=Notes

When I go into UPLOADS folder I can see all the files that are working, but none of not working. Simply files are not placed on file system.
SuiteCRM 7.8.23, 7.8.25
PHP 7.2
LinuxMint 18.1

[pgorod]

I am suspicious of that message in the logs

https://github.com/salesagility/SuiteCRM/blame/7.8.x/modules/InboundEmail/InboundEmail.php#L3583

The code there looks quite different from what we have in 7.11, there has been work on this and I wonder if some fix needs to be made also in 7.8.x

I should be working on something else right now, can any of you hook up a debugger to this and check what's going on?

[fcorluka]

Hi guys,

my colleague and I have FOUND THE PROBLEM !!!! (also have a quick fix for CRM 7.8.23)

Problem is located in modules/InboundEmail/InboundEmail.php , function 'saveAttachmentBinaries'.
There are 2 times IF clause with the same condition "if(!file_exists($uploadDir.$fileName))". Because of second condition the attachment file is never saved to file system.
Second one is not neccesarry and saving is placed in 'elseif' part?!
Fix for this:
(old)

   if (!file_exists($file)) {
                            LoggerManager::getLogger()->warn('InboundEmail::saveAttachmentBinaries: file not found: ' . $file);
                        } elseif(file_put_contents($file, $msgPart)) 

(new)

 if (/*!*/ file_exists($file)) {
                            LoggerManager::getLogger()->warn('InboundEmail::saveAttachmentBinaries: file not found: ' . $file);
                        } elseif(file_put_contents($file, $msgPart)) 

Please can someone of developers look at it, and compare it to version 7.10.12. (this is how we discovered it).
Br,
Filip

[evgu]

Tested this on 7.8.24 and resolves the issue. Emial attachments can now be viewed and uploaded correctly.
@Dillon-Brown can you please check with developers to take in this change in next release?

[Dillon-Brown]

Hi @fcorluka, thanks for the assistance! i'll take a look at this today and see about getting this merged in for the next release.

[TeNNoX]

Great that this is fixed - but I have the question: how can I fix attachments of emails that have been loaded with the buggy code?

Delete them and mark them as unread again in the mailbox? Does that work?
Or is there another way to trigger re-downloading of attachments?