Suitecrm: Email-Template/Confirm Opt in: Email Template Variables not replaced in "Confirmed Opt In EMail"

[edvinkuric]

Hello,
currently i have the issue that my Email-Template "Confirmed Opt In Template" does not replace all variables correctly.
When a new user signs up via my WebToPerson-Form, a Lead is getting created and a Confirmed Opt-In Mail is sent, but most variables within this template are ignored, even though the correct values are set in the contact, as seen in the following screenshot:

In this template, i use default-fields as well as custom-fields.
I created a custom field called "anrede", which should represent the text which is sent in the mail, based on gender (which is filled by the form + an additional workflow-process when a new lead-entry is added).

The WebToPerson-Form fills the following fields:
-) Salutation
-) First_name
-) Last_name
-) Department
-) "anrede_c" (translates to "email_salutation_c")

The moduleDir for the WebToPerson-Form is "Leads".

For debugging-purposes, i tried to set the following:

Issue

Here is a screenshot of my current Template:

As you can see, multiple Variables from "contact"- and "lead"-module are used (for debugging purposes)

When the mail is sent automatically by the system (by submitting the WebToLead-Form), then the email only populates "contact_first_name" and "contact_last_name", like seen in the screenshot:

When the mail is sent manually (via UI-Menu), the "lead"-fields are populated (instead of contact), as seen in this screenshot:

The Issue is:

• Custom fields should be populated correctly in the email-template, when sending auto-emails like "confirmed opt-in"
• Default fields should be populated correctly in the email-template (firstname + lastname already working, but department isn't), when sending auto-emails like "confirmed opt-in"

Expected Behavior

These fields should be populated in the Email-Template in the same way, independently of the creation (WebToPerson vs. Manual Insert)

Actual Behavior

Incoherent Implementations of EMail-template-parsing in different places (sometimes "Leads"-fields are used, sometimes "Contacts"-fields are used)

The log contains no useful informations about this issue.

Possible Fix

No idea how to fix this issue - i can help if you can point me to some files

Steps to Reproduce

1. Setup SuiteCRM + Working Email-Server within SuiteCRM + Activate "Confirmed Opt-In" Setting in Administration Settings + Create Email-Opt-In-Template like described in the screenshot

Szenario A
A1. Setup a WebToPerson-Form, which successfully adds an entry to your Leads
A2. Check Email, when submitting the WebToPerson-Form --> you should get mail instantly
A3. used Variables within the Template should be replaced (department, anrede_c)

Szenario B
B1. Create Lead manually via SuiteCRM-UI and fill in the required fields for the variable-substitution
B2. Send Opt-In-Email.
B3. Check Email, only "lead"-variables are replaced within the template.

C. Compare Differences between A3 and B3.

Context

I am trying to create an email-template-text based on person-field-values.
The custom-field in the email-template is only needed because i cannot do conditional logic in the email-template directly.
Even though my person-data is saved in the system, i cannot use it by any means.
It would be great, if i somehow could create e.g. a gender-based text-template for Mails (German salutations differ based on gender)

Your Environment

Your Environment
SuiteCRM Version used: 7.11.15, Sugar Version 6.5.25 (Build 344) ( also tried 7.11.13 )
Browser: Google Chrome Version 83.0.4103.116 (Official build) (64-Bit)
Env: Docker-Engine 19.03.8, docker-compose 1.25.5
PHP 7.3.14 (cli) ( NTS ), Zend Engine v3.3.14, with Zend OPcache v7.3.14
DB: mysql Ver 15.1 Distrib 10.4.11-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2
OS: Ubuntu 18.04, SuiteCRM used in a Docker-Container

It would be great if someone could confirm this issue and maybe point me to some files, where/how this problem could be fixed.

[pgorod]

Some background reading related to this: #8315

Those gender-specific expressions you ask for, that would require some mechanism to evaluate conditions like what I've been doing for an add-on.

A work-around would be to prepare a custom field before-hand, with a value set by a workflow as appropriate (_Damen_ / _Herren_ or whatever) and then use it directly in the template.

[edvinkuric]

@pgorod thanks for the reply.
I tried already the custom-field-thing with the field "email_anrede_c" (as mentioned in the initial description), but the data is never populated in the template (even when setting this data via the WebToPerson-form - so your proposed workaround didn't work for me :(.
When this didn't work, i tried to save my data into the "department"-field (which isn't custom) - but this field also wasn't populated in the email-template when using the WebToPerson-form.

Maybe your mentioned tool is a good replacement for that issue - but not sure how i can test it, since this tool you mention is still in alpha and released with SuiteCRM 7.9?

[pgorod]

I _think_ it's going through here
https://github.com/salesagility/SuiteCRM/blob/master/modules/EmailMan/EmailMan.php#L1468-L1475
which in fact doesn't replace any variables except contact name... 👎

The code just isn't there to do any more replacements.

Do you mind asking me about PowerReplacer in the SuiteCRM forums? Just open a new thread here, for example.

It's just because here on GitHub it should be only about Core development stuff. Thanks

[edvinkuric]

Thank you for your reply, this helps a lot :-)

Can you point me to the code for the "normal" email-sending process? That would be awesome.
i would like to try to copy some code into the optIn-Email-code - maybe somewhere in https://github.com/salesagility/SuiteCRM/blob/master/modules/EmailMan/EmailMan.php#L852 ?

I'll create a seperate topic for PowerReplacer in the suitecrm Forums.

[chris001]

@edvinkuric

Can you point me to the code for the "normal" email-sending process? That would be awesome.
i would like to try to copy some code into the optIn-Email-code - maybe somewhere in...

FYI, to prevent more future bugs: Copy-and-paste programming, which is when you copy the same code into different areas of the code base, with slight variation, is a very bad method of solving the problem.

It's best(*) to factor out one library class which will do the work, replace the variables, including custom variables, in any/all email message that the CRM software will generate.

(*) "Best" means: reduces current bugs, makes fixing current bugs/issues easier, and prevents future bugs from getting added.

[edvinkuric]

@chris001 the refactored method still needs to be called twice afterwards, which ends in copy/paste ;-)

back to topic:
sure, implementing with the proper design patterns is great, but that should have been done by the suitecrm-team in the first place.
I don't have the time, internal knowledge or dev-environment to fix this appropriately, so i would be already happy to fix this in any way it fits my use-case.

Since i haven't got any reply in my first bug-issue from 4 months ago, i assumed my PRs wouldn't be accepted anyways - so why bother going "the long road"?
I can image that this feature could be completely rewritten in V8 or later since this isn't the only bug.

Obviously it's better to write clean/proper code - but if nobody uses it, it kinda is worthless.

So, maybe you can point me to some lines, which the "correct replacement" takes place? That would be awesome.

[pgorod]

I can share with you the notes I made when I was preparing to work on the new code. You can see how simple it is 🤮

It's actually a perfect example of what Chris is describing - code copied all over the place with some changes, causing bugs...

Code flow 1 (Compose)
-----------------
EmailsController actions_save 
> replaceEmailVariables 
> EmailTemplate::parse_email_template
> preg_match_all
> substr_replace

Code flow 2 (newer code) Campaigns, queue, wizard
--------------------------------------
modules/EmailMan/EmailManDelivery.php (no class)
EmailMan::sendEmail
> EmailTemplateParser::parseVariables
> preg_match_all
> str_replace

Code flow 3 (pre-7.9 email code? not called anywhere?)
--------------------------------
EmailUIAjax, $_REQUEST['emailUIAction']='sendEmail'
> Email::email2send (mentions drafts... ?)
> EmailTemplate::parse_template
> EmailTemplate::parse_template_bean (repeated code for specific modules)
> EmailTemplate::add_replacement
> EmailTemplate::_parseUserValues
> handle_body (replaces images)
> str_replace

  But the EmailTemplate::parse_template_bean is called from
    - FP_Events send invites 

  But the PDF templateParser::parse_template_bean is called from...
    - AOW_Workflow sendemail
    - AOS_PDF_Templates template::Parser

  Then there's modules/AOP_Case_Updates/util.php aop_parse_template, used in Surveys, Case Updates

So you can take your pick and choose which code you prefer to use... 😃

[chris001]

A colossal mess of repeated, yet different, code, and different for no legitimate reason! Yes, this is the inhierited legacy of SugarCRM 6.5 Community Edition, which must be fixed and improved upon so that it will start to work reliably!
This "replacing of CRM database variables (parsed words that begin with a dollar sign in the text of the email)" functionality, should be done with the exact same library class, everywhere that the CRM does this feature for the user! For maximum dependability, and minimum bugs!

[pgorod]

I've been doing just that in the past two months. It's a big, big project. When I am finished with variable substitutions in email, I will go and open that Pandora's box that is PDF Templates... and I also need to work on the stuff coming from en_us.notify_template.html, I haven't done those yet...

Any significant change to the code base also raises unexpected problems that complicate things tremendously. I've spent more time fixing the Compose window (making it use the user-configured Editor, and making sure it let's you put in your text and doesn't change it), than on the actual variable substitution code (which is incredibly powerful due to using Twig).

If all goes well, my SponsorWare model takes off and in a year or so I will be contributing this code to Core, which is where it belongs. But I must say it's not looking very promising... :-(