Mautic: Updated configuration parameters in UPGRADE-3.0.md are incomplete

[dennisameling]

IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII
Before you create the issue:
IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII
Search for similar report among other reported issues.
Learn how to troubleshoot at https://www.mautic.org/docs/en/tips/troubleshooting.html
Use drag&drop to attach images or other files )

Bug Description

While working on the M2 > M3 upgrade script, I came across some settings in local.php that have changed in Mautic 3, but are not mentioned at https://github.com/mautic/mautic/blob/3.x/UPGRADE-3.0.md#configuration. This is leading to problems (e.g. old update URL (not the new GitHub one)), so I decided to compare a full local.php file (generated when going to Configuration > Save & Close in the UI) between the two versions.

It is absolutely crucial that we properly migrate settings to their new versions, as otherwise we'll very likely break existing Mautic installations post-upgrade.

Please find below all the parameters that differ between clean installations of Mautic 2 and Mautic 3. Please note the "Comment" column in the table as well (vertically scroll to the right)!!

I'll need to have a decision made on every single config parameter below in order to prevent issues when users upgrade their Mautic instances from 2 to 3.

Google Sheets version of the table below (only editable by Mautic's Product team for security reasons): https://docs.google.com/spreadsheets/d/1Fr_VBuiDbYy_YdjPMCFsFIyUP5vz6nexiNg9BNarluM/edit?usp=sharing

| Mautic 2 local.php | Mautic 3 local.php | Comment
|---|---|---|
| 'mailer_transport' => 'mail' | 'mailer_transport' => 'sendmail' | The mail option that was present in v2 now leads to Not a built-in Swiftmailer transport: mail errors in v3. Not sure if there are any other values that were present in v2 but not in v3 anymore! (TODO check)
| 'mailer_spool_path' => '%kernel.root_dir%/spool' | 'mailer_spool_path' => '%kernel.root_dir%/../var/spool' | Should I only replace this value when the M2 value exactly matches what is shown on the left, or should I replace %kernel.root_dir% with %kernel.root_dir%/../var?
| 'cache_path' => '/var/www/html/app/cache' | 'cache_path' => '/var/www/html/app/../var/cache' | Probably the smartest to replace app/cache with app/../var/cache so that we don't mess with this path if folks have set a custom cache path?
| 'log_path' => '/var/www/html/app/logs' | 'log_path' => '/var/www/html/app/../var/logs' | Probably the smartest to replace app/logs with app/../var/logs so that we don't mess with this path if folks have set a custom logs path?
| 'tmp_path' => '/var/www/html/app/cache' | 'tmp_path' => '/var/www/html/app/../var/tmp' | (Note this was written to the cache folder previously). Probably the smartest to replace app/cache with app/../var/tmp so that we don't mess with this path if folks have set a custom tmp path?
| 'dev_hosts' => null | 'dev_hosts' => array() | Replace null with array
| 'rememberme_lifetime' => '31536000' | 'rememberme_lifetime' => 31536000 | Replace string with int? Or leave as-is?
| N/A | 'cookie_httponly' => false | add cookie_httponly parameter? Or leave as-is (non-existent)?
| N/A | 'track_private_ip_ranges' => false | add track_private_ip_ranges parameter? Or leave as-is (non-existent)?
| N/A | 'batch_campaign_sleep_time' => false | add batch_campaign_sleep_time parameter? Or leave as-is (non-existent)?
| 'system_update_url' => 'https://updates.mautic.org/index.php?option=com_mauticdownload&task=checkUpdates' | 'system_update_url' => 'https://api.github.com/repos/mautic/mautic/releases' | Replace update URL with new one
| 'theme' => 'Mauve' | 'theme' => 'blank' | Only replace theme if it equals Mauve (otherwise we'd potentially reset the theme the user has selected)
| N/A | 'debug' => false | add debug parameter? Or leave as-is (non-existent)?
| N/A | 'stats_update_url' => 'https://updates.mautic.org/stats/send' | add stats_update_url parameter? Or leave as-is (non-existent)?
| N/A | 'install_source' => 'Mautic' | add install_source parameter? Or leave as-is (non-existent)?
| 'api_oauth2_access_token_lifetime' => 60 | 'api_oauth2_access_token_lifetime' => 3600 | update to 3600 if current value equals 60?
| 'api_oauth2_refresh_token_lifetime' => 14 | 'api_oauth2_refresh_token_lifetime' => 1209600 | update to 1209600 if current value equals 14?
| 'api_rate_limiter_cache' => [ 'type' => 'filesystem'] | 'api_rate_limiter_cache' => [ 'adapter' => 'cache.adapter.filesystem'] | Already in UPGRADE-3.0.md. But, do we only account for the filesystem case? Or do we need to migrate this config in other cases as well (like with memcached, as mentioned in the Mautic docs)?
| 'email_frequency_number' => null | 'email_frequency_number' => 0 | if null then update to 0 or leave as-is?
| 'email_frequency_time' => null | 'email_frequency_time' => 'DAY' | if null then update to DAY or leave as-is?
| 'track_by_fingerprint' => 0 | N/A | Remove or leave as-is?
| 'rabbitmq_host' => 'localhost', 'rabbitmq_port' => '5672', 'rabbitmq_vhost' => '/', 'rabbitmq_user' => 'guest', 'rabbitmq_password' => 'guest', 'beanstalkd_host' => 'localhost', 'beanstalkd_port' => '11300', 'beanstalkd_timeout' => '60' | N/A | Remove or leave as-is?
| 'report_export_batch_size' => '1000' | 'report_export_batch_size' => 1000 | convert string to int or leave as-is?
| 'report_export_max_filesize_in_bytes' => '5000000' | 'report_export_max_filesize_in_bytes' => 5000000 | convert string to int or leave as-is?
| N/A | 'sms_enabled' => false | add sms_enabled parameter? Or leave as-is (non-existent)?
| 'sms_frequency_number' => null | 'sms_frequency_number' => 0 | Convert null to 0 or leave as-is?
| 'sms_frequency_time' => null | 'sms_frequency_time' => 'DAY' | convert null to DAY or leave as-is?
| 'saml_idp_own_password' => null | N/A | Remove or leave as-is?
| 'webhook_start' => '0' | N/A | Remove or leave as-is?
| 'webhook_limit' => '10', 'webhook_log_max' => '1000', 'webhook_disable_limit' => '100', 'webhook_timeout' => '15' | 'webhook_limit' => 10, 'webhook_log_max' => 1000, 'webhook_disable_limit' => 100, 'webhook_timeout' => 15 | Convert string to int or leave as-is?

| Q | A
| --- | ---
| Mautic version | 3.0.0-beta2
| PHP version | 7.2
| Browser | N/A

Steps to reproduce

1. Download & install a fresh Mautic 2 instance.
2. Go through the installation process, click Configuration > Save & Close. This will generate a full-blown app/config/local.php file with many parameters in it.
3. Repeat step 1 and 2 for Mautic 3 beta2
4. Compare both local.php files (attached 2 example files to this issue for convenience, you can show them side by side in e.g. a text editor): local_m2_and_m3_files.zip

Log errors

N/A (multiple, e.g. when trying mautic:update:find while the old update URL is still in place)

Please check for related errors in the latest log file in [mautic root]/app/log/ and/or the web server's logs and post them here. Be sure to remove sensitive information if applicable. )

[escopecz]

Good catch Dennis! Here is my brain dump:

The mail option was removed from the new version of the Swiftmailer library, so it cannot exist in M3. It used to be a default option in M2, so it may be painful for some that this option no longer exists.

spool, cache, logs, tmp dirs were moved from app/ directory to var/ directory as it's Symfony's standard. But it should not cause any troubles keeping the old locations.

All config param values are now type sensitive due to use of Symfony's environment variables. That's why some default values were changed from null to [] or from string to int. It is now important to keep the type accurate.

Theme Mauve was deprecated and hidden in M2. It should not have been a default value anymore. In M3 the Mauve theme was removed completely, so the default value must be different.

The api_rate_limiter_cache used to use Doctrine cache bundle that was used only for this. Doctrine project deprecated that bundle and advices to use different solutions. So that's what we did. It is important to change this to the new format or delete it from the local.php file otherwise Mautic will break if it's present.

track_by_fingerprint Fingerprint tracking was removed as it wasn't really usable. The option usually just let users shoot themselves to their foot. It could merge thousands real contacts into one Mautic contact. So we removed it.

Alan may know more details.

[dennisameling]

That's already super helpful information! Thanks @escopecz! I'd opt for setting the spool/cache/logs/tmp dirs to their new locations. This will help us to keep documentation consistent for end users that haven't changed the default values. I'll make sure to only replace when the default values were in place, e.g. exactly matches %kernel.root_dir%/spool, because if users have set custom locations we don't want to mess with those settings.

What the upgrade script does (I'll create proper documentation tonight) is basically to move the current M2 installation into mautic-2-backup-files, then moves the mautic-3-temp-files into the root directory, and then copy settings/custom plugins/custom themes/etc. from the old M2 installation over to the new one. This is what it looks like while the process is running (note that only upgrade_v3.php will remain as that's the script that moves everything to the right places):

So, since we're doing so many movements already, it should be absolutely fine to take the opportunity and set spool/cache/logs/tmp dirs to their new locations if the user had it set to the default value in the M2 installation :+1:

[dennisameling]

For all configuration parameters in the table above, a decision has been made on what to do with it (see https://docs.google.com/spreadsheets/d/1Fr_VBuiDbYy_YdjPMCFsFIyUP5vz6nexiNg9BNarluM/edit#gid=0) and all necessary migrations are in place in the upgrade script.

Last thing that's left is to update the https://github.com/mautic/mautic/blob/3.x/UPGRADE-3.0.md#configuration file with updated config settings.

Note to self: add something about custom api_rate_limiter_cache configs in UPGRADE-3.0.md (found the info below after digging the Symfony docs):

=======================

If you had a custom API rate limiter other than the filesystem (default), you'll need to specify a Symfony cache adapter as of Mautic 3. For example, if you had Memcached set up with the following configuration:

'api_rate_limiter_cache' => array(
  'memcached' => array(
    'servers' => array(
      '0' => array(
        'host' => 'memcached.local',
          'port' => '12345'
      )
    )
  )
),

... it should now be:

'api_rate_limiter_cache' => array(
  'adapter' => 'cache.adapter.memcached',
  'provider' => 'memcached://memcached.local:12345'
),