Charts: [bitnami/mariadb] slave can't connect through socket

[theAkito]

Which chart:
bitnami/mariadb
version: 7.3.13
appVersion: 10.3.22

Describe the bug
I am running a custom version of this chart, that makes use of the master/slave approach regarding the MariaDB backend. The MariaDB master has no errors and seems to run properly, but the slave constantly restarts with below error. When I turn off the master, the slave still restarts.

Slave's Pod log:

mariadb 10:07:05.16

mariadb 10:07:05.16 Welcome to the Bitnami mariadb container

mariadb 10:07:05.16 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-mariadb

mariadb 10:07:05.17 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-mariadb/issues

mariadb 10:07:05.17 Send us your feedback at [email protected]

mariadb 10:07:05.17

mariadb 10:07:05.17 INFO ==> ** Starting MariaDB setup **

mariadb 10:07:05.24 INFO ==> Validating settings in MYSQL_*/MARIADB_* env vars

mariadb 10:07:05.25 INFO ==> Initializing mariadb database

mariadb 10:07:05.27 INFO ==> Using persisted data

mariadb 10:07:05.31 INFO ==> Running mysql_upgrade

mariadb 10:07:05.32 INFO ==> Starting mariadb in background

mariadb 10:07:07.42 INFO ==> Stopping mariadb

Slave's Pod events:

Readiness probe failed: Enter password: mysqladmin: 
connect to server at 'localhost' failed error: 
'Can't connect to local MySQL server through socket
'/opt/bitnami/mariadb/tmp/mysql.sock' (2)'
Check that mysqld is running and that the socket:
'/opt/bitnami/mariadb/tmp/mysql.sock' exists!

I am using a custom version of the chart, but for the most part it is left with the default settings. I turned on replication mode i.e. master/slave mode and set every password, etc. and it should work, as the master seems to work fine. However I probably missed something, so I hope you can give me hints.

To Reproduce
Steps to reproduce the behavior:

1. Turn on replication mode.
2. Set all passwords manually (no generated passwords).
3. Deploy chart.
4. MariaDB master works.
5. MariaDB slave does not work. For error message, see above.

Expected behavior
MariaDB slave should work, as well.

Version of Helm and Kubernetes:

• Output of helm version:

Helm v2.14.3-rancher1

• Output of kubectl version:

Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.1", GitCommit:"d647ddbd755faf07169599a625faf302ffc34458", GitTreeState:"clean", BuildDate:"2019-10-02T17:01:15Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.3", GitCommit:"b3cbbae08ec52a7fc73d334838e18d17e8512749", GitTreeState:"clean", BuildDate:"2019-11-13T11:13:49Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}

Additional context
Issue migrated from official forum, as requested by michiel.

[javsalgar]

Hi,

I was unable to reproduce the issue, this is what I tested:

helm install support-mariadb bitnami/mariadb --set rootUser.password=test1238b --set db.user=bitnami_db --set db.password=test111122 --set replication.password=repl123test --set slave.replicas=2

 1 ❯ kubectl get pods -w
NAME                       READY   STATUS    RESTARTS   AGE
support-mariadb-master-0   1/1     Running   0          3m39s
support-mariadb-slave-0    1/1     Running   0          3m39s
support-mariadb-slave-1    1/1     Running   0          2m36s

Could it be because of the passwords you are using?

[theAkito]

@javsalgar

Thanks for helping.

I made a customized version of the chart by adding features to it. I did not really change anything in the existing base, though maybe I missed something.

That said, I use plain text passwords from the values.yaml for testing purposes. I have triple checked if the passwords are read correctly from the values.yaml, so that shouldn't be an issue. The only problem I had with a password is that the rootUser.password wasn't allowed to be longer than 32 characters. The others are still longer, while I changed the latter one. Could that be an issue? However, I would expect a proper error message if such an issue would arise.

The weird thing I don't understand:

1. I use replication mode.
2. All passwords (including replication password) are set.
3. Master works!
4. Slave does not work!
5. I triple checked if Master and Slave have the same settings. They do.

[javsalgar]

Could you please share with us a set of passwords that triggered the issue in your side? I want to reproduce with the exact set.

[theAkito]

@javsalgar

So in the meantime I proceeded much further in the journy of the failing slave.

At first I noticed, that I accidentally had two if statements in the slave template at reversed places. Now they are ordered properly. After fixing this, I was researching further and learning about MySQL/MariaDB errors and what can go wrong with this database.

After all that, I arrived at a point, where I get the following error, when I manually try to start mysqld in the slave container:

...
2020-04-02 19:47:26 0 [ERROR] InnoDB: Operating system error number 13 in a file operation.
2020-04-02 19:47:26 0 [ERROR] InnoDB: The error means mysqld does not have the access rights to the directory.
2020-04-02 19:47:26 0 [ERROR] InnoDB: Operating system error number 13 in a file operation.
2020-04-02 19:47:26 0 [ERROR] InnoDB: The error means mysqld does not have the access rights to the directory.
2020-04-02 19:47:26 0 [ERROR] InnoDB: Cannot open datafile './ibdata1'
2020-04-02 19:47:26 0 [ERROR] InnoDB: Could not open or create the system tablespace. If you tried to add new data files to the system tablespace, and it failed here, you should now edit innodb_data_file_path in my.cnf back to what it was, and remove the new ibdata files InnoDB created in this failed attempt. InnoDB only wrote those files full of zeros, but did not yet use them in any way. But be careful: do not remove old data files which contain your precious data!
2020-04-02 19:47:26 0 [ERROR] InnoDB: Database creation was aborted with error Cannot open a file. You may need to delete the ibdata1 file before trying to start up again.
...

I had to start it manually and see for myself, to see the actual error, as the logs didn't say anything... (Room to improve on this, though that's something for another issue.)

As it says in the error message, which is also confirmed by the online information I could find about this type of error, is that wrong permissions (usually the wrong user) are assigned to files and folders. Usually the user mysql or in this case 1001 should be the owner of the MariaDB related locations.

The permissions for the data volume are handled by the initContainer by default, so can't be related to that (theoretically).
However, everything inside /opt/bitnami is owned by root:root. This additionally seems to create permissions issues, as shown here:

...
2020-04-03  9:41:59 0 [Note] InnoDB: Starting shutdown...
2020-04-03  9:41:59 0 [ERROR] Plugin 'InnoDB' init function returned error.
2020-04-03  9:41:59 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2020-04-03  9:41:59 0 [ERROR] mysqld: Can't create/write to file '/opt/bitnami/mariadb/data/aria_log_control' (Errcode: 13 "Permission denied")
2020-04-03  9:41:59 0 [ERROR] mysqld: Got error 'Can't create file' when trying to use aria control file '/opt/bitnami/mariadb/data/aria_log_control'
2020-04-03  9:41:59 0 [ERROR] Plugin 'Aria' init function returned error.
2020-04-03  9:41:59 0 [ERROR] Plugin 'Aria' registration as a STORAGE ENGINE failed.
2020-04-03  9:41:59 0 [Note] Plugin 'FEEDBACK' is disabled.
2020-04-03  9:41:59 0 [ERROR] Could not open mysql.plugin table. Some plugins may be not loaded
2020-04-03  9:41:59 0 [ERROR] Unknown/unsupported storage engine: InnoDB
2020-04-03  9:41:59 0 [ERROR] Aborting

Which I cannot handle with an initContainer, as this is not a volume. Additionally, I shouldn't create what feels like 100 workarounds for parts that should just work out of the box, in any scenario.

So researching for hours and reading all the logs I could find plus testing stuff manually, I have come to being stuck at this point and wonder why this is an error in the first place. Because

1. can't change permissions and the permissions on the actual data volume are handled already, anyway,
2. the Master works with these permissions!

What's going on? Really need this fixed to proceed further with my project.

[javsalgar]

Hi,

The thing is, MariaDB should not write in that folder /opt/bitnami/mariadb/data/aria_log_control, but in /bitnami/mariadb/data/aria_log_control. Does this error appear with the default MariaDB settings or you are using different configuration settings? I would like to reproduce the issue so I would need to have the minimal steps that lead you to that issue.

[theAkito]

@javsalgar

These folders are standard _bitnami/mariadb_ folders, set as default within this chart. If they are wrong then the upstream is wrong. I assumed they have to be correct, as I couldn't assume that upstream could provide such a major mistake.

The problem is, that I already checked if these folder could be merged. It doesn't work though, at least a full merge wouldn't work, as two instances need their own lock files, etc., I assumed. So I assumed, they need to be separate for 2 parallel instances working on the same objects to be possible.

[javsalgar]

Hi,

As you can see here

https://github.com/bitnami/bitnami-docker-mariadb/blob/master/10.4/debian-10/rootfs/opt/bitnami/scripts/libmysql.sh#L67
https://github.com/bitnami/charts/blob/master/bitnami/mariadb/templates/slave-statefulset.yaml#L100

Everything is configured to write in /bitnam/mariadb, that's why I need to understand better what happened. Could you please provide all the customizations you performed? Otherwise I will not be able to fully reproduce the issue.

[theAkito]

@javsalgar

Same here. Did not change anything about that. I read all the templates, line by line.
As you can see yourself, there are a LOT of folders, pointing to _/opt/binami/..._. That should be the issue, as that is the folder that always pops up in any debugging process. _/bitnami/..._ is clean and has correct permissions.

I can give you an excerpt from my customization:
master-statefulset.yaml:

...
# original
        {{- if and .Values.volumePermissions.enabled .Values.master.persistence.enabled }}
        - name: volume-permissions
          image: {{ template "mariadb.volumePermissions.image" . }}
          imagePullPolicy: {{ .Values.volumePermissions.image.pullPolicy | quote }}
          command: ["chown", "-R", "{{ .Values.securityContext.runAsUser }}:{{ .Values.securityContext.fsGroup }}", "{{ .Values.master.persistence.mountPath }}"]
          securityContext:
            runAsUser: 0
          resources: {{ toYaml .Values.volumePermissions.resources | nindent 12 }}
          volumeMounts:
            - name: data
              mountPath: {{ .Values.master.persistence.mountPath }}
        {{- end }}
# custom
        {{- if and .Values.volumePermissions.enabled .Values.master.hostpath.enabled }}
        - name: volume-permissions
          image: {{ template "mariadb.volumePermissions.image" . }}
          imagePullPolicy: {{ .Values.volumePermissions.image.pullPolicy | quote }}
          command: ["chown", "-R", "{{ .Values.securityContext.runAsUser }}:{{ .Values.securityContext.fsGroup }}", "{{ .Values.master.hostpath.mountPath }}"]
          securityContext:
            runAsUser: 0
          resources: {{ toYaml .Values.volumePermissions.resources | nindent 12 }}
          volumeMounts:
            - name: data
              mountPath: {{ .Values.master.hostpath.mountPath }}
        {{- end }}
...
# original
            {{- if not (empty .Values.db.user) }}
            - name: MARIADB_USER
              value: "{{ .Values.db.user }}"
            {{- if .Values.db.injectSecretsAsVolume }}
            - name: MARIADB_PASSWORD_FILE
              value: "/opt/bitnami/mariadb/secrets/mariadb-password"
# custom
            {{- else if .Values.db.plain }}
            - name: MARIADB_PASSWORD
              value: {{ .Values.db.password }}
            {{- else }}
            - name: MARIADB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: {{ template "mariadb.secretName" . }}
                  key: mariadb-password
            {{- end }}
...
# original
          volumeMounts:
            - name: data
              {{- if .Values.master.persistence.enabled }}
              mountPath: {{ .Values.master.persistence.mountPath }}
# custom
              {{- else if .Values.master.hostpath.enabled }}
              mountPath: {{ .Values.master.hostpath.mountPath }}
              {{- end }}
              {{- if .Values.master.persistence.subPath }}
              subPath: {{ .Values.master.persistence.subPath }}
              {{- end }}

values.yaml:

...
  persistence:
    enabled: false
    mountPath: /bitnami/mariadb
...
  hostpath:
    enabled: true
    mountPath: /bitnami/mariadb
    hostPath: /mariadb/database
...

As you can see yourself, the deployments use the exact same folder structure. Permissions are handled properly.
.Values.master.hostpath.mountPath = .Values.master.persistence.mountPath

I did not even touch any line to point to _/opt/bitnami_ or change any folder pointing to _/opt/bitnami_ to something else. It all is the same as in the standard chart.
The more this issue is addressed, the more it seems like a big bug. Maybe I should pull a newer version and apply my customizations there...

[javsalgar]

Looking at the changes, I don't see anything that should trigger an issue trying to write in /opt/bitnami/mariadb/data. Do you have a git repo with the chart customized?

[stale[bot]]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[theAkito]

No, not solved. So far this and other problems led to us using a different solution, so this is obsolete for us, for now, but others will encounter this issue, too, as seen in the referenced issue.

I know, that if it can't be confirmed, then it is hard to just plainly accept. The point is still: I did change nothing within the actual default part of the chart. I just added stuff to it. As everyone can see in my previous comment, I just copy-pasted the defaults and changed them a bit. My chart is using those new parts. The point is, they are just copy-pasted and the defaults are still the same! The best example to demonstrate that is the part where the PVC is replaced with a hostPath. It's literally the same template part, except it points to an actual path on your Node instead of a path leading to a PVC. Every other addition is exactly like this representative change.
So there has to be something wrong with the Chart and/or the image, because the Chart I am using uses the defaults, just for e.g. a different location like a hostPath. That is it. Nothing changed, from the perspective of the template. It behaves exactly like it did before. All variables stay the same, just their values changed.

So as already mentioned, I can't put in more time into fixing this, because this is now obsolete for us and the time taken by this issue can't be taken back, in the end. However, the next one encountering this issue might be of help.

[javsalgar]

Hi,

Thanks for the feedback. We test our chart automatically in all the major Kubernetes distributions: GKE, AKS, TKG, EKS... before releasing the container and the chart. We are sorry for the time spent on debugging the issue. Let us see if another person comes across the issue and we can find more information about what the issue was.

[stale[bot]]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[stale[bot]]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[stale[bot]]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.

[codeNgamer]

Hey so I definitely have this issue as well. It's come up before in another project and I simply nuked the whole thing and started clean. This time I can't really do that cause I've got considerable data at stake. Below is my current yaml as at now

---
  global: 
    storageClass: "aws-volume"
  master: 
    persistence: 
      existingClaim: "mariadb-volume"
  rootUser: 
    password: "*********"
  securityContext: 
    fsGroup: "0"
    runAsUser: "1001"
  slave: 
    persistence: 
      size: "20Gi"
      storageClass: "aws-volume"
    replicas: "3"
  volumePermissions: 
    enabled: "true"

kubectl describe pod mariadb-master-0 outputs the following:
Warning Unhealthy 6m20s (x329 over 115m) kubelet, ip----.us-east-2.compute.internal Readiness probe failed: mysqladmin: connect to server at 'localhost' failed
error: 'Can't connect to local MySQL server through socket '/opt/bitnami/mariadb/tmp/mysql.sock' (2)'
Check that mysqld is running and that the socket: '/opt/bitnami/mariadb/tmp/mysql.sock' exists!
Warning BackOff 76s (x220 over 93m) kubelet, ip----.us-east-2.compute.internal Back-off restarting failed container

[codeNgamer]

Also tried this: https://github.com/helm/charts/issues/20645#issuecomment-614215801 with no luck

[codeNgamer]

Also (sorry to piecemeal this - I'm just working realtime) kubectl logs --tail=20 mariadb-master-0 shows this now:
The mariadb configuration file '/opt/bitnami/mariadb/conf/my.cnf' is not writable. Configurations based on environment variables will not be applied for this file.

Not sure if related or helpful

[marcosbc]

Hi @codeNgamer, I don't know what may be going on but let's enable debugging to see if we can find anything.

I'd first ask to enable image.debug in a deployment to see if there's any relevant error.

If that did not help, then disable liveness/readiness probes to be able to enter the MariaDB container shell:

--set master.livenessProbe.enabled=false --set master.readinessProbe.enabled=false

Enter the shell via kubectl exec -ti POD bash and check if mysqld is running (via ps aux for example), and try to connect manually to it via your known credentials. Do you get any error? Does the container shut down due to a failed initialization? Is there any error via container output?

Also (sorry to piecemeal this - I'm just working realtime) kubectl logs --tail=20 mariadb-master-0 shows this now:
The mariadb configuration file '/opt/bitnami/mariadb/conf/my.cnf' is not writable. Configurations based on environment variables will not be applied for this file.

That is fine. By default the chart deploys a ConfigMap and mounts it, so it is not readable/writable by the container image.

[codeNgamer]

Okay so updated yaml is:

---
  db: 
    name: "charlotte_wordpress"
  global: 
    storageClass: "aws-volume"
  image: 
    debug: "true"
  master: 
    livenessProbe: 
      enabled: "false"
    persistence: 
      existingClaim: "mariadb-volume"
    readinessProbe: 
      enabled: "false"
  rootUser: 
    password: "*********"
  securityContext: 
    enabled: "false"
    fsGroup: "1001"
    runAsUser: "1001"
  slave: 
    persistence: 
      size: "20Gi"
      storageClass: "aws-volume"
    replicas: "3"
  volumePermissions: 
    enabled: "true"

output from kubectl logs -f --tail=20 mariadb-master-0

020-08-10 16:55:46 0 [Note] InnoDB: Completed initialization of buffer pool
2020-08-10 16:55:46 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2020-08-10 16:55:46 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=19874019533
2020-08-10 16:55:46 0 [Note] InnoDB: Last binlog file './mysql-bin.000085', position 667442528
2020-08-10 16:55:46 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
2020-08-10 16:55:46 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2020-08-10 16:55:46 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2020-08-10 16:55:46 0 [ERROR] InnoDB: preallocating 12582912 bytes for file ./ibtmp1 failed with error 28
2020-08-10 16:55:46 0 [ERROR] InnoDB: Could not set the file size of './ibtmp1'. Probably out of disk space
2020-08-10 16:55:46 0 [ERROR] InnoDB: Unable to create the shared innodb_temporary
2020-08-10 16:55:46 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
2020-08-10 16:55:46 0 [Note] InnoDB: FTS optimize thread exiting.
2020-08-10 16:55:51 0 [Note] InnoDB: Starting shutdown...
2020-08-10 16:55:51 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2020-08-10 16:55:51 0 [ERROR] Plugin 'InnoDB' init function returned error.
2020-08-10 16:55:51 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2020-08-10 16:55:51 0 [Note] Plugin 'FEEDBACK' is disabled.
2020-08-10 16:55:51 0 [ERROR] Unknown/unsupported storage engine: InnoDB
2020-08-10 16:55:51 0 [ERROR] Aborting

ps aux outputs the following (mysqld doesn't appear to be up):

ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
1001         1  0.0  0.0   4672  3888 ?        Ss   16:55   0:00 /bin/bash /opt/bitnami/scripts/mariadb/entrypoint.sh /opt/bitnami/scripts/mariadb/run.sh
1001        12  0.0  0.0   4672  3900 ?        S    16:55   0:00 /bin/bash /opt/bitnami/scripts/mariadb/setup.sh
1001       452  0.0  0.0   2400   776 pts/0    Ss+  17:02   0:00 /bin/sh -c TERM=xterm-256color; export TERM; [ -x /bin/bash ] && ([ -x /usr/bin/script ] && /usr/bin/script -q -c "/bin/bash" /dev/null || exec /bin/bash) || exec /bin/sh
1001       458  0.0  0.0   2400   120 pts/0    S+   17:02   0:00 /bin/sh -c TERM=xterm-256color; export TERM; [ -x /bin/bash ] && ([ -x /usr/bin/script ] && /usr/bin/script -q -c "/bin/bash" /dev/null || exec /bin/bash) || exec /bin/sh
1001       459  0.0  0.0   2604  1868 pts/0    S+   17:02   0:00 /usr/bin/script -q -c /bin/bash /dev/null
1001       460  0.0  0.0   2400   716 pts/1    Ss   17:02   0:00 sh -c /bin/bash
1001       461  0.0  0.0   4012  3276 pts/1    S    17:02   0:00 /bin/bash
1001       465  0.0  0.0   2308   768 ?        S    17:02   0:00 sleep 2
1001       466  0.0  0.0   7652  2788 pts/1    R+   17:02   0:00 ps aux

mysqld outputs the following:

mysqld
2020-08-10 17:04:58 0 [Note] mysqld (mysqld 10.3.23-MariaDB) starting as process 644 ...
2020-08-10 17:04:58 0 [Warning] Can't create test file /opt/bitnami/mariadb/data/mariadb-master-0.lower-test
2020-08-10 17:04:58 0 [Note] InnoDB: Using Linux native AIO
2020-08-10 17:04:58 0 [Note] InnoDB: The first innodb_system data file 'ibdata1' did not exist. A new tablespace will be created!
2020-08-10 17:04:58 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2020-08-10 17:04:58 0 [Note] InnoDB: Uses event mutexes
2020-08-10 17:04:58 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2020-08-10 17:04:58 0 [Note] InnoDB: Number of pools: 1
2020-08-10 17:04:58 0 [Note] InnoDB: Using SSE2 crc32 instructions
2020-08-10 17:04:58 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
2020-08-10 17:04:58 0 [Note] InnoDB: Completed initialization of buffer pool
2020-08-10 17:04:58 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2020-08-10 17:04:58 0 [ERROR] InnoDB: Operating system error number 13 in a file operation.
2020-08-10 17:04:58 0 [ERROR] InnoDB: The error means mysqld does not have the access rights to the directory.
2020-08-10 17:04:58 0 [ERROR] InnoDB: Operating system error number 13 in a file operation.
2020-08-10 17:04:58 0 [ERROR] InnoDB: The error means mysqld does not have the access rights to the directory.
2020-08-10 17:04:58 0 [ERROR] InnoDB: Cannot open datafile './ibdata1'
2020-08-10 17:04:58 0 [ERROR] InnoDB: Could not open or create the system tablespace. If you tried to add new data files to the system tablespace, and it failed here, you should now edit innodb_data_file_path in my.cnf back to what it was, and remove the new ibdata files InnoDB created in this failed attempt. InnoDB only wrote those files full of zeros, but did not yet use them in any way. But be careful: do not remove old data files which contain your precious data!
2020-08-10 17:04:58 0 [ERROR] InnoDB: Database creation was aborted with error Cannot open a file. You may need to delete the ibdata1 file before trying to start up again.
2020-08-10 17:04:59 0 [Note] InnoDB: Starting shutdown...
2020-08-10 17:04:59 0 [ERROR] Plugin 'InnoDB' init function returned error.
2020-08-10 17:04:59 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2020-08-10 17:04:59 0 [ERROR] mysqld: Can't create/write to file '/opt/bitnami/mariadb/data/aria_log_control' (Errcode: 13 "Permission denied")
2020-08-10 17:04:59 0 [ERROR] mysqld: Got error 'Can't create file' when trying to use aria control file '/opt/bitnami/mariadb/data/aria_log_control'
2020-08-10 17:04:59 0 [ERROR] Plugin 'Aria' init function returned error.
2020-08-10 17:04:59 0 [ERROR] Plugin 'Aria' registration as a STORAGE ENGINE failed.
2020-08-10 17:04:59 0 [Note] Plugin 'FEEDBACK' is disabled.
2020-08-10 17:04:59 0 [ERROR] Could not open mysql.plugin table. Some plugins may be not loaded
2020-08-10 17:04:59 0 [ERROR] Unknown/unsupported storage engine: InnoDB
2020-08-10 17:04:59 0 [ERROR] Aborting

Warning: Memory not freed: 520

[codeNgamer]

Looking closely it appears that the volume is out of space. I'll try to increase it and see what happens

[codeNgamer]

Okay so the issue was resolved after resizing the volume. Others can look at this possibility as well.

[marcosbc]

That's great! Thanks for sharing. It looks like something that can be a common issue, so it could be the same issue as what @theAkito found.

For now, BITNAMI_DEBUG seems to show enough info to debug most errors, so we can consider this solved for now. Feel free to ask to re-open this issue if you find the same problem in the future, or create a new one linking to here.