Error while checking certificates

jcaballeroa

I just installed Mango 5 Free to try it out since we are regular users of Mango 3 and we want to see if we can migrate our system, but when starting up, I continuously get the following error when renewing certificates.

(I'm uploading the trace information in text mode since uploading the images gives an error.)

ERROR 2024-10-23T07:57:49,179 (com.infiniteautomation.mango.spring.pki.DefaultCertificateMonitor.lambda$initialize$0:80) - Error while renewing certificates
org.springframework.core.convert.ConversionFailedException: Failed to convert from type [java.lang.String] to type [double] for value '25%'; nested exception is java.lang.NumberFormatExcep
tion: For input string: "25%"

Every minute and a half or so, it restarts, showing the following message,without displaying any other error messages. Could you tell me what might be happening? Any installation issues?

INFO 2024-10-23T07:59:05,364 (com.serotonin.m2m2.rt.DataSourceGroupTerminator.process:35) - Terminating 1 LAST priority data sources in 1 threads
INFO 2024-10-23T07:59:05,371 (com.serotonin.m2m2.rt.RuntimeManagerImpl.stopDataSource:356) - Data source (name=status, id=2, type=MetaDataSourceRT) stopped in 5ms
INFO 2024-10-23T07:59:05,372 (com.serotonin.m2m2.rt.DataSourceGroupTerminator.process:40) - Termination of 1 LAST priority data sources in 1 threads took 8 ms
INFO 2024-10-23T07:59:05,372 (com.serotonin.m2m2.rt.DataSourceGroupTerminator.process:35) - Terminating 1 NORMAL priority data sources in 1 threads
INFO 2024-10-23T07:59:05,375 (com.serotonin.m2m2.rt.RuntimeManagerImpl.stopDataSource:356) - Data source (name=Mango Internal, id=1, type=InternalDataSourceRT) stopped in 1ms
INFO 2024-10-23T07:59:05,377 (com.serotonin.m2m2.rt.DataSourceGroupTerminator.process:40) - Termination of 1 NORMAL priority data sources in 1 threads took 5 ms
INFO 2024-10-23T07:59:05,380 (com.infiniteautomation.mango.io.serial.JsscSerialPortManager$1.run:74) - Mango Serial Port Event Processor Terminated
INFO 2024-10-23T07:59:05,385 (com.serotonin.m2m2.rt.maint.BackgroundProcessingImpl.joinTermination:452) - All low priority tasks exited gracefully.
INFO 2024-10-23T07:59:05,385 (com.serotonin.m2m2.rt.maint.BackgroundProcessingImpl.joinTermination:459) - All medium priority tasks exited gracefully.
INFO 2024-10-23T07:59:05,386 (com.serotonin.m2m2.rt.maint.BackgroundProcessingImpl.joinTermination:466) - All high priority tasks exited gracefully.
INFO 2024-10-23T07:59:05,423 (com.infiniteautomation.mango.spring.components.executors.MangoExecutors.destroy:93) - Shutting down shared executor and scheduled executor
INFO 2024-10-23T07:59:05,425 (com.infiniteautomation.mango.spring.components.executors.MangoExecutors.awaitTermination:150) - Executors shutdown successfully
INFO 2024-10-23T07:59:05,431 (com.serotonin.m2m2.db.H2Proxy.terminateImpl:408) - Terminating database.
INFO 2024-10-23T07:59:05,817 (com.serotonin.m2m2.Lifecycle.terminate:507) - Mango Lifecycle terminated: SHUTDOWN_HOOK

BG

@jcaballeroa
I am sorry I can't help with your error, but I am very interested to know how you get on with your tests with mango 5. We are also considering the move from 3 to 5 ourselves and I haven't had the time to test 5 yet.

Cheers
Brian

joeamiraglia

Hello @jcaballeroa, I'm sorry to hear you're experiencing issues. To better understand the problem, I will need you to provide more details.

Which Mango version are you using in v3, v3.7.12?
Did this error start after upgrading to v5.2.1, or did you start with a clean install and your overrides added?
Please share your SSL properties settings (those you feel comfortable sharing).
Also, can you confirm that you are running Java 17 for the v5 Mango?

Sharing your ma.log file would greatly help in diagnosing the issue. Thank you.

BR,

Joe Miraglia
SAE/Radix IoT

jcaballeroa

Hi @BG, thanks for your quick response.

Which Mango version are you using in v3, v3.7.12? We usually use Mango v3.7.7
Did this error start after upgrading to v5.2.1, or did you start with a clean install and your overrides added? I downloaded the latest version of Mango Free v5.2.1 from the Radix website and installed it on a clean version of Ubuntu 24.04.
Please share your SSL properties settings (those you feel comfortable sharing).
I kept the default SSL configurations that come in the mango.properties file, as they seemed coherent and sufficient to me:

# SSL/TLS setup

# Note: Enabling SSL/TLS also turns on HSTS by default, see the ssl.hsts.enabled setting below
#ssl.on=true
#ssl.port=8443

# Configure the key store from which to load X.509 certificate chain and private key.
# All key store settings are reloaded dynamically.
# If the keystore file does not exist, a temporary self-signed certificate is used instead.

# Path to a PKCS #12 or JKS key store, relative to ${paths.data}
# By default, use the path to the key store created by the Mango PKI service.
#ssl.keystore.location=${pki.keyStore}
# Key store password
#ssl.keystore.password=${pki.keyStorePassword}
# Key password (if not set, it is assumed to be the same as the key store password)
#ssl.key.password=
# Watch the key store file for changes and reload the certificates/keys for SSL/TLS when it changes
#ssl.keystore.watchFile=true

#Time socket can be idle before being closed (ms)
#ssl.socketIdleTimeout=70000

# Enable ALPN (Application-Layer Protocol Negotiation) for HTTP/2 support.
# Current browsers only support HTTP/2 for SSL/TLS connections.
#ssl.alpn.on=true

# Configure HSTS (HTTP Strict Transport Security)
# Enabled by default when ssl.on=true
# Sets the Strict-Transport-Security header, web browsers will always connect using HTTPS when they
# see this header and they will cache the result for max-age seconds
#ssl.hsts.enabled=true
#ssl.hsts.maxAge=31536000
#ssl.hsts.includeSubDomains=false

Also, can you confirm that you are running Java 17 for the v5 Mango?
This is my Java version:
openjdk 17.0.12 2024-07-16
OpenJDK Runtime Environment (build 17.0.12+7-Ubuntu-1ubuntu224.04)
OpenJDK 64-Bit Server VM (build 17.0.12+7-Ubuntu-1ubuntu224.04, mixed mode, sharing)
Sharing your ma.log file would greatly help in diagnosing the issue. Thank you.
I tried to attach the ma.log file, but it tells me I don't have permission to do so. I've uploaded it to Wetransfer, link attached. If you prefer another method to receive it, I have no problem doing it another way.

https://we.tl/t-snLX1Og2Do

Thanks and Best Regards

joeamiraglia

@jcaballeroa It appears the issue is related to this property:

pki.monitor.expiringSoonThreshold=25%

The problem may stem from unusual locale settings on your computer that do not properly recognize the percentage symbol (%). To address this, follow these steps in order:

Step 1: Modify the property

Edit the configuration file containing pki.monitor.expiringSoonThreshold=25%. Change this setting to the decimal format (0.25) instead of using a percentage, pki.monitor.expiringSoonThreshold=0.25.
Restart the Mango service to apply the new settings.

If the issue persists, proceed with Step 2:

Step 2: Check your locale settings

Ensure your system uses a standard locale, such as en_US, where the percentage symbol (%) is recognized correctly. Adjust your locale settings if necessary.
Restart the Mango service again to apply these changes.

Please implement these steps and let us know on the forum whether this resolves the issue. Your feedback is invaluable and can assist others experiencing similar issues.

jcaballeroa

@joeamiraglia

I changed the value from 25% to 0.25 as you suggested, and indeed Mango no longer throws an error at startup—at least, no errors appear in the ma.log file.

The locale issue seems strange to me because we have the Spanish locale (es_ES) configured, so the % symbol should also be valid in this language.

LANG=es_ES.UTF-8
LANGUAGE=
LC_CTYPE="es_ES.UTF-8"
LC_NUMERIC="es_ES.UTF-8"
LC_TIME="es_ES.UTF-8"
LC_COLLATE="es_ES.UTF-8"
LC_MONETARY="es_ES.UTF-8"
LC_MESSAGES="es_ES.UTF-8"
LC_PAPER="es_ES.UTF-8"
LC_NAME="es_ES.UTF-8"
LC_ADDRESS="es_ES.UTF-8"
LC_TELEPHONE="es_ES.UTF-8"
LC_MEASUREMENT="es_ES.UTF-8"
LC_IDENTIFICATION="es_ES.UTF-8"
LC_ALL=

However, it continues to restart every 1-2 minutes, this time without any apparent reason, as no error traces appear. From what I’ve seen, it seems that the issue lies with the service; if I manually start Mango via the start-mango.sh script, it works perfectly without restarting, but if I start it as a service through systemctl, that’s when the problem occurs. The script we use for the service is the one provided on the installation webpage:

[Unit]
Description=Mango Automation
After=mysqld.service
StartLimitIntervalSec=0
[Service]
EnvironmentFile=/etc/environment
Type=forking
WorkingDirectory=/opt/mango
PIDFile=/opt/mango/bin/ma.pid
ExecStart=/opt/mango/bin/start-mango.sh
SuccessExitStatus=0 SIGINT SIGTERM 130 143
Restart=always
RestartSec=5s
User=mango
NoNewPrivileges=true
[Install]
WantedBy=multi-user.target

joeamiraglia

@jcaballeroa You mentioned that you are testing Mango 5 in free mode. Could you please tell me how many data points you are using, and send the complete ma.log file again?

Additionally, assuming the server writes logs to /var/log/syslog, you might find more log entries related to Mango’s issues there. I often find the information I need in the syslog when Mango fails to start up correctly.

Thank you.

jcaballeroa

@joeamiraglia

I've uploaded the ma.log and syslog files to the following WeTransfer link:

https://we.tl/t-jpLw4GG0nl

I currently have 3 data sources, 50 data points, and 25 publishers solely for testing purposes.

Looking at the following extract from the syslog file, I understand that the mango service restarts the application because it detects that it hasn’t finished starting up. A timer expires, interpreting that mango hasn't completed all necessary startup steps to mark it as fully initialized, which is why systemctl relaunches it. Is that correct?:

2024-10-24T15:29:30.776837+00:00 navarraserver3 start-mango.sh[257343]: INFO  2024-10-24T15:29:30,775 (com.infiniteautomation.mango.spring.MangoRuntimeContextConfiguration.contextRefreshed:168) - Spring context 'rootWebContext' refreshed: Root WebApplicationContext
2024-10-24T15:29:31.223949+00:00 navarraserver3 start-mango.sh[257343]: INFO  2024-10-24T15:29:31,222 (com.infiniteautomation.mango.spring.MangoRuntimeContextConfiguration.contextRefreshed:168) - Spring context 'restDispatcherContext' refreshed: WebApplicationContext for namespace 'REST_DISPATCHER-servlet'
2024-10-24T15:29:33.664510+00:00 navarraserver3 start-mango.sh[257343]: INFO  2024-10-24T15:29:33,663 (com.infiniteautomation.mango.spring.MangoRuntimeContextConfiguration.contextRefreshed:168) - Spring context 'restV3Context' refreshed: WebApplicationContext for namespace 'restV3DispatcherServlet-servlet'
2024-10-24T15:30:00.171984+00:00 navarraserver3 systemd[1]: Starting sysstat-collect.service - system activity accounting tool...
2024-10-24T15:30:00.221033+00:00 navarraserver3 systemd[1]: sysstat-collect.service: Deactivated successfully.
2024-10-24T15:30:00.222035+00:00 navarraserver3 systemd[1]: Finished sysstat-collect.service - system activity accounting tool.
2024-10-24T15:30:08.789340+00:00 navarraserver3 systemd[1]: Reloading requested from client PID 257711 ('systemctl') (unit session-386.scope)...
2024-10-24T15:30:08.789874+00:00 navarraserver3 systemd[1]: Reloading...
2024-10-24T15:30:09.153917+00:00 navarraserver3 systemd[1]: Configuration file /run/systemd/system/netplan-ovs-cleanup.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway.
2024-10-24T15:30:09.320285+00:00 navarraserver3 systemd[1]: Configuration file /run/systemd/system/systemd-networkd-wait-online.service.d/10-netplan.conf is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway.
2024-10-24T15:30:09.764634+00:00 navarraserver3 systemd[1]: Reloading finished in 974 ms.
2024-10-24T15:30:48.652351+00:00 navarraserver3 systemd[1]: mango.service: start operation timed out. Terminating.
2024-10-24T15:30:48.659307+00:00 navarraserver3 start-mango.sh[257343]: INFO  2024-10-24T15:30:48,656 (com.serotonin.m2m2.Lifecycle.terminate:371) - Mango Lifecycle terminating...
2024-10-24T15:30:48.726801+00:00 navarraserver3 start-mango.sh[257343]: INFO  2024-10-24T15:30:48,725 (com.serotonin.m2m2.rt.RuntimeManagerImpl.stopPublisher:606) - Publisher (name=Alsasua, id=1, type=HttpSenderRT) stopped in 3ms
2024-10-24T15:30:48.728599+00:00 navarraserver3 start-mango.sh[257343]: INFO  2024-10-24T15:30:48,727 (com.serotonin.m2m2.rt.RuntimeManagerImpl.stopPublisher:606) - Publisher (name=Etxegarate, id=2, type=HttpSenderRT) stopped in 1ms

jcaballeroa

@joeamiraglia Have you had a chance to review the traces I sent you? Any idea why Mango keeps restarting continuously when launched as a service? For us to consider migrating our applications to Mango 5, we would need it to work properly as a service so that it starts correctly in case of a restart or failure.

joeamiraglia

Hello @jcaballeroa, sorry for the delay on this topic but could you please open a support request at Radix IoT - Support? I would like to better assist on this topic but access to the files have expired. Also, have you tried to start a clean Mango v5 to see if you have any issue starting up Mango before upgrading?