Poll times generally indicate the time it takes for the device to respond to all the request for data. The information here probably isn't enough for us to look at especially since we never see this on any other systems so it is quite odd. Generally our modbus networks run at 19.2k baud.
You could capture the following system metrics during a time when the polls are slow just to see if something is going wrong on the MangoES:
Screen shot from the Internal Metrics tab and work items tab
Download JSON from the Threads tab on logging console
Screenshot of running "top" in a ssh console
JSON export of your Modbus Serial Data source and data points so we can review all the settings.
These items will help us see if any kind of performance in Mango is causing your poll times to slow down.