Technically the free / unlicensed core in 3.x is only to be used for personal, educational, and other non-commercial uses, so I hope that limitation is being honored along with the 300 point limit.
There are other ways to do it that would not use a data point, and then not get you closer to the limit. Still operating under the assumption this is Modbus, the Modbus4J code is open source and you could create the right requests from a script and submit them to the Modbus master. A script solution is also completely possible (where you read the registers as a different data type and then do bitmasking in a script as you suggest. You would just have to encode what to do to the Modbus point into the point that updates the script's context, and do something like,
//untested var data = input.value.split("-"); var mask = 1 << parseInt(data[1]); if( data[0] === "0" ) modbusPoint.set( modbusPoint.value & ~mask ); else modbusPoint.set( modbusPoint.value | mask );Or for that matter scripts can use the JsonEmport tool to modify points during script runtime.