Mebius,
Do you have a single data source that is getting points from the BACnet device, or do you have multiple? If the latter, you might have multiple simultaneous requests going to the device, which it may not be able to handle, causing timeouts and the appearance of a slowdown.
I know at least that even the BACnet MS/TP is capable of polling for thousands of points without problems, subject to the speed of the connection. (I know you are using I/P, which should be much faster than MS/TP.)