No, the problem is not having to support when the input framerate changes.
Even when the input framerate stays constant for the duration of the input stream, the output stream should have the packet duration filled in properly, and not left blank like it is currently.
Can you do this?