I've been dealing with timing and latency problems for years, Nick. I don't have an answer for your problem. But, I can offer you the work around that I use.
I insert a single note on channel 10 drums at 1:0:0 in the midi file.
Then, I render the tracks separately inclusive of the channel 10 note (most times a cowbell hit because of the quick attack on/off). After all the tracks are rendered, you have the click note in them to assist in lining everything up in your audio program. It is not always the case, but most times latency is constant throughout playing/recording. Meaning that a track shouldn't drift once it is started.
Good luck with a common problem in using multiple sources.
BW@