Bounce in place issue with Logic and VEPro

[aldous]

Yes I know that’s the case for Logic itself but I’m wondering if the off-line render is maybe harder for the networked PC/VEP to keep up. I was always able to bounce in real time to multiple audio tracks within Logic using this setup, however as I said I couldn’t bounce offline at all until I upgraded my computer. So I can only assume the resource-saving process on the Mac/Logic somehow makes a bigger demand on the slave in order to bounce offline. But perhaps I’m wrong.

I don't think you're wrong at all: that does seem to be happening... I just don't think it should be. Logic will wait for the VEP plugin, so we just need the rest of the chain - VEP plugin <-> VEP server <-> the instrument plugin - to behave the same way.

Freezing is an option alright but I ultimately need the audio tracks in order to do the mix. At least that’s how I work, rather than mixing in midi.

If freeze works, though, you then have the audio. You can find it in Finder under "Media", "Freeze Files" inside the project folder (or package, by Control+click and choosing "Show Package Contents...") The files' names include their channel strip type/number, so the easiest way to match them up is to enable "Additional Name Column" under "Track", "Configure Track Header..." and choose "Channel strip type/number". (For some reason, audio tracks are frozen to "Track 42" instead of "Audio 42" as I'd expected.)

Still, it's not perfect. It would be nice if Logic allowed you to bounce a frozen track - i.e. basically just re-coding the frozen audio and putting it on a correctly-configured track - but it doesn't.

 

[MarcusMaximus]

I don't think you're wrong at all: that does seem to be happening... I just don't think it should be. Logic will wait for the VEP plugin, so we just need the rest of the chain - VEP plugin <-> VEP server <-> the instrument plugin - to behave the same way.

Yes, maybe they don't like the fact that Logic is asking them to behave so quickly and all together. Bouncing fewer tracks, or even just one, at a time might shed some light on this.

If freeze works, though, you then have the audio.

I used to freeze tracks when working on the one computer. However if I un-froze them, the respective audio files were automatically deleted. I presume that still happens. I need to keep them for mixing. Anyway, I'll play around with it and see if it is a viable solution but when I tried it on the old master-slave setup I still got only silence on the frozen tracks. Maybe it will work on the new one though, as bouncing does (more or less).

Still, it's not perfect. It would be nice if Logic allowed you to bounce a frozen track.

Yes that would certainly be good. As it is if I wanted separate audio tracks I'd probably have to record them in real time which would be the same process I have used up to now except with the original midi tracks.

Hmm. Plenty to think about and try out. Thanks!

 

[aldous]

However if I un-froze them, the respective audio files were automatically deleted. I presume that still happens. I need to keep them for mixing.

Yes; you just need to take copies, e.g. into the "Audio Files" folder, so Logic knows it doesn't "own" them. (Closing the project before copying would be sensible; it looks like Logic keeps them memory-mapped after writing them, perhaps anticipating further changes.)

 

[MarcusMaximus]

Ok I'll try that thanks.

 

[MarcusMaximus]

Some interesting though inconclusive findings with one track. Freeze yielded the same sort of result with some truncated notes. Bouncing a track in place again did the same as before. Then I tried turning Scripter off and got no dropouts but when I turned it back on again I also got no dropouts! I was going to say that Scripter was causing the issue but it doesn't seem to be, at least not with this test.

 

[aldous]

Were you playing a reasonable number of tracks/notes in the test? A sampled instrument will run "artificially well" if you only play a small selection of samples repeatedly, as each sample will stick around in the cache. During normal playback, they'll get evicted by samples from the much wider variety of notes/instruments.

Or, it may not be the problem - just something to be aware of when testing.

 

[MarcusMaximus]

I tested it with just one track over the length of the piece. English horn. It uses a few different articulations and a reasonable number of notes throughout the track, although it doesn’t play constantly. Next I’ll try it a few tracks at a time and see how that works out.

 

[MarcusMaximus]

I'm still trying to get this to work with one track and then will move on to multiple ones once it does. If I bounce a track or all the regions of a track I still get many truncated notes. The only way it seems to work is if I bounce a single region and that is going to be too unwieldy over the course of the whole project. Strangely, as I said before I did have success with one track but not since.

I have confirmed that the bounces are not being normalised by using aldous' gain test as described above.

It looks like I'm going to have to go back to my old method of recording all mid tracks to audio in real time but that is a pain to set up in terms of routing etc. so I'd much rather get bounce-in-place to work if that's at all possible.

 

[aldous]

Firstly, I just want to summarise, to make sure we're on the same page. Please correct me if I'm wrong:

• Based on what you've found so far, it looks/sounds like the player is abruptly dropping voices because it can't get sample data quickly enough during off-line bouncing. We haven't absolutely proved that, though. We also haven't pinned the problem down to a single piece of software. Both would help before (a) tweaking settings; or (b) opening support tickets.
• Your set-up is: Logic -> VEP plugin -> ethernet(?) -> VEP server -> HOOPUS instruments in OPUS plugin(?)
• You're not using the VEP Event or Audio plugins.

A few things to check - apols if you've answered before and I've just missed it:

1. If you're using one instrument per instance of VEP (?), then do you have the "Multiprocessing" setting set to one thread in VEP server?
2. Did you confirm the problem still happens with Scripter disabled?
3. Does it also keep happening when freezing tracks, instead of bouncing?
4. Can you avoid a bad bounce by playing the project through normally first, before bouncing?
5. Have you tried raising the pre-load buffer in Opus?

Some ideas:

1. It might help to get a screen shot of performance monitors - on both machines - for both "bad bounce" and a "good bounce". Is that possible? That might help prove the problem is on one machine - I suspect PC - and may show which resource is running out (CPU, memory, disk.)
   1. If the player and/or VEP server have a performance monitor too, getting that in the shot might be useful. They show different quantities.
2. All my other ideas at the moment are just about methodically swapping elements of your set-up to prove the issue is in a particular place.
   1. Can you replicate the problem with any non-Opus instrument hosted in VEP server?
   2. ...how about using the Opus instrument hosted directly in Logic; is that practical to try?

 

[MarcusMaximus]

Thanks very much aldous. It’s 1.00 in the morning here so I’m off to bed but I’ll reply properly on the morrow.

 

[MarcusMaximus]

Ok here we go:

Firstly, I just want to summarise, to make sure we're on the same page. Please correct me if I'm wrong:

Yes all those points are correct.

A few things to check - apols if you've answered before and I've just missed it:

Multiprocessing is set to one thread. The problem still happens with Scripter disabled, also when freezing. Bouncing after a normal play through makes no difference. I have Streaming (Preload) set to SSD/SATA in Opus as that reflects my system, changing it to both other options made no difference.

1. Some ideas:

Here's a screenshot of task manager during a bounce (they're all bad!):



Here's a link to a video showing the different panels during bouncing in Activity Monitor. I can't find any performance monitors in Opus or VEP.

I tried a bounce using a VSL woodwind patch in Synchron Player and guess what, it came out fine! The same when I used another instrument in Opus directly in Logic. It would be too complicated trying it with the same HOOPUS instrument I'm using in VEP because I'd have to move the licence etc. but I imagine it would be fine as well.

I think that's all my homework done!

So it looks like the particular combination that I am using is not doing offline bounces properly. The same used to happen when I was using the Play version of HO. So Logic meets VEP meets Eastwest player gives me this problem. I wonder if you, or anyone else, can successfully bounce offline using the same setup?

I think I am going to revert to real time bouncing if we can't get this to work. At least it is a tried and tested method that works, even if it is a bit labour-intensive.

Many thanks once again!

 

[aldous]

Thanks for confirming everything!

Here's a screenshot of task manager during a bounce (they're all bad!):

I didn't immediately spot anything that would bother me on the mac. The PC doesn't look especially taxed in general, though that spike in HDD activity is a worry, and the commit charge looks fairly substantial to me too - though not terrifying. (Disclaimer: my knowledge of Windows internals is certainly rustier than macOs.)

I assume your OPUS sample libraries are meant to be on one of the SSDs; is that correct? It almost looks like they're coming from your HDD, which would explain the playback issue.

This could happen if the samples were preloaded (from SSD) and then swapped to the page file (guessing you've left it on C:, the HDD?) I'm slightly sceptical because I'd normally expect a sample player to prevent its memory being paged this way.

Ideally, it would be nice to confirm that is / is not happening; or, otherwise, explain what that HDD spike is about. Do you know Task Manager / Resource Monitor well enough to see the VEP/OPUS process(es) and the memory they're using? And also whether or not there's a spike in Page Faults when you bounce? Also, are you able to confirm where your pagefile is located?

Otherwise, you could try treating the problem and seeing if it helps. That would involve finding ways to cut (dramatically, if possible) the amount of memory being used on the PC overall. If paging is the issue, that should cut that HDD spike and - hopefully - the dropped notes. For testing purposes, then, is it possible to shut down or purge a big chunk of instruments/mics you're not using?

 

[MarcusMaximus]

Thanks for confirming everything!

You're welcome.

I didn't immediately spot anything that would bother me on the mac. The PC doesn't look especially taxed in general, though that spike in HDD activity is a worry, and the commit charge looks fairly substantial to me too - though not terrifying. (Disclaimer: my knowledge of Windows internals is certainly rustier than macOs.)

Very interesting observation, and your knowledge about all these matters seems pretty considerable to me!

I assume your OPUS sample libraries are meant to be on one of the SSDs; is that correct?

Yes the samples are all on SSD.

This could happen if the samples were preloaded (from SSD) and then swapped to the page file (guessing you've left it on C:, the HDD?) I'm slightly sceptical because I'd normally expect a sample player to prevent its memory being paged this way.

Again, very interesting. I think you're probably onto something here, although I would also expect sample player designers to take this possibility into account and make sure to avoid it.

Do you know Task Manager / Resource Monitor well enough to see the VEP/OPUS process(es) and the memory they're using? And also whether or not there's a spike in Page Faults when you bounce? Also, are you able to confirm where your pagefile is located?

I don't really know them that well but I can investigate!

T.M. says that VEP is using about 25 of 30gb ram in my current project. That's all down to Opus. I have seen the figure for Compressed memory go up considerably as I run the sequencer and work on the piece. When I bounced one track just now, the average hard page faults per second went up from 0 to 2. Does that constitute a spike? I can confirm that the page file is located on my C drive, the HDD.

For testing purposes, then, is it possible to shut down or purge a big chunk of instruments/mics you're not using?

I deleted all instances except one which contains all the articulations of a clarinet and then bounced that track offline. Interestingly, it showed no page faults and very little HDD activity but the resulting audio track was the same as before with the same truncated notes. So does that suggest that it may not be a paging issue?

 

[aldous]

One of my vaguer posts, as need to look up a couple of things tomorrow

T.M. says that VEP is using about 25 of 30gb ram in my current project. That's all down to Opus. I have seen the figure for Compressed memory go up considerably as I run the sequencer and work on the piece.

Hmm. I don't really like the sound of that: not so much because memory compression is necessarily "bad" - it's better than being paged! - but because it's on the path to the page-file... so, it sounds like the Opus memory is being treated like pages of dirty data rather than read-only pages of memory-mapped files. (i.e. I'm worried that the authors have not avoided the issue as I had hoped.) But let's not base any decisions on that just yet: I need to check the details in a dead-tree reference, as the web is just too full of noise on this stuff.

When I bounced one track just now, the average hard page faults per second went up from 0 to 2. Does that constitute a spike? I can confirm that the page file is located on my C drive, the HDD.

Hmm. No, 2 PFs/second doesn't constitute a spike.

Assuming you're back to your normal 'working state', could you keep an eye on the task manager and see if the HDD spike is a feature of every bounce? It's possible we're being led astray by a one-off spike that has nothing to do with it. (Also, I haven't forgotten the spike disappeared when you unloaded all but the clarinet.)

If it *is* a feature of every bounce, it'd be good to find out what the HDD is doing if it's not servicing page faults.

It would also be good to get more detailed composition of your system's and VEP/OPUS's memory - in terms of what's resident, what's committed (backed by the page file, whether resident or not), what's memory mapped (i.e. backed by the source file on the SSDs, not the page on the HDD), etc. Resource Monitor should give you that information.

I deleted all instances except one which contains all the articulations of a clarinet and then bounced that track offline. Interestingly, it showed no page faults and very little HDD activity but the resulting audio track was the same as before with the same truncated notes. So does that suggest that it may not be a paging issue?

It's starting to look that way, yes, though I'd say the court is still out either way. The HDD chart was only showing throughput rather than latency. Your sample player only needs to end up waiting on a single byte from the HDD in order to be disrupted.

Incidentally, did you notice your ethernet usage with the single instrument? (It was showing outbound data throughput equivalent to about 10 streams of 24-bit, stereo, 48kHz PCM audio - if it had been real-time - in your screenshot. So just curious if that's a good proxy for estimating how much render work it's trying to do.)

 

[MarcusMaximus]

Hmm. I don't really like the sound of that: not so much because memory compression is necessarily "bad" - it's better than being paged! - but because it's on the path to the page-file... so, it sounds like the Opus memory is being treated like pages of dirty data rather than read-only pages of memory-mapped files. (i.e. I'm worried that the authors have not avoided the issue as I had hoped.) But let's not base any decisions on that just yet: I need to check the details in a dead-tree reference, as the web is just too full of noise on this stuff.

Oh dear, that doesn't sound so good but maybe is something configurable within Windows? I will look forward to hearing what you turn up.

Hmm. No, 2 PFs/second doesn't constitute a spike.

Ok, that's good I think..

Assuming you're back to your normal 'working state', could you keep an eye on the task manager and see if the HDD spike is a feature of every bounce?

Ok will do.

If it *is* a feature of every bounce, it'd be good to find out what the HDD is doing if it's not servicing page faults.

It would also be good to get more detailed composition of your system's and VEP/OPUS's memory.. Resource Monitor should give you that information.

That's pretty technical stuff but I'll have a look around in R.M. and see if I can identify those factors.

Incidentally, did you notice your ethernet usage with the single instrument?

I didn't but I'll have another look at that pane while bouncing.

Ok, back to it and I'll let you know what I observe.

 

[MarcusMaximus]

Here is a short video of the ethernet usage/throughput when bouncing one track in the single instrument project. Strangely enough this bounce turned out perfect though I didn't do anything different from before. Once I loaded the full project again I got the old issue.

The HDD is not spiking during bounces - it seems to spike at other times when apparently nothing is happening, must be some sort of activity going on. So I think we were being led astray.

In terms of the memory question, with the full project: the system is using 27.26 GB, VEP is using 21.84. Committed is 34.26 for VEP. I can't see how to tell what's memory mapped but here is a screenshot of R.M. with the project loaded but not running/bouncing. I hope that tells you what you need to know:

 

[aldous]

The HDD is not spiking during bounces - it seems to spike at other times when apparently nothing is happening, must be some sort of activity going on. So I think we were being led astray.

That's all useful stuff, thank you.

I did my own homework: hard page faults for compressed pages are allocated to the system, and not the application whose pages were compressed (the application sees a soft fault of the newly decompressed page.) So it's possible the hard faults I expected are showing up elsewhere in Task Manager.

Separately(-ish!) I'm interested in the rise you see in compressed memory when starting the sequencer. When that happens, do you by any chance see a reduction in the "cached" memory?

When was the last time you restarted the PC? After a restart, does it work well for a bit before degrading?

Oh dear, that doesn't sound so good but maybe is something configurable within Windows?

It's possible to disable memory compression, but I don't think that'd be a good idea. If this is related to your render issue at all - and I'm keeping an open mind - the system is almost certainly reacting correctly to the way applications are using memory. It's those applications that'll be the problem.

 

[MarcusMaximus]

Separately(-ish!) I'm interested in the rise you see in compressed memory when starting the sequencer. When that happens, do you by any chance see a reduction in the "cached" memory?

Actually I wasn't clear enough. What I meant was overall I see a rise in compressed memory while working on the piece in general. At the moment, and that's without doing anything beyond loading the project, it's showing 14.5 compressed and 11.5 cached. Those numbers stay the same when I run the sequencer.

When was the last time you restarted the PC? After a restart, does it work well for a bit before degrading?

I always shut both computers down when I finish for the day so it's a fresh boot every day. Today I did get that good bounce after starting everything up for the first time so yes, it might.

It's possible to disable memory compression, but I don't think that'd be a good idea. If this is related to your render issue at all - and I'm keeping an open mind - the system is almost certainly reacting correctly to the way applications are using memory. It's those applications that'll be the problem.

Yes I had guessed that interfering with the way Windows handles memory might not be such a good idea. So the problem is likely to be more inside the apps and I'm guessing it's the particular interaction between Logic, VEP and Opus that's the real issue.

 

[aldous]

Actually I wasn't clear enough. What I meant was overall I see a rise in compressed memory while working on the piece in general. At the moment, and that's without doing anything beyond loading the project, it's showing 14.5 compressed and 11.5 cached. Those numbers stay the same when I run the sequencer.

OK - so you see the compressed amount drift upwards, but not at any particular time? I was wondering about a file handle leak, but if you're regularly restarting then the compressed memory will tend to drift upwards (i.e. it's common for most pages in a system to be "cold" - rarely used and on the queue to be paged out - after a period of up-time, but on start-up they all start "hot".)

Yes I had guessed that interfering with the way Windows handles memory might not be such a good idea.

Good I'm still a bit surprised that instinct isn't more common - there are a lot of posts advising people to turn off memory compression (along with installing this or that memory/CPU "cleaner" and other associated snake oil.)

So the problem is likely to be more inside the apps and I'm guessing it's the particular interaction between Logic, VEP and Opus that's the real issue.

It's possible you're hitting a bug that requires a syzygy of all three applications, but my money is on two more generic issues: (1) incorrect coordination of offline bouncing by one link in the Logic<->VEP client<->VEP server<->Opus player chain, so the Opus player is being driven faster than resources allow; and (2) maybe Opus is dropping voices that it should be able to serve, possibly because of poor memory management, insufficient or excessive preload.

The evidence of problem (1) is simply the time it takes to render vs the time it takes to play back. If the ethernet traffic is anything to go by, it looks like it's rendering at about 4x real-time. Sound right?

As for (2), it looks like VEP/Opus is using an unhelpful amount of memory - since preloading an amount that causes memory pressure is always unhelpful - but it can play back your project without issue, so it's hard to argue it's "wrong" during a bounce. We haven't so far found the bottleneck for sample playback, but looking into memory more deeply is going to be quite involved. Currently I can't see damning evidence of anything particularly buggy.

I'm not sure I've asked how Opus's own multithreading is set-up: is multi-threaded voice renderer disabled, e.g.? Otherwise, I'm running low on inspiration for investigations (at least, ones I can describe at all concisely.) If you haven't already, my next step would be to approach EW support to ask about any known issues with offline render, and - if not - VSL support after that.

 

[MarcusMaximus]

OK - so you see the compressed amount drift upwards, but not at any particular time?

Yes that's right.

Good I'm still a bit surprised that instinct isn't more common - there are a lot of posts advising people to turn off memory compression (along with installing this or that memory/CPU "cleaner" and other associated snake oil.)

I've always been pretty conservative when dealing with OS's and tend to avoid the likes of CleanMyMac etc.

If the ethernet traffic is anything to go by, it looks like it's rendering at about 4x real-time. Sound right?

Hard to tell for sure but judging by the time it takes to bounce I'd say that's about right.

I'm not sure I've asked how Opus's own multithreading is set-up: is multi-threaded voice renderer disabled, e.g.? Otherwise, I'm running low on inspiration for investigations (at least, ones I can describe at all concisely.) If you haven't already, my next step would be to approach EW support to ask about any known issues with offline render, and - if not - VSL support after that.

Yes that is disabled and Voice Renderer Threads is set to medium, though that's probably irrelevant if it is disabled. Would it help to enable it?

That's fine and you have been a great help - thank you for sticking with it. I will contact support as you suggest.