Mar 21, 2010
labs: Render droplets and Qmaster workflow. . .
Here is a quick demo showing the workflow for using the different render droplets on my site. PLUS! A very thorough demo and explanation of using Apple’s Qmaster application that is part of Final Cut Studio for rendering with Maya, After Effects, and Nuke, and some detail on how it works and what it is doing.
A lot of the Qmaster info and initial motivation for the different render droplets comes from Hugo Guerra’s post on vfxtalk.com.
The vimeo version of the demo is kind of low resolution so check out the QuickTime below if you want to see more detail:
QuickTime Version 1152×720 (93MB)
iPhone VersionAgain, if there are questions, definitely check out the Qmaster documentation or contact me at: andoruLABS.support
Here are links to the aerender docs, maya docs, and nuke user guide.
EDIT: A lovely commenter named David did some very thorough testing of the render droplet and Qmaster setup and posted his findings in the comments.
Hey Andrew, absolutely incredible site! thanks man! I’ve got the droplets up and running beautifully, specially the AE one and the AEae is fab too.
I’m really thrilled about the Qmaster workflow though, and have followed your tute in detail and double checked with Hugo G. too.
However although i’ve run the same settings as you, I have problems getting all the cores to get busy rendering.
I’m running a MacPro 8 core (Xeon) and have selected to run only 4 instances in the Qmaster pref pane, but still it tells me it’s only running one instance of aerender core and the processors don’t seem anywhere as busy as when I;m rendering with Nucleo Pro.
I’m already a very happy visitor here, but If you would have any tips to throw my way on how I can tweak the Qmaster settings or if there is something I need to look out for to get several aerender cores to run i’d super grateful.
THanks again for all your work and for sharing!
cheers /david
David, thanks for the kind words, it is nice to have the time spent on this stuff recognized a little bit.
You may need to add the “- mp” flag to your Generic Render After Effects template to see if that better utilizes the system. It might just be a matter of what your rendering as well but I would give that multi-processing flag a try.
So your template would be:
[EXECUTABLE] -mp -project [INPUT]
Andrew, thanks man for the quick response on that! will try that first thing tomorrow.
I did try a rather heavy HD project in AE with lots of motion blur. It took about 1.45 with only AE. 45 mins with Nucleo Pro and about 1.10 with Qmaster and the settings mentioned above, so it’d be nice to get it revving a bit. I am looking to get Qmaster going with a few Macpro’s though too, but I have to get going with some version syncing between the machines i guess, as after a bit of reading I understand that Qmaster is very picky with it’s own version between machines, QT etc..
Can I ask, do you find a big difference in behaviour of multi processors when rendering, depending on using Nuke or AE? I’m a complete AE devotee at the moment, but i’m beginning to look at Nuke and node based script progs… any thoughts on that? sorry, maybe off topic here… thanks again for your work and help! cheers /d
I would imagine Nucleo Pro is getting into a bit more multi-processor management than AE does so that is why it is probably quicker. If you get multiple machines running though, a multi-machine sequence is absolutely the way to go.
Qmaster will be fantastic for a multi-machine farm setup, AE even by itself is very efficient at managing what machines are rendering what frames and even just using the render queue it is quite scalable. It works the same way with the render droplets it a more manual farm setup. Have the droplets on each machine. Save the project file on the workstation, open it up on the other machines directly off of the networked workstation, no need to copy it really, and drag it into the droplet, each machine should always move to the next frame that needs rendering. Qmaster should be easier once it is setup though.
Nuke is quite good at utilizing multi-processor machines in the right circumstances. As shown in the links I posted, with Qmaster on a 8 core Mac Pro render times can increase dramatically. I really like Nuke and AE for different reasons. Nuke doesn’t really dumb anything down showing you every single node and connection which can be a huge benefit for what it is made for. It doesn’t have an editing toolset built-in as it is more a shot to shot based compositing system, but you can still get into sequencing shots. The best part about Nuke is its proper 3D system though, and its dedication to high-end vfx compositin, it is a lot more focused than AE.
AE really works in the same way as Nuke in the end it just puts more GUI around everything to make some aspects of the workflow easier. Don’t be reluctant to learn how Node based systems work though, it is easy to have a bias when it comes to software but you just have to keep an open mind and try things out.
Check out all the training and tutorials The Foundry has up on their site for Nuke:
http://www.thefoundry.co.uk/pkg_examples.aspx?ui=CBC2593A-2C9F-4EF9-84BE-C198B0171453
Andrew, great stuff! thanks for your suggestions. Everything makes complete sense and it sounds great about render farming with the droplet.. I’ll definitely give that a go! …the question I have then, for that, is whether I need to set up the render output in AE as a multi-machine render or just a regular image sequence?
Actually come to think of it… I’m unsure if I should set up the AE out put module as a multi-machine sequence if I use a Qmaster too? I suppose there is no need for that if I use Qmaster with only one machine, but what if I intend to connect a few machines?
Been checking out the tutes at the Foundry.. I really feel i need to start embracing the node system any day now. I had a short rumble with Shake a while back, but cowardly crawled back to AE.
Thanks again.. cheers /david
Hey Andrew, ok sorry for going crazy here, but as this was a full days extensive testing and serious amounts of trial and error, maybe and hopefully someone else can have some use for my notes on render farming with AE and Qmaster.
(feel really free to delete/edit this if it doesn’t make sense or I’ve got it all wrong.)
First a big thank you to Andrew for opening the door to this. This is a life changer for me!! 🙂
– The main problem I had with Qmaster was to figure out what settings to set in the AE render queue and what settings to control with flags inside Qmaster’s render settings.
– The faults, which the various non-working setting configurations caused for me, were that frames rendered over each other and that the qmaster couldn’t divide frame sections properly on the various processors and be efficient, which was the main point to achieve to begin with.
1. First off, while running one 8 core xeon MacPro and one 4 core xeon MacPro, I got by far the best power and speed, by running render instances for all processors on each machine in the Qmaster sys prefs pane.
2. I set AE’s internal prefs to use multi-processor rendering to render several frames simultaneously, left 2gb for each processor but to only use 6 processors for AE and two for other apps on the 8 core and 3 for AE on the 4 core. Then i played with the -mp (multi processor) flag in the Qmaster render commands. In the end i decided not to use it as it seemed to confuse the processors and sometimes cause all processors to render every frame.
3. The next gotcha was the -s and -e flags (start and end frame) in the Qmaster. I found that trying to override these with settings from the actual Render Queue inside AE did not work for me properly for some silly reason. During tests, more often than not, frames started to overwrite each other again. So I used -s [START FRAME] -e [END FRAME] and to typed the numbers in correctly in the fields in Qmaster’s render settings.
5. Also for a muli-machine render farm I used the -comp “RENDER QUEUE ITEM HERE” flag, otherwise there will be trouble. Again not sure why. It still seems to me a bit random which flags can over-run AE render queue settings and which cannot. I’m still confused on this, but what I’m jotting down here are settings that worked for me.
6. One of the last things I discovered is to remember to ruse the RESET button in your Qmaster sys pref pane often. This doesn’t reset any changes you’ve done to your settings, but it does reset the sharing status and seem to clear out some kind of cache somewhere. Very useful if things seem to go wonky for no particular reason.
7. The output settings I kept in the AE render queue and leaved the Item field in Qmaster blank – which worked fine all along, hence it’s left out of the Qmaster flags below.
8. OK so FINAL SETTINGS for AE multi machine render farm:
[Executable] -project [Input] -comp “AE RENDER QUE ITEM” -s [Start Frame] -e [End Frame]
Simple enough, but remember these settings need an AE Render Queue Item with Render Settings to: “Multi-Machine Setting” and Output Module to: “Any Image Sequence”
Also obviously important is to set you AE Render Executable and Input field at the top in the Qmaster render settings window.
And so the Speed resulsts:
A 450 frame HD 1080 25p sequence with motion blur, cameras and lights rendered with:
– only AE = 2h20min (on 8 core machine only)
– AE with Nucleo Pro = 1h (on 8 core machine only)
– Andrew’s AE droplet as fast as nucleo pro!!! 1h !! (on 8 core machine only)
– Qmaster render farm with 8core and 4 core 30mins!!
I hope all this isn’t all complete mumbojumbo and that works for someone else who’s having trouble with Qmaster.
And again, a big up for Andrew’s tutorial on this topic and even more his AE Render Droplet. The Droplet on it’s own took half the render time compared to AE only.
Good luck all hope all this helps someone!
Cheers. david
Hey Andrew and David, Wow, just Wow. I finally got this working last night and I’m thrilled to death. I haven’t had time to try it out on a heavy render but I had it working successfully on 5 (iMac i7, 2 MBPs – 09&08 – and two dual core mac minis – cerca 06), count it 5 computers at home and half of them were old ones at that! I told someone about it at work and they were instantly grabbed and wanted me to show them right away. She’s always wanted a use for her old machines and this seems to be not only a future safe method but also one that can work on dated machines! Thank you Both for this, I’ve Net Rendered with C4D forever and it’s been so easy but here at work I’m using AE more than anything and until this haven’t found a viable multi-machine management system, I’ve learned so much and i’m super happy that I can just tap into Qmaster to do it all.
Over a year later and I’m wondering why I never heard of it before.
– Adios – CWF
Also I wanted to know are you still using this method? what have you learned from it since? I also found last night that Compressor 4 mucks with a lot of the settings as Qmaster is only in Compressor now and not as independant, any experience with that? Really easy fix though but still.
I gave all my cores but two on the main iMac to the batch but what really is the benefit of decreasing the number instances? I’m still a bit fuzzy on that part of the whole AE/Qmaster deal.
Chris, glad the information is useful! To be honest I use Qmaster less and less and use my own applescript tools like RenderQ more and more. It requires a little more work on the users part, but if you have a strong understanding of how file sharing works from machine to machine something like RenderQ works very well.
Qmaster is pretty unsupported at the moment unfortunately, so I prefer a bit more of a homegrown setup for rendering.
The ideal setup for me for a simple render farm is a bunch of Macs using bonjour networking to talk to each other, the same way it works with something like iChat, and fire off renders across multiple machines, sending frames back to a central location on the network use later. I am looking to take RenderQ to this level at some point, but I am not really a software developer. Always closer and closer to happening though.
Finally i quit my day job, now i earn decent money online you should try too, just search in google –
blackhand roulette system