Nexrender: Using AWS/Google/Azure as a render node?

[dzg]

Does anyone have experience/success running aerender and nexrender on a cloud machine?

If success ... would love to know the specs of the virtual machine (cores, os, etc.) and what was needed to get it running.

Thanks!

[inlife]

One of the members of our community also recommended this one to me quite recently: https://www.paperspace.com/core

[K-JBoon]

I've had limited issues on a standard Windows AMI on EC2, without accelerated graphics, you should mostly be looking for reasonable CPU power (8+ cores) and good IOPS on your disk. It's a bit of a hassle to set it up, but there's no major blockers, though I did only end up using @nexrender/core

[bluematter]

Yup, I am happy to share.

I am running nexrender in production with users. I am still learning about the bottlenecks and figuring out ways to scale with cutting costs. Currently, this is what I am using.

AWS EC2
t2.xlarge
4 vCPUs
16 GB RAM
200 GB SSD

August Costs
EC2 = $198.48
S3 = $75.39
RDB = $378.00

Total = $651.87

I have $25k in AWS for this year so I am pushing it hard. I plan to add some more CPUs (8 cores) and I am looking into upgrading my machine to have better GPU capabilities.

[limzykenneth]

I have a render server setup on Azure but stability of the render is not great at the moment, AE occasionally hangs and I'll need to kill it after it stop progressing with the render for a set amount of time. It could be my setup as I'm running on a minimal budget.

What I find hardware provision wise is that if speed is not a factor, 2vCPUs + 16 GB RAM is about the minimum you can get away with (there's extra computing budget to run a simple web server on it too). Lower than that it will not work stably at all. I personally will look towards scaling up (+GPU for example) before scaling out with this starting point. Cost of this minimal setup adds up to just over $100 per month with a render speed of 1 minute video in about 1 minute provided AE didn't hang. It's not fast but it works for me.

OS wise, you are stuck with Windows. I'm using Windows 10 on Azure but I see no reason why Windows Server won't work. Nexrender wise I'm using a combination of nexrender-server + nexrender-worker with nexrender-server running with NSSM. nexrender-worker could have been ran from NSSM but for reason I still couldn't decipher, if running from NSSM AE won't read user added fonts so I have to keep a command prompt running nexrender-worker open at all times.

There are limits both technical and budget wise for me but it certainly is a good area to explore for Nexrender.

[brassel]

@limzykenneth, did you happen to find out why "AE won't read user added fonts"? We encountered the same problem in a different setting without nexrender-server + worker.

[limzykenneth]

@brassel I've looked around and tried many different things but so far I just cannot figure out why. It only happens when I run the worker through NSSM but not when I run the worker directly in command prompt.

What is your setup like? Maybe we can narrow down the potential problem by comparing setups.

[inlife]

Regarding the command prompt and rendering, maybe it is something related to the configured PATH settings. At least it sounds like something that potentially could be the reason. Is there a way to detect differences between PATH values for both cases @limzykenneth?

[brassel]

What is your setup like? Maybe we can narrow down the potential problem by comparing setups.

Yes, good idea, @limzykenneth. I would say that our setting is a bit simpler than yours. We have our own process running which listens for render commands. When a command comes up it will download a zip with an AE-project and also a folder of fonts. For each fonts it checks if it is installed and installs it when necessary. Then nexrender is called.
This works well, but then we installed some of the fonts in a different shell window (but with the same script). And then we got very weird behaviour: Our script will find the installed fonts and will not install them again. But AE insists that it cannot find those fonts.

But as far as we could make out, the other shell was started with the same rights, especially the same user. And so we are still baffled by this effect. I will start to look at the problem some more and keep you posted here with any findings.

@inlife checking the paths is a good idea although I cannot imagine how a difference in paths could come up in our setting.

[limzykenneth]

@inlife The path has no difference between running from command prompt or from NSSM, I got the value from a simple node.js script printing the value of process.env.PATH and they matched.

@brassel How does your script checks for installed fonts? Does AE throws errors when you attempt to render with those fonts? For the AE does not throw and error, it just renders but substitute the fonts with Times New Roman.

[atitkothari]

Yup, I am happy to share.

I am running nexrender in production with users. I am still learning about the bottlenecks and figuring out ways to scale with cutting costs. Currently, this is what I am using.

AWS EC2
t2.xlarge
4 vCPUs
16 GB RAM
200 GB SSD

August Costs
EC2 = $198.48
S3 = $75.39
RDB = $378.00

Total = $651.87

I have $25k in AWS for this year so I am pushing it hard. I plan to add some more CPUs (8 cores) and I am looking into upgrading my machine to have better GPU capabilities.

@bluematter I'm facing a long wait time of about 10 seconds for after effects to load before it can even start rendering. Is it the same for your machine? If not, what can I bump (CPU clock cycle, RAM, etc) to reduce AE load time? Thanks!

[brassel]

I'm facing a long wait time of about 10 seconds for after effects to load before it can even start rendering.

@atitkothari, theoretically, the --reuse is the only way to help with this. However, there are some issues with that, discussed in other issues around here.

[limzykenneth]

Just an update, I've managed to solve the fonts issue. Turns out the font needs to be installed for all users using the context menu, install for all users. After that the nexrender-worker instance running via NSSM is able to use the installed fonts to render text in AE.

One last final hurdle for me to scale the system is the problem of AE hanging occasionally, requiring a kill command to stop the rendering.

[brassel]

Sorry that I didn't come back about the fonts yet.
But about the hanging AE: We have that problem as well, occasionally. It seems to be connected to size of RAM.
Especially, importing videos in AE in a certain format seems to need a lot of RAM.
(Can't remember the format. I can find out next week. But maybe this quick remark is already helpful.)
So, increasing RAM and having a closer look at input formats helped us - at least so far. :o)

[limzykenneth]

@brassel Would be great to know what the format is, my setup is using exclusively H264 and mp4 (probably) and I wasn't expecting it to pose a problem but if they are I can try switching things out. Can't really test on my on my setup right now as it is a production environment so I'd prefer not to change things up like that at the moment.

[brassel]

Hi @limzykenneth, I have discussed this with the video team and it seems that the only two formats which made trouble with the memory footprint seem to have been "wmv" and "m4v".
I also got some more reasons which might be connected to hanging AE:

• sometimes AE will hang if it cannot find a footage item (e.g. we wanted to replace it and deleted it from footage to get a small project but forgot to add it to the job file). Now we always make sure that a (small) footage item is always included in the project
• AE allocating too much memory of the machine - I am not the AE expert but it seems that you can set AE to only use a certain percentage of available memory in the AE UI. I think we set this to about 80%
• having said this, running low on memory is the most often reason to hang - allocate a good amount to your VMs

One last point we have sometimes noticed that an AE instance which is kept running is steadily growing in memory. (Memory Leaks?) But we did not really succeed in getting --reuse working in our automatic pipeline. Did you?

[limzykenneth]

Thanks @brassel. I'll need to set up a resource monitor to see if it is indeed RAM issues on my end as well.

I have tried the --reuse flag but it doesn't seem to work, even when I'm using aerender directly, I think this is a bug on Adobe's side so I'm not hopeful it will get fix quickly if at all. Shame though as it would speed things up a whole lot for me.