McNeel Wiki

I'm pretty sure the “Out of Memory” error is coming from the AVI compressor - probably because the AVI compression settings are set to Full Frames (Uncompressed). This codec seems to fail on many machines with certain sized images, and unfortunately it is beyond our control. On the other hand, you shouldn't be using it anyway as it doesn't produce good animations because they can't be displayed fast enough.

If you must use AVI, I would advise you to change the compression settings on the Render Animation dialog to something like DivX if you have it installed. If not you can download the codec from www.divx.com

However, a much better option in my opinion is simply to switch to MPEG output. It's much more compatible than AVI and usually better quality. There's a lot of information on this in the User Manual under “Render Animation”.

You can also test all of this by using the Video Encoder in “Interactive mode.” Just choose “Video Encoder” from the Bongo menu and point it at your single frames to re-encode the video.

Bongo FAQ