Basically, you are running out of RAM. One way to reduce the memory usage for the render elements is to write the image to a .vrimg file and then use the vrimg2exr converter to extract the channels. 
(*) Make sure you are using the V-Ray VFB; 
(*) In the Common tab of the Render scene dialog set the resolution to something like 320x240; 
(*) In the V-Ray VFB settings, turn OFF the "Get resolution from Max" option for the VFB, and specify the actual resolution that you need; 
(*) Turn off the "use memory frame buffer" option; 
(*) Turn on the "write raw image file" option and specify a .vrimg file in the field below it. 
(*) Turn on the "generate preview" option. 
(*) After the rendering is complete, use the vrimg2exr tool to extract the render elements as .exr files. You can find the vrimg2exr tool in the Start menu > Programs > Chaos Group > V-Ray for 3dsmax > Tools. There is also a GUI for it posted in the off-topic section of the forum, I think. 

Alternatively, you can simply render on a 64-bit machine with 64-bit 3dsmax and V-Ray. 

Best regards, 
Vlado