Slowdown over time on large files

I start at 30fps doing a 19G blue-ray file with 9600GT, slowly the "system cache memory" goes from 500,000 up and up to 1,500,000 while the frame rate goes down and down to 12fps. Even though total physical memory is 2G and always has 1.5G available.

I am rendering it to 1280X720, delinterlaced, about 12MBS.

What can I do to stay at the 30fps that I start out with?