Question : Problem: General questions regarding multi-core CPU's and memory utilization

I recently started a new job where I'm in the process of researching hardware possibilities for a high performance computational cluster.  We currently have a research cluster consisting of 40 dual CPU servers that is about 4 years old.  One question that has been brought up recently is if, now that you can get quad-core CPU's, is there a significant trade off between CPU processing and memory access and utilization?

Our current idea is to build the new cluster with 320 CPU's to start and leave room for future growth.  Companies like IBM, Dell, etc. are now making servers available that support two quad-core CPU's each, so theoretically we could reach this number with 40 dual quad-core CPU servers.  What I'm not so sure about is whether or not packing 8 CPU's into a single server with 32gb of RAM makes sense.  The cluster will be running commercial products like Mathematica as well as home-brew MPI-based applications.  Our current cluster is bound much more by memory utilization than I/O so we're anticipating seeing similar behavior in the new cluster.

So the really big question is whether the CPU's will become memory-starved if all 8 CPU's on a given node are running memory-intensive applications.  I know it's a bit of a "religious" issue but is Intel or AMD considered better in such an environment, or are they pretty much on equal footing?  Are more cores per chip always better, or is there a reason why we would want fewer cores per chip, or perhaps (in the quad-core case) single-socket instead of dual-socket systems to prevent RAM bandwidth starvation?  Bottom line - is it worth the additional cost (not just for servers but physical space, power, cooling, etc) of splitting CPU's out into more physical servers to improve computational throughput?

Answer : Problem: General questions regarding multi-core CPU's and memory utilization

I'm agreeing with the AMD call on the CPUs.  Personally I prefer Intel for a number of different applications, but for what you're doing AMD is better.  The AMD chips have the memory controller on the CPU/die which leads to better access times when going to RAM.

Now, as for memory, with all these CPUs why would you onlly go with 32GB of RAM?  In this type of environment I'd really hope that you're running a 64-bit OS, in which case you should seriously consider going to at least 64GB on each server.  Windows 2003 Enterprise (R2) can support up to 2TB of RAM - get as much per server as you can afford, but honestly going with 32 could potentially cause a bottleneck if you're running memory intensive applications.

That being said, I assume your applications are coded for actually utilizing all these available CPUs?  It they haven't been optimized for this type of environment you're really putting the cart before the horse.  One other thing to keep in mind is the size of the L1, L2, and possibly if they have it L3 cache sizes on the die - and what's actually available to each CPU.  Many new processors have shared caches between all the CPU's on each physical die, look for a die that will either give you a larger shared cache or provide multiple caches specific to each CPU core.

Building an environment like this is a pretty interesting project (which I've had some experience with), but your time spent doing background research and thinking out all layers of the design will be rewarded and easily paid back when you build out an environment that works well and gives blazing performance.

Are you looking at any particular processors?  If so, which ones?  i'd be happy to look at specs and offer some input.