No the size should not make any difference. The class 6 should be faster than a class 4. Class 6 has a large options of speed that it can run at, but it should still be faster than class 4. However, they do just refer to the minimum transfer rate, so you could rate a really fast card as class 4, although it wouldnt make sense.
you can see some info about the speeds at:
http://en.wikipedia.org/wiki/Secure_Digital_cardI would look for reviews on your specific cards and see what others found