You will have a few issues here. First you need to get terminal server licensing. The default for 2003 is 2 remote desktop connections for administrator use. The next thing you need to determine is how many users will be on at once. This will help determine disk usage(how many disk you want in the array running their stuff) and memory(you need a lot more then a gig).
Also is this server acting as your domain controller? Your best setup is a second server doing nothing but terminal services.