Fraps settings:

Use 25 fps because it is high enough speed for most games, and mainly because it keeps the file size smaller compared to the quality. Also make sure you have it on full size or w/e.
UPDATE:
*Some have a preference for 30fps because they are able to notice the slight difference. Keep this in mind, but also remember the file size.
**If you plan be adjusting the velocity (doing slow-motion effects) you should record/fraps at a much higher FPS (50-60+)
------
Code:
25fps vs 30fps:
1 second = 3kbits/s (example)
3000bits / 30frames = 100bits (12.5bytes) per frame
3000bits / 25frames = 120bits (15bytes) per frame
25fps = 1.2 times the quality of 30fps
Quality of 30fps @ 3kb/s = Quality of 25fps @ 2.5kb/s
Quality of 25fps @ 3kb/s = Quality of 30fps @ 3.6kb/s
Project Properties for Vegas (do this before when you open vegas, before you do any editing or importing):
To Open: " ALT+ENTER " on keyboard, or " File -> Properties " on top menu

i use 1440x840 but choose whatever resolution you fraps at
When you are setting the name and such of your rendered file, make sure you set it to be a *.avi file.
Render settings:
"File" -> "Save" .
"File" -> "Render as..." -> "Custom" ->

once again, i use 1440x840 but set it to whatever you fraps at
you can set the bit rate higher if you want but it will make the file bigger, 2-3kb/s should be pretty good for most games. With larger resolutions and lots of smaller details you should increase this. If you use a high resolution for your videos, or you want your videos quality to be higher (if the video is shorter?) , you can go higher on the bitrate, but keep in mind how it will effect the file size. (Do the math. Length of video in seconds * Bitrate). 20minutes video : 3kb/s = ~550mb , 5kb/s = ~750mb , 8kbs = ~1.3gb
------
you can do multi-pass rendering if you want to, it makes the video higher quality. i don't usually do it unless I am rendering a long term project, because it takes about twice as long to render and can be buggy + have problems.
------
** If you dont have that codec on your list then install:
Free-Codecs.com : Download K-Lite Codec Pack 6.2.0 FULL, K-Lite Codec Pack 6.2.0 STANDARD, K-Lite Codec Pack 6.2.0 BASIC : K-Lite Codec Pack is a collection of codecs and related tools
Download the full one, and when installing it do the setting "lots of codecs + player".
h264 has a higher quality picture than XviD at the same file size.
*** Fraps Version 2.8+ , Sony Vegas Version 6.0+
-------
UPDATE:
For all the people with the problem of H264 option not appearing on the list of codecs in sony vegas download:
Browse x264vfw Files on SourceForge.net
Notice there is one for 32bit and one for 64bit.
-------
There can be a lot of problems with h264 + sony vegas. If you really get stuck I recommend rendering as .WMV or using XviD for .AVI.
-------
UPDATE:
For uploading to youtube it is best to use this mp4 alternative

*1440x840 is just my resolution, set it to the same way you set your project settings
**For youtube it is best to stick to their resolutions also.
1280*720 (720p)
1920*1080 (1080p)
***Bitrate can be higher/lower, this is just the my recommended amount. If you change it keep the file size increase in mind.:
-------
If you have any suggestions or think anything is wrong with this guide, post and I will update it.
- Not using H264/Avi for youtube recommended by Dragonshadow
- Recording at higher FPS recommended by Dragonshadow
------