You have to think about it in terms of digital and analog. Analog requires a wire for each channel. So your RCA cables are a channel each, optical, which tx's a digital signal can carry 5 channels (plus) via one cable. What this means to you: is that just the red and white cable = stereo, and not Dolby 5.1, however you can have 5 channels through RCA cables through a 5 channel interface between your player and receiver.
1) Use an optical or coax digital cable from your source (DVD player, cable/satellite box, etc) to your receiver. In this case, your receiver is doing the Dolby Digital decoding.
2) Connect your source (DVD player, SACD, DVD-A player, etc.) to your receiver via 6 RCA cables (usually labelled "multi-channel in" on your receiver). In this case, your source equipment is doing the decoding.
With just the red and white cables, you're only getting two channels (left and right). You may use Dolby Pro Logic or any other DSP mode on your receiver on those two channels to get surround sound, but this is not Dolby Digital.