I am running my suround sound system through a optical cable as it dosnt decode sound from HDMI, and the suround works fine for blu ray movies and games in 5.1 (my receiver can reproduce it in 7.1) DTS and DD 5.1.
You can definitely run 5.1 through your optical cable.
My receiver is 1 year old pioneer THX 7.1 ($1500)
I think originaly I had to play with some settings in the PS3 menu. I think there is a setting to make the optical out your main audio out, and selection for audio formats supported by your reciever.