I can try a few suggestions:
1) Before you try a wireless setup, if you have an ethernet cable, plug it in to the PS3 and the iMac.
2) In System Preferences, go to Sharing, and then click on 'Internet Sharing'. From here, you can choose the connection you want to share (and I'm assuming the USB connection appears in this dropdown?
3) Now choose 'Ethernet' in the 'computers using' list box. Once done, you should be able to switch this on.
4) On your PS3, you will need to make sure the network settings are correct. For example, the gateway ip address will be the ip address of your iMac. The IP address will also need to be on the same network as your iMac. For example, if your iMac has an ip address of 10.0.0.1, then give your PS3 an IP address of 10.0.0.250 (the reason I have made this number so high is in case anything on the network is allocating IP addresses via DHCP, usually this number is high enough to be out of the DHCP range)
With a bit of trial and error, there's no reason this setup wouldn't work, it's supposed to with an Xbox 360.