I don't, since it can possibly create a spark, and that wire is connected to the amp. With electronics, you never know what could happen. So just disconnect the ground to be safest.
I've always removed the fuse without a problem. I have several batteries, and disconnecting several terminals all going to that one power wire wouldn't have been easy. You remove the fuse from the holder, (not the amp), you break the circuit, and there is no more power going to that wire or amp. Simple as that. Just as big a chance of the batt sparking as the fuse..