I've cleaned a bunch of belts over the years. Best to use hot water and a lot of soap. Sometimes I've let em soak in a bucket full of hot water and some Oxyclean. Other times I've filled the bathtub up with hot water and some dawn dishwashing soap . Then took a stiff bristle brush and scrubbed each belt individually really good on both sides from one end to the other. Then turn on the hot water and rinse them real good one at a time. You'll be suprised at how dirty they really are. Take and hang them out to dry if its nice warm weather outside. It might take a while for them to dry completely, usually a couple days.
Then you can mask off the metal ends and spray some good quality vinyl dye on them to bring the color back to life, if they are faded.