Yes, remove the belt and see if the noise stops. That's a good way to narrow down the source of the noise. If it stops, check each pulley for noise, roughness or lateral play. Suspects are, as Mike suggested, the idlers, tensioner, water pump and a/c compressor. Knowing your mileage can...