About fifteen years ago, when I was in a supermarket near my Tokyo hotel, an American woman came up to me with two large bottles of water. Raising them both so I could see, she said she couldn't figure out which one to buy.
Suspecting I knew the answer, I asked her what she intended to do with it and she explained that she'd heard so much about not drinking the water when one travels, she thought she'd better buy these for brushing her teeth and for drinking.
I told her that they were both just water, so either would be fine, but so would the water in the hotel.
I've been drinking it now for 34 years, and have never experienced a bad taste or odor, and never heard anyone say anything about the water being unsafe. Typically, the water pressure is even quite good. It certainly is in our current apartment.
So, even if you decide you don't want to drink the Kool Aid, the water itself is fine.