Someone's always going to be disappointed at your decision. There is no way that we can please everyone. Therefore your choice is, do you disappoint a lot of people a little bit, or a few people a lot?

On the other hand, you could think of it as a positive. Do you please a lot of people a little bit, or a few people a lot?

Who are you going to disappoint is another way to say who are you going to serve. But I wouldn't dwell on the people who will be disappointed, be aware of their existence, and ignore them as you focus on who you want to serve.