Latency is the measure of how long it takes data to go from Host A to Host B. Latency under 100ms is considered good. It really only becomes a problem when you start getting up to 250ms of latency. At this point the delay in the audio becomes apparent to the caller and agent. It’s possible for agents to learn to work with the delay but it’s not desirable. In situations where you have US callers with agents in the Phillipines there’s nothing you can do but work with it. It is quite impressive how well some agents can be working around the delay.
Latency mostly tracks with geographical distance since the speed of light is only so fast. So the latency from New York to the phillipines will always be much higher then the latency from New York to Los Angeles. If you are using a large regional ISP like a cable modem provider then there is really nothing you can do. Latency is not much of an issue for what most of the internet is used for. If you are in an enterprise network space like a Data Center or Colocation Facility then theres a chance they might be able to help. It will depend upon if they have any alternate routes with better network conditions.