The speed of your server hardware, its network connection and its distance to your users dictate the response time experienced by users.
So, server response time directly influences load time, as the browser always needs to wait for the HTML before it can continue downloading additional resources. If you reduce server response time by 1 second, you will reduce total load time by 1 second.
Usually, server response time is only a problem for the page itself, not for external resources, because the page itself is dynamically generated and the external resources are static files. Most servers manage to server static files quickly, but often have trouble serving dynamic pages.
We define server response time as the time it takes the browser to download the HTML code of the page. Alternatively, you can use a definition that looks only at the time it takes the server to start sending the HTML code, the time to first byte (TTFB).
For most web applications, and when the user has a fast connection, the time to first byte and the time needed to download the entire document will be almost the same.
A reasonable target for server response time is 200 ms (0.2 seconds).
200 ms is Google's guideline as well, and the point at which Google PageSpeed Insights will stop complaining about response time.
Some applications are so heavy that it is hard to get them to load in less than 500 ms, but usually it is possible to achieve this goal.
Measure response time with this KeyCDN tool. Enter your site URL and
response time will be measured from 14 worldwide locations. Add a query string
?123 to the URL to measure response time without caching.
Check the results for those locations nearest to your users. If most of your users are from the same region, there's no need to worry about response times on the other side of the world.
Fixing server response time
The easiest server speed fix for most websites is faster hardware. Your site is probably hosted with a shared hosting provider or on a VPS (virtual private server). Both usually provide pretty disappointing performance, because they are optimized for scale (hosting many sites) rather than performance (making a single site fast). Some providers are worse than others, but hosting your site on fast and dedicated hardware is always better.
Have you reached your server response time goal? You can now start looking at the weight of your web pages to shave even more off the load time of your site. Continue to fixing heavy pages.