Isolating Network Problems

How do I determine if slow performance is due to the network?

PROBLEM

How do you determine if slow Perforce response time is caused by network problems?

SOLUTION

First, review Perforce log files and use standard system administration tools to diagnose slow response times.  A network issue would be suspected if Perforce commands run quickly on the local machine but run slowly across a network.  Or compare the lapse time against the usage time, for example, if the information is available in the Perforce logs.

While it is possible that an extremely large user operation could be causing the Perforce Server to respond slowly, persistent slow responses to Perforce commands are usually due to network troubles. Any of the following may cause slow responses:

1. Misconfigured domain name server (DNS)
2. Misconfigured Windows name server (WINS) or Windows domain
3. Slow network response

Notes

Although solving network problems is beyond the scope of this note, here are some suggestions for troubleshooting them.

p4 info

A good initial test is to run the p4 info command. If this does not respond nearly immediately then there is a network related problem. The p4 info command uses the P4PORT setting to contact the Perforce Server. If the P4PORT setting uses the hostname of the server machine, a DNS lookup is required to fetch the server IP address. If the DNS server is failing or the network is slow, the lookup process will take time. You can use the IP address directly to avoid the DNS lookup.

Hostname vs. IP Address

On a client machine, try using the Perforce Server's IP address in the P4PORT setting. This will avoid the DNS lookup used to convert the hostname to an IP address. Here is a P4PORT example which uses the server hostname:

	P4PORT=hostname:1666

Using the IP address directly, the P4PORT setting then looks like this:

	P4PORT=1.2.3.4:1666

If do not have the Perforce Server machine's IP address handy, you can use the ping command to find it. Here is an example of how you would run the ping command:

	ping hostname

The output of the ping command will list the IP address for the hostname.

If the p4 info command responds immediately when you use the IP address in the P4PORT setting, you have a misconfigured DNS.

"p4 info" vs. P4Win

The p4 info command is processed by the Perforce Server. As the Perforce Server compiles the output information for the command, it does a reverse DNS lookup on the client's IP address. A forward DNS lookup may be fast, while a reverse DNS lookup is slow. One way to determine if a reverse DNS lookup is slow is by using P4Win, the Perforce Windows Client. The "Show Connection Info" operation in P4Win does not perform a revese DNS lookup.

You can compare the response of P4Win's "Show Connection Info" with the response from the command line p4 info. If a "Show Connection Info" operation is fast while a p4 info is slow, there is a reverse DNS lookup problem on the Perforce Server machine.

Note: This test is only valid for Release 99.1 and newer Perforce Servers. In releases prior to 99.1, the Perforce Server always did a reverse name lookup, whether the request was coming from p4 info or P4Win.

The "hosts" file

You can work around DNS problems by adding hostname IP address entries to the hosts file. This task is typically performed by a Systems Administrator; be sure to follow your company's standard procedures.

Rather than using an IP address for the P4PORT setting, you can add a hostname IP address entry in your hosts file. The host file can be tricky to find. Here are a few places you can look to find it.

1. NT/2000/XP: C:\Winnt\System32\Drivers\etc\hosts
2. Win98: C:\Windows\hosts and hosts.sam
3. Win95: C:\Windows\hosts and hosts.sam
4. Unix: /etc/hosts

On Windows 95 and 98, it might be necessary to copy the hosts file to the hosts.sam file.

Here is an example entry for the hosts file:

	1.2.3.4		hostname

If you have determined there is a reverse DNS problem, you'll need to add an entry for your client hostname in the Perforce Server machine's hosts file.

Wildcards on Windows

In some cases, p4 commands using unquoted filepatterns with a combination of depot syntax or client syntax with wildcards can cause delays.

	p4 files //depot/*
	p4 files //client/*

The delay occurs because the wild card expansion routines for Windows will mistake the depot name or client name for a network system name. You can prevent the delay by putting double quotes around the filepattern:

	p4 files "//depot/*"

See technical Note041 for more information on special handling of wild cards.

Network Topology

Many of today's networks use 100 Mbit technology. The maximum theoretical transfer rate of this type of network is 10 Mbytes per second. In most cases slower transfer rates will be experienced. If a network is saturated, transfer rates can drop off significantly, maybe as low as 4 Mbytes per second. In these cases use of network routers or switches can help.

Client on a Network Filesystem

It's possible that the p4 executable itself is on a networked file system which is performing very poorly. To check this, try running:

	p4 -V 

This command simply prints out the version information and does not attempt to do any network access. If you get a slow response, network access to the p4 executable itself may be the problem. Try copying or downloading the p4 executable onto a local filesystem.

Using a network filesystem for the Perforce client root can also cause slow command response. Every byte transferred in a Perforce client command must go through the network twice.   Once from the Perforce Server to the Perforce client application. Once from the Perforce client application to the network filesystem. If the network is saturated and the client workspace contains large files or a large number of files, you can expect commands like p4 sync to be slow. Try moving the Perforce client workspace to a local filesystem.

Server on a Network Filesystem

Using a network filesystem for the Perforce Server must be thought through very carefully. Just as in the scenario above with the Perforce client, data to and from the server application must go through the network twice. The Perforce Server also uses file locking on vital data files. Not all network filesystems have efficient locking implementations and some are buggy.

If the network is saturated and the transfer rate is down to about 4 Mbytes per second, the Perforce Server will not be able to satisfy client requests. Each client request will use part of the network, the Perforce Server must share that resource in order to access the network filesystem. A cheap hard drive these days will provide a 20 Mbyte per second transfer rate. A good SCSI hard drive can transfer as much as 160 Mbytes per second. If at all possible try to avoid a network filesystem related server bottle neck.

If you have a specialized network filesystem like a NetApp filer, consider using a private Gigabit fiber connection directly to the Perforce server machine. It is reasonable to expect transfer rates of 100 Mbytes per second on an isolated gigabit connection.