Performance tuning
Because overall system performance depends on so many different variables, performance tuning is part art and part science.
Be sure to also the read the section titled "Performance Tuning" in the repository(8) man page
Measurement tools
Before making changes to try and improve the performance of your system, you should start by taking measurements to determine where you have the most opportunity for improvement. You should also be prepared to measure again after making changes to see how much performance has improved.
There are many different tools and methods for measuring system performance and we can't cover them all here. However, here are a few suggestions:
- Vesta-specific tools:
- vreposmonitor(8) provides information that can help diagnose the repository's performance
- The repository also includes an optional feature that is normally removed at compile time to gather very detailed fine-grained timing data. It's probably only interesting to people willing to work on the repository source code.
- Cross-platform tools:
ntop can be used to measure network usage.
The Bonnie++ benchmark programs can be used to measure disk performance. You can use it in /vesta-work and you could even run it as a tool under the control of the Vesta evaluator (though you would probably want to use the -noaddentry flag).
- OS-specific methods:
- Linux
- You can see how much raw network traffic there is by looking at the "RX bytes" and "TX bytes" numbers in the output of ifconfig(8).
You can count a number of interesting things about a specific process by looking at /proc/<process ID>/stat. For multi-threaded programs like the Vesta server daemons you may need to look at /proc/<process ID>/task/*/stat. (See the proc(5) man page.)
Here's a simple script to count virtual memory page faults of a process: process-page-faults.pl
- You can also get the total amount of CPU time a process has accumulated there, or by using ps(1).
- You can get information on the rate of I/O traffic to a disk with iostat(1)
- Linux
Minimizing I/O Resource Contention
As is often the case with server systems, you may be able to increase total throughput by reducing contention for the same I/O resource (either storage or network). There are several things you can do in the configuration of your Vesta server(s) to reduce contention for limited I/O resources.
Put the cache daemon and repository daemon on two different hosts. In general the repository daemon has higher resource requirements than the cache daemon, but the cache daemon's needs can be significant (depending on how you're using the system). Putting them on two separate hosts can help. Keep in mind that each of the daemons should have its own files stored on a disk local to the host where it runs.
Use multiple disks. Aside from using RAID, you can reduce contention by separating different storage needs into multiple volumes (ideally different physical volumes, if possible on different adapters):
The repository's shortid pool ("[Repository]metadata_root/[Repository]sid_dir", normally /var/lib/vesta/sid) where all the source and derived files are stored.
The repository's transaction log ("[Repository]metadata_root/[Repository]log_dir", normally /var/lib/vesta/repos/log) which must be written to on every change to the directory structure under /vesta and /vesta-work.
The cache daemon's storage ("[CacheServer]MetaDataRoot/[CacheServer]MetaDataDir", normally /var/lib/vesta/cache) where all the information on build work that has been done previously is stored (other than the file contents). (This could be a non-issue if you have your cache and repository on separate hosts.)
- Virtual memory swap space. Both the repository and cache daemons can use significant amounts of memory (depending on how the system is used). If you find that your system is doing a significant amount of virtual memory swapping and you can't add more physical RAM, it can help to have the swap space on a different physical volume from other storage.
- The operating system. In a large installation, the operating system volume will probably need much less I/O than the other disks already mentioned, but it may still be worth separating it from them.
Use multiple network interfaces on your repository server.
- A channel bonded setup (also called "bonded Ethernet") would allow you to simply multiply the amount of network bandwidth available.
Another option would be splitting the way clients contact the server across different IPs on different network interfaces. For example, you could set [Repository]NFS_host to a hostname for one of the server's network interfaces and [Repository]VestaSourceSRPC_host to a hostname for a different network interface on the same server. If your users make significant use of the ability to ship build results as symlinks, it might make more sense to use one network interface for the OS-native NFS interface used to serve the shortid directory to the clients and use another for all the services provided by the Vesta repository.