Introduction

Vesta supports distributed operation. Once set up correctly, vcheckout can be used to transparently check out a package from a remote repository. While the user works on their change all operations happen locally. When they're done, vcheckin will replicate their changes back to the remote repository.

This page describes what you'll need to do to set this up.

Remote checkout is built on top of operations between peer repositories: replication and mastership transfer. Setting up for remote checkout is really a matter of setting up the access controls to allow these operations between particular repositories. After that, the repository tools take care of the details.

Experienced users may want to skip to the quick reference at the end.

Background Information Crash Course

It's assumed that you understand how hostnames, hostname resolution, and the basics of how TCP/IP works. It's assumed that you understand and have some control over the networks between the sites you want to set up. (Be sure to be aware of any firewalls or special packet handling such as NAT or VPNs.) Here's a few background resources on these topics:

It's assumed that you understand the basics of client/server networking. It's assumed that you understand that Vesta uses a repository server which can be accessed by multiple clients.

It's assumed that you understand what the term "realm" means in a Vesta context and how a realm is used to form global identifiers for users and groups. If not, see the repository(8) man page.

It's assumed that you understand how the basic Vesta operations like checkout and checkin work. Presumably you already have one Vesta server up and running and are familiar with its basic operation.

Here are some basic Vesta informational resources:

It's assumed that you have secured the network connection between the peer sites. This document doesn't cover anything about encryption and relies upon simple host-based authentication.

Examples

Throughout this page examples will use a fictional company with two geographically distributed sites.

The fictional company's domain is "example.com".

The two sites are represented by the sub-domains "foo.example.com" and "bar.example.com". These sub-domains are also used as the realms for Vesta

The Vesta servers have the hostnames "vesta.foo.example.com" and "vesta.bar.example.com".

Client hosts (workstations used by individual contributors) have hostnames "client.foo.example.com" and "client.bar.example.com".

When not using a special account such as vadmin, we'll use the fictional user "jsmith". (In the two realms his global identities are of course "jsmith@foo.example.com" and "jsmith@bar.example.com".

Preliminaries

Before we talk about the setup in detail, let's make sure the basic networking infrastructure is working correctly and that each site has simple access to the peer repository.

Hostnames, Addresses, Reachability

Each repository server must have a hostname which can be resolved to an IP address by its peer repositories. In our example, the servers are "vesta.foo.example.com" and "vesta.bar.example.com". We should start by logging into each of those and making sure that we can get an IP address for the other.

vesta.foo.example.com% host vesta.bar.example.com
vesta.bar.example.com has address 10.2.1.1

vesta.bar.example.com% host vesta.foo.example.com
vesta.foo.example.com has address 10.1.1.1

We should also make sure that each server can send packets to and receive from the other. One way we can do this is by using ping(8):

vesta.foo.example.com% ping vesta.bar.example.com
PING vesta.bar.example.com (10.2.1.1) 56(84) bytes of data.
64 bytes from vesta.bar.example.com (10.2.1.1): icmp_seq=1 ttl=64 time=0.232 ms
64 bytes from vesta.bar.example.com (10.2.1.1): icmp_seq=2 ttl=64 time=0.225 ms
64 bytes from vesta.bar.example.com (10.2.1.1): icmp_seq=3 ttl=64 time=0.225 ms
64 bytes from vesta.bar.example.com (10.2.1.1): icmp_seq=4 ttl=64 time=0.232 ms

--- vesta.bar.example.com ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3002ms
rtt min/avg/max/mdev = 0.225/0.228/0.232/0.015 ms

vesta.bar.example.com% ping vesta.foo.example.com
PING vesta.foo.example.com (10.1.1.1) 56(84) bytes of data.
64 bytes from vesta.foo.example.com (10.1.1.1): icmp_seq=1 ttl=64 time=0.232 ms
64 bytes from vesta.foo.example.com (10.1.1.1): icmp_seq=2 ttl=64 time=0.225 ms
64 bytes from vesta.foo.example.com (10.1.1.1): icmp_seq=3 ttl=64 time=0.225 ms
64 bytes from vesta.foo.example.com (10.1.1.1): icmp_seq=4 ttl=64 time=0.232 ms

--- vesta.foo.example.com ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3002ms
rtt min/avg/max/mdev = 0.225/0.228/0.232/0.015 ms

In the examples on this page, we're going to use reverse hostname mappings (aka reverse DNS) to determine access. We should check that the reverse mappings are in place and give us the right hostnames.

vesta.foo.example.com% host 10.2.1.1
1.1.2.10.in-addr.arpa domain name pointer vesta.bar.example.com.

vesta.bar.example.com% host 10.1.1.1
1.1.1.10.in-addr.arpa domain name pointer vesta.foo.example.com.

Repository Access

Now that we've made sure have basic addressing and network connectivity, let's grant access to the peer sites. Each repository has an export file which controls which client hosts it will accept requests from. We'll need to add entries to the export file on each of our two repositories to allow the hosts at the other site access.

In the export file on "vesta.foo.example.com" we'll add these lines:

# Allow any host in the bar.example.com access with the relam bar.example.com
*.bar.example.com: allow global bar.example.com

Similarly, in the export file on "vesta.bar.example.com" we'll add these lines:

# Allow any host in the foo.example.com access with the relam foo.example.com
*.foo.example.com: allow global foo.example.com

Now that we've granted some access, let's check that the remote repository is allowing us in. We can do this with the vid utility.

vesta.foo.example.com% vid -R vesta.bar.example.com
User names and aliases:
  jsmith@foo.example.com
Groups:
Unix (NFS) user ID:          1002
Unix (NFS) primary group ID: 1002

vesta.bar.example.com% vid -R vesta.foo.example.com
User names and aliases:
  jsmith@bar.example.com
Groups:
Unix (NFS) user ID:          1002
Unix (NFS) primary group ID: 1002

Repository Addresses

Each repository has a public address which peer sites use to contact it. This is a hostname and a port and is written as a string: "host:port". Assuming we're using the default ports, our two example repositories addresses are:

vesta.foo.example.com:21776

vesta.bar.example.com:21776

Specifying Your Public Address

Ideally all repository clients will use the public address to access the repository. If this is the case, the repository's public address will simply be taken from the config file settings giving the repository's host and port.

However, this may not be feasible in all cases. For example, a laptop installtion will want to use "localhost" to access the repository yet may still have a public address which is valid on a certain network. In cases like this, you can set the configuration variable [Repository]master_hint to explicitly specify the public address of your repository.

Automatic Recording

When you create a new master appendable locally (e.g. a new top-level directory created by vwizard), the repository automatically stores the public address of your repository in a "master-repository" attribute on that object. This attribute can be propagated (along with the object) to a replica.

Whenever you transfer mastership of an object between repositories, "master-repository" attributes get added/updated.

This is why it's important to get your repository's public address set correctly (e.g. with the [Repository]master_hint configuration setting). If the wrong address gets recorded, or if you don't have any "master-repository" attributes, you may need to set them explicitly.

Suppose for example that the top-level directory /vesta/foo.example.com was created in the repository at vesta.foo.example.com. Maybe the initial setup didn't use a fully-qualified hostname (and [Repository]master_hint wasn't set either). The master-repository attribute would have recorded this incomplete address:

vesta.foo.example.com% vattrib -g master-repository /vesta/foo.example.com
vesta:21776

We can simply correct this with vattrib:

vesta.foo.example.com% vattrib -s master-repository vesta.foo.example.com:21776 /vesta/foo.example.com
vesta.foo.example.com% vattrib -g master-repository /vesta/foo.example.com
vesta.foo.example.com:21776

If /vesta/foo.example.com has already been replicated to the peer repository at vesta.bar.example.com, the attribute will be wrong there as well:

vesta.bar.example.com% vattrib -g master-repository /vesta/foo.example.com
vesta:21776

You can either correct it with vattrib the same way as above, or you can use vrepl to propagate the attribute change from the other repository:

vesta.bar.example.com% vrepl -v -s vesta.foo.example.com -e+ /vesta/foo.example.com -e- "/vesta/foo.example.com/*"
exists  /vesta/foo.example.com
attribs /vesta/foo.example.com

Use For Access Control

There are three special access control attributes which are used during distributed operations:

The values of each of these attributes are repository addresses. Each will be described in more detail below.

Replication

Replication is the process by which objects under /vesta are propagated from one repository to another.

There's a short overview of replication on the vrepl(1) man page. If you want to understand it in depth, see "Partial Replication in the Vesta Software Repository" (which also explains mastership, which we'll talk about a little later).

Access Checks

Several access control checks are performed in the course of a replication:

To support remote checkout/checkin, we'll need to allow bi-directional replication in both push and pull modes. The local repository (where the checkout is performed) will need to replicate objects from the remote repository. When checking in, the local repository must replicate the new version back.

We'll go through each of the access checks listed above and make sure that we've set up the access that we need.

Finding Objects to Replicate

The first step in performing a replication is checking both the source and destination repositories to find objects which need to be replicated. Sometimes a replication may specify a complex pattern to be matched (e.g. using the replicator's directive syntax). Also, the object(s) might already be present at the destination.

To do this we need read access to both repositories from any client host that might perform a replication. In our example, the remote site is bar.example.com. We need to make sure that users at that site can access the objects we're interested in replicating in both repositories. Earlier we checked that the repository server would allow them to make requests with vid, but that doesn't guarantee that they will have sufficient permission to access packages and versions at the repository on vesta.foo.example.com.

We'll assume that there's a package named /vesta/foo.example.com/a which contains a version 1. Let's eaxmine its attributes from the remote site:

client.bar.example.com% vattrib -R vesta.foo.example.com /vesta/foo.example.com/a/1
vattrib: /vesta/foo.example.com/a/1: No permission

We'll need to solve this before we can replicate. There are several possible ways we could solve this:

After doing at least one of these, our user should have permission to access the package and its version, so let's try our test again:

client.bar.example.com% vattrib -R vesta.foo.example.com /vesta/foo.example.com/a/1
master immutableDirectory
message
        initial version of package a
content
        /vesta/foo.example.com/a/checkout/1/5
checkin-time
        Mon Oct 29 14:33:10 EST 2005
checkin-by
        jsmith@foo.example.com
#owner
        jsmith@foo.example.com

We should also repeat this test in the opposite direction, having a user at foo.example.com try to access objects in the repository at vesta.bar.example.com (similarly modifying access control information to grant access). In our example we haven't yet replicated anything to vesta.bar.example.com for us to access yet. However, it would be a good idea to revisit this later.

Requesting the Replication

Earlier, we granted each site full access to the peer repository. (That is, in the repository export file we used "allow global" rather than "readonly global".) This should allow us to perform replications in either direction.

Trusting the Source Repository

Each repository can control which other repositories it trusts as replication souces. Vesta relies on the full path to an immutable version always referring to identical contents in all repositories. However, this is not strictly enforced, so you should only choose to allow replication from repositories which you are reasonably certain you can trust.

The #replicate-from attribute lists the addresses of repositories to be trusted as replication sources. This can be placed on /vesta or subdirectories of it. In order to determine whther a particular replication is allowed, the repository searches upward from the path to be replicated for the closest enclosing directory with a #replicate-from attribute. The source repository's address must be listed in that attribute or permission for the replication will be denied.

In our example, we first need to allow replication from the master repository (vesta.foo.example.com) to the remote site (vesta.bar.example.com). So we'll add a #replicate-from attribute which allows this:

client.bar.example.com% vattrib -a #replicate-from vesta.foo.example.com:21776 /vesta

Now we'll try a replication of the version we inspected above (/vesta/foo.example.com/a/1):

client.bar.example.com% vrepl -v -s vesta.foo.example.com -e+ /vesta/foo.example.com/a/1
create  /vesta/foo.example.com
create  /vesta/foo.example.com/a
copy    /vesta/foo.example.com/a/1
attribs /vesta/foo.example.com/a/1
latest  /vesta/foo.example.com/a/latest
attribs /vesta/foo.example.com/a
attribs /vesta/foo.example.com

Now we're all set to replicate from the remote master. Hwoever, we also need to be able to replicate back to the remote master so that we we check in a new version from bar.example.com it will appear in the repository at vesta.foo.example.com. If we try to replicate the same version back, we'll run into trouble:

client.foo.example.com% vrepl -v -s vesta.bar.example.com -e+ /vesta/foo.example.com/a/1
exists  /vesta/foo.example.com/a/1
attribs /vesta/foo.example.com/a/1
vrepl: No permission, copying attribs for /vesta/foo.example.com/a/1

This is because we haven't yet granted permission to replicate in the other direction. We'll have to add a #replicate-from over in the repostiory at vesta.foo.example.com.

vesta.foo.example.com% vattrib -a #replicate-from vesta.bar.example.com:21776 /vesta

Now we should be able to perform the replication in the other direction:

client.foo.example.com% vrepl -v -s vesta.bar.example.com -e+ /vesta/foo.example.com/a/1
exists  /vesta/foo.example.com/a/1
attribs /vesta/foo.example.com/a/1

Just to convince ourselves that this is working, let's make an attribute change locally and make sure that it gets propagated:

client.bar.example.com% vattrib -a "test-attribute" "jsmith@bar was here" /vesta/foo.example.com/a/1

client.foo.example.com% vrepl -v -s vesta.bar.example.com -e+ /vesta/foo.example.com/a/1
exists  /vesta/foo.example.com/a/1
attribs /vesta/foo.example.com/a/1
client.foo.example.com% vattrib -g "test-attribute" /vesta/foo.example.com/a/1
jsmith@bar was here

Access from Destination as Requestor

When performing a replication, the destination makes requests to the source repository to read the objects being replicated. These requests are made with the identity of the user requesting the replication.

When the local repository is the destination and the source is a remote rpeository, this seems perfectly natural. For example, above the user ran the replicator on client.bar.example.com to copy an object from the remote repository vesta.foo.example.com to the local repository. The requests from the destination repository host used the global identity of the user running the replicator (jsmith@bar.example.com). This matches the line we added to the repository export file earlier.

If the local repository is the replication source and the destination is a remote repository, things are a little different. If the user ran the replicator on client.bar.example.com to copy an object from the local repository to the remote repository, requests would arrive from vesta.foo.example.com with the global identity of the user running the rpelicator (jsmith@bar.example.com). This would not match the export file entries we added earlier.

To support replication in from the local repository to a remote repository, we'll need to add some entries to the export files of our two repositories. In the export file on "vesta.foo.example.com" let's add these lines:

# Allow peer repository in with our realm to support "push" replication
vesta.bar.example.com: allow global foo.example.com

Similarly, in the export file on "vesta.bar.example.com" we'll add these lines:

# Allow peer repository in with our realm to support "push" replication
vesta.foo.example.com: allow global bar.example.com

Now let's test this:

client.foo.example.com% vrepl -v -d vesta.bar.example.com -e+ /vesta/foo.example.com/a/1
exists  /vesta/foo.example.com/a/1
attribs /vesta/foo.example.com/a/1

client.bar.example.com% vrepl -v -d vesta.foo.example.com -e+ /vesta/foo.example.com/a/1
exists  /vesta/foo.example.com/a/1
attribs /vesta/foo.example.com/a/1

Mastership

Each object under /vesta has a master flag. You can check the master flag with vattrib which prints "master" or "nonmaster".

Only a single repository can have the master flag set for an object at once. That is the master repository for that object. The replica of an object at its master repository is called the master copy of the object. Changes to an object can only be made at the master repository. For example, if you want to check out a package, its master repository must grant you permission to do so.

Finding the Master

The repository tools locate the master copy of an object using several methods:

Transferring Mastership

Mastership transfer is the process by which the master flag can be moved from one repository to another. vmaster can be used to explicitly transfer mastership, but it's much more common for tools like vcheckout to automatically transfer mastership for the user.

Access Controls

The #mastership-to attribute lists the addresses of repositories we're willing transfer masterhip to.

The #mastership-from attribute lists the addresses of repositories we're willing to accept a mastership transfer from.

Networking Details

Mastership transfer is a handshake protocol. It proceeds in several steps:

  1. The client locates the master repository.
    • Let's suppose that the master is vesta.foo.example.com.

  2. The client contacts the repository it wants to become the master and asks it to acquire mastership from the current master.

    • Let's suppose the client wants vesta.bar.example.com to become the master. It tells vesta.bar.example.com "acquire mastership from vesta.foo.example.com:21776"

  3. After deciding whether it will accept this transfer, the repository acquiring mastership contacts the current master and asks it to cede mastership to it.

    • In our example, the repository at vesta.bar.example.com contacts the repository at vesta.foo.example.com and asks "please cede mastership to vesta.bar.example.com:21776".

    • Note that the request to cede mastership goes from the destination to the current master with the identity of the user requesting the transfer. (This is similar to the way replication works.) Normally this is a user local to that repository, so the export file at the current master probably already allows this.
  4. After deciding whether it will accept this transfer, the repository ceding mastership turns off the master flag and contacts the new master and and tells it that it has given up mastership.
    • In our example, the repository at vesta.foo.example.com contacts the repository vesta.bar.example.com and says "I've given up mastership, you can have it now"

    • Note that this also uses the identity of the requesting user. This user probably isn't local to the repository making the request, so you'll need to allow their realm in from the peer repository. In our example, this means that the export file on vesta.bar.example.com must allow vesta.foo.example.com with realm bar.example.com. (This is similar to what we need for "push" replication.)

  5. Finally, the repository acquiring mastership turns on its master flag and indicates to the client that it has acquired mastership.

mastership-transfer.png

Automatic Recovery

Mastership transfer is designed to maintain the special property that at most one repository has the master flag set even if something bad happens during the transfer such as a loss of network connectivity or one or both repository servers going down (e.g. from a power loss). The repository server will automatically retry partially completed mastership transfers.

Normally this is completely transparent and users never have to be aware of it. However, it's worth mentioning because if a repository permanently goes off line peer repositories may continue trying to contact it.

Remote Checkout in Action

Now that we've got all the necessary permissions in place, let's perform an actual remote checkout. Our user jsmith simply uses vcheckout the same way they would for a package that originated in the local repository:

client.bar.example.com% vcheckout /vesta/foo.example.com/a
Reserving version /vesta/foo.example.com/a/2 at vesta.foo.example.com:21776
Creating session /vesta/foo.example.com/a/checkout/2 at vesta.foo.example.com:21776
Making working directory /vesta-work/jsmith/a

The only difference visible to the user is the "at vesta.foo.example.com:21776". That tells the user that a remote repository was contacted for those parts of the operation. After creating the version reservation and the session directory at the remote master, vcheckout automatically replicates them to and transfers mastership of them to the local repository. We can see that the local repository now has mastership of the reservation with vattrib:

client.bar.example.com% vattrib /vesta/foo.example.com/a/2
master stub
work-dir
        /vesta-work/jsmith/a
session-dir
        /vesta/foo.example.com/a/checkout/2
old-version
        /vesta/foo.example.com/a/1
master-repository
        vesta.bar.example.com:21776
checkout-time
        Fri Dec  2 13:27:46 EDT 2005
checkout-by
        jsmith@bar.example.com
#mode
        775
checkout-to
        vesta.bar.example.com:21776
checkout-from
        vesta.foo.example.com:21776

Also worth noting are the last two attributes:

These attributes are also used by other tools. For example, a user at foo.example.com running vwhohas will see that a user at bar.example.com has the package checked out:

client.foo.example.com% vwhohas /vesta/foo.example.com
/vesta/foo.example.com/a/2  jsmith@bar.example.com  vesta.bar.example.com:21776

All the user's activities before checking in happen locally. Editing and taking snapshots with vadvance only use the local repository. Even filling in the reserved version on checkin happens locally. After doing the local part, vcheckin will replicate the version back to the repository in the checkout-from attribute:

client.bar.example.com% vcheckin /vesta-work/jsmith/a
Checking in /vesta/foo.example.com/a/2
Deleting /vesta-work/jsmith/a
Replicating /vesta/foo.example.com/a/2 to vesta.foo.example.com:21776

Summary / Quick Reference

Skipping over all the detailed exaplantion and examples, here's everything you need to have set up for remote checkout to work: