• ReposUIModifiers

Overview

The repository tools have a variety of different arguments that can be specified explicitly with command-line arguments or use default values. Sometimes users would prefer a value different from the built-in default but would also rather not be forced to specify the value by hand on the command line each time.

At this time, if users want to alter the default behavior of the repository tools they must write a wrapper such as a script or a shell alias. This typically includes some code for determining a different parameter value and specifying it explicitly when invoking the actual repository tool. Unfortunately, this tends to lead to a divergent user interface which must be maintained. Users become confused about where Vesta ends and their layered user interface begins. They have difficulty corresponding their user experience to the Vesta documentation and can have difficulty moving between projects which have different layered UIs. Also, such wrappers tend to have insufficient documentation and subtle bugs.

This page proposes a design way of replacing defaults with a user-supplied way of determining the values for each parameter. This is essentially another "plug-in" system for the repository tools similar to the repository tool triggers. The goal is to both make such modifications easier to implement (as triggers make before and after actions easier) and to avoid the need to wrappers that have a non-standard user-interface where possible.

(Of course for activities that perform multiple steps users may still need scripts that call the repository tools, but the goal is to avoid the need for wrappers for basic operations like vcheckout and vcheckin.)

Examples

This section lists some examples of parameter modifications. Many of these have been implemented with wrapper scripts.

Checkout Working Directory Name

In projects with a hierarchy of packages, it's not uncommon to have two packages which have the same name. This results in working directory name collisions which can make it difficult for users to immediately identify which package a given working directory corresponds to. In cases like this, users would prefer to have their working directories named based on more than just the package name.

Suppose for example that a project has a number of different modules organized into different directories and each of these modules has separate packages for code, documentation, and tests. A user might regularly deal with a set of packages with such as:

/vesta/example.com/foo/pdi/cin/doc
/vesta/example.com/foo/pdi/cin/code
/vesta/example.com/foo/pdi/cin/tests

/vesta/example.com/foo/ici/dfi/doc
/vesta/example.com/foo/ici/dfi/code
/vesta/example.com/foo/ici/dfi/tests

/vesta/example.com/foo/ici/fmc/doc
/vesta/example.com/foo/ici/fmc/code
/vesta/example.com/foo/ici/fmc/tests

The user might want to automatically generate a working directory name from the package path. Or they might prefer to make a decision when creating their hierarchy of packages about the preferred working directory name for each package, perhaps they would record it with an attribute on each package:

% vattrib -g default-working-dir-name \
  /vesta/example.com/foo/pdi/cin/doc \
  /vesta/example.com/foo/ici/dfi/code \
  /vesta/example.com/foo/ici/fmc/tests
/vesta/example.com/foo/pdi/cin/doc
  pdi-cin-doc
/vesta/example.com/foo/ici/dfi/code
  ici-dfi-code
/vesta/example.com/foo/ici/fmc/tests
  ici-fmc-tests

Deeper Session Directory Hierarchy

On large projects, some packages wind up with a lot of session directories. Consider a project with a central build package which users regular check out non-exclusively to make build overrides for their work. Suppose the project has hundreds of users and the build package has thousands of versions over the course of the project. That could easily become hundreds of thousands of session directories to contain individual user builds. This can become a performance problem as listing or performing a lookup within a directory takes time proportional to the number of entries. Even determining whether a name can be used for a new session directory requires looking it up in the directory which will contain the session directory.

To avoid this, it's better to separate the non-exclusive session directories by user, and possibly basis version. For example, instead of the session directory:

/vesta/example.com/foo_build/checkout/1398.jsmith_example.com.1

It might be better to create one of these session directory (implicitly creating the enclosing directories):

/vesta/example.com/foo_build/checkout/jsmith/1398_example.com.1
/vesta/example.com/foo_build/checkout/jsmith_example.com/1398.1
/vesta/example.com/foo_build/checkout/jsmith/1398/example.com.1
/vesta/example.com/foo_build/checkout/jsmith_example.com/1398/1

Avoiding Unstable Basis Version

Suppose a project is using an automated build and testing system which marks package versions with an attribute when they have passed a series of tests.

% vattrib -f passed_tests `vglob '/vesta/example.com/foo/ici/fmc/code/[LAST-5,LAST]'`
/vesta/example.com/foo/ici/fmc/code/18
        Sun Jun  8 20:09:43 EDT 2008
/vesta/example.com/foo/ici/fmc/code/19
        Tue Jun 17 23:07:51 EDT 2008
/vesta/example.com/foo/ici/fmc/code/20
        Mon Jun 23 13:51:02 EDT 2008
/vesta/example.com/foo/ici/fmc/code/21
/vesta/example.com/foo/ici/fmc/code/22

A user performing a non-exclusive checkout might prefer to start from the latest version which has been marked as "good" in this way, rather than simply the latest version of the package. In this case, they would want version 20 to be automatically selected for them because version 21 and 22 hasn't been marked as having passed tests yet.

Path Shortcuts

A Vesta repository that is used across more then one large project might end up with a directory structure similar to this:

/vesta/example.com/foo/pdi/ici/doc
/vesta/example.com/foo/pdi/ici/code
/vesta/example.com/bar/pdi/ici/doc
/vesta/example.com/bar/pdi/ici/code

And a user might want to save keystrokes and have vcheckout automatically select the correct package for them. The command line might be vcheckout ici/code and the modifier would consult some portion of the user's environment to determine that /vesta/example.com/bar/pdi/ici/code is the correct package to check out.

Per-Tool Notes

vcheckout

Modifiers must run before triggers, as triggers need variables that tell them what vcheckout is going to do and these would change what vcheckout will do

Modifiers should get the same variables as triggers for the current choices of what to do.

Order of operations:

1. Parse command line and determine defaults
2. Run behavior modifiers to alter choices
3. Run pre triggers
4. Make repository changes
5. Run post triggers

Settings (in the section [vcheckout modifiers]):

• modify_package
  • A wrapper script in use at Intel makes a branch in some cases and then checks out the branch rather than the package. (Intel people will know this as the "stepping methodology".)
• modify_new_version
• modify_old_version
• modify_session_dir
• modify_work_dir

In what order should the modifiers be run? It could matter in some cases. We could make a vesta.cfg setting to change the order of the modifiers, but that could make things much more complicated. This order seems like it should work most of the time:

1. modify_package
2. modify_old_version
3. modify_new_version
4. modify_session_dir
5. modify_work_dir

The standard output of the invoked command will be the replacement for the value (old version, session dir, work dir). The standard error will be left to go to the standard error of the repository tool.

We should add a trigger variable to tell whether the current setting (old version, session dir, work dir) was defaulted or specified explicitly on the command line. It would still be up to the behavior modifier to decide whether or not to change the value. Presumably, most would not change the value when it had been explicitly specified.

There should be a switch to disable the behavior modifiers (i.e. "No, really, do what I said")

If a modifier exits with failure status, vcheckout should fail. This makes it possible for a modifier to cancel an operations that's somehow invalid.

If a modifier does not want to change the value, it should print the current value to standard output.

If a modifier produces no output (i.e. its output is the empty string) and exist with successful status, it's equivalent to "omit X"

• modify_new_version => -N

• modify_old_version => -O

• modify_session_dir => -S

• modify_work_dir => -W

Uniquification:

• The working directory is always be uniquified, even if you pass it with -w on the command line. If the modifier doesn't change it, we don't need to re-uniqify it, but if they do we need to uniqify it.
• Should a session directory provided by a modifier be uniqified? The modifier could uniquify a session directory itself if it wants it to be unique, but it would be nice if it didn't have to. For consistency, let's have the session directory always get uniquified like the working directory.
• The new version should *not* be uniquified. (Obviously a modifier could uniquify it if that was really desired, but vcheckout shouldn't do that.)

We can allow an unverified old-version (or other value) to be passed through to the modifier to allow for something like "-o blessed", "-o tested", "-o golden".

• A checkout wrapper script in use at Intel allows you to say "-version=blessed". It will look in the config file for a setting that tells it what attribute corresponds to "blessed" and then find the latest version with that attrbute. This could be done with modify_old_version.
• A checkout wrapper script in use at Intel allows you to specify a partial path to a package. It then searches the repository (or rather a saved list of known packages) for a match. This could be done with modify_package.

vcheckin

Settings (in [vcheckin modifiers]):

• modify_new_version
• modify_content
• modify_session_dir
• modify_work_dir

Order in which modifiers run:

1. modify_work_dir
   • Set new-version, session-dir, content based on modified work-dir
2. modify_new_version
3. modify_session_dir
4. modify_content

Examples:

• modify_new_version
  • Checkout was non-exclusive (i.e. currently there is no new version) but was based on the highest current version. Reserve the next version now and use that.
• modify_work_dir
  • User has a method for organizing the working directories into sub-directories under /vesta-work/user. Find the right one.
• modify_content
  • Do "clean up" or finalization operations that modify the working directory, vadvance a new snapshot, and use that instead. (This is ugly, and means checking in something that probably hasn't been built or tested, but for practical reasons is needed in some cases.)

vbranch

Settings (in [vbranch modifiers]):

• modify_branch
• modify_old_version

Order in which modifiers run

1. modify_parent
2. modify_old_version
3. modify_branch

Here's an example based on the Intel "stepping methodology" scripts:

• Command: vbranch beh/ubox/pizza
• Original package: /vesta/example.com/foo/beh/ubox/pizza
  • Eager branch: /vesta/example.com/foo_a0/beh/ubox/pizza
    • old-version is some /vesta/example.com/foo/beh/ubox/pizza/N
  • Lazy branch: /vesta/example.com/foo/beh/ubox/pizza/foo_a0.N
    • old-version is /vesta/example.com/foo/beh/ubox/pizza/N

vcreate

Settings (in [vcreate modifiers]):

• modify_package
• modify_old_version

vmkdir

Settings (in [vmkdir modifiers]):

• modify_dir

vrm

Settings (in [vrm modifiers]):

• modify_path