Could we Make Importing a New Tool Easier?
Setting up a new tool to be run as part of a Vesta build is difficult.
Setting up the encapsulated filesystem ./root requires knowing all the files needed by the tool.
Setting up the environment variables ./envVars requires knowing all the settings needed by the tool.
SDL the language takes some getting used to, especially the various details of the _run_tool primitive function.
In contrast, most people that want to import some new tool into Vesta already know how to run that tool outside of Vesta. They may not know all the details, but they probably know how to get their system and environment into a state where the tool can be run.
What if there were a way for the user to simply type a command at their shell prompt and have all the files it uses discovered automatically? What if this also created a boilerplate SDL file which set up all the environment variables and invoked _run_tool with the command line the user typed?
How Filesystem Encapsulation Works
When the evaluator runs a tool, it creates a volatile directory. It does this by calling the function VDirSurrogate::createVolatileDirectory. This directory becomes the root filesystem for the tool (see the chroot(2) man page).
The evaluatoir does not transmit the contents of this filesystem to the repository immediately. Instead, it tells the repository a host and port where it can call back to the evaluator for informaiton about the contents of the volatile directory. (These are the "host" and "port" arguments to VDirSurrogate::createVolatileDirectory.) The evaluator's side of this network protocol is implemented in ToolDirectoryServer.C, and the repository's side is implemented in VDirEvaluator.C. (Some documentation of the protocol is in Evaluator_Dir_SRPC.H.)
An important fact about this is that whenever a tool tries to access a file or directory, the repository first calls back to the evaluator. This is how the evaluator records information about which files/directories a tool run depends upon.
Discovering Rather Than Recording
Currently, the only program which calls VDirSurrogate::createVolatileDirectory, the only program which implements the network protocol for providing the contents of a volatile directory to the repository is the evaluator. But what if there was another one? What if this alternate program did something different when receiving a call-back from the repository?
Suppose that when this program receives a call-back for a particular file, it behaves differently. Rather than taking the file contents from a data structure like the binding in ./root, it would take the file from the normal filesystem. This would require copying it into the repository, as files in a volatile directory must be backed by a shortid. It could also create a mutable directory containing the same root filesystem structure.
After the tool finishes, this new program could create an SDL file which:
Brings in the root filesystem created in a mutable directory while the tool was running and puts it in ./root
Sets up ./envVars with the environment variables which were set at the time the tool was started
Calls _runt_tool with the original command-line
This would be a very crude way of getting started with a new tool inside Vesta. It would not be anything like a bridge. But it would make it easier to get started. It would be helpful to both new and experienced users.
(JohnVk): i like this, but there needs to be a way to search existing bridges first, and try them first, and if they work, just add the import and/or to std_env build list for this package. Next, if no such bridge exists, check if the file already exists somewhere in vesta, and import it from there. Then if none of those work, get it from the use's environment.
Prototype Implementation
There is a branch with some initial work on an implementation: