The collection of information about a target schema, its structure, connection details, data source and defined loader rules is called a loader set. A loader set is built and maintained by the DataBee Set Designer application.
Physically, loader sets are stored on disk as standard XML text file. So yes, technically the loader set files are user editable via a text editor. Please be aware that making manual adjustments is unsupported - the Set Designer application is the only supported solution. We do recognize that situations may sometimes arise where a quick search and replace on the loader set file may save hours of work. If you do edit the loader set files by hand, be sure to save a backup copy first and perhaps email Support@DataBee.com for some advice before proceeding.
Loader sets are built with the DataBee Set Designer application and executed with the Set Loader application. The Set Loader needs to have access to a list of ROWIDs of the rows in the source schema which will need to be copied to the target. The DataBee Set Extractor application builds the ROWID list during the execution of an extraction set. The two tools work together to create a referentially correct subset database. It is possible to load many target schemas from the results of a single extraction run.
Loader sets are simple to build - much simpler than extraction sets. The recommended method of building a loader set is to:
The list above is just a summary. The DataBee Quick Start Guide provides a detailed, practical, step-by-step walk through of the DataBee design, extract and load methodology. The Extraction and Load Process help file provides a summary the subset database creation and population process.
As discussed above, a loader set is a container for the source and target connection information, the table/index/foreign key structure details and for the loader rules which control the load activities. All of this information is encoded in the loader set in the form of rules.
One key rule, called a Controller Rule, describes the schema into which the data will be loaded and also provides the contact information for the schema which contains the extracted source data. The Rule Controller also contains the schema structure (tables, indexes foreign keys) and has dependent rules which describe the load actions. Every loader set has at least one Rule Controller (you can have more than one). Every other type of loader rule has a parent Rule Controller and will execute in the schema defined within that Rule Controller.
Other than the Rule Controller and the Command rules, all of the other rules which can be configured in a loader set are of a type known as manager rules. Manager rules encapsulate a large number of actions and perform those actions in parallel. If you are unfamiliar with DataBee manager rules we strongly advise reading the short What Is A Rule Manager help file.
The following is a brief summary of the rules which can be configured in a loader set. Each rule has its own detailed help file and a slightly longer summary can be found on the New Loader Rule Form help page.
The Set Loader application is multi-threaded. It can, and will, run multiple rules simultaneously. The number of rules which can run in parallel is determined by the Number of Loader Threads setting on the Misc. Setup tab. Unless strictly specified (see below), each loader rule is considered by the DataBee software to be an independent entity and its order of execution is not guaranteed.
In addition, certain types of loader rule (Foreign Key Manager, Trigger Manager, Truncate Manager, and Load Manager) encapsulate a large number of subsidiary actions. If the manager rules run simultaneously, then their actions will also run simultaneously and this may not be appropriate. For example, the Truncate Manager rule will truncate the tables in the target schema and the Load Manager will populate them with extracted data. It would not be acceptable for a table to be loaded by the Load Manager only to have the Truncate Manager remove all rows from it at a later time. The order of execution of the two rules is important and clearly the Truncate Manager rule must complete its actions before the Load Manager rule activates.
An Example Loader Set Rule Blocks
Since the rules in a loader set can, and will, execute in parallel the DataBee software implements a concept called Rule Blocks to explicitly control the execution order. A rule block is the two-digit numeric prefix listed before the rule number. Rule blocks are processed in strict numeric order and all rules in a given rule block will complete before any rule in the next highest rule block begins. Inside a rule block the rules execute in random order as determined by the optimization routines and the availability of worker threads.
In the image above, six rules are listed. In this case each rule has been given its own rule block. For example, rule 10-0002 disables the foreign keys. Because it is the only rule in rule block 10 there will be no other rules operating at that time. The foreign key disable operations will proceed in parallel according to the number of worker threads set on the Set Loader Misc. Setup tab.
Once rule 10-0002 has completed, the rules in the next highest rule block will be activated. In the example above this is rule 15-0003. After that rule completes, the the chain of rules 20-0004, 30-0005, 40-0006 and 45-00007 will all execute sequentially one after the other because they are in separate rule blocks.
Note that the decision to place rule 0002 in rule block 10 and rule 0003 in rule block 15 is somewhat arbitrary. Since these rules only disable foreign keys and triggers they could reasonably execute at the same time and hence they both could be placed in the same rule block. Doing so would have no effect on the results of the loader set. Sometimes the rule block really matters though - certainly rule 0004 needs to be in a lower rule block than rule 0005 since the truncate must happen before the load. The decision of where in the execution sequence to place a rule is up to the implementer of the loader set. For example, if a Command Rule was added to the loader set it would be necessary to carefully adjust the rule block of that rule so the SQL command it contains executes at the appropriate point in the loader set.
