BatchPipes
Encyclopedia
In IBM mainframe
s, BatchPipes is a batch job processing
utility designed for the MVS/ESA operating system, and all later incarnations—OS/390
and z/OS
.
and BSAM
) data set
on disk or tape, they cannot be at the same time read back in by another job. Hence, these two jobs—"writer" and "reader"—cannot run at the same time. This is termed file-level interlock or data-set-level interlock.
With BatchPipes an installation can arrange for the data to be "piped" between the two jobs. The advantage is that the jobs can run concurrently and it is possible, and very usual, to avoid the time to write the data to secondary storage and to read it back. The combination of these two characteristics, if used judiciously, leads to a reduction in the combined elapsed time of the two jobs, as measured from the start of the writer job to the end of the reader job.
BatchPipes maintains a short queue of records being passed between the writer and the reader. The writer adds records to the back of the queue and the reader takes them from the front. This is deemed record-level interlock and allows the reader and the writer to run concurrently.
A sort
is a special case: all the input records must be read before the first output record can be written. Hence there can be no overlap between the input and output phases of a sort. But the input phase can be overlapped with the previous job's output phase. Similarly, the output phase of sort can be overlapped with a downstream job that reads the sorted data.
to pipe data between different members of a Parallel Sysplex
, using the BatchPipePlex facility.
's CMS Pipelines product developed for another operating system, VM/ESA. These stages provide additional processing, without the need for additional batch jobs in the pipeline.
A Monaco
based company, APT International, produced a competitive product trademarked as WARP
. A few months after launching this product, IBM renamed their OS/2
product OS/2 Warp 4, conflicting with the marketing of the performance product that was the only competitor to BatchPipes, resulting in 7 years of litigation at Tribunal de grande instance in Paris
Subsequently BatchPipes Version 2 was released, incorporating BatchPipes Version 1 and some additional features from SmartBatch: BatchPipePlex and BatchPipeWorks. BatchPipes Version 2 is still a marketed IBM product.
IBM mainframe
IBM mainframes are large computer systems produced by IBM from 1952 to the present. During the 1960s and 1970s, the term mainframe computer was almost synonymous with IBM products due to their marketshare...
s, BatchPipes is a batch job processing
Batch processing
Batch processing is execution of a series of programs on a computer without manual intervention.Batch jobs are set up so they can be run to completion without manual intervention, so all input data is preselected through scripts or command-line parameters...
utility designed for the MVS/ESA operating system, and all later incarnations—OS/390
OS/390
OS/390 is an IBM operating system for the System/390 IBM mainframe computers.OS/390 was introduced in late 1995 in an effort, led by the late Randy Stelman, to simplify the packaging and ordering for the key, entitled elements needed to complete a fully functional MVS operating system package...
and z/OS
Z/OS
z/OS is a 64-bit operating system for mainframe computers, produced by IBM. It derives from and is the successor to OS/390, which in turn followed a string of MVS versions.Starting with earliest:*OS/VS2 Release 2 through Release 3.8...
.
Core function
In traditional processing, if data records are written out to sequential (QSAMQueued Sequential Access Method
In IBM mainframe operating systems, Queued sequential access method is an access method to read and write datasets sequentially. QSAM is available on OS/360, OS/VS2, MVS, z/OS, and related high-end operating systems....
and BSAM
Basic sequential access method
In IBM mainframe operating systems, Basic sequential access method is an access method to read and write datasets sequentially. BSAM is available on OS/360, OS/VS2, MVS, z/OS, and related high-end operating systems....
) data set
Data set (IBM mainframe)
data set , dataset , is a computer file having a record organization. The term pertains to the IBM mainframe operating system line, starting with OS/360, and is still used by its successors, including the current z/OS. Those systems historically preferred this term over a file...
on disk or tape, they cannot be at the same time read back in by another job. Hence, these two jobs—"writer" and "reader"—cannot run at the same time. This is termed file-level interlock or data-set-level interlock.
With BatchPipes an installation can arrange for the data to be "piped" between the two jobs. The advantage is that the jobs can run concurrently and it is possible, and very usual, to avoid the time to write the data to secondary storage and to read it back. The combination of these two characteristics, if used judiciously, leads to a reduction in the combined elapsed time of the two jobs, as measured from the start of the writer job to the end of the reader job.
BatchPipes maintains a short queue of records being passed between the writer and the reader. The writer adds records to the back of the queue and the reader takes them from the front. This is deemed record-level interlock and allows the reader and the writer to run concurrently.
A sort
Mainframe sort merge
The Sort/Merge utility is a mainframe program to sort records in a file into a specified order, merge pre-sorted files into a sorted file, or copy selected records...
is a special case: all the input records must be read before the first output record can be written. Hence there can be no overlap between the input and output phases of a sort. But the input phase can be overlapped with the previous job's output phase. Similarly, the output phase of sort can be overlapped with a downstream job that reads the sorted data.
Advanced pipe topologies
More complex topologies than "one reader one writer" are possible.- "Two readers one writer" is a good example of an attempt to balance reader's speed against a writer's speed. Because the queue is short a faster writer will often be forced to wait for a slower reader to take records off the queue before the writer can continue processing. Using two readers helps to utilize writers capabilities.
- "One job as a reader from one pipe and a writer to another" is often seen where this job edits the records. While traditional batch streams often contain such jobs, this kind of processing can be introduced using, for example IBMIBMInternational Business Machines Corporation or IBM is an American multinational technology and consulting corporation headquartered in Armonk, New York, United States. IBM manufactures and sells computer hardware and software, and it offers infrastructure, hosting and consulting services in areas...
's DFSORT product or BatchPipeWorks (part of BatchPipes).
Criticism
One of the key implementation considerations is scheduling the reader and writer jobs to run together. In practical batch schedules this might not be feasible. Furthermore if any job in the pipeline fails, recovery actions will be wider than just recovering this single job. For these reasons some installations have found it difficult to implement BatchPipes.BatchPipePlex
BatchPipes can use the IBM mainframe Coupling FacilityCoupling Facility
In IBM mainframe computers, a Coupling Facility or CF is a piece of computer hardware which allows multiple processors to access the same data.A Parallel Sysplex relies on one or more Coupling Facilities...
to pipe data between different members of a Parallel Sysplex
IBM Parallel Sysplex
In computing, a Parallel Sysplex is a cluster of IBM mainframes acting together as a single system image with z/OS. Used for disaster recovery, Parallel Sysplex combines data sharing and parallel computing to allow a cluster of up to 32 systems to share a workload for high performance and high...
, using the BatchPipePlex facility.
BatchPipeWorks
BatchPipes includes a set of pipeline stages based on IBMIBM
International Business Machines Corporation or IBM is an American multinational technology and consulting corporation headquartered in Armonk, New York, United States. IBM manufactures and sells computer hardware and software, and it offers infrastructure, hosting and consulting services in areas...
's CMS Pipelines product developed for another operating system, VM/ESA. These stages provide additional processing, without the need for additional batch jobs in the pipeline.
History
BatchPipes Version 1 was developed in the late 1980s and early 1990s simply as a technique to speed up MVS/ESA batch processing. In 1997 the functionality of BatchPipes was integrated into a larger IBM product - SmartBatch (which incorporated two BMC Corporation product features: DataAccelerator and BatchAccelerator). However SmartBatch was discontinued in April 2000.A Monaco
Monaco
Monaco , officially the Principality of Monaco , is a sovereign city state on the French Riviera. It is bordered on three sides by its neighbour, France, and its centre is about from Italy. Its area is with a population of 35,986 as of 2011 and is the most densely populated country in the...
based company, APT International, produced a competitive product trademarked as WARP
WARP (software)
WARP is a now-defunct IBM mainframe performance product from APT International, a former Monaco based company. Its launch was compromised by the near simulataneous launch of OS/2 Warp by IBM in 1994....
. A few months after launching this product, IBM renamed their OS/2
OS/2
OS/2 is a computer operating system, initially created by Microsoft and IBM, then later developed by IBM exclusively. The name stands for "Operating System/2," because it was introduced as part of the same generation change release as IBM's "Personal System/2 " line of second-generation personal...
product OS/2 Warp 4, conflicting with the marketing of the performance product that was the only competitor to BatchPipes, resulting in 7 years of litigation at Tribunal de grande instance in Paris
Paris
Paris is the capital and largest city in France, situated on the river Seine, in northern France, at the heart of the Île-de-France region...
Subsequently BatchPipes Version 2 was released, incorporating BatchPipes Version 1 and some additional features from SmartBatch: BatchPipePlex and BatchPipeWorks. BatchPipes Version 2 is still a marketed IBM product.
See also
- Pipeline (Unix)Pipeline (Unix)In Unix-like computer operating systems , a pipeline is the original software pipeline: a set of processes chained by their standard streams, so that the output of each process feeds directly as input to the next one. Each connection is implemented by an anonymous pipe...
, an original concept - CMS Pipelines
- ESA/390ESA/390ESA/390 was introduced in September 1990 and is IBM's last 31-bit-address/32-bit-data mainframe computing design, copied by Amdahl, Hitachi, and Fujitsu among other competitors...
, a mainframe architecture