Job scheduler
Encyclopedia
A job scheduler is a software application that is in charge of unattended background executions, commonly known for historical reasons as batch processing
.
Synonyms are batch system, Distributed Resource Management System (DRMS), and Distributed Resource Manager (DRM). Today's job schedulers typically provide a graphical user interface and a single point of control for definition and monitoring of background executions in a distributed network of computers. Increasingly job schedulers are required to orchestrate the integration of real-time business activities with traditional background IT processing, across different operating system
platforms and business application environments.
Job scheduling should not be confused with process scheduling
, which is the assignment of currently running processes to CPUs by the operating system
.
If software from a completely different area includes all or some of those features, this software is considered to have job scheduling capabilities.
Most operating system
s (such as Unix
and Windows
) provide basic job scheduling capabilities, for example: cron
. Web hosting services provide job scheduling capabilities through a control panel
or a webcron solution
. Many programs such as DBMS
, backup
, ERPs
, and BPM
also include relevant job-scheduling capabilities. Operating-system (OS) or point program supplied job-scheduling will not usually provide the ability to schedule beyond a single OS instance or outside the remit of the specific program. Organizations needing to automate highly complex related and un-related IT workload may also leverage more-advanced features from a job scheduler, such as:
These advanced capabilities can be written by in-house developers but are more often provided by suppliers who specialize in systems-management
software.
Beyond the basic, single OS instance scheduling tools there are two major architectures that exist for Job Scheduling software.
Some widely used cluster batch systems are Oracle Grid Engine
, Portable Batch System
, LoadLeveler
, Condor, OAR
and Simple Linux Utility for Resource Management
.
.”
From a historical point of view, we can distinguish two main eras about Job Schedulers:
In terms of the type of scheduling there are also distinct eras:
Batch processing
Batch processing is execution of a series of programs on a computer without manual intervention.Batch jobs are set up so they can be run to completion without manual intervention, so all input data is preselected through scripts or command-line parameters...
.
Synonyms are batch system, Distributed Resource Management System (DRMS), and Distributed Resource Manager (DRM). Today's job schedulers typically provide a graphical user interface and a single point of control for definition and monitoring of background executions in a distributed network of computers. Increasingly job schedulers are required to orchestrate the integration of real-time business activities with traditional background IT processing, across different operating system
Operating system
An operating system is a set of programs that manage computer hardware resources and provide common services for application software. The operating system is the most important type of system software in a computer system...
platforms and business application environments.
Job scheduling should not be confused with process scheduling
Scheduling (computing)
In computer science, a scheduling is the method by which threads, processes or data flows are given access to system resources . This is usually done to load balance a system effectively or achieve a target quality of service...
, which is the assignment of currently running processes to CPUs by the operating system
Operating system
An operating system is a set of programs that manage computer hardware resources and provide common services for application software. The operating system is the most important type of system software in a computer system...
.
Overview
Basic features expected of job scheduler software include:- interfaces which help to define workflows and/or job dependencies
- automatic submission of executions
- interfaces to monitor the executions
- priorities and/or queues to control the execution order of unrelated jobs
If software from a completely different area includes all or some of those features, this software is considered to have job scheduling capabilities.
Most operating system
Operating system
An operating system is a set of programs that manage computer hardware resources and provide common services for application software. The operating system is the most important type of system software in a computer system...
s (such as Unix
Unix
Unix is a multitasking, multi-user computer operating system originally developed in 1969 by a group of AT&T employees at Bell Labs, including Ken Thompson, Dennis Ritchie, Brian Kernighan, Douglas McIlroy, and Joe Ossanna...
and Windows
Microsoft Windows
Microsoft Windows is a series of operating systems produced by Microsoft.Microsoft introduced an operating environment named Windows on November 20, 1985 as an add-on to MS-DOS in response to the growing interest in graphical user interfaces . Microsoft Windows came to dominate the world's personal...
) provide basic job scheduling capabilities, for example: cron
Cron
Cron is a time-based job scheduler in Unix-like computer operating systems. Cron enables users to schedule jobs to run periodically at certain times or dates...
. Web hosting services provide job scheduling capabilities through a control panel
Control panel (computer)
Many computer user interfaces use a control panel metaphor to give the user control of software and hardware features.-Computer History:The term control panel was used for the plugboards in unit record equipment and in the early computers of the 1940s and '50s...
or a webcron solution
Webcron
webcron is the term for a time-based job scheduler hosted on a web server. The name derives its roots from the phrase web server and the Unix daemon cron...
. Many programs such as DBMS
Database management system
A database management system is a software package with computer programs that control the creation, maintenance, and use of a database. It allows organizations to conveniently develop databases for various applications by database administrators and other specialists. A database is an integrated...
, backup
Backup
In information technology, a backup or the process of backing up is making copies of data which may be used to restore the original after a data loss event. The verb form is back up in two words, whereas the noun is backup....
, ERPs
Enterprise resource planning
Enterprise resource planning systems integrate internal and external management information across an entire organization, embracing finance/accounting, manufacturing, sales and service, customer relationship management, etc. ERP systems automate this activity with an integrated software application...
, and BPM
Business process management
Business process management is a holistic management approach focused on aligning all aspects of an organization with the wants and needs of clients. It promotes business effectiveness and efficiency while striving for innovation, flexibility, and integration with technology. BPM attempts to...
also include relevant job-scheduling capabilities. Operating-system (OS) or point program supplied job-scheduling will not usually provide the ability to schedule beyond a single OS instance or outside the remit of the specific program. Organizations needing to automate highly complex related and un-related IT workload may also leverage more-advanced features from a job scheduler, such as:
- real-time scheduling based on external, un-predictable events
- automatic restart and recovery in event of failures
- alerting and notification to operations personnel
- generation of incident reports
- auditAuditThe general definition of an audit is an evaluation of a person, organization, system, process, enterprise, project or product. The term most commonly refers to audits in accounting, but similar concepts also exist in project management, quality management, and energy conservation.- Accounting...
trails for regulatory complianceCompliance (regulation)In general, compliance means conforming to a rule, such as a specification, policy, standard or law. Regulatory compliance describes the goal that corporations or public agencies aspire to in their efforts to ensure that personnel are aware of and take steps to comply with relevant laws and...
purposes
These advanced capabilities can be written by in-house developers but are more often provided by suppliers who specialize in systems-management
Systems management
Systems management refers to enterprise-wide administration of distributed systems including computer systems. Systems management is strongly influenced by network management initiatives in telecommunications....
software.
Main concepts
There are many concepts that are central to almost every job scheduler implementation and that are widely recognized with minimal variations:- Jobs
- Dependencies
- Job Streams
- Users
Beyond the basic, single OS instance scheduling tools there are two major architectures that exist for Job Scheduling software.
- Master/Agent architecture — the historic architecture for Job scheduling software. The Job Scheduling software is installed on a single machine (Master) while on production machines only a very small component (Agent) is installed that awaits commands from the Master, executes them, and returns the exit code back to the Master.
- Cooperative architecture — a decentralized model where each machine is capable of helping with scheduling and can offload locally scheduled jobs to other cooperating machines. This enables dynamic workload balancing to maximize hardware resource utilization and high availabilityHigh availabilityHigh availability is a system design approach and associated service implementation that ensures a prearranged level of operational performance will be met during a contractual measurement period....
to ensure service delivery.
Batch queuing for HPC clusters
An important niche for job schedulers is managing the job queue for a cluster of computers. Typically, the scheduler will schedule jobs from the queue as sufficient resources (cluster nodes) become idle.Some widely used cluster batch systems are Oracle Grid Engine
Sun Grid Engine
Oracle Grid Engine, previously known as Sun Grid Engine , previously known as CODINE or GRD , is an open source batch-queuing system, developed and supported by Sun Microsystems...
, Portable Batch System
Portable Batch System
Portable Batch System is the name of computer software that performs job scheduling. Its primary task is to allocate computational tasks, i.e., batch jobs, among the available computing resources...
, LoadLeveler
LoadLeveler
LoadLeveler is a job scheduler written by IBM, to control scheduling of batch jobs. LoadLeveler matches the job requirements with the best available computer resource for execution...
, Condor, OAR
High-performance computing
High-performance computing uses supercomputers and computer clusters to solve advanced computation problems. Today, computer systems approaching the teraflops-region are counted as HPC-computers.-Overview:...
and Simple Linux Utility for Resource Management
Simple Linux Utility for Resource Management
Simple Linux Utility for Resource Management is an opensource job scheduler used by many of the world's supercomputers and computer clusters. It provides three key functions. First it allocates exclusive and/or non-exclusive access to resources to users for some duration of time so they can...
.
History
Job Scheduling has a long history. Job Schedulers are one of the major components of the IT infrastructure since the early mainframe systems. At first, stacks of punched cards were processed one after the other, hence the term “batch processingBatch processing
Batch processing is execution of a series of programs on a computer without manual intervention.Batch jobs are set up so they can be run to completion without manual intervention, so all input data is preselected through scripts or command-line parameters...
.”
From a historical point of view, we can distinguish two main eras about Job Schedulers:
- The mainframe era
- Job Control LanguageJob Control LanguageJob Control Language is a scripting language used on IBM mainframe operating systems to instruct the system on how to run a batch job or start a subsystem....
(JCL) on IBM mainframeIBM mainframeIBM mainframes are large computer systems produced by IBM from 1952 to the present. During the 1960s and 1970s, the term mainframe computer was almost synonymous with IBM products due to their marketshare...
s. Initially based on JCL functionality to handle dependencies this era is typified by the development of sophisticated scheduling solutions forming part of the systems management and automation toolset on the mainframe.
- Job Control Language
- The open systemOpen system (computing)Open systems are computer systems that provide some combination of interoperability, portability, and open software standards. The term was popularized in the early 1980s, mainly to describe systems based on Unix,...
s era- Modern schedulers on a variety of architectures and operating systems. With standard scheduling tools limited to such as CronCronCron is a time-based job scheduler in Unix-like computer operating systems. Cron enables users to schedule jobs to run periodically at certain times or dates...
, the need for mainframe standard job schedulers has grown with the increased adoption of distributed computing environments.
- Modern schedulers on a variety of architectures and operating systems. With standard scheduling tools limited to such as Cron
In terms of the type of scheduling there are also distinct eras:
- Batch processingBatch processingBatch processing is execution of a series of programs on a computer without manual intervention.Batch jobs are set up so they can be run to completion without manual intervention, so all input data is preselected through scripts or command-line parameters...
- the traditional date and time based execution of background tasks based on a defined period during which resources were available for batch processing (the batch window). In effect the original mainframe approach transposed onto the open systems environment. - Event-driven process automation - where background processes cannot be simply run at a defined time, either because the nature of the business demands that workload is based on the occurrence of external events (such as the arrival of an order from a customer or a stock update from a store branch) or because there is no / insufficient batch window.
- Service Oriented job scheduling - recent developments in Service Oriented Architecture (SOA) have seen a move towards deploying job scheduling as a reusable IT infrastructure service that can play a role in the integration of existing business application workload with new Web Services based real-time applications.
Scheduling
Various schemes are used to decide which particular job to run. Parameters that might be considered include:- Job priority
- Compute resource availability
- License key if job is using licensed software
- Execution time allocated to user
- Number of simultaneous jobs allowed for a user
- Estimated execution time
- Elapsed execution time
- Availability of peripheral devices
- Occurrence of prescribed events
Common batch systems
- Univa Grid EngineUniva Grid Engine- History :The roots of Grid Engine as a commercial product date back to 1993 . A more comprehensive genealogy of the product is described in Sun Grid Engine. Grid Engine was first distributed by Genias Software and from 1999, after a company merger, by Gridware In. In 2000, Sun Microsystems...
- Oracle Grid Engine
- Platform LSF
- Maui Cluster SchedulerMaui Cluster SchedulerMaui Cluster Scheduler is a job scheduler for use on clusters and supercomputers initially developed by Cluster Resources, Inc.. Maui is capable of supporting multiple scheduling policies, dynamic priorities, reservations, and fairshare capabilities...
- Moab Cluster SuiteMoab Cluster SuiteThe Moab Cluster Suite is a cluster workload management package, available from Adaptive Computing, Inc., that integrates the scheduling, managing, monitoring and reporting of cluster workloads. Moab Cluster Suite simplifies and unifies management across one or multiple hardware, operating system,...
- TORQUE Resource ManagerTORQUE Resource Manager-External links:* *...
- Portable Batch SystemPortable Batch SystemPortable Batch System is the name of computer software that performs job scheduling. Its primary task is to allocate computational tasks, i.e., batch jobs, among the available computing resources...
- Condor High-Throughput Computing System
- Simple Linux Utility for Resource ManagementSimple Linux Utility for Resource ManagementSimple Linux Utility for Resource Management is an opensource job scheduler used by many of the world's supercomputers and computer clusters. It provides three key functions. First it allocates exclusive and/or non-exclusive access to resources to users for some duration of time so they can...
(SLURM) - Optimal Allocation of Resources (OAR)
See also
- Business Process AutomationBusiness process automationBusiness process automation, or BPA, is the strategy a business uses to automate processes in order to contain costs. It consists of integrating applications, restructuring labor resources and using software applications throughout the organization....
- Computational resourceComputational resourceIn computational complexity theory, a computational resource is a resource used by some computational models in the solution of computational problems....
- Distributed computingDistributed computingDistributed computing is a field of computer science that studies distributed systems. A distributed system consists of multiple autonomous computers that communicate through a computer network. The computers interact with each other in order to achieve a common goal...
- List of job scheduler software
Further reading
- Scott, M: The Shortcut Guide to IT Workload Automation and Job Scheduling. Realtime Publishers, 2009.