DTrace
Encyclopedia
DTrace is a comprehensive dynamic tracing
framework created by Sun Microsystems
for troubleshooting
kernel
and application problems on production systems in real time. Originally developed for Solaris, it has since been released under the free Common Development and Distribution License
(CDDL) and has been ported to several other Unix-like
systems.
DTrace can be used to get a global overview of a running system, such as the amount of memory, CPU time, filesystem and network resources used by the active processes. It can also provide much more fine-grained information, such as a log of the arguments with which a specific function is being called, or a list of the processes accessing a specific file.
Tracing programs (also referred to as scripts) are written using the D programming language (not to be confused with other programming languages named "D"). The language is a subset of C
with added functions and variables specific to tracing. D programs resemble awk
programs in structure; they consist of a list of one or more probes (instrumentation points), and each probe is associated with an action. These probes are comparable to a pointcut
in aspect-oriented programming
. Whenever the condition for the probe is met, the associated action is executed (the probe "fires"). A typical probe might fire when a certain file is opened, or a process is started, or a certain line of code is executed. A probe that fires may analyze the run-time situation by accessing the call stack
and context variables and evaluating expressions; it can then print out or log some information, record it in a database, or modify context variables. The reading and writing of context variables allows probes to pass information to each other, allowing them to cooperatively analyze the correlation of different events.
Special consideration has been taken to make DTrace safe to use in a production environment. For example, there is minimal probe effect
when tracing is underway, and no performance impact associated with any disabled probe; this is important since there are tens of thousands of DTrace probes that can be enabled. New probes can also be created dynamically.
Scripts can also be written which can reach hundreds of lines in length, although typically only tens of lines are needed for advanced troubleshooting and analysis. Over 200 examples of open source DTrace scripts can be found in the DTraceToolkit, created by Brendan Gregg
(author of the DTrace book), which also provides documentation and demonstrations of each.
project to have its source code released under the Common Development and Distribution License
(CDDL).
DTrace has been ported to FreeBSD
and NetBSD
.
Apple added DTrace support in Mac OS X 10.5 "Leopard", including a GUI called Instruments
. Over 40 DTrace scripts from the DTraceToolkit are included in /usr/bin, including tools to examine disk I/O (iosnoop) and process execution (execsnoop). Unlike other platforms that DTrace is supported on, Mac OS X has a flag (P_LNOATTACH) that a program may set that disallows tracing of that process by debugging utilities such as DTrace and gdb. In the original Mac OS X DTrace implementation, this could affect tracing of other system information, as unrelated probes that should fire while a program with this flag set was running would fail to do so. This problem was addressed a few months later in the Mac OS X 10.5.3 update.
The Linux
port of DTrace has been available since 2008; work continues actively to enhance and fix issues. Standard core providers are available (fbt, syscall, profile), plus a special "instr" provider (some of the Solaris providers are not yet available). The Linux DTrace implementation is a loadable kernel module, which means that the kernel itself doesn't have to be modified, and also allows DTrace to avoid CDDL vs. GPL licensing conflicts.
DTrace is also being developed to support QNX
6.
Oracle added DTrace support for Oracle Linux in 2011.
, Mike Shapiro, and Adam Leventhal
.
The authors received recognition in 2005 for the innovations in DTrace from InfoWorld
and Technology Review
. DTrace won the top prize in the Wall Street Journal's 2006 Technology Innovation Awards competition. The authors were recognized by USENIX
with the Software Tools User Group (STUG) award in 2008.
Tracing (software)
In software engineering, tracing is a specialized use of logging to record information about a program's execution. This information is typically used by programmers for debugging purposes, and additionally, depending on the type and detail of information contained in a trace log, by experienced...
framework created by Sun Microsystems
Sun Microsystems
Sun Microsystems, Inc. was a company that sold :computers, computer components, :computer software, and :information technology services. Sun was founded on February 24, 1982...
for troubleshooting
Troubleshooting
Troubleshooting is a form of problem solving, often applied to repair failed products or processes. It is a logical, systematic search for the source of a problem so that it can be solved, and so the product or process can be made operational again. Troubleshooting is needed to develop and...
kernel
Kernel (computing)
In computing, the kernel is the main component of most computer operating systems; it is a bridge between applications and the actual data processing done at the hardware level. The kernel's responsibilities include managing the system's resources...
and application problems on production systems in real time. Originally developed for Solaris, it has since been released under the free Common Development and Distribution License
Common Development and Distribution License
Common Development and Distribution License is a free software license, produced by Sun Microsystems, based on the Mozilla Public License , version 1.1....
(CDDL) and has been ported to several other Unix-like
Unix-like
A Unix-like operating system is one that behaves in a manner similar to a Unix system, while not necessarily conforming to or being certified to any version of the Single UNIX Specification....
systems.
DTrace can be used to get a global overview of a running system, such as the amount of memory, CPU time, filesystem and network resources used by the active processes. It can also provide much more fine-grained information, such as a log of the arguments with which a specific function is being called, or a list of the processes accessing a specific file.
Description
DTrace is designed to give operational insights that allow users to tune and troubleshoot applications and the OS itself.Tracing programs (also referred to as scripts) are written using the D programming language (not to be confused with other programming languages named "D"). The language is a subset of C
C (programming language)
C is a general-purpose computer programming language developed between 1969 and 1973 by Dennis Ritchie at the Bell Telephone Laboratories for use with the Unix operating system....
with added functions and variables specific to tracing. D programs resemble awk
AWK (programming language)
The AWK utility is a data extraction and reporting tool that uses a data-driven scripting language consisting of a set of actions to be taken against textual data for the purpose of producing formatted reports...
programs in structure; they consist of a list of one or more probes (instrumentation points), and each probe is associated with an action. These probes are comparable to a pointcut
Pointcut
In aspect-oriented computer programming, a pointcut is a set of join points. Whenever the program execution reaches one of the join points described in the pointcut, a piece of code associated with the pointcut is executed. This allows a programmer to describe where and when additional code...
in aspect-oriented programming
Aspect-oriented programming
In computing, aspect-oriented programming is a programming paradigm which aims to increase modularity by allowing the separation of cross-cutting concerns...
. Whenever the condition for the probe is met, the associated action is executed (the probe "fires"). A typical probe might fire when a certain file is opened, or a process is started, or a certain line of code is executed. A probe that fires may analyze the run-time situation by accessing the call stack
Call stack
In computer science, a call stack is a stack data structure that stores information about the active subroutines of a computer program. This kind of stack is also known as an execution stack, control stack, run-time stack, or machine stack, and is often shortened to just "the stack"...
and context variables and evaluating expressions; it can then print out or log some information, record it in a database, or modify context variables. The reading and writing of context variables allows probes to pass information to each other, allowing them to cooperatively analyze the correlation of different events.
Special consideration has been taken to make DTrace safe to use in a production environment. For example, there is minimal probe effect
Probe effect
Probe effect is unintended alteration in system behavior caused by measuring that system.In code profiling and performance measurements, the delays introduced by insertion/removal of code instrumentation may result in a non-functioning application, or unpredictable behavior.-Examples:In...
when tracing is underway, and no performance impact associated with any disabled probe; this is important since there are tens of thousands of DTrace probes that can be enabled. New probes can also be created dynamically.
Command line examples
DTrace scripts can be invoked directly from the command line, providing one or more probes and actions as arguments. Some examples:
- New processes with arguments,
dtrace -n 'proc:::exec-success { trace(curpsinfo->pr_psargs); }'
- Files opened by process,
dtrace -n 'syscall::open*:entry { printf("%s %s",execname,copyinstr(arg0)); }'
- Syscall count by program,
dtrace -n 'syscall:::entry { @num[execname] = count; }'
- Syscall count by syscall,
dtrace -n 'syscall:::entry { @num[probefunc] = count; }'
- Syscall count by process,
dtrace -n 'syscall:::entry { @num[pid,execname] = count; }'
- Disk size by process,
dtrace -n 'io:::start { printf("%d %s %d",pid,execname,args[0]->b_bcount); }'
- Pages paged in by process,
dtrace -n 'vminfo:::pgpgin { @pg[execname] = sum(arg0); }'
Scripts can also be written which can reach hundreds of lines in length, although typically only tens of lines are needed for advanced troubleshooting and analysis. Over 200 examples of open source DTrace scripts can be found in the DTraceToolkit, created by Brendan Gregg
Brendan Gregg
Brendan Gregg is a kernel and performance engineer who worked at Sun Microsystems and later at Oracle Corporation following its acquisition by Sun. He left Oracle in October, 2010, to become the Lead Performance Engineer at Joyent....
(author of the DTrace book), which also provides documentation and demonstrations of each.
Supported platforms
DTrace was first made available for use in November 2003, and was formally released as part of Sun's Solaris 10 in January 2005. DTrace was the first component of the OpenSolarisOpenSolaris
OpenSolaris was an open source computer operating system based on Solaris created by Sun Microsystems. It was also the name of the project initiated by Sun to build a developer and user community around the software...
project to have its source code released under the Common Development and Distribution License
Common Development and Distribution License
Common Development and Distribution License is a free software license, produced by Sun Microsystems, based on the Mozilla Public License , version 1.1....
(CDDL).
DTrace has been ported to FreeBSD
FreeBSD
FreeBSD is a free Unix-like operating system descended from AT&T UNIX via BSD UNIX. Although for legal reasons FreeBSD cannot be called “UNIX”, as the direct descendant of BSD UNIX , FreeBSD’s internals and system APIs are UNIX-compliant...
and NetBSD
NetBSD
NetBSD is a freely available open source version of the Berkeley Software Distribution Unix operating system. It was the second open source BSD descendant to be formally released, after 386BSD, and continues to be actively developed. The NetBSD project is primarily focused on high quality design,...
.
Apple added DTrace support in Mac OS X 10.5 "Leopard", including a GUI called Instruments
Instruments (application)
Instruments is an application performance analyzer and visualizer, integrated in Xcode 3.0 and later versions of Xcode. It is a developer tool included in Apple Mac OS X v10.5 and later versions of Mac OS X, built on top of the DTrace tracing framework from OpenSolaris and ported to Mac OS...
. Over 40 DTrace scripts from the DTraceToolkit are included in /usr/bin, including tools to examine disk I/O (iosnoop) and process execution (execsnoop). Unlike other platforms that DTrace is supported on, Mac OS X has a flag (P_LNOATTACH) that a program may set that disallows tracing of that process by debugging utilities such as DTrace and gdb. In the original Mac OS X DTrace implementation, this could affect tracing of other system information, as unrelated probes that should fire while a program with this flag set was running would fail to do so. This problem was addressed a few months later in the Mac OS X 10.5.3 update.
The Linux
Linux
Linux is a Unix-like computer operating system assembled under the model of free and open source software development and distribution. The defining component of any Linux system is the Linux kernel, an operating system kernel first released October 5, 1991 by Linus Torvalds...
port of DTrace has been available since 2008; work continues actively to enhance and fix issues. Standard core providers are available (fbt, syscall, profile), plus a special "instr" provider (some of the Solaris providers are not yet available). The Linux DTrace implementation is a loadable kernel module, which means that the kernel itself doesn't have to be modified, and also allows DTrace to avoid CDDL vs. GPL licensing conflicts.
DTrace is also being developed to support QNX
QNX
QNX is a commercial Unix-like real-time operating system, aimed primarily at the embedded systems market. The product was originally developed by Canadian company, QNX Software Systems, which was later acquired by Canadian BlackBerry-producer Research In Motion.-Description:As a microkernel-based...
6.
Oracle added DTrace support for Oracle Linux in 2011.
Authors and awards
DTrace was designed and implemented by Bryan CantrillBryan Cantrill
Bryan M. Cantrill is an engineer who worked at Sun Microsystems and later at Oracle Corporation following its acquisition by Sun. He left Oracle on July 25, 2010 to become the Vice President of Engineering at Joyent....
, Mike Shapiro, and Adam Leventhal
Adam Leventhal (programmer)
Adam Leventhal is an American software engineer, and one of the three authors of DTrace, a dynamic tracing facility in Solaris 10 which allows users to observe, debug and tune system behavior in real time...
.
The authors received recognition in 2005 for the innovations in DTrace from InfoWorld
InfoWorld
InfoWorld is an information technology online media and events business operating under the umbrella of InfoWorld Media Group, a division of IDG...
and Technology Review
Technology Review
Technology Review is a magazine published by the Massachusetts Institute of Technology. It was founded in 1899 as "The Technology Review", and was re-launched without the "The" in its name on April 23, 1998 under then publisher R. Bruce Journey...
. DTrace won the top prize in the Wall Street Journal's 2006 Technology Innovation Awards competition. The authors were recognized by USENIX
USENIX
-External links:* *...
with the Software Tools User Group (STUG) award in 2008.
External links
- DTrace Hands On Lab - A step by step course to learn DTrace
- DLight Tutorial — DLight is an interactive graphical observability tool for C/C++ developers based on DTrace technology from the Oracle Solaris StudioSun Studio (software)The Oracle Solaris Studio compiler suite is Oracle's flagship software development product for Solaris and Linux. It was formerly known as Sun Studio...
- Exploring Leopard with DTrace -- how to use DTrace for debugging and exploration
- Google Tech Talk on DTrace given by Bryan Cantrill