Evaluation approaches
Encyclopedia
Evaluation approaches are conceptually distinct ways of thinking about, designing and conducting evaluation
efforts. Many of the evaluation approaches in use today make unique contributions to solving important problems, while others refine existing approaches in some way. Classification systems intended to sort out unique approaches from variations on a theme are presented here to help identify some basic schools of thought for conducting an evaluation. After these approaches are identified, they are summarized in terms of a few important attributes.
Since the mid 1960s, the number of alternative approaches to conducting evaluation efforts has increased dramatically. Factors such as the United States Elementary and Secondary Education Act
of 1965 that required educators to evaluate their efforts and results, and the growing public concern for accountability of human service programs contributed to this growth. In addition, over this period of time there has been an international movement towards encouraging evidence based practice
in all professions and in all sectors. Evidence Based Practice (EBP) requires evaluations to deliver the information needed to determine what is the best way of achieving results.
House considers all major evaluation approaches to be based on a common ideology
, liberal democracy
. Important principles of this ideology include freedom of choice
, the uniqueness of the individual
, and empirical
inquiry
grounded in objectivity
. He also contends they all are based on subjectivist ethics
, in which ethical conduct is based on the subjective or intuitive experience of an individual or group. One form of subjectivist ethics is utilitarian
, in which “the good” is determined by what maximizes some single, explicit interpretation of happiness for society as a whole. Another form of subjectivist ethics is intuitionist
/ pluralist, in which no single interpretation of “the good” is assumed and these interpretations need not be explicitly stated nor justified.
These ethical positions have corresponding epistemologies—philosophies
of obtaining knowledge
. The objectivist epistemology is associated with the utilitarian ethic. In general, it is used to acquire knowledge capable of external verification (intersubjective agreement) through publicly inspectable methods
and data
. The subjectivist epistemology is associated with the intuitionist/pluralist ethic. It is used to acquire new knowledge based on existing personal knowledge and experiences that are (explicit) or are not (tacit) available for public
inspection.
House further divides each epistemological approach by two main political
perspectives. Approaches can take an elite
perspective, focusing on the interests of managers and professionals. They also can take a mass perspective, focusing on consumer
s and participatory
approaches.
Stufflebeam and Webster place approaches into one of three groups according to their orientation toward the role of value
s, an ethical
consideration. The political orientation promotes a positive or negative view of an object regardless of what its value might actually be. They call this pseudo-evaluation. The questions orientation includes approaches that might or might not provide answers specifically related to the value of an object. They call this quasi-evaluation. The values orientation includes approaches primarily intended to determine the value of some object. They call this true evaluation.
Table 1 is used to classify fifteen evaluation approaches in terms of epistemology, major perspective (from House), and orientation (from Stufflebeam & Webster). When considered simultaneously, these three dimensions produce twelve cells. Only seven of the cells contain approaches, although all four true evaluation cells contain at least one approach.
Two pseudo-evaluation approaches, politically controlled and public relations studies, are represented. They are based on an objectivist epistemology from an elite perspective.
Six quasi-evaluation approaches use an objectivist epistemology. Five of them—experimental research, management information systems, testing programs, objectives-based studies, and content analysis—take an elite perspective. Accountability takes a mass perspective.
Seven true evaluation approaches are included. Two approaches, decision-oriented and policy studies, are based on an objectivist epistemology from an elite perspective. Consumer-oriented studies are based on an objectivist epistemology from a mass perspective. Two approaches—accreditation/certification and connoisseur studies—are based on a subjectivist epistemology from an elite perspective. Finally, adversary and client-centered studies are based on a subjectivist epistemology from a mass perspective.
Table 2 is used to summarize each approach in terms of four attributes
—organizer, purpose, strengths, and weaknesses. The organizer represents the main considerations or cues practitioners use to organize a study. The purpose represents the desired outcome
for a study at a very general level. Strengths and weaknesses represent other attributes that should be considered when deciding whether to use the approach for a particular study. The following narrative highlights differences between approaches grouped into the same cell of Table 1.
controlled and public relations studies are based on an objectivist epistemology from an elite perspective. Although both of these approaches seek to misrepresent value interpretations about some object, they go about it a bit differently. Information
obtained through politically controlled studies is released or withheld to meet the special interests of the holder.
Public relations
information is used to paint a positive image of an object regardless of the actual situation. Neither of these approaches is acceptable evaluation practice, although the seasoned reader can surely think of a few examples where they have been used.
Experimental research
is the best approach for determining causal relationships
between variables. The potential problem with using this as an evaluation approach is that its highly controlled and stylized methodology
may not be sufficiently responsive to the dynamically changing needs of most human service programs.
Management information system
s (MISs) can give detailed information about the dynamic operations of complex programs. However, this information is restricted to readily quantifiable
data usually available at regular intervals.
Testing programs are familiar to just about anyone who has attended school, served in the military, or worked for a large company. These programs are good at comparing individuals or groups to selected norms in a number of subject areas or to a set of standards of performance. However, they only focus on testee performance and they might not adequately sample what is taught or expected.
Objectives-based approaches relate outcomes to prespecified objectives, allowing judgments to be made about their level of attainment. Unfortunately, the objectives are often not proven to be important or they focus on outcomes too narrow to provide the basis for determining the value of an object.
Content analysis
is a quasi-evaluation approach because content analysis judgments need not be based on value statements. Instead, they can be based on knowledge. Such content analyses are not evaluations. On the other hand, when content analysis judgments are based on values, such studies are evaluations.
is popular with constituents because it is intended to provide an accurate accounting of results that can improve the quality of products and services. However, this approach quickly can turn practitioners and consumer
s into adversaries when implemented in a heavy-handed fashion.
-oriented studies are designed to provide a knowledge base for making and defending decisions. This approach usually requires the close collaboration
between an evaluator and decision-maker, allowing it to be susceptible to corruption
and bias
.
Policy studies
provide general guidance and direction on broad issues by identifying and assessing potential costs and benefits
of competing policies
. The drawback is these studies can be corrupted or subverted by the politically motivated actions of the participants.
-oriented studies are used to judge the relative merits of goods and services based on generalized needs and values, along with a comprehensive range of effects. However, this approach does not necessarily help practitioners improve their work, and it requires a very good and credible evaluator to do it well.
programs are based on self-study and peer review of organizations, programs, and personnel. They draw on the insights, experience, and expertise of qualified individuals who use established guidelines to determine if the applicant should be approved to perform specified functions. However, unless performance-based standards are used, attributes of applicants and the processes they perform often are overemphasized in relation to measures of outcomes or effects.
Connoisseur
studies are designed to provide a knowledge base for making and defending decisions. This approach usually requires the close collaboration between an evaluator and decision-maker, allowing it to be susceptible to corruption and bias.
issues through quasi-legal proceedings
. This helps ensure a balanced presentation of different perspectives on the issues, but it is also likely to discourage later cooperation and heighten animosities between contesting parties if “winners” and “losers” emerge.
Client-centered studies address specific concerns and issues of practitioners and other clients of the study in a particular setting. These studies help people understand the activities and values involved from a variety of perspectives. However, this responsive approach can lead to low external credibility
and a favorable bias toward those who participated in the study.
Evaluation
Evaluation is systematic determination of merit, worth, and significance of something or someone using criteria against a set of standards.Evaluation often is used to characterize and appraise subjects of interest in a wide range of human enterprises, including the arts, criminal justice,...
efforts. Many of the evaluation approaches in use today make unique contributions to solving important problems, while others refine existing approaches in some way. Classification systems intended to sort out unique approaches from variations on a theme are presented here to help identify some basic schools of thought for conducting an evaluation. After these approaches are identified, they are summarized in terms of a few important attributes.
Since the mid 1960s, the number of alternative approaches to conducting evaluation efforts has increased dramatically. Factors such as the United States Elementary and Secondary Education Act
Elementary and Secondary Education Act
The Elementary and Secondary Education Act , is a United States federal statute enacted April 11, 1965. It was passed as a part of President Lyndon B. Johnson's "War on Poverty" and has been the most far-reaching federal legislation affecting education ever passed by Congress...
of 1965 that required educators to evaluate their efforts and results, and the growing public concern for accountability of human service programs contributed to this growth. In addition, over this period of time there has been an international movement towards encouraging evidence based practice
Evidence based practice
Evidence-based practice is an interdisciplinary approach gaining ground after 1992. It started in medicine as evidence-based medicine and spread to other fields such as nursing, psychology, education, library and information science and other fields...
in all professions and in all sectors. Evidence Based Practice (EBP) requires evaluations to deliver the information needed to determine what is the best way of achieving results.
Classification of approaches
Two classifications of evaluation approaches by House and Stufflebeam & Webster were combined by Frisbie into a manageable number of approaches in terms of their unique and important underlying principles. The general structures of these classification systems are discussed first. The structures then are combined to present a more detailed classification of fifteen evaluation approaches.House considers all major evaluation approaches to be based on a common ideology
Ideology
An ideology is a set of ideas that constitutes one's goals, expectations, and actions. An ideology can be thought of as a comprehensive vision, as a way of looking at things , as in common sense and several philosophical tendencies , or a set of ideas proposed by the dominant class of a society to...
, liberal democracy
Liberal democracy
Liberal democracy, also known as constitutional democracy, is a common form of representative democracy. According to the principles of liberal democracy, elections should be free and fair, and the political process should be competitive...
. Important principles of this ideology include freedom of choice
Choice
Choice consists of the mental process of judging the merits of multiple options and selecting one of them. While a choice can be made between imagined options , often a choice is made between real options, and followed by the corresponding action...
, the uniqueness of the individual
Individual
An individual is a person or any specific object or thing in a collection. Individuality is the state or quality of being an individual; a person separate from other persons and possessing his or her own needs, goals, and desires. Being self expressive...
, and empirical
Empiricism
Empiricism is a theory of knowledge that asserts that knowledge comes only or primarily via sensory experience. One of several views of epistemology, the study of human knowledge, along with rationalism, idealism and historicism, empiricism emphasizes the role of experience and evidence,...
inquiry
Inquiry
An inquiry is any process that has the aim of augmenting knowledge, resolving doubt, or solving a problem. A theory of inquiry is an account of the various types of inquiry and a treatment of the ways that each type of inquiry achieves its aim.-Deduction:...
grounded in objectivity
Objectivity (philosophy)
Objectivity is a central philosophical concept which has been variously defined by sources. A proposition is generally considered to be objectively true when its truth conditions are met and are "mind-independent"—that is, not met by the judgment of a conscious entity or subject.- Objectivism...
. He also contends they all are based on subjectivist ethics
Ethics
Ethics, also known as moral philosophy, is a branch of philosophy that addresses questions about morality—that is, concepts such as good and evil, right and wrong, virtue and vice, justice and crime, etc.Major branches of ethics include:...
, in which ethical conduct is based on the subjective or intuitive experience of an individual or group. One form of subjectivist ethics is utilitarian
Utilitarianism
Utilitarianism is an ethical theory holding that the proper course of action is the one that maximizes the overall "happiness", by whatever means necessary. It is thus a form of consequentialism, meaning that the moral worth of an action is determined only by its resulting outcome, and that one can...
, in which “the good” is determined by what maximizes some single, explicit interpretation of happiness for society as a whole. Another form of subjectivist ethics is intuitionist
Ethical intuitionism
Ethical intuitionism is usually understood as a meta-ethical theory that embraces the following theses:# Moral realism, the view that there are objective facts of morality,...
/ pluralist, in which no single interpretation of “the good” is assumed and these interpretations need not be explicitly stated nor justified.
These ethical positions have corresponding epistemologies—philosophies
Philosophy
Philosophy is the study of general and fundamental problems, such as those connected with existence, knowledge, values, reason, mind, and language. Philosophy is distinguished from other ways of addressing such problems by its critical, generally systematic approach and its reliance on rational...
of obtaining knowledge
Knowledge
Knowledge is a familiarity with someone or something unknown, which can include information, facts, descriptions, or skills acquired through experience or education. It can refer to the theoretical or practical understanding of a subject...
. The objectivist epistemology is associated with the utilitarian ethic. In general, it is used to acquire knowledge capable of external verification (intersubjective agreement) through publicly inspectable methods
Methodology
Methodology is generally a guideline for solving a problem, with specificcomponents such as phases, tasks, methods, techniques and tools . It can be defined also as follows:...
and data
Data
The term data refers to qualitative or quantitative attributes of a variable or set of variables. Data are typically the results of measurements and can be the basis of graphs, images, or observations of a set of variables. Data are often viewed as the lowest level of abstraction from which...
. The subjectivist epistemology is associated with the intuitionist/pluralist ethic. It is used to acquire new knowledge based on existing personal knowledge and experiences that are (explicit) or are not (tacit) available for public
Public
In public relations and communication science, publics are groups of individuals, and the public is the totality of such groupings. This is a different concept to the sociological concept of the Öffentlichkeit or public sphere. The concept of a public has also been defined in political science,...
inspection.
House further divides each epistemological approach by two main political
Politics
Politics is a process by which groups of people make collective decisions. The term is generally applied to the art or science of running governmental or state affairs, including behavior within civil governments, but also applies to institutions, fields, and special interest groups such as the...
perspectives. Approaches can take an elite
Elite
Elite refers to an exceptional or privileged group that wields considerable power within its sphere of influence...
perspective, focusing on the interests of managers and professionals. They also can take a mass perspective, focusing on consumer
Consumer
Consumer is a broad label for any individuals or households that use goods generated within the economy. The concept of a consumer occurs in different contexts, so that the usage and significance of the term may vary.-Economics and marketing:...
s and participatory
Participation (decision making)
Participation in social science refers to different mechanisms for the public to express opinions - and ideally exert influence - regarding political, economic, management or other social decisions. Participatory decision making can take place along any realm of human social activity, including...
approaches.
Stufflebeam and Webster place approaches into one of three groups according to their orientation toward the role of value
Value (personal and cultural)
A personal or cultural value is an absolute or relative ethical value, the assumption of which can be the basis for ethical action. A value system is a set of consistent values and measures. A principle value is a foundation upon which other values and measures of integrity are based...
s, an ethical
Ethics
Ethics, also known as moral philosophy, is a branch of philosophy that addresses questions about morality—that is, concepts such as good and evil, right and wrong, virtue and vice, justice and crime, etc.Major branches of ethics include:...
consideration. The political orientation promotes a positive or negative view of an object regardless of what its value might actually be. They call this pseudo-evaluation. The questions orientation includes approaches that might or might not provide answers specifically related to the value of an object. They call this quasi-evaluation. The values orientation includes approaches primarily intended to determine the value of some object. They call this true evaluation.
Table 1 is used to classify fifteen evaluation approaches in terms of epistemology, major perspective (from House), and orientation (from Stufflebeam & Webster). When considered simultaneously, these three dimensions produce twelve cells. Only seven of the cells contain approaches, although all four true evaluation cells contain at least one approach.
Table 1
| Classification of approaches for conducting evaluations based on epistemology, major perspective, and orientation |
||||
|---|---|---|---|---|
| Epistemology (Ethic) |
Major perspective | Orientation | ||
| Political (Pseudo-evaluation) |
Questions (Quasi-evaluation) |
Values (True evaluation) |
||
| Objectivist (Utilitarian) |
Elite (Managerial) |
Politically controlled Public relations |
Experimental research Management information systems Testing programs Objectives-based Content analysis |
Decision-oriented Policy studies |
| Mass (Consumers) |
|
Accountability | Consumer-oriented | |
| Subjectivist (Institutionalist/ Pluralist) |
Elite (Professional) |
|
|
Accreditation/ certification Connoisseur |
| Mass (Participatory) |
|
|
Adversary Client-centered |
|
| Note. Epistemology and major perspective from House (1978). Orientation from Stufflebeam & Webster (1980). | ||||
Two pseudo-evaluation approaches, politically controlled and public relations studies, are represented. They are based on an objectivist epistemology from an elite perspective.
Six quasi-evaluation approaches use an objectivist epistemology. Five of them—experimental research, management information systems, testing programs, objectives-based studies, and content analysis—take an elite perspective. Accountability takes a mass perspective.
Seven true evaluation approaches are included. Two approaches, decision-oriented and policy studies, are based on an objectivist epistemology from an elite perspective. Consumer-oriented studies are based on an objectivist epistemology from a mass perspective. Two approaches—accreditation/certification and connoisseur studies—are based on a subjectivist epistemology from an elite perspective. Finally, adversary and client-centered studies are based on a subjectivist epistemology from a mass perspective.
Summary of approaches
The preceding section was used to distinguish between fifteen evaluation approaches in terms of their epistemology, major perspective, and orientation to values. This section is used to summarize each of the fifteen approaches in enough detail so that those placed in the same cell of Table 1 can be distinguished from each other.Table 2 is used to summarize each approach in terms of four attributes
Abstraction
Abstraction is a process by which higher concepts are derived from the usage and classification of literal concepts, first principles, or other methods....
—organizer, purpose, strengths, and weaknesses. The organizer represents the main considerations or cues practitioners use to organize a study. The purpose represents the desired outcome
Outcome (Game theory)
In game theory, an outcome is a set of moves or strategies taken by the players, or their payoffs resulting from the actions or strategies taken by all players. The two are complementary in that, given knowledge of the set of strategies of all players, the final state of the game is known, as are...
for a study at a very general level. Strengths and weaknesses represent other attributes that should be considered when deciding whether to use the approach for a particular study. The following narrative highlights differences between approaches grouped into the same cell of Table 1.
Table 2
| Summary of approaches for conducting evaluations | ||||
|---|---|---|---|---|
| Approach | Attribute | |||
| Organizer | Purpose | Key strengths | Key weaknesses | |
| Politically Politics Politics is a process by which groups of people make collective decisions. The term is generally applied to the art or science of running governmental or state affairs, including behavior within civil governments, but also applies to institutions, fields, and special interest groups such as the... controlled |
Threats | Get, keep or increase influence, power or money. | Secure evidence advantageous to the client in a conflict. | Violates the principle of full & frank disclosure. |
| Public relations Public relations Public relations is the actions of a corporation, store, government, individual, etc., in promoting goodwill between itself and the public, the community, employees, customers, etc.... |
Propaganda needs | Create positive public image. | Secure evidence most likely to bolster public support. | Violates the principles of balanced reporting, justified conclusions, & objectivity. |
| Experimental research Experiment An experiment is a methodical procedure carried out with the goal of verifying, falsifying, or establishing the validity of a hypothesis. Experiments vary greatly in their goal and scale, but always rely on repeatable procedure and logical analysis of the results... |
Causal relationships | Determine causal relationships between variables. | Strongest paradigm for determining causal relationships. | Requires controlled setting, limits range of evidence, focuses primarily on results. |
| Management information system Management information system A management information system provides information needed to manage organizations efficiently and effectively. Management information systems involve three primary resources: people, technology, and information. Management information systems are distinct from other information systems in that... s |
Scientific efficiency | Continuously supply evidence needed to fund, direct, & control programs. | Gives managers detailed evidence about complex programs. | Human service variables are rarely amenable to the narrow, quantitative definitions needed. |
| Testing programs | Individual differences | Compare test scores of individuals & groups to selected norms. | Produces valid & reliable evidence in many performance areas. Very familiar to public. | Data usually only on testee performance, overemphasizes test-taking skills, can be poor sample of what is taught or expected. |
| Objectives-based | Objectives | Relates outcomes to objectives. | Common sense appeal, widely used, uses behavioral objectives & testing technologies. | Leads to terminal evidence often too narrow to provide basis for judging to value of a program. |
| Content analysis Content analysis Content analysis or textual analysis is a methodology in the social sciences for studying the content of communication. Earl Babbie defines it as "the study of recorded human communications, such as books, websites, paintings and laws."According to Dr... |
Content of a communication | Describe & draw conclusion about a communication. | Allows for unobtrusive analysis of large volumes of unstructured, symbolic materials. | Sample may be unrepresentative yet overwhelming in volume. Analysis design often overly simplistic for question. |
| Accountability Accountability Accountability is a concept in ethics and governance with several meanings. It is often used synonymously with such concepts as responsibility, answerability, blameworthiness, liability, and other terms associated with the expectation of account-giving... |
Performance expectations | Provide constituents with an accurate accounting of results. | Popular with constituents. Aimed at improving quality of products and services. | Creates unrest between practitioners & consumers. Politics often forces premature studies. |
| Decision Decision making Decision making can be regarded as the mental processes resulting in the selection of a course of action among several alternative scenarios. Every decision making process produces a final choice. The output can be an action or an opinion of choice.- Overview :Human performance in decision terms... -oriented |
Decisions | Provide a knowledge & value base for making & defending decisions. | Encourages use of evaluation to plan & implement needed programs. Helps justify decisions about plans & actions. | Necessary collaboration between evaluator & decision-maker provides opportunity to bias results. |
| Policy studies Policy analysis Policy analysis is "determining which of various alternative policies will most achieve a given set of goals in light of the relations between the policies and the goals". However, policy analysis can be divided into two major fields. Analysis of policy is analytical and descriptive—i.e., it... |
Broad issues | Identify and assess potential costs & benefits of competing policies. | Provide general direction for broadly focused actions. | Often corrupted or subverted by politically motivated actions of participants. |
| Consumer Customer A customer is usually used to refer to a current or potential buyer or user of the products of an individual or organization, called the supplier, seller, or vendor. This is typically through purchasing or renting goods or services... -oriented |
Generalized needs & values, effects | Judge the relative merits of alternative goods & services. | Independent appraisal to protect practitioners & consumers from shoddy products & services. High public credibility. | Might not help practitioners do a better job. Requires credible & competent evaluators. |
| Accreditation / certification Professional certification Professional certification, trade certification, or professional designation, often called simply certification or qualification, is a designation earned by a person to assure qualification to perform a job or task... |
Standards & guidelines | Determine if institutions, programs, & personnel should be approved to perform specified functions. | Helps public make informed decisions about quality of organizations & qualifications of personnel. | Standards & guidelines typically emphasize intrinsic criteria to the exclusion of outcome measures. |
| Connoisseur Connoisseur A connoisseur is a person who has a great deal of knowledge about the fine arts, cuisines, or an expert judge in matters of taste.Modern connoisseurship must be seen along with museums, art galleries and "the cult of originality"... |
Critical guideposts | Critically describe, appraise, & illuminate an object. | Exploits highly developed expertise on subject of interest. Can inspire others to more insightful efforts. | Dependent on small number of experts, making evaluation susceptible to subjectivity, bias, and corruption. |
| Adversary | “Hot” issues | Present the pro & cons of an issue. | Ensures balances presentations of represented perspectives. | Can discourage cooperation, heighten animosities. |
| Client-centered | Specific concerns & issues | Foster understanding of activities & how they are valued in a given setting & from a variety of perspectives. | Practitioners are helped to conduct their own evaluation. | Low external credibility, susceptible to bias in favor of participants. |
| Note. Adapted and condensed primarily from House (1978) and Stufflebeam & Webster (1980). | ||||
Pseudo-evaluation
PoliticallyPolitics
Politics is a process by which groups of people make collective decisions. The term is generally applied to the art or science of running governmental or state affairs, including behavior within civil governments, but also applies to institutions, fields, and special interest groups such as the...
controlled and public relations studies are based on an objectivist epistemology from an elite perspective. Although both of these approaches seek to misrepresent value interpretations about some object, they go about it a bit differently. Information
Information
Information in its most restricted technical sense is a message or collection of messages that consists of an ordered sequence of symbols, or it is the meaning that can be interpreted from such a message or collection of messages. Information can be recorded or transmitted. It can be recorded as...
obtained through politically controlled studies is released or withheld to meet the special interests of the holder.
Public relations
Public relations
Public relations is the actions of a corporation, store, government, individual, etc., in promoting goodwill between itself and the public, the community, employees, customers, etc....
information is used to paint a positive image of an object regardless of the actual situation. Neither of these approaches is acceptable evaluation practice, although the seasoned reader can surely think of a few examples where they have been used.
Objectivist, elite, quasi-evaluation
As a group, these five approaches represent a highly respected collection of disciplined inquiry approaches. They are considered quasi-evaluation approaches because particular studies can legitimately focus only on questions of knowledge without addressing any questions of value. Such studies are, by definition, not evaluations. These approaches can produce characterizations without producing appraisals, although specific studies can produce both. Each of these approaches serves its intended purpose well. They are discussed roughly in order of the extent to which they approach the objectivist ideal.Experimental research
Experiment
An experiment is a methodical procedure carried out with the goal of verifying, falsifying, or establishing the validity of a hypothesis. Experiments vary greatly in their goal and scale, but always rely on repeatable procedure and logical analysis of the results...
is the best approach for determining causal relationships
Causality
Causality is the relationship between an event and a second event , where the second event is understood as a consequence of the first....
between variables. The potential problem with using this as an evaluation approach is that its highly controlled and stylized methodology
Methodology
Methodology is generally a guideline for solving a problem, with specificcomponents such as phases, tasks, methods, techniques and tools . It can be defined also as follows:...
may not be sufficiently responsive to the dynamically changing needs of most human service programs.
Management information system
Management information system
A management information system provides information needed to manage organizations efficiently and effectively. Management information systems involve three primary resources: people, technology, and information. Management information systems are distinct from other information systems in that...
s (MISs) can give detailed information about the dynamic operations of complex programs. However, this information is restricted to readily quantifiable
Numerical data
Numerical data is data measured or identified on a numerical scale. Numerical data can be analyzed using statistical methods, and results can be displayed using tables, charts, histograms and graphs. For example, a researcher will ask a questions to a participant that include words how often, how...
data usually available at regular intervals.
Testing programs are familiar to just about anyone who has attended school, served in the military, or worked for a large company. These programs are good at comparing individuals or groups to selected norms in a number of subject areas or to a set of standards of performance. However, they only focus on testee performance and they might not adequately sample what is taught or expected.
Objectives-based approaches relate outcomes to prespecified objectives, allowing judgments to be made about their level of attainment. Unfortunately, the objectives are often not proven to be important or they focus on outcomes too narrow to provide the basis for determining the value of an object.
Content analysis
Content analysis
Content analysis or textual analysis is a methodology in the social sciences for studying the content of communication. Earl Babbie defines it as "the study of recorded human communications, such as books, websites, paintings and laws."According to Dr...
is a quasi-evaluation approach because content analysis judgments need not be based on value statements. Instead, they can be based on knowledge. Such content analyses are not evaluations. On the other hand, when content analysis judgments are based on values, such studies are evaluations.
Objectivist, mass, quasi-evaluation
AccountabilityAccountability
Accountability is a concept in ethics and governance with several meanings. It is often used synonymously with such concepts as responsibility, answerability, blameworthiness, liability, and other terms associated with the expectation of account-giving...
is popular with constituents because it is intended to provide an accurate accounting of results that can improve the quality of products and services. However, this approach quickly can turn practitioners and consumer
Consumer
Consumer is a broad label for any individuals or households that use goods generated within the economy. The concept of a consumer occurs in different contexts, so that the usage and significance of the term may vary.-Economics and marketing:...
s into adversaries when implemented in a heavy-handed fashion.
Objectivist, elite, true evaluation
DecisionDecision making
Decision making can be regarded as the mental processes resulting in the selection of a course of action among several alternative scenarios. Every decision making process produces a final choice. The output can be an action or an opinion of choice.- Overview :Human performance in decision terms...
-oriented studies are designed to provide a knowledge base for making and defending decisions. This approach usually requires the close collaboration
Collaboration
Collaboration is working together to achieve a goal. It is a recursive process where two or more people or organizations work together to realize shared goals, — for example, an intriguing endeavor that is creative in nature—by sharing...
between an evaluator and decision-maker, allowing it to be susceptible to corruption
Political corruption
Political corruption is the use of legislated powers by government officials for illegitimate private gain. Misuse of government power for other purposes, such as repression of political opponents and general police brutality, is not considered political corruption. Neither are illegal acts by...
and bias
Bias
Bias is an inclination to present or hold a partial perspective at the expense of alternatives. Bias can come in many forms.-In judgement and decision making:...
.
Policy studies
Policy analysis
Policy analysis is "determining which of various alternative policies will most achieve a given set of goals in light of the relations between the policies and the goals". However, policy analysis can be divided into two major fields. Analysis of policy is analytical and descriptive—i.e., it...
provide general guidance and direction on broad issues by identifying and assessing potential costs and benefits
Cost-benefit analysis
Cost–benefit analysis , sometimes called benefit–cost analysis , is a systematic process for calculating and comparing benefits and costs of a project for two purposes: to determine if it is a sound investment , to see how it compares with alternate projects...
of competing policies
Policy
A policy is typically described as a principle or rule to guide decisions and achieve rational outcome. The term is not normally used to denote what is actually done, this is normally referred to as either procedure or protocol...
. The drawback is these studies can be corrupted or subverted by the politically motivated actions of the participants.
Objectivist, mass, true evaluation
ConsumerCustomer
A customer is usually used to refer to a current or potential buyer or user of the products of an individual or organization, called the supplier, seller, or vendor. This is typically through purchasing or renting goods or services...
-oriented studies are used to judge the relative merits of goods and services based on generalized needs and values, along with a comprehensive range of effects. However, this approach does not necessarily help practitioners improve their work, and it requires a very good and credible evaluator to do it well.
Subjectivist, elite, true evaluation
Accreditation / certificationProfessional certification
Professional certification, trade certification, or professional designation, often called simply certification or qualification, is a designation earned by a person to assure qualification to perform a job or task...
programs are based on self-study and peer review of organizations, programs, and personnel. They draw on the insights, experience, and expertise of qualified individuals who use established guidelines to determine if the applicant should be approved to perform specified functions. However, unless performance-based standards are used, attributes of applicants and the processes they perform often are overemphasized in relation to measures of outcomes or effects.
Connoisseur
Connoisseur
A connoisseur is a person who has a great deal of knowledge about the fine arts, cuisines, or an expert judge in matters of taste.Modern connoisseurship must be seen along with museums, art galleries and "the cult of originality"...
studies are designed to provide a knowledge base for making and defending decisions. This approach usually requires the close collaboration between an evaluator and decision-maker, allowing it to be susceptible to corruption and bias.
Subjectivist, mass, true evaluation
The adversary approach focuses on drawing out the pros and cons of controversialControversy
Controversy is a state of prolonged public dispute or debate, usually concerning a matter of opinion. The word was coined from the Latin controversia, as a composite of controversus – "turned in an opposite direction," from contra – "against" – and vertere – to turn, or versus , hence, "to turn...
issues through quasi-legal proceedings
Lawsuit
A lawsuit or "suit in law" is a civil action brought in a court of law in which a plaintiff, a party who claims to have incurred loss as a result of a defendant's actions, demands a legal or equitable remedy. The defendant is required to respond to the plaintiff's complaint...
. This helps ensure a balanced presentation of different perspectives on the issues, but it is also likely to discourage later cooperation and heighten animosities between contesting parties if “winners” and “losers” emerge.
Client-centered studies address specific concerns and issues of practitioners and other clients of the study in a particular setting. These studies help people understand the activities and values involved from a variety of perspectives. However, this responsive approach can lead to low external credibility
Credibility
Credibility refers to the objective and subjective components of the believability of a source or message.Traditionally, modern, credibility has two key components: trustworthiness and expertise, which both have objective and subjective components. Trustworthiness is based more on subjective...
and a favorable bias toward those who participated in the study.