Subject: Re: Formative/Summative & Process/Outcome 2*2 Matrix? Date: Tue, 28 Dec 2004 12:24:10 -0500 From: Eileen Stryker Reply-To: American Evaluation Association Discussion List To: EVALTALK@BAMA.UA.EDU References:

I don't remember whether or where Dan Stufflebeam might have written this,

"Evaluation may be done to provide feedback to people who are trying to improve something (formative evaluation); or to provide information for decision-makers who are wondering whether to fund, terminate, or purchase something (summative evaluation)." (Scriven, 1980, S. 6-7)

Problematik des Begriffpaars

Der Begriff formative Evaluation (nicht das Konzept) geht auf Scriven (1972) zurück und bildet mit seinem Gegenstück summative Evaluation wohl das prominenteste Begriffspaar in der Evaluationsliteratur. Dennoch handelt es sich um einen problematischen Begriff, da er ungenau definiert, theoretisch unstimmig und in seiner praktischen Verwendung oft entsprechend beliebig ist (vgl. dazu etwa die Beiträge von Patton, Chen und Wholey in Evaluation Practice, 1996, Vol. 17, No. 2).

Empfehlungen zum Gebrauch von formativ und summativ

Da sich das Begriffspaar wegen seiner hohen Anmutungsqualität trotz dieser Probleme mit Sicherheit halten wird, scheint mir folgende Begriffsverwendung sinnvoll:

Die Begrifflichkeiten formativ/summativ werden ausschließlich zur Bezeichnung intendierter Evaluationszwecke verwendet, so wie es das obige Zitat Scrivens andeutet. Auf alle anderen von Scriven und Apologeten vertretenen Addenda wird verzichtet. Darunter fallen:

• Konfundierung von Evaluationszweck und Zeitpunkt der Evaluation (während der Maßnahme - nach Abschluss der Maßnahme)
• Konfundierung von Evaluationszweck und Evaluationsgegenstand (Programmkonzept - Realisierung des Programmkonzepts)
• Präskriptive Präferierung einer der beiden Seiten (nämlich der summativen)
• Anspruch, mit dem Begriffspaar eine erschöpfende Taxonomie möglicher Evaluationstypen abzubilden ("A world larger than formative and summative", M.Q.Patton)

fgs dhjkfgösdljkfg ösldfkjg ösdflgkj södflkgj sdöfglk sjödf

Date:    Wed, 21 Jul 2004 08:29:49 +0100
From:    bill fear 
Subject: Agendas, decisions and using evaluation


The debate about the role of the evaluator in relation to getting the
evaluation used has had a long and perennial history re-emerging, as ever,
about once every five years.  There are a couple, or more, important points
that are consistent (IMESHO):


1)  No evaluator has the right to assume that their findings will, or
should, be used as a number of people have just recently noted.  This right
is the preserve of auditors.
2)  There are two ways to maximise the value of an evaluation:  a) involve
stakeholders from the off (Patton); b) link evaluation to budgets
(Australia; the Netherlands).
3)  Most interestingly, a piece of work by by the NAO (probably by Chelimsky
and published around 5-8 years ago; sorry, I have a real problem remembering
references) showed that high quality evaluations tend to be rejected
initially.  However, these same evaluations usually have an impact around
five years - that's 5 years - later, usually at the conceptual level.  Ergo,
an acid test of a good evaluation that has been carried out independently of
stakeholders may well be the degree of initial resistance and rejection.
Indeed, it may be that an evaluation has more impact if the evaluator does
not try to get it taken account of.  Just think through what we know about
decision making.


On that point, any good evaluator surely must, surely absolutely must, have
an understanding of decision making from the individual level to the
organisational level.


Helpful references are:


at an individual level


www.bps.org.uk then click on 'publications' then 'the psychologist' then
'search the psychologist online' then 'volume 15 (2002)' then 'volume 15
part 2(February 2002)' then look at articles 4, 5, 6, 7.  Easy reading to a
high standard (mostly).


and


Gilbert, D. and Wilson, T.  'Miswanting.'
www.wjh.harvard.edu/~dtg/Gilbert%20&%20Wilson%20(Miswanting).pdf  (or put
'miswanting' into google)


At an organisational level it is still, for me, the stock in trade
publication of 'Organsiations:  Structures, processes and outcomes' by Hall.


We might also want to consider that US Senators apparently spends just 7
minutes a day reading on average and that for a GP to keep up to date with
current relevant medicine they need to read for 17 hours a week (mostly
non-fiction, or at least not knowingly fiction).


And then of course there is the values of the evaluator.  Our values tend to
drive our behaviour - although they don't have to.  Not judging others on
the basis of their values, which may conflict heavily with our own, is
immensely difficult.  So, we may assume that our evaluation should be taken
account of according to our values, but the values of the person on the
other side may be different.  And somehow we have to find a way not to let
that influence our behaviour and to respect the values of the other/s.
After all, there is no moral 'right' or 'wrong', and ethics are consensus of
agreed rules depicting right and wrong, and not a universal absolute, and
there is no known set of universal values.