
Interpreting
Model Performance:
Use the “Smart” Decile Analysis Bruce Ratner, Ph.D. Data analysts use the
decile analysis – based on the scores of the response model at hand –
for creating a solicitation list of the most likely
individuals to obtain an advantage over a random selection
of individuals. The decile analysis involves a brute (“dumb”) division
of a database into
ten equalsized contiguous groups (deciles) without regard for
the
shape of the distribution of model scores. The assumption of this
“dumb” decile analysis – individuals within a decile have
equivalent model scores, which are different from the model scores of
the aboveandbelow neighboring deciles – is not always tenable,
as the distribution of model scores is not always "smooth" but often
characterized by "clumps" or "gaps". Deciles with these characteristics
lodge extreme response segments, which reflect what the model is doing
and how to implement the model to obtain a greater advantage over a
random selection. The purpose of this article is to present the correct
approach to interpreting model performance via the "smart"
decile analysis, which provides a division of a database taking into
account the clumps and gaps, for identifying extreme response segments
to aid in understanding what the response model is doing and how to
best implement the response model.
(Point of Note: The smart decile analysis is employed and enjoyed
within the GenIQ Model
Software.) Two
Illustrations of
Dumb and Smart Decile Analyses
Illustration #1 for a Response
Model #1
How to Read the Smart Decile Analysis The quasi Ntile analysis
(smart
decile analysis) is used to helplessly show that the dumb decile
analysis is
misleading in its display of model performance. Although, the quasi
10tile analysis
of Illustration #1, below, produces 10 divisions or tiles (which is not
always the
case; see Illustration #2, below), its display is not like the
corresponding decile
analysis. Therefrom, the quasi Ntile analysis shows that the decile
analysis assumption
– individuals within a given decile have equivalent model scores,
reflecting
equivalent likelihoods of responding – is not met, and therefore, the
estimates
from the dumb decile analysis are not honest. (Let’s not concern
ourselves with
rounding off individuals in the deciles for now: Who wants to discuss
919.5
individuals anyway?!)
I use the quasi 10tile
analysis
to parse the Top decile to show that the model scores form three
clusters of
individuals, each with nonequivalent responsiveness. That is, the Top
decile
consists of individuals of three levels of response rates – 20.00%,
12.40%, and
9.84%. This is inferred as follows:
Illustration #2 for a Response Model #2 How to Read the Smart Decile Analysis The quasi Ntile analysis (smart decile analysis) for Illustration #2 is read similarly to that in Illustration #1. But, note that the quasi 10tile for Illustration #2 only has (showing) six tiles. The four "missing" tiles (2^{nd}, 4^{th}, 6^{th}, and 8^{th}) are actually suppressed as their model scores are nonobservable "gap" scores, and indicate there are no individuals in these tiles. Clearly, this smart decile analysis demonstrates that the estimates from the dumb decile analysis are not honest. Using 50tiles Using 100tiles Using 200tiles 1 800 DM STAT1, or email at br@dmstat1.com. 