Blake Riley

Scoring Rules for Self-Interested Experts

leave a comment »

While many people are curious about the future, few are ready to pay for expert predictions unless that information is relevant to their lives and decisions. Similarly, experts often have a stake in these decisions, not just in how much they are paid. Judgment-elicitation mechanisms should be robust to the possibility of experts with outside interests. Standard scoring rules are incentive-compatible only when experts are neutral to how the information is used.

In a forthcoming paper in the AAMAS proceedings, Craig Boutilier introduces the concept of a compensation rule, which augments typical scoring rule payments to form a net proper scoring rule. One proper compensation rule adds a payment equal to the expert’s loss in utility between the principal’s optimal decision and the expert’s preferred decision at that probability reported. This turns out to be more generous than necessary to guarantee expected expert utility is non-negative, but it is the only compensation rule that ensures expert prefer participation over the principal’s default policy. If experts are uncertain about the policy mapping reports to decisions, compensation can be reduced, but not eliminated.

Developing any proper compensation rule depends on the principal having full knowledge of the expert’s utility. Due to the strength of this assumption, the paper helpfully provides bounds on an expert’s incentive to misreport, the degree of misreporting, and the resulting expected utility loss of the decision-maker. With these bounds in hand, compensation rules can be developed to minimize the expected damage of misreporting without explicitly conditioning on the expert’s bias.

Unlike in other recent papers addressing decision markets, Boutilier assumes a single underlying random variable that can be observed regardless of the decision taken. This works well for events like the weather, where rain can be observed whether or not a wedding is held in the park or in a banquent hall. If instead, a company wanted to choose which state to open a new branch in based on expected sales, the sales in Maryland are never observed when the branch is opened in Massachusetts. This restriction in setting means the decision-maker can rely on a deterministic policy, mapping forecasts to decisions, without incentive issues. Being free from unobservable counterfactuals also simplifies the implementation of this scheme as a market scoring rule. I suspect these market scoring rules could be implemented as cost-function-based market makers without much difficulty, though Boutilier doesn’t address this.


Written by blakeriley

2012.01.24 at 14:04

Posted in Uncategorized

Market Scoring Rules

leave a comment »

Decision-makers in need of information face the dual tasks of finding experts and then motivating them to give accurate forecasts. If there is an obvious expert to rely on, proper scoring rules are a well-understood means of eliciting honest probabilities. Alternatively, if there is a large enough pool of people willing to participate in a market, prices from a continuous double auctions of contingent securities do well at aggregating information, without any need to screen for expertise. However, most prediction tasks are stuck between these two methods, with only a few, hard-to-identify individuals who can meaningfully give input. Market scoring rules bridge this gap, working with an arbitrary amount of agents without becoming deadlocked or breaking the bank of the decision-maker.

Market scoring rules, and their equivalent formulation as cost-function-based market makers, debuted in “Logarithmic market scoring rules for modular combinatorial information aggregation” by Robin Hanson, first circulated as a working paper in 2002 and published somewhat perfunctorily in 2007. Mechanisms that solved similar problems, like David Pennock’s dynamic pari-mutuel markets, came out around the same time, but Hanson’s innovation has shaped up to be the seminal advance in prediction market design.

At first glance, a market scoring rule is an almost trivial extension of typical scoring rule: each participant receives the difference between the score of his report and the score of the previous participant. This doesn’t affect incentive-compatibility or willingness to participate, because in the worse case, a participant could match the report of the previous agent and have no net payment. As a result, the sum of all the payments to participants telescope, leaving the sponsor of the market liable only for the difference in the scores of the last participant and some initial report.

Although developed in the context of a sequentially applied scoring rule, this system turns out to be equivalent to an automated market maker that sells shares of contingent securities. This feels more like a prediction market, but with some striking advantages. First, the prices of securities always form a coherent probability distribution by construction, simplifying interpretation. Second, the market has infinite liquidity because all transactions are conducted through the market-maker. Third, prices for all securities are updated whenever a sale or purchase is made. Together, these advantages mean markets for conjunctive or conditional events can be feasibly priced. Even if no one else ever trades on a joint security that Obama wins the 2012 presidential election and it snows in Washington DC on inauguration day, this security can be bought and the information expressed in the purchase percolates out to all other combinations of events.

The modern prediction market literature largely revolves around market-makers inspired by Hanson. A decade later, the logarithmic market-maker now has a air of classic elegance to it, in contrast to the seemingly primeval prior literature and the complex refinements that have followed.

Written by blakeriley

2012.01.23 at 23:29

Posted in economics

Tagged with ,

Incentives to Exercise (2009) – Charness and Gneezy

leave a comment »

Monetary incentives are tricky things. At one point, economists thought payments could only increase the frequency of the behavior being rewarded. Following the psychological literature, the possibility of extrinsic motivations (e.g. cash) crowding out intrinsic motivation is now widely accepted. Especially if a reward is given and then removed, people can be much less inclined to engage in the behavior. Deci (1971) is the classic paper on intrinsic motivations being displaced.

Alternatively, cash rewards could give a person a boost in adopting a habit. If the payment is removed, but the habit has been established, rates of engagement could be higher than the baseline. Given the public policy interest in making or breaking habits like regular exercise or smoking, respectively, knowing whether payments have a positive or negative effects is vital. I’m agnostic whether public policy should encourage better lifestyles, but at the very least it should do no harm.

Charness and Gneezy (working paper) address crowding-out vs habit-formation by paying undergrads at two universities for gym attendance. In addition to a control, one group was required to attend the gym once to receive payment, while another was required to attend eight times in a month. If a participant did not attend the gym regularly prior to the intervention and was in the eight-time treatment, they went to the gym about 0.75 times more per week than the control after the reward period. This group also showed some health improvements. For everyone else, regulars or one-time, the intervention had no or slightly negative effect. So, with a positive, lasting effect for some and no apparent downside, monetary incentives for exercise might be worth considering. This may be due to the subjects acquiring a habit rather than losing one, because incentive programs for quitting smoking for instance lose effect once the payments stop.

Written by blakeriley

2011.02.1 at 20:31

Posted in economics, psychology

Microfoundations of the city

leave a comment »

Geoffery West, a physicist associated with the Santa Fe Institute, concluded that cities scale superlinearly: when cities double in size, per-capita wealth, crime, innovation, traffic, construction spending, AIDS cases, etc, increases by 15%. Firms, on the other hand, scale sublinearly. While cities and firms are integral parts of economics, this looks nothing like modern economic theory. Where are the optimizing agents with preferences and budget constraints? Theoretical economics is something more like a method than a subject area, with little room for laws established by observation. Economists are happy users of statistics, but any overarching patterns should be reinforced by a model suggesting how the relationship emerges from interacting agents. Cosma Shalizi doubts whether macroeconomic theories really need these microfoundations. Certainly it would be great to know exactly how the law arises, but how necessary is it?

As I make the transition from economics student to economics researcher, questions like these are at the front of my mind. What exactly are economists trying to accomplish? My current opinion is that, contra Friedman, economic theory is not about prediction. Instead economics models are frameworks for ex ante understanding and evaluation of counterfactuals. I see strong ties between economics and evolutionary biology. If anything, economic tools like game theory are better suited to evolution than their original domain.

Written by blakeriley

2011.01.20 at 20:52

Division of labor at the LHC

leave a comment »

By my rough calculation, this paper from the LHC has 3,146 coauthors.

(ht: MR)

Written by blakeriley

2011.01.19 at 18:12

Posted in economics

Microplane graters and socialist calculation

leave a comment »

Today I was listening to an EconTalk episode about Ludwig von Mises, including his kicking-off the socialist calculation debate. Especially as expanded by Hayek, the argument is that prices aggregate distributed knowledge, guiding resources in ways that are very hard to predict, much less plan. Case in point: the Microplane grater.

“I didn’t set out to make cheese graters,” Mr. Grace, an engineer by education, said recently… “I thought I was making serious woodworking tools,” he continued. “To see them used in the kitchen, that was frankly a personal disappointment.”

“We laughed when people told us they were using our products in their kitchens,” recalled Microplane’s Web site and woodworking-products manager, Maria Grace, one of Richard Grace’s daughters. “But we didn’t turn down their orders.”

(hat tip: Kottke)

Written by blakeriley

2011.01.19 at 17:51

Posted in economics

Tagged with , ,

Psychology of Intimate Relationships video course

with 3 comments

Lectures from a UCLA course on love, attraction, marriage, and conflict are available free online. As a way to escape everything else on my plate right now, I started listening to these yesterday. The course is taught by Benjamin Karney, with assistance from Thomas Bradbury. Karney in particular is an entertaining speaker, and who doesn’t want to have a better understanding of love and relationships? My notes on the first four lectures follow.

Flash videos:
RealPlayer videos and mp3s:
Additional accompanying videos at: (Based on the first few, these videos are not that interesting and don’t add much. Despite being less than ten minutes each, my attention wandered quickly.)

*   *   *

Lecture 1: Course Introduction

In this lecture, Karney defines intimate relationships, points out how important they are in our lives, and identifies the big mystery in the field. To him, the really interesting question about love is why it changes over time,  possibly despite our desires for it to stay the same.

04:00 — Interdependence as a measure of whether a relationship exists; the degree to which the actions of one entity affects the outcomes of another determines the degree the two are related.
06:30 — Dimensions of interdependence: frequency of contact, duration of contact, diversity of types of interactions, direction of influence (uni- vs. bi-directional), and strength of influence
13:30 — Hence, a close relationship is one of strong, frequent, and diverse interdependence that lasts over a considerable period of time.
14:30 — Close relationships include family, friends, and intimate relationships, but this course only focuses on intimate ones, so what is intimacy?
16:30 — Sexual potential is a necessary, but not sufficient, component.
18:30 — Fidelity: you don’t care as much if your friends have other friends, but intimate partners are treated as special and unique, with a possessive element.
23:00 — Intimate relations are really, really powerful. Constitute our highest highs and lowest lows, to the point of affecting our health and longevity.
27:30 — Lying and deception: white lies are told to strangers, but the big lies are saved for close partners. If a wife is murdered, the first place to look is the husband.
31:00 — Holding a partner’s hand reduced the threat of an electric shock delivered in a brain scanner, and the stronger the relationship, the greater the effect.
33:00 — Couples that are nicer to each other tend to live longer.
35:00 — Why do some relationships last and some don’t?
36:30 — Over 50% of first marriages dissolve through divorce or legal separation.
37:40 — Divorce rate is highest in the second year, and declines steadily from there.
39:00 — Younger age of marriage correlates with higher rates of divorce, as well as remarriages.
41:00 — Rather than ask what prevents divorce, maybe we should care about what makes a relationship satisfying.
43:00 — However, we have a good sense of what makes a marriage satisfying: trust, humor, good sex, chemistry. The huge mystery is why things change.

*   *   *

Lecture 2: Methods of Studying Families and Couples

To objectively study an abstract topic like love, researchers must identify a concrete representation to measure. Methods of study include self-reports and direct observation, each with their own benefits and drawbacks. Because relationships are so important, research must be conducted carefully to remain ethical, but the potential benefits of understanding are enormous.

09:30 — Relationship science measures constructs, i.e. abstract ideas like love, commitment, or satisfaction. These can’t be studied directly, so a construct is operationalizalized by translating it into concrete terms.
14:30 — Health is a construct, but number of visits to a doctors is an operationalization.
15:00 — Self-reports are one possible measure of love. Only way to discover some things.
17:30 — Zick Rubin constructed one of the first love scales, a list of 13 questions for the respondent to rate agreement on a scale from 1 to 9.
23:00 — Self-reports are easy to conduct, cheap, and quick, but people are apt to lie or simply not be aware of the truth.
26:00 — Couples were asked to check a list of activities they did together over the last 24 hours, and agreement only occurred 50% of the time.
29:00 — Another strategy is systematic observation of verbal communication, emotional expressions, length of eye gaze, or biological responses.
47:00 — Observation can be very relevant (if you choose the metric well), has lots of detail, and requires concrete definitions.
48:45 — On the other hand, it’s very expensive, must be done with the subject aware they are being observed, and we don’t always know what a behavior means to the people involved.
54:30 — Cross-sectional research helps with description of what traits are associated with others. Longitudinal research helps with prediction.
57:00 — Type of research can lead to contradictory results.
59:00 — Cross-sectional data showed marital happiness increased steadily from 20 years on, but longitudinal data showed satisfaction declining through married life.
61:00 — Who are we studying?
65:00 — How ethical is the research?
68:00 — A study involving one group being repeatedly questioned and another only at the beginning and the end, and intense study magnified existing feelings. Good relationships became better and bad ones became worse.

*   *   *

Lecture 3: Theories of Intimate Relationships, Part I

Theories are like maps: they are better to the extent they correspond with reality, but also leave out details. They provide direction for further research by organizing existing knowledge. Two prominent theories are based on evolutionary history and childhood attachment. Evolutionary psychology looks to selection pressures in our ancestral environment that are carried over into the modern world. Attachment theory says that the mental models we build in childhood of how loved ones treat us are also applied in adulthood to romantic partners.

Note: A UC San Diego class was mentioned where the students did exercises designed to make them fall in love with paired partner. The class was taught by Robert Epstein, who was profiled in this article. Epstein wrote an article in Scientific American Mind on how we fall in love, along with descriptions of the exercises. Some include embracing and synchronizing breathing, gazing into each others’ eyes for two minutes, mimicking each others’ actions, sharing secrets, standing four feet apart and moving slightly closer every ten seconds until almost touching, and placing palms close, but not touching, and trying to sense the other person. Each is inspired by research, and said to increase closeness even with complete strangers.

02:00 — We develop personal theories about love to navigate our own relationships.
03:30 — “Science is built up of facts, as a house is built of stones; but an accumulation of facts is no more a science than a heap of stones is a house.” — Henri Poincaré
07:00 — What does a good theory do? Organizes existing knowledge, explains in a parsimonious way, makes specific predictions, and guides measurement decisions.
12:30 — A good theory of relationships should encompass the full range of predictors, specify mechanisms of change, and account for variability between between couples and within couples over time.
17:00 — Theories can be organized by how far they look back into human history.
17:30 — Evolutionary psychology looks for motives when seeking and dealing with a partner in ancestral environments. “Human seek specific mates to solve specific adaptive problems that their ancestors confronted during the source of human evolution.” — David Buss.
24:30 — Psychological mechanisms are the tendency to think and respond to a certain situation in a certain way, and are passed down just like physical traits.
30:00 — Our psychological mechanisms is adapted to living on the African savannah, not modern society.
33:00 — Theory of Parental Investment posits that the particular facts of human reproduction — women bearing a small number of children that require a high investment from the mother, but not necessarily the father — exerts the most influence on sexual preferences.
37:00 — Women should prefer men capable of protecting them and men should prefer women who are more fertile because of the differential costs.
40:40 — David Buss showed women find emotional infidelity worse than sexual infidelity, and vice versa for men.
45:00 — This result held internationally, across cultures and levels of development.
49:00 — The evolutionarily instilled psychological mechanisms are not necessarily conscious.
54:40 — Does the theory only pick out obvious predictions? A striking result: Ovulating women should be more sensitive to cues of reproductive fitness. To test this, college men were tested for physical symmetry, a known marker of genetic health. They were given clean t-shirts to wear while sleeping. Women who were ovulating preferred the scent of more symmetric men, even though other women couldn’t discern the difference.
61:00 — However, evolutionary theory doesn’t account well for differences within genders or how relationships change once they are formed.
62:00 — Attachment theory looks into our personal past, to the relationships we formed as children with our parents.
66:00 — From those early relationships, we develop mental models about whether others will be available to care for us, which are also applied to relationships.
67:00 — Mary Ainsworth showed babies tend to have three styles of attachment: secure, avoidant, and anxious/ambivalent. These ways of coping continue into adulthood.
69:00 — Same system might have evolved to deal with parent-child and lover relationships.
72:00 — Attachment theory can explain where our standards and expectations for intimacy come from and why we end up in the same types of relationships.

*   *   *

Lecture 4: Theories of Intimate Relationships, Part II

Other theories of love include Social Exchange Theory, Social Learning Theory, and Social Ecological Theory. The first says partners evaluate relationships based on their net rewards and costs. The second focuses on how partners respond to each others’ actions, with behavior being conditioned by repeated interaction. The last looks to external influences like culture or major events to explain how relationships differ and why they change.

05:30 — Best prediction of poverty is recent divorce.
09:30 — Social Exchange Theory posits partners evaluate the rewards and costs ssociated with being in the relationship, along with the alternatives. Recognizes that satisfaction with a relationship is not the only net outcome of the relationship.
14:00 — Rewards include material gains, status, comfort …
15:00 — Costs include pain, lost opportunities …
16:30 — Satisfaction is the difference between the outcome and expectations. Dependency is the difference between the outcome and the alternatives. The two are correlated, but could be very different.
23:00 — The theory predicts break-ups and divorce. Between 20-40% of people who file for divorce never follow through, possibly because the costs become more salient.
26:30 — Explains why some distress relationships continue, but still doesn’t address change well.
32:30 — Social Learning Theory says our behavior is molded and shaped within relationships. Rewarding and punishing interactions affect later behavior and judgments.
37:00 — Behaviors accumulate. Negative behaviors might be met with rewards in the short-term, but bring long-term costs. Nagging might bring desired actions right now, but in the future, communication might jump to resistance and yelling.
43:00 — Capability to spontaneously generate positive emotion can sustain a relationship.
45:00 — Reactions to behaviors mediated by cognitive models; same action might be interpreted differently at different times or by different people.
48:00 — Social Learning suggests researchers should observe actual interactions.
51:00 — Good couples have conflict, but can prevent negative feedback loops that make the conflict grow and continue.
53:00 — The theory suggests interventions can successfully improve relationships.
55:30 — Explains how relationships change through accumulated negativity, but leaves out where negative behavior originally comes from and how spontaneous improvement improves.
58:00 — Last theory is Social Ecological Theory, which looks to the stresses, supports, and constraints in the surrounding environment.
66:30 — Suggests culture and SES should be taken seriously.

Written by blakeriley

2010.03.14 at 20:46

Posted in notes, psychology