Blake Riley

Scoring Rules for Self-Interested Experts

leave a comment »

While many people are curious about the future, few are ready to pay for expert predictions unless that information is relevant to their lives and decisions. Similarly, experts often have a stake in these decisions, not just in how much they are paid. Judgment-elicitation mechanisms should be robust to the possibility of experts with outside interests. Standard scoring rules are incentive-compatible only when experts are neutral to how the information is used.

In a forthcoming paper in the AAMAS proceedings, Craig Boutilier introduces the concept of a compensation rule, which augments typical scoring rule payments to form a net proper scoring rule. One proper compensation rule adds a payment equal to the expert’s loss in utility between the principal’s optimal decision and the expert’s preferred decision at that probability reported. This turns out to be more generous than necessary to guarantee expected expert utility is non-negative, but it is the only compensation rule that ensures expert prefer participation over the principal’s default policy. If experts are uncertain about the policy mapping reports to decisions, compensation can be reduced, but not eliminated.

Developing any proper compensation rule depends on the principal having full knowledge of the expert’s utility. Due to the strength of this assumption, the paper helpfully provides bounds on an expert’s incentive to misreport, the degree of misreporting, and the resulting expected utility loss of the decision-maker. With these bounds in hand, compensation rules can be developed to minimize the expected damage of misreporting without explicitly conditioning on the expert’s bias.

Unlike in other recent papers addressing decision markets, Boutilier assumes a single underlying random variable that can be observed regardless of the decision taken. This works well for events like the weather, where rain can be observed whether or not a wedding is held in the park or in a banquent hall. If instead, a company wanted to choose which state to open a new branch in based on expected sales, the sales in Maryland are never observed when the branch is opened in Massachusetts. This restriction in setting means the decision-maker can rely on a deterministic policy, mapping forecasts to decisions, without incentive issues. Being free from unobservable counterfactuals also simplifies the implementation of this scheme as a market scoring rule. I suspect these market scoring rules could be implemented as cost-function-based market makers without much difficulty, though Boutilier doesn’t address this.

Written by blakeriley

2012.01.24 at 14:04

Posted in Uncategorized

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: