Legible decision procedure

A computer program is making a decision based on some criteria.

The criteria are complicated, and the decision is important, so at least some of the time the humans’ reaction is to say “that’s wrong, why did it do that?”, yet end up agreeing with the decision after looking at it in detail. In addition the decision procedure changes over releases of the software, tending to increase in complexity, and further confusing humans interacting with it, as they’re not constantly up to date on the changes.

As an example, here’s some python implementing a mortgage lending decision:

def should_issue(applicant, property):
    return (
        property.is_insurable()
        and applicant.can_afford_payments(property.interest_payments(applicant.down_payment()) + property.insurance_costs() + property.utilities() + property.maintenance_costs() + property.taxes())
        and (
            applicant.down_payment() >= property.purchase_price * 0.2
            or applicant.down_payment() >= property.purchase_price - property.foreclosure_resale_value()
        )
        and (
           applicant.has_good_credit_score() or applicant.has_cosigner()
        )

The goal is to present to the humans the following:

• A representation of the decision procedure in the abstract.
• A representation of a past instance of the decision procedure on some specific facts, with a focus on making it easy to answer the following:
  • Which decision was taken.
  • Which factors, if changed, could change the overall decision.
• A representation of the overall set of decisions taken, with the # of times each particular path was taken.

This should be done without making the code itself too hard to read or test.