But why not become an expert liar, if that’s what maximizes expected utility?  Why take the constrained path of truth, when things so much more important are at stake?

Because, when I look over my history, I find that my ethics have, above all, protected me from myself.  They weren’t inconveniences.  They were safety rails on cliffs I didn’t see.

I made fundamental mistakes, and my ethics didn’t halt that, but they played a critical role in my recovery.  When I was stopped by unknown unknowns that I just wasn’t expecting, it was my ethical constraints, and not any conscious planning, that had put me in a recoverable position.

You can’t duplicate this protective effect by trying to be clever and calculate the course of “highest utility”.  The expected utility just takes into account the things you know to expect.  It really is amazing, looking over my history, the extent to which my ethics put me in a recoverable position from my unanticipated, fundamental mistakes, the things completely outside my plans and beliefs.

Eliezer Yudkowsky, Protected From Myself
(via raginrayguns)

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s