commission voting 2004, the debriefing

Hi everybody,

This year, the Rhizome Net Art Commissions used a membership-driven
voting system to help determine who would win an award. Here are some
of my notes from the process; I look forward to hearing feedback &
discussion from anyone who participated in the process, either as a
candidate or a voter or just an onlooker.

The full voting rules can be found at
http://rhizome.org/commissions/2004_voting.rhiz .

ISSUES ABOUT MAKING THE PROCESS PUBLIC
When we kicked off the voting process I sent an email to all the
candidates, asking them to follow a few ground rules:
+ Candidates can follow along with any discussions about the proposals,
but shouldn't take part.
+ Candidates should not change their web sites in an effort to address
criticisms that could come up during the discussion.
+ Candidates _can_ vote, in the same way that, say, John Kerry will be
allowed to vote for himself in the U.S. Presidential election this
November.

At least a few artists, it seemed, had not been aware of the public
nature of this competition, and asked me to remove their works from
consideration. Also, one artist expressed a concern about ideas in his
proposal being plagiarized, but wasn't concerned enough to withdraw his
proposal.

The full list of proposals can be seen at
http://rhizome.org/commissions/all_proposals.rhiz .

APPROVAL STAGE
During the approval stage, Rhizome members were asked to view
proposals, five at a time, and give a simple Yes/No vote. Members could
vote for as few or as many as they liked. At the end the proposals were
ranked by the percentage of Yes votes they received.

In order to make sure that each proposal got enough Yes or No votes,
the system rotated what proposals to vote on in such a way that the
proposal with the least Yes or No votes was viewed next.

I was uncertain about this part of the process, since it's a new
strategy for a new voting problem: My quick research on voting methods
didn't turn up anything that dealt with this situation well. When this
process was done, I was personally satisfied that the finalists
represented the opinion of the voting members, but that's a pretty
subjective assessment to make.

51 members voted in this phase, registering a total of 1282 individual
votes. Each proposal received 26 or 27 Yes or No votes.

The list of finalists can be seen at
http://rhizome.org/commissions/all_finalists.rhiz .

RANKING STAGE
During the ranking stage, Rhizome members were asked to rank the
finalists from most favorite to least favorite. Tallying the votes
would involve repeatedly eliminating the candidate with the fewest
first-place votes, until one remaining candidate receives the majority
of the first-place votes. This is called Instant Runoff Voting, and
it's a well-established voting system used in many places, including in
national elections in some countries in western Europe.

The online ballot for this phase involved an area above for
already-ranked proposals, and an area below for proposals that have yet
to be ranked. (This interface, FYI, is shamelessly stolen from
Netflix.) After the voting was finished, somebody pointed out to me
that the order that candidates are listed in may have an affect on how
they are ranked. This ballot ordered the already-ranked proposals by
rank, but didn't explicitly set the order for the proposals to be
ranked–I think they were displayed by the order they were originally
submitted.

19 members voted in this phase, registering a total of 286 votes. Each
proposal received from 9 to 15 votes. Carlo Zanni's piece, which was
awarded the winner, received 3 first-place votes the first round,
winning by process of elimination. By way of comparison, 1 other
proposal also received 3 first-place votes (but didn't do as well in
the elimination process), and 4 other proposals received 2 first-place
votes.

One concern of this process is that there are too few voters for too
many candidates. Other than getting more people to vote, I'm not
certain of the best solution for this. Personally the votes seemed to
work out okay; generally speaking the proposals that I would've
expected to do well ranked pretty well. (Again, this is pretty
subjective.)

TURNOUT
Turnout was really low. Out of roughly 3000 eligible voting members, 51
(1.7%) members voted in the approval stage, and 19 (0.6%) voted in the
ranking stage. I'm not certain why it was so low or how such turnout
can be improved next year. Maybe the interface was too slow or
confusing? Maybe people didn't know about it? Maybe some Rhizomers just
don't care?

SECURITY & VALIDITY
Mindful of the fact that Rhizome is based in a country that doesn't
have a very good track record when it comes to holding elections, I
took a number of steps to ensure that the results could be validated if
necessary.

+ Every time somebody submits a vote, it's logged in a separate log
file so the votes can theoretically be reconstructed from those logs by
hand or by an automated script.
+ At the end of each phase, I hand-checked the results to make sure
they were in line with the results that the software gave me.
+ I also looked over votes to ensure that people weren't voting with
duplicate accounts, trying to notice any cases where two voters'
opinions looked suspiciously similar. I didn't find any such cases.

IMPROVEMENTS FOR NEXT YEAR
I can easily think of a few things that will improve the process next
time:

+ Improve the proposal submission tool. This year it was a sort of a
one-time submission process, which means if you changed your mind later
you had to resubmit and then email me telling me to fix it. Hassle for
me, hassle for the artist. Next year it would be good to have a tool
where you create a proposal and then can edit it constantly until the
voting begins.
+ Make it very explicitly to the artist that their proposal will be
viewed by the Rhizome community. I can even imagine this taking the
form of an opt-in form. Artists should know what they're getting into
from the start.
+ For the ranking process, unranked proposals should probably be
presented in some randomized order to prevent any proposal from gaining
a subtle advantage.

What do people think? Was the process clear? Fair? Easy-to-use?

Francis