Hello,
So this has been a rocky but ultimately positive month for
Rhizome. :-/ Two major outtages disrupted service and delayed our
programs. However, I used these problems as an opportunity to make
Rhizome stronger. Additionally, some very positive things have
happened, including the launch of a new Advanced Search tool.
Problems detailed below…
=== Disk Error
Early in the morning on March 28th, the Rhizome server crashed with a
severe disk error. While the database and mailing lists were backed
up, we still had to rebuild Rhizome.org on a fresh server. This
contributed to most of our delays, as we rebuilt each service
individually. Among the services that were disrupted were:
+ Rhizome internal email
+ Rhizome.org website
+ Search and Advanced Search
+ RAW and RARE email lists
+ All syndications
+ The payment process, including online Credit Card submissions
and the tools for processing mailed checks
+ The commission submission forms
=== Mailia
On the weekend of April 7th, RAW began receiving a massive flow of
email from the "Mailia" project. These emails overloaded the server
and caused several outages. Unfortunately, this coincided with the
time that users were finalizing their commissions proposals. I
unsubscribed the Mailia project from RAW and blocked email from
Mailia to Rhizome.
I have since been in touch with the artist responsible for Mailia.
He told me that Mailia itself was the victim of a mail bomb, and it
bounced messages with a forged "[email protected]" address back to
Rhizome.org. He has forwarded me examples of the emails that Mailia
received and his story appears credible. Furthermore, the artist has
agreed to block any email from Mailia to the Rhizome lists
([email protected], [email protected]).
I would like to restore the artist's access, pending these actions:
+ additional investigation of the emails that Mailia received
+ configuration of RAW to prevent a repeat of the mail bomb /
outage situation
It is not in the interest of Rhizome to block or ban users,
especially from RAW. I will have more updates on this situation later.
=== Artbase
There have continued to be problems with artbase submissions. This
service was disrupted when the new site went up in December, and was
hit again when the server went down in late March. The tools have
been fixed, but we still have to catch up with the many submissions
that have been made.
Additionally, I need to review cloned artwork before posting them
online. Due to the issues over the last month I have not had the
time to post uploaded cloned artworks. I shall be focusing my
efforts in this direction.
===== Positive Notes
While we have encountered difficulty this month, we have still
accomplished several important projects:
+ structured Rhizome.org server for planned load balancing
+ fixed several key performance bottlenecks on Rhizome.org website
+ our site traffic is up to a new record, over 5.4 million page views
+ our commissions program received a record 180 exciting submissions
+ launched new Search and Advanced Search tools
=== Commissions
As noted, we received a record number of submissions. The voting
phase has begun:
http://rhizome.org/commissions/voting/
Another email with more details will follow shortly.
=== Search and Advanced Search
We have released a new search and advanced search:
http://rhizome.org/search/
http://rhizome.org/search/advanced.rbw
These tools have an entirely new search back end. Much of this
development had been completed before the server crash in late
March. We went ahead and pushed this out instead of installing the
old search. These tools are still somewhat beta, so please let us
know how they work for you.
===== Summary
With the recent restoration of RARE and launch of new Advanced
Search, we have completed the restoration of Rhizome services. Over
the last month, a number of individuals have contacted us about
various issues with Rhizome. Now that the server is stabilizing, our
focus is turning to these issues and we hope to have them resolved in
the coming week(s). I'd like to take a moment to thank all our
members, who have been so patient during this difficult time.
Cheers,
Patrick
–
Patrick May
Director of Technology
Rhizome.org
phone: (212) 219-1288 x202
AIM: cyclochew
+ + +
Director of Technology's report, April 2006
-
Type: discussion