Tantalus rebuilt

[rasiel]

It's still early in the restoration process but I wanted to get a
status update on what's happened so far.

Sometime late on June 27th Tantalus went offline suddenly. The firs
thought was that this was due to a programming error in one of the
scripts that overwrote the main index. However, when we managed to log
in to the server all the site files were gone. As far as we can tell
this is the analysis:

-After having a number of minor outages we asked to switch servers to
improve performance.
-The migration however was not fully completed. The software that
manages the site was updated to reflect the new location but the data
was still on the old hardware
-Since there was no data yet at this location each night the software
was creating an empty backup
-Meanwhile, as far as we can tell, the staff brought the old server
offline and reformatted the hard drive; not realizing, of course, that
there was a live site on it which was not being backed up.

Adding to this list of errors is my own fault for relying solely on
the hoster's backups and not creating a set of my own. The reason is
that it had grown to over 60 gigabytes in size and a file of this size
takes days to download.

Fortunately, most of the database was recoverable. The database
contains all the data about the users and the records and keeps it
separate from the images. At this point it looks like we're missing
all of June and the last week of May. The images at this point seem
only partly recoverable. An old copy had stored up to December 2009
and a few more may have been saved from testing. It's too early to
tell how we can patch everything together but we should have a better
idea by the end of the weekend.

Now this situation has caused me a lot of distress. I know many users
and guests have come to rely on Tantalus over the years and data loss
is a huge blow for everyone involved. No matter what the succession of
events as site owner the responsibility nevertheless falls on me. Once
the dust settles I will see how I can compensate everyone for their
inconvenience. In the meantime please accept my sincere apology.

Ras

[rasiel]

At this point most of the field data has been recovered (we're missing 30 days'
worth from May 27-June 27) but most images have been wiped out.

If you used the import/export tool to make a backup of your site please send me
the file so I can restore your account. You can upload it somewhere and email me
the link or use the Tantalus ftp account to send it straight there:

ftp://tantaluscoins.com
Username: restore
Password: restoreme

File repair at the server level is still going on but it may not get much better
than how it is now (except for fixing programming bugs like the ebay sniper,
import/export, etc.)

Again, I want to repeat how deeply deeply sorry I am for the data loss and the hassle. If there is anything I can personally do to make this up to you just email me please. I know it's cold
comfort but I'm putting into effect an independent backup system to ensure
something like this can never happen again. Also, if you have a paying account I
have already made it so that it never expires or charges your credit card/paypal
account again.

For those who already have free accounts I will make the site's premium features
free for a year on request and as soon as the site is stable I will make it so
for everyone signing on new as well.

Ras

[rasiel]

Data now recovered up to June 19. Only a fraction of images have been salvageable unfortunately.

If you have any backups stored on your hard drive please let me know as I should be able to recover those images.

Ras

[kerux]

Hang in there, Ras. We appreciate all your efforts!

[rasiel]

Today exergue symbols were restored.

In addition to the nightly backups (now verified that they're working properly) there will be a monthly offsite backup as well.

Still plenty of work to do...

Ras