What happened?

At 5am on Friday 2nd March the datacentre where the FishFood server is housed suffered a catastrophic failure from one of its network suppliers. From that point until 6am today (Monday 5th March) the FishFood server has been offline. 

Background

I perhaps need to explain some background about the physical set up of the FishFood hosting platform. Contrary to popular belief I do not run a web server from my back bedroom or from under my desk. No, the FishFood server is a top end purpose built web server housed in a multi-million dollar data centre in the United States. Yes, that's right, your website is served from America! 

I lease the hardware, engineer support and network connectivity from a server management company. They are responsible for ensuring the server works. However, they themselves lease all the network service to the outside world from a telecommunications company who provide huge data connections into the datacentre. So I'm responsible for what goes on the server, my server management company are responsible for ensuring the server works, and the network provider is responsible for ensuring the working server can connect to the internet. The breakdown occured with the network service provider.

Some details of what went on

On Friday morning it seems that the network provider from whom my server company lease internet connectivity, suffered some catastrophic breakdown. This didn't just affect the FishFood server but hundreds of other servers in the datacentre who use the same network provider. This is out of the hands of my engineers and server company. They are not responsible for the huge datapipe connecting their servers to the outside world and pay good money to that network provider to ensure a continuous service provision. When it broke down they had no option but to wait for the network provider to fix their problem. At this point the FishFood server, like all the others, was entirely entact and running perfectly well, it was just cut off from the outside world. After waiting for most of Friday my server company took the unprecedented decision to physically move all their servers to a new datacentre. I believe, but am not certain, that they simply took the hard drives out of my server and carried them 5 miles down the road to a waiting bank of new servers in the new datacentre. 

When they plugged the hard drives in the server refused to recognise them. This represented a complete change of scenario. Whereas the network was now working fine in the new datacentre, the server wasn't. The issue was that the hard drives are configured in a RAID configuration. Put simply this allows for much higher data trasnsfer and a fail safe mechanism, whereby if one hard drive fails there is another identical hard drive with all the data on working alongside it. The dead drive can be remove and a new one put back in to the RAID array without interrupting the service. The trouble with RAID arrays is that once set up in a server they need the same server to continue to work, in some cases. So when the drives were put in to the new server there was enough of a hardware discrepancy to cause a fault. What I am not clear on is why this affected the FishFood server and not others, since the FishFood server was no different to the other hundreds of servers my server company lease out. 

After trying several different servers with no success my management company decided the only solution was to recreate an identical, chip for chip replica of the previous server the drives had come out of. The issue now was finding exactly the right parts. This proved problematic over the weekend and so the delay continued. 

Happily, this morning they finally managed to get the hard drives to work in the new server they had built and all is well. They have apologised profusely for the delay. I know too that the man in charge has not left the server bank since the trouble started and is currently asleep on the floor of the datacentre having virtually not slept for three days. 

What's next for FishFood Media?

In short, I'm shutting up shop! I've got to. There is no way I can afford to provide a better system than is already in place and I believe each of you deserve better than the service I am able to provide. I am also unable to personally provide this amount of time when issues such as this arise. I have lost three days from my full time church ministry as I have sought to deal with this emergency.

As you all know FishFood Media is meant to be a ministry and service to the church, not a business. It was always intended to be that and mostly has been. Some of you have been with me for years, for which I am very grateful, and it breaks my heart to have made the decision that I have been forced to make. You all also know that alongside running FishFood Media I am also in full time ministry. In January this year I was appointed to a new work planting churches in the West Midlands. This itself is a massive task and a very important one within the Church of England. It requires all my energy each day and many of you are painfully aware that my own capacity for support and resolving issues has steadily declined over the last couple of months certainly, and really for many months before that as my church work has become more time consuming. 

I can only apologise to each of you for the poor service you have received when you've needed it. To honour each of you, and the vision of FishFood media of providing affordable high quality hosting to the church, I need to invite each of you to find a new home for your websites. 

Money

Now this is the big issue. A few of you will have paid for a year's worth of hosting which you will not yet have received. I will happily issue each of you in such a position with a full refund for each month of hosting you have paid for but haven't used yet. So if you've paid for 12 months hosting and have only recieved 6 months, I will issue a cheque for half the hosting cost. Alternatively, I have set up a new hosting service with capacity for some of you. I will happily offer those of you who have already paid for this year's hosting service to come with me to that new hosting platform. It is actually a better hosting platform, but shared with others and without me in charge of the server. It will be more robust and has more resources than my server could provide. It will serve you well while you finish your year with FishFood Media if you choose to come with me to that new home. After your year is up though I will ask that you find a new hosting service. This new hosting service is meant just for my own websites, but I can host an unlimited number of websites on it. The issue is more a case of my personal capacity to deal with support issues and admin. But if you need a new home, I will give you one temporarily. Please talk to me about this.

Most of you however will realise that you have not paid any renewal costs for year on year hosting fees or domain renewals. This is not your fault. This again is my fault for being great at preaching a gospel message, leading people to Jesus, and all manner of other ministry type things, but being wholly useless at administration! Hilary and I have always been fully committed to providing a home for church or christian websites so we have always paid personally for the server costs. While I was doing web design too the money I got in for that work always covered the costs of the server, so it was never a pressing issue and I was never overly concerned with making a profit, just ensuring a service to the church. This has meant though that most of you have got away with effectively having free hosting for at least the last year, if not much longer. Indeed, we've even paid for many of you to have your domain names renewed and then never sent on an invoice to recoup the money. 

The cost to us personally has been many thousands of pounds if I'm honest. In all honesty I would love to sit down and go back through the paying in book stubs and recoup the money owed since your intitial payments. But I don't have time for it and the book keeping is hopelessly out of date. The task is enormous. I would like to simply draw a line under it all and work with each of you in finding a new home.

If any of you feel honour bound to make a donation to Hilary and myself then that would be great. But at this time we will not be calling in any debts.  We hope though that this also serves to soften the blow of the end to FishFood Media's hosting service.

An offer

If any of you feel called to carry on the FishFood hosting service then that would be an option I would consider. You will need to be technically competent though and prove yourself able to manage a server and the client, better than I've been able to. The server has everything set up and a management company looking after it. Someone could still make me an offer to take over the server and all the clients. That would perhaps be the best option for everyone. If administered properly it would be a very viable financial concern and profitable business, with about 150 clients at the present time.

Timescale

I will not turn off the hosting service until the last client has left the building as it were. Last one out switch off the lights please!

I hope that over the next month most of you will have found alternative accommodation and between me and you working together will have moved all your websites and email setups to your new host.  

I aim to turn off the FishFood web server in 2 months time. If we can do it within 1 month that would be great.

There are plenty of other web hosts out there, Christian and non-Christian. I would recommend www.godsweb.co.uk as a Christian alternative to FishFood Media, or for secular alternatives please have a look at www.123-reg.co.uk or www.1and1.co.uk.

Domain Names

One issue is the transfer of domain names to each of you to manage yourselves. This will be a difficult and costly exercise. Again though, for most of you there is actually a cost for annual renewal of your domain names which Hilary and I have personally paid for on your behalf and then stupidly not invoiced you for. So most of you owe us money already for your domain renewals. What I invite each of you to do is to go to www.123-reg.co.uk and create your own free account. When you have done that I can transfer your domain name over to you for your own management. I will provide full support while you do this. 123-reg actually provides domain names cheaper than I do. But there will be a transfer fee in most cases as most of my domains are managed on an american site called ENOM and there is a cost to transfer from Enom into a different registrar.  I would ask that you pay for this fee in lieu of what you ought to have paid but haven't been invoiced for. If any of you have paid for domain fees and hosting this year then I will continue your domain management for this year, and seek to transfer the domain when renewal comes around. Domain management is very simple, and 123-reg automate the whole process for you. 

You may find that your new host also provides this service and you ought to check with them about this issue. 

Although there is a fee for domain transfers, that fee normally includes a year's worth of domain registration, so in effect it is free, sort of.

The king is dead, long live the king!

Although I have taken this difficult decision to close FishFood Media's hosting service, I will not be closing FishFood Media completely. The vision for FishFood Media as a website was not to be first and foremost a web hosting service. In fact, right back at the beginning I sort of decided I wouldn't do web hosting because I knew how time consuming it would be. But so many people kept asking me for it and for web design that I carried on doing it as I needed the money! So it grew and grew and the original vision of FishFood never took off because I never had time to focus my efforts on it. IN my new role though there is a really strong tie in with that original vision. In my new work I will be doing a lot of media production, and not just on my own, but with a much larger team of people. So there should be continuous output of new media resources for the church to use, and FishFood will be the place to come and download them! That's what FishFood was meant to be in the first place. So the website and the community (which has also never really taken off very well) will all continue. Hooray! 

Thank you!

Last but by no means least, I wish to say again to each of you - THANK YOU! It has been a wonderful pleasure and  privilege to work with you all and provide in the amazing way that I've been honoured to. Through FishFood Media  (and before that Jireh Media, and before that theVestry.net) I've set up more church or Christian ministry websites than I can remember, and I've hosted over 150 different sites. There have been good times and bad in it all, but on the whole it has been great and a real privilege to serve the church in this way. Every blessing to each of you, and may your web presence been even greater blessed as you move on to pastures new.

God bless! 

Evan