Tuesday, June 18, 2013

Between a rock and a hosting company

I have been in this IT racket for a number of years now and the story that unfolded last week for one of my clients is the exact reason why I started my own IT company....

I have decided to not shame the company that created this mess, but the names of the staff are all real, so lets call the hosting company 'happy hosting' (just hint of irony there) or HH for short.
Only because I have clients still on them and don't need a repeat disaster.....

So My client calls to say they have noticed a load of traffic across the site in their HH stats package, this traffic is not being picked up on Google analytics.

After some hunting and scratching of heads we find all of the traffic was on 2 pages;
  • /rayban.asp
  • mulberry.asp
Both had  spam HTML content, images all relating to their own brand and links to where you can buy said products.
Both pages were being indexed by google from pages like this one http://www.bizzybassrecordings.co.uk/index.php/component/kunena/2-general-forums/5076-vtpvvlmhttpwwwidropoitray-banasp?Itemid=0

Now being a up market publishing company these sorts of products wouldn't normally get mentioned anywhere on their site...

So the firefighting started, anyone who found a bad page was told to delete it via the CMS, because the CMS had root access.
No one at the client was creating these pages, so as the alarm bells started their ominous ringing in my head, we raised a support ticket at mysupport.HH.com ...

24 hours later with no response to HH (my client spends around £1000 a year with HH) I did some invetigation online and found that it appeared we had been hacked by something similar to china chopper 
Notes added to the ticket...

Finally we get a response, "it's not us it's your site", "or it could be someone else on your server as you are on a shared server", " we patch  our servers etc etc"

The server we were being hosted on was a Server 2003 box, running .Net 2.0 (thanks to http://browserspy.dk/webserver.php for some help with this)
I asked them to check the security settings on the site as if it was china chopper then it would need to get in somehow,

- "it's not us it's you"
"right thanks, can you send over any IIS logs for us to work out where the access is coming from"
- "we don't do logs for shared hosting"
"thanks again Dave, can you check IIS for me to see if there are any noticeable alteration to security settings on our site then?"

This is when it all went wrong, 3 hours later I get a call from the client, "the site is yellow and white and says "error" - bugger

So it is 530 on a Thursday evening and the support lines are open 24x7.... ring ring ring.... ring ring ring....
New support ticket added as this is not the same as the first problem, obviously something has happened at a server level as its completely dead returning a 500 error mentioning security privileges.
Wake up the the same error and no support updates, you have to love the HH support team dedication to the meaning of "24x7"

We start on the phone, support are in but the guy we need 'Jim' or 'James' to his friends is not in until 930, he will call you.
11'o'clock and all is quiet, site has blipped a few different errors so we assume Jim is rebooting the server (if all else fails!).
We call Jim, he is unavailable but the best guy to fix the problem , he will call you back
1pm, the site error now says .Net 4.0, so they have failed to fix the problem and are hoping that patching the server and upgrading the .Net framework will fix the problem.

Another call, this time I offer some advice "Can we move the site to a new server while we wait for you to fix the current one?"
- "no, well maybe, or what if we do restore a backup of the site from last week?"
"Dave that sounds like a good idea, you should get a medal, no really, mate you are a genius, wonder why no one thought of that earlier, I am going to recomend you for a promotion" (I did not say any of that to Dave, if I had been close enough I would have driven over there and...

So they restore the site at 430pm on a friday night, my client has been without their site (or what I like to call their only form of income) for going on 40 hours, the nooses are removed from the rafters and everyone goes home, leaving me to pick up the pieces because the CMS they have is not compatible with .net 4.0 and although HH have restored the site they have not restored IIS to its former glory.
There are still weird errors all over the place, the home page doesn't work, the stats package doesn't work, the backend bespoke admin system doesn't work.

A week later and it is all held together with sticky tape and number 8 wire, no one wants to tell HH about any of their services that don't work in case they try and fix them.

The support team were crap (or are under resourced and cant be bothered), HH are crap, they were one of the best in the UK until they were bought out by iomart and are now part of easyspace

 This is what I think happened when Dave went off to look at the IIS settings;
  • Dave restarted IIS - fail
  • Dave tried a reboot -fail
  • Dave tried to patch the server - fail
  • Dave upgraded the 2003/IIS6 server . net 4.0 - fail
  • Dave then did more updates and left to go home at 5 - fail
  • Something broke on the upgrade, or a window on the console 'click to continue' was left?
  • Dave came in the next day and made sure no-one else noticed the ticket and hoped it would all go away - fail
  • The shit hit the fan
  • My client lost thousands - fail
  • I have to fix it - fail
  • HH are still in business - fail
  • I am moving all of my client sites hosted on HH to a new host, proabbaly only cost HH a few grand, but hey a - success

There is a lot to be said for good customer service, if my client who have zero technical staff had had to deal with HH directly the problem would not have been solved (or may be it would have).
My client lost valuable sales and business because someone decided to fiddle with IIS6 (not for the faint hearted) without telling us that it might break and not hanging around to check it.

Most importantly there has never been any mention of compensation (I know 2 days is under the 99.9 SLA but come on) or anything to say how it happened or that they will make sure it soesn't again.

All in all I would never use HH again, make sure when you select a web host you do some research on them, the most expensive are not always the best.

And we never did get to the bottom of china chopper

No comments: