Friday, June 21, 2013

Kids vs Clients - who wins you decide

I spent the day with my 2 kids yesterday (4yrs and 2yrs) and had an awesome if not tiring day with them.
What I realised after looking back on the day was that I think anyone ho wants to run their own business should start by managing young kids for a day, they make the best on the job training money can buy.

My oldest is quite happy to sit and play on her own with her princess castle and entertain herself in short 15-30 minute bursts before deciding it is time to do something different.
She will then come and find me asking for a different toy or if she can watch TV etc, once setup with this new mini project we get 30 minutes of clear time before the next project needs to be found and started. Projects rarely get finished, movies are always half watched etc.
If she does something embarrassing and it is noticed (a trip up over the corner of a rug for example), all hell breaks loose and all projects are put on hold, or dropped from the master

My son is the complete opposite, he likes to follow you around checking up on you, closing cupboard doors you have left open to make putting the dishes away possible, luckily he announces the closure by pointing, yelling 'door' and slamming it closed while you aren't looking.
He like to change the rules constantly, after picking his sandwich ingredients and watching it be made, he decides one or two or all of the ingredients are not up to spec and removes them to be eaten or dropped on the floor.

And I guess anyone who freelances will see some similarities in this behavior in some of their clients, well I can in some of mine and this is not a bad thing.
As someone who tries to build software/web to model business processes for a living, I need to understand how the people who are going to use the software think and deal with all aspects of what is built.
Yes I know it can be a pain to have a client looking over your should while you work and there can be a lot of 'hold your tongue' moments, but in the long run I think they have to be (hopefully) happier with the end result as they have been there every step of the way.
Obviously this is not always the case....

Customers who give you a brief spec and say good luck, then disappear until they come up with the next idea are great for developers as we can get on with the project and just build it. At lest until the customer comes back with the next best idea they have ever had and ask for it to be added...

Both have good and bad points, there is no such thing as a perfect customer and at least most of the time time they door drop to their stomachs and starting kicking and thumping the floor while wailing.... well .....

Wednesday, June 19, 2013

Happy Hosting gone wild

Rule 1 of the IT world, never say in public 'it is all working'....

The site mentioned in the previous post is offline now and guess what I am on the phone to the Happy Host team...

I am guessing thier system sattus notice does not cover this server:

Last modified: June 18, 2013 @ 9:08 am
All systems and services are operating as normal.

Oddly enough the site was online at 908am... Dave, Dave, Dave put down the screwdriver...
I wonder if they have caller ID, I would have thought that being no.2 in the queue would have been a good thing...

Support member on phone, he has gone off to check...

He is back (5 minutes later), "oh yeah that box is definitely off, we will turn it back on and see whay it happened, probably be back up in an hour or so"


Tuesday, June 18, 2013

Between a rock and a hosting company

I have been in this IT racket for a number of years now and the story that unfolded last week for one of my clients is the exact reason why I started my own IT company....

I have decided to not shame the company that created this mess, but the names of the staff are all real, so lets call the hosting company 'happy hosting' (just hint of irony there) or HH for short.
Only because I have clients still on them and don't need a repeat disaster.....

So My client calls to say they have noticed a load of traffic across the site in their HH stats package, this traffic is not being picked up on Google analytics.

After some hunting and scratching of heads we find all of the traffic was on 2 pages;
  • /rayban.asp
  • mulberry.asp
Both had  spam HTML content, images all relating to their own brand and links to where you can buy said products.
Both pages were being indexed by google from pages like this one

Now being a up market publishing company these sorts of products wouldn't normally get mentioned anywhere on their site...

So the firefighting started, anyone who found a bad page was told to delete it via the CMS, because the CMS had root access.
No one at the client was creating these pages, so as the alarm bells started their ominous ringing in my head, we raised a support ticket at ...

24 hours later with no response to HH (my client spends around £1000 a year with HH) I did some invetigation online and found that it appeared we had been hacked by something similar to china chopper 
Notes added to the ticket...

Finally we get a response, "it's not us it's your site", "or it could be someone else on your server as you are on a shared server", " we patch  our servers etc etc"

The server we were being hosted on was a Server 2003 box, running .Net 2.0 (thanks to for some help with this)
I asked them to check the security settings on the site as if it was china chopper then it would need to get in somehow,

- "it's not us it's you"
"right thanks, can you send over any IIS logs for us to work out where the access is coming from"
- "we don't do logs for shared hosting"
"thanks again Dave, can you check IIS for me to see if there are any noticeable alteration to security settings on our site then?"

This is when it all went wrong, 3 hours later I get a call from the client, "the site is yellow and white and says "error" - bugger

So it is 530 on a Thursday evening and the support lines are open 24x7.... ring ring ring.... ring ring ring....
New support ticket added as this is not the same as the first problem, obviously something has happened at a server level as its completely dead returning a 500 error mentioning security privileges.
Wake up the the same error and no support updates, you have to love the HH support team dedication to the meaning of "24x7"

We start on the phone, support are in but the guy we need 'Jim' or 'James' to his friends is not in until 930, he will call you.
11'o'clock and all is quiet, site has blipped a few different errors so we assume Jim is rebooting the server (if all else fails!).
We call Jim, he is unavailable but the best guy to fix the problem , he will call you back
1pm, the site error now says .Net 4.0, so they have failed to fix the problem and are hoping that patching the server and upgrading the .Net framework will fix the problem.

Another call, this time I offer some advice "Can we move the site to a new server while we wait for you to fix the current one?"
- "no, well maybe, or what if we do restore a backup of the site from last week?"
"Dave that sounds like a good idea, you should get a medal, no really, mate you are a genius, wonder why no one thought of that earlier, I am going to recomend you for a promotion" (I did not say any of that to Dave, if I had been close enough I would have driven over there and...

So they restore the site at 430pm on a friday night, my client has been without their site (or what I like to call their only form of income) for going on 40 hours, the nooses are removed from the rafters and everyone goes home, leaving me to pick up the pieces because the CMS they have is not compatible with .net 4.0 and although HH have restored the site they have not restored IIS to its former glory.
There are still weird errors all over the place, the home page doesn't work, the stats package doesn't work, the backend bespoke admin system doesn't work.

A week later and it is all held together with sticky tape and number 8 wire, no one wants to tell HH about any of their services that don't work in case they try and fix them.

The support team were crap (or are under resourced and cant be bothered), HH are crap, they were one of the best in the UK until they were bought out by iomart and are now part of easyspace

 This is what I think happened when Dave went off to look at the IIS settings;
  • Dave restarted IIS - fail
  • Dave tried a reboot -fail
  • Dave tried to patch the server - fail
  • Dave upgraded the 2003/IIS6 server . net 4.0 - fail
  • Dave then did more updates and left to go home at 5 - fail
  • Something broke on the upgrade, or a window on the console 'click to continue' was left?
  • Dave came in the next day and made sure no-one else noticed the ticket and hoped it would all go away - fail
  • The shit hit the fan
  • My client lost thousands - fail
  • I have to fix it - fail
  • HH are still in business - fail
  • I am moving all of my client sites hosted on HH to a new host, proabbaly only cost HH a few grand, but hey a - success

There is a lot to be said for good customer service, if my client who have zero technical staff had had to deal with HH directly the problem would not have been solved (or may be it would have).
My client lost valuable sales and business because someone decided to fiddle with IIS6 (not for the faint hearted) without telling us that it might break and not hanging around to check it.

Most importantly there has never been any mention of compensation (I know 2 days is under the 99.9 SLA but come on) or anything to say how it happened or that they will make sure it soesn't again.

All in all I would never use HH again, make sure when you select a web host you do some research on them, the most expensive are not always the best.

And we never did get to the bottom of china chopper