Paid IT support and revenge
I've recently had some very bad experiences with paid technical support and I'm wondering if anyone else shares my dismay.
I'm responsible for about 40 servers which support services for approximately 15,000 people so you might imagine that good 24x7 technical support is fairly important to me. The last several months of dealings with two companies in particular have my blood pressure way higher than it need be.
On several occasions I have had to repeatedly call or email companies multiple times just to get a call back. Many times the stipulated minimum response time was long past by the time I was able to get help. On several other occasions, one company in particular told me that the support agreement ID I had provided did not exist and that we would have to pay for support when the contract had months left on it. On at least two occasions the solution has taken from 6 months to a year to arrive at. On other occasions, the tech I was dealing with left for day without passing the trouble ticket on to another tech, so I'm up at 1:00am explaining the entire problem yet again to a new person.
Today I was passed like a hot potato through three people till I found a fourth person that actually knew what I was talking about. I had to have the tech repeat about 40% of everything he said and I had to phonetically spell part of my side of the conversation. Another problem I was dealing with today, took most of the day but when I finally hooked up with a back-line person in Atlanta it took about 10 minutes to prove my theory and the hardware is on the way.
These are no longer anomalies, this is the norm for me over the last year and it's getting progressively worse. Admittedly, when I call tech support it's usually a whopper, but it should not take days to get to the back line people for what we're paying for support.
I've been fighting back though. About 78% of the servers in my charge have gone from proprietary hardware and software to commodity rack-mount Linux boxes. This will be 87% in the next couple weeks and in a few months, I will have only Linux servers. We are saving thousands on hardware, software, and maintenance costs, and problems requiring support have decreased dramatically. We're also using virtualization to further reduce server count and utilization.
One of the field engineers for the worst offender put it best, "I don't know how much longer we can afford to annoy our customers and remain in business".
That's him!!! That's the guy!!!
That's the guy I've been dealing with all week :-)
From the other side
In my work I sometimes get involved in solving severely escalated support incidents. I also work on the development of high volume software products. Some thoughts on this :
IMHO the product has to be built to 'just work'. If it isn't, then
all is lost. Attempting to fix the problems in the field with
support will always fail, or will certainly cost orders of magnitude
more than the cost to engineer the thing correctly in the first place.
So the rule is : use and buy products that are already in use by
large numbers of other folk. By definition they aught to work,
since if they didn't, long ago most folks would have quit using them.
btw, the vendor not returning your call is just plain lame.
Vendors have whole departments of people who have the sole task
of calling customers back. They won't do anything to solve your
problem, but they sure as heck will call you back pronto !
In the end, the fact that you are paying for support doesn't
really help : all that does is to pay for the guy to answer
the phone and call you back (or not, as applicable!). The real
cost to debug and fix a tricky problem is always going to be
much more than the yearly support income from any single
customer. For example I spent literally weeks investigating a
problem for a client of ours that affected a large hotel chain
in Europe. Sometimes their e-mail messages were lost. I had to
make countless code changes just to get enough logging captured
to begin to see what was happening. This involved deploying
many special builds on the customer servers, analysis of the logs,
invention of theories as to what was happening that had to be
tested against the production servers, and so on. The entire
exercise was extremely expensive. Field debugging projects like
this are rare, because it usually takes the threat of a lawsuit
or other severe consequences, before a vendor will divert the
necessary resources from other areas.
Re: The other side
True enough and very good points.
The products we buy are totally main stream from a huge world-wide company (think initials, there's not too many to chose from). The only thing the field people do is show up and swap parts - the diagnosis is done before they leave home.
I agree on the support costs. I used to be part owner of a board level repair shop back in the 80's and our contract costs never covered the cost of single repairs. We just tried to sell enough contracts to cover the cost of the equipment that actually did break plus some profit. You don't play nice on the contracts and they don't renew, tell their friends, and soon you're talking real money. Hence my starting to blog about this. If nothing else it's cathartic and keeps my blood pressure down.
The support problems we have are constant and getting worse by the day while they try to get us to buy "uplifted" support. Six months to diagnose they sold us 12 HBAs from a bad manufacturing batch. A year to get us a customer specific patch, and only after the threats you mentioned, to fix a bad patch they gave us that could not be backed out. Now we can't patch those systems without vetting each patch with engineering and that took almost two months last time. They even told us they had a cluster file system but when it came time to implement it they said oops, sorry, it's still in beta and we can't give it to you. That cost them almost $0.5M so you see how bloody proud they must be of their stuff. There are dozens of smaller instances that add to the pile.
I'm a pretty understanding guy having been in their shoes once, but that last problem put me over the edge. That's why 78% of my systems are now on Linux and commodity hardware, possibly 100% by the end of the year. That vendor is losing hundreds of thousands of dollars in business from us. Elvis has left the building, the ship has sailed, and I won't be lookin' back. :-)
I feel your pain ... I have
I feel your pain ... I have been dealing with Dell for a school lab and it's just about enough to drive me to drink!
And I only wish that I could move to linux based but the boss is a windows only man
Re: How about those surveys?
The surveys are usually a phone interview, favor the company, and go on the field techs record if one has to come and replace a part. In short I have to be careful how I respond. The company's closest tech is a 4-hour round trip drive from here and they have a policy of having the field guys do the board swap or it jeopardizes the service contract. I've gotten good at convincing them that paying a field tech four hours for a 10-minute board swap is asinine so they usually waive the policy and drop ship me the part. That way when it takes me two bloody days to convince them I have an intermittent HBA failure I can say what I really think on the survey without the field guy taking the rap :-)
The field guys aren't my problem really and are hard to come by in Montana so I want to protect them as much as I can. There's occasions where a fix is particularly sticky or involves a $400K piece of equipment and I want a company rep doing the actual repair so if it goes south, the company is responsible for fixing it. My first responsibility is to my employer after all.
How about those surveys?
I've only had to deal directly with one company but I've heard the stories and I feel your pain. I've been lucky as I haven't had any critical problems and the service I've gotten has been good. They always email me a link to a survey after every encounter... so I'm guessing you fill out the surveys too? I'm also sure you gave them an honest assessment. Did you ever hear anything back as a result of filling out a survey? I'm just curious.
re: Paid IT support and revenge
Thomas