• Beach Tweets

  • Archives

  • RSS PEER 1 Blog

    • An error has occurred; the feed is probably down. Try again later.

ServerBeach Fusion Post Mortem

In early December 2009, all of us at ServerBeach embarked on an ambitious project to upgrade our entire back-end systems. This was a massive project with a lot of moving pieces. We did this for several reasons, but most importantly, to better serve you. We’ve created a consolidated platform for all of our PEER 1 Hosting and ServerBeach customers to allow for the ease-of-use and access to the multiple web hosting products we offer across all of our brands. This robust foundation enables us to offer you more and improved automation tools well into the future.

As you know, the transition to the new platform did not go as we planned. We made several mistakes during the process, and we would like to take this opportunity to share our postmortem findings with you. I’ve highlighted them here:

  • We created a large burden on our team with hard and fast deadlines that allowed little room for movement or errors.
  • We missed some milestones along the way and instead of moving deadlines, we cut some corners with training, testing and QA.
  • We were too focused on replacing the system we had as the initial step, instead of focusing on what new features and requirements our customers needed the most. We were too concerned with scope creep.
  • We did not spend enough time gathering customer requirements and feedback. We gathered some, but needed more.
  • We underestimated the impact of some of the gaps between the old DNS system and the new one.
  • We had several failures in our DNS data migration for a subset of our customers.
  • Our tested rollback processes and procedures were invalidated by the failure to identify some of the DNS issues identified gradually after going live.
  • We underestimated the branding piece on the new customer portal, and as a result, created confusion around the relationship between PEER 1 Hosting and ServerBeach – a relationship that has existed for over five years, since the acquisition of ServerBeach in 2004.
  • We based the portal and tool decisions on customers who use both PEER 1 Hosting Managed Hosting services, along with ServerBeach Dedicated Servers. In retrospect, we should have focused more on the services an individual ServerBeach customer would seek, and the portal experience related to them. We are committed to correcting this.
  • Our usability and interface functionality tests were insufficient and we should have gathered requirements and feedback from a much broader spectrum of customers.
  • We conducted limited beta testing – in hindsight, this was not sufficient.

We did a thorough investigation of every one of these and walked away with several lessons learned. Below are the highlights:

  • Build a more flexible system for developing. This includes provisions for setback, which will allow for deadline evaluation and movement of tasks to avoid cutting corners.
  • Gather better and broader customer feedback on system requirements before upgrading in the future.
  • Improve our testing process to include beta and customer participants.

As you can see, we made mistakes. Based on what we’ve learned, we have aggressive initiatives to close the gap, to improve our process and services, and to make ServerBeach the best place to host your sites. Here are some of our initiatives:

  • Create a Customer Council – a round table of our customers to gather feedback on the ServerBeach experience.
  • Utilize forums and customer communication for feedback on what’s working and what’s not, screen change evaluation, and to bounce new ideas off the community.
  • Implement a better beta testing program for future upgrades.
  • Implement a series of strategy sessions to create new ideas on how to better connect with you and how to involve you in our strategy decisions.

We are humbled by this experience and sincerely apologize for the challenges we’ve put you through. It’s been tough on all of us, but more importantly, even tougher on you and we know that. We have a lot of work to do but we’re up for it. That’s a promise. We’re ready to put the “Geek” back in the Beach and we expect you to hold us to it. We will be sending out notifications on how you can participate in our processes to help improve them to meet your needs.

Thank you for your patience, understanding and support. If you have any questions, comments or feedback, please feel free to email me directly at rmiggins@serverbeach.com.

Robert Miggins
SVP, Business Development

15 Responses

  1. Three months after the fact, we get a blog update? I am not one to complain, but your DNS mistake did effect me directly. I would have expected this sort of response within 30 days of your mistake; 90 days … too little, too late.


  2. I agree. 90 days and all we get is a blog post and a 14 minute video?

    I’m not feeling the love here. We were directly affected by the DNS mistake, and the removal of features from the self serve DNS continues to impact our experience. And all we get is “Sorry, we screwed up, this is how we’re going to fix it.” No service credit, no nothing.

  3. Do you really think it boils down to, “We were too strict with our deadlines.?” What happened to “By Geeks for Geeks”? Maybe it should be “By Management for Geeks”. This was just a disaster. Also, you’ve been working on a DNS tool now for well over 2+ years and at the end of the day it was and is basically worthless. Apparently that is a hot topic and one that can easily be addressed.

    I’m not a heavy user of your systems, but I do have a sub 1,000 customer ID and I’ve been around the block with you. All I ask for is:

    A reasonably priced, feature rich box
    A method to manage it and my account
    An ability to automate it
    An ability to monitor it

    Pretty basic, straight forward needs – the first 2 of which you have met quite well. The next 2 have a long way to go.

    I wish you luck and speed on Fusion as it can’t get here fast enough.

  4. Good thing that we don’t host DNS too with ServerBeach, although our invoicing is still a guess as to what will happen.

    This is a far better confession than the engineering blunders that left us without systems. Too bad it is posted on some hidden blog.

  5. This is all well and good, but the lack of communication with customers still continues with everything from billing to customer support. (NOT the tech support people – they have been great) People promise to get back to you and don’t. The portal still has major issues in my opinion. The billing system is still horrid.

    I remain loyal to server beach because the actual servers/bandwidth and set up is for the most part reliable. The tech staff have been responsive. But as to management, customer service and billing departments, I have simply learned not to expect much.

  6. I would have to agree with the customer above! The billing system is a case lesson in what NOT to do. The interface is clunky and fails every user expectancy test I could dream up. I’ve taken to ignoring any emails I get from the billing portal as they are probably either late or incorrect. Isn’t an upgrade usually supposed to make life easier? Seems like this upgrade took pages from the playbook of Windows Vista.

    • First of all i normally do not vent on blogs or in public —


      Well we do not use their DNS and we have many servers with SB / Peer1 — after spending hours and hours with billing and myself wasting hours and hours of time – and being told that this was an isolated incident and then reading and getting this post — make me wonder why lie to me and make me feel like i was the only one with billing errors.

      here is the best one — I PAID AN INVOICE ON DEC 26 – they charged my credit card 3 extra times – and then applied the payment to Jan, Feb and March invoices — I asked how can that be when the invoices were not even generated yet — the response – we can allocate those payments anywhere –

      oh ok –

      so you applied them to invoices that did not exist and some how the answer was yes — after it was all over i am sure i paid for 1 month in Feb for a server that was canceled in December and was taken off line at the end of January.

      I got tired of going in circles —

      Robert – you can call me personally – We have met a few times and I would gladly have a discussion with you on this entire process and cut over. –

      I have a great account Manager at Serverbeach – his hands are now tied I am sure by Peer1 – billing errors continue, the portal still does not notify us properly of anything related to billing and it is the most complex thing to sort and find invoices as they relate to a server –

      THE PORTAL FROM A BILLING, INVOICE, Point of View – Disaster really really is —
      THE END!!!!

  7. The whole migration has been a complete disaster. We have been SB customers since 2007 and right now we cannot move out fast enough. The DNS problem is a joke compared to the billing issues we are experiencing. This is just an example of how NOT to do a migration or a merger, SB was a superb company before this, now, it is just a complete disaster.

  8. to Domenic, No you were not the only billing issue. They billed us for 2 months on 2 servers we cancelled on top of that they didn’t charge our debit card for 2 months and didn’t alert us of any mistake then his us for 3 months at once for 7 servers, over 1000 of which was over billed, Ouch… A notice would have been nice.

    The DNS debacle affected us but not bad as we only have a few site using the dns services. The biggest issue we found was that our servers stopped communicating with each other and they were all in the same data center. When we complained about this we were told it was our firewall… They were fine in November, we made no changes.

    Needless to say If the migration process of over 120 sites most of which are custom RoR with multiple DBs was not so painfully time consuming and scary I would no longer be at the beach.

  9. I miss the old plain-text e-mail invoices that I could simply foward to accounting and forget about. Now I have to dig them out of this awful portal and hope that it can actually be retrieved without having to open a support ticket. It’s such a chore that I don’t bother until accounting comes looking for the invoice.

    Is the DNS tool still not capable of creating new DNS zones as slave zones? I resorted to not using SB’s DNS servers at all and haven’t bothered to check whether or not they fixed that important omission.

    Despite this solution in search of a problem, I can’t find a better unmanaged/self-managed host at a better price. If you’ve got any tips, tweet me.

  10. Thank you for all your comments. We have taken careful note of all of these areas (DNS, billing woes, etc) and will be in touch with our customers soon on next steps and plans to improve our service as promised.

    If any of you have specific questions for me, once again, I can be reached at RMigginsATserverbeach.com.

  11. Thanks Robert and Charnell…

  12. What are you guys talking about? These people actually do very good job, they might just be the best hosting company on the market. No gimmicks, no hidden fees, very straight forward, honest and easy to deal with. My suggestion is to try working with other companies and compare before making any premature decisions, like switching. PEER 1 team, apologies accepted, although it seemed like you could have done a better recruitment work: 60 programmers? 18 months? Really?

Comments are closed.

%d bloggers like this: