Big Black Hole of communication

Tonight was not a good experience. X( From my point of view, everything “jodo” was completely down. You might as well have been out of business, and absconded with our money. I would not have known the difference. I got back from a day away from the office, I could not get my mail on any of my jodo accounts. I could not get to any jodo web site. A cold chill ran down my spine, dreading the flood of phone calls I was about to get from my reseller clients. 8o What could I possibly tell them? Oh God.

Now, it turns out that you were very much on top of the situation and working to get it fixed, and within a few hours it was fixed… but I (as I assume 90% of other interested users and resellers) had no way to know what the heck was going on (that I could think of). I did not have your phone number handy, and could not look it up on your wesite (down). Also as noted in another thread, and as logic would dictate, you could not have possibly answered phone calls from every client to explain the same thing over and over. That would be a great task for a website, like a forum… hmm… like THIS forum… but alas, it was down (or unreachable at least) like everythig else.:evil:

You get where I am going with this?

It is insane that ALL of your DNS and emergency support are all hosted by YOU at the same place where any system-wide problem would also prevent support services and communication with your clients.

In my humble opinion, you need backup DNS services (AT LEAST FOR YOUR “JODO” URL’s, if not all your client’s sites) hosted somewhere completely unrelated to your network, so that when your whole datacenter is down, at least some DNS server somewhere can locate stuff.

Second, of course, is that these very forums (as well as other jodo web pages.) sould be hosted (or mirrored?) somewhere outside your datacenter and network.

Am I just way out in left field here? Isn’t this reasonable? Can anyone argue that it is unreasonable? Or that it is more reasonable to just leave it the way it is? Would love to hear discussion.

This is outside our network and it was up, people(including us) were posting on it during the outage :slight_smile:

We also posted on webhostingtalk.com (I called up Yash to have him go there as my phone takes too long to load it up)

This forum was up the entire time of the outage.

Jodopulse.com is also external and has chat links.

I Won’t say communications were perfect, as I was unable to stay online in a reliable fashion from the datacenter due to low cellular signal, and when I did post I was using the security desk DSL connection(and it worked just fine), I was able to get one post up from the phone but it is just too tedious.

I could not get to the forums. Once I did, I immediately found the posts about the issue, and that it was “resolved”. I can only guess that I could not reach the forums because the DNS info was not cached anywhere on my system as it probably was on yours (or the DNS servers were “local” to your connection, and not mine). Perhaps the main problem from my end was DNS. Still, whatever it was, it was unreachable.

No, there were posts here:
http://support.jodohost.com/showthread.php?p=57640#post57640
http://support.jodohost.com/showthread.php?p=57644#post57644
http://support.jodohost.com/showthread.php?p=57633#post57633

http://support.jodohost.com/showthread.php?t=10989 ← customer only section

in addition I was using the security desk DSL connection that I had never used before no way it was cached :slight_smile:

jodopulse is outside network and outside DNS 100% and it has the live chat links, we can handle many live chats at once, many more than phone

I also was unable to access the forums for some time (I have 3 independent internet
connections here) but glad everything is working now. :slight_smile:

Our Team continously informed status to our customers through LiveChat, phone calls, this Customer Support Forums and Web Hosting Industry discussion forums. There was no break in these communication means.

jodopulse.com and this forum are offsite, and were being updated.
We were infact chatting with many customers during this outage, using our livechat system.

I apologise for the fact you couldn’t reach us, and we will look to improve that

I understand, and already aknowledge that you were dealing with the situation the best that you could, and the updates here were good to see. But, I couldn’t see them. There was a break, and denial is not helpful.

My point (and CNX’s) is that WE COULD NOT GET TO THE FORUMS during the outage. And for me, at least, with email also down, I couldn’t think of another way to reach you.

Maybe totally a DNS issue, and maybe the people who WERE able to get here were more regular (daily) users who had some local access to your DNS servers, or were using some DNS server that had your info cached, or something.

I did not remember about jodopulse.com. Even if I did, the LINK from there to these forums would not have worked for me, because the link uses a support.jodohost.com URL, and that was not working for me at all. (I agree, I probably could have started a chat from there though). [edit: Also, in case I do remember about jodopulse.com next time, could you list your phone numbers on that page as well? Thanks.]

One other note… during the outage, I did go to webhostingtalk.com and searched “jodohost” I did not find any discussion there.

I seem to recall other incidents in the past where a backup, external, registered DNS server would have saved the day. Maybe that is worth looking into?

jph,

webhostingtalk discussion was in the outages forum which they hide from users not logged in for some reason, so login and it will then work.

Odd, I had no problem reaching the forum during the course of the outage, neither did many others who posted here during that time.

It is the responsibility of the reseller to bookmark essential URLs i.e., jodopulse.com, live chat, enter contact phone phone numbers in their address book and use them when needed. This does not indicate any shortcoming on the part of Jodohost.

This is the point of jodopulse; it allows you to see the status of all systems in realtime, automatically refreshed every 30 seconds. A major network break, such as the one that happened yesterday severs all communication with the datacenter. Even an alternate DNS would not help in this situation.

Kudos to Jodohost for handling this matter in a professional , expeditious and forthright manner. As usual, communication was excellent (regardless of the statements of detractors) and inconvenience was kept to a minimum.

Just wanted to add that I was able to access the forums during the outage. Perhaps it might have went down a few times, but I was checking the forums at least 10 times within the hour.

(Totally forgot about jodopulse though, so didn’t check it).

Hafa,

“As usual, communication was excellent (regardless of the statements of detractors) and inconvenience was kept to a minimum.”

Does everything have to be coddling, kudos, and defensiveness around here? I am not a detractor, I am a realist and a customer. I said they did a good job generally speaking with the crisis situation.

You did not have the same problem that I had. OK, so WHAT? we should only address problems that effect everyone?

I was not using my usual computer at the time, I did not have my bookmarks or email where I may have had other contact info, if you must know. Whatever YOU feel should be my “responsibility” I don’t care much about. However I think it IS my responsibility to help improve things, and I think things could easily be improved in advance of any future incident.

For example, here is another suggestion.
Jodo, could you please add DNS records for support.jodopulse.com and forum.jodopulse.com
and point them to the same IP as support.jodohost.com
and make the forum link on www.jodopulse.com also use one of those URLs (instead of support.jodohost.com).

That would truly make a way for people to access the forums without depending on Jodo’s own DNS servers (or any of their network) at all. Now i just have to go out and get a tattoo of “jodopulse.com” put on my body so I can remember where to look in an emergency (or you could put DNS records for jodohost.com on some external DNS server, and I would have one less thing to remember when I am already about to panic).

I like jodohost, that’s why I want to see it continually review and improve small things like these that can make a HUGE difference at some critical moment in the life of a customer.

jph,

your comments were noted as well :slight_smile:

Just to support jph
I couldnt access anything on jodo either during that outage although I was able to monitor the situation via WHT.

I also agree that jodo did a great job getting everything up and running as quick as possible, for my customers this happened at 2pm on a thursday afternoon – OUCH

was bad timing in any case, you never want to go down.
During the outage it you REALLY wanted to get to most any site it would still work if you refreshed a few times.
since the connection was “flapping” (meaning going on and off) it would have times even that it would be near normal for 10 seconds, then drop to not working, come back, it was a bad sitaution and the engineers actually had given me an apology at the end because they were looking at the wrong switch (the one that was alerting on their monitors) instead of the one that ended up being responsible. Sometimes monitors are good to tell you is a problem, but not to pinpoint the problem and that was the issue this time as well.

OK, it’s been a few days now, and this will be forgotten about if we let it.

Jodo, could you please update the JODOPULSE.com page so that the link for “Network Outages” does NOT use a “jodohost.com” URL? It makes no sense. That link won’t work in a true network outage. At least please add a link that uses the IP address of the forum server.

It would also be convenient if you would add a DNS record for “support.JODOPULSE.com” which also pointed to the forums.

Also, could you put your phone number(s) on the jodopulse page somewhere.

Thanks.

jph,

We are working on far more than that with the upgrades and all, it is not forgotten.

I won’t be putting the phone numbers however, as during an outage it is the last thing we need, each tech can handle 10+ chats in such case but taking a phone call is 1 on 1 and takes away from chat too, may be a bit of an issue for you, but chat is by far the best manner to handle it in such a crisis.

It is a 2 minute job, and it was forgotten.

The forum was up, but the first message about this issue was 5 HOURES AFTER the issue was (when Stephen wrotte in). Before, NOBODY knew anything about this issue. Yes, the comunication was break during 4-5 houres.

And about Livechat, I couldn’t open a chat sesion in 4 houres.

Please, let’s go to say the things as it was.

Thanks.