FTP Upload failure 80% of the time

G

gsaunders

Perch
I have put in a ticket a couple of weeks ago, but I wanted to pass this along to the forums and see if I can get some feedback.

Basically I have client that cannot reliably upload to server at Jodohost. 80% of the time he gets a failure where the bytes per second go from a respectable rate down to 0 and simply never finishes.

Here is what I posted in my last ticket update:

******
Sorry for delay getting back on this, but we have been doing a lot of testing and have had the cable company come out multiple times to check things at home... as well as had them replace wiring, modems, and routers and we get the same result.

Updates on tests:

The client has Cox cable. It was failing for him 80% of the time. He thought it was working off of his neighbors Cox link when he used wireless, but when I got involved in testing I found that it failed a majority of the time both from his own Cox link and from the neighbors cox link.

We have taken his unit and tested it from another ISP and it works 100% of the time. So we know it is not the PC.

We have tested it from 4 other ISP's and they all succeed 100% of the time as well.

So everything points back to COX as being the problem. However we have had cox come on-site and they have replaced the wiring, the cable modem, the router (even remove router from equation), and some external devices out at the pole and still same results. It fails 80 plus percent of the time. And when it fails it just stops transmitting as if the bytes uploaded per second go to 0.

Cox claims it is happening outside their network as all ping tests appear to be fine and tracerts appear to be fine within their network. Now I realize ping tests are not definitive answers, but they have escalated this to upper levels at Cox and they are standing by that it is not them.

So I want to send you what we have so you can examine and give feedback. The only difference I see between Cox and the other ISPs is the route in which the traffic takes. I am going to post tracerts below. I have marked in the header of each tracert which one has problems and which do not. Basically just the 2 cox trace routes have the problems with upload to ftp. Maybe you will find something or see something with one of the backbones upline. Here goes:

TraceRoute LogFor
(64.187.101.31)
(FROM Comcast - GOOD)

0: 0ms [192.168.250.1]
1: 15ms [96.178.58.1]
2: 16ms ge-4-1-ur01.salem.va.richmond.comcast.net [68.86.127.141]
3: 0ms te-9-1-ur01.martinsville.va.richmond.comcast.net [68.86.173.25]
4: 16ms te-1-1-ur02.danville.va.richmond.comcast.net [68.86.173.21]
5: 0ms te-9-1-ur01.danville.va.richmond.comcast.net [68.86.173.17]
6: 15ms te-1-1-ur01.lynchburg.va.richmond.comcast.net [68.86.173.13]
7: 16ms te-9-1-ur02.amherst.va.richmond.comcast.net [68.86.173.9]
8: 0ms te-1-1-ur01.amherst.va.richmond.comcast.net [68.86.173.5]
9: 15ms te-8-1-ar01.charlvilleco.va.richmond.comcast.net [68.86.173.1]
10: 16ms po-10-ar01.chesterfield.va.richmond.comcast.net [68.86.172.129]
11: 15ms te-0-4-0-0-cr01.charlotte.nc.ibone.comcast.net [68.86.91.113]
12: 31ms pos-2-8-0-0-cr01.atlanta.ga.ibone.comcast.net [68.86.85.213]
13: 47ms pos-0-6-0-0-cr01.miami.fl.ibone.comcast.net [68.86.86.65]
14: 141ms mojo-cr01.miami.fl.ibone.comcast.net [75.149.228.50]
15: 46ms [64.59.80.1]
16: 47ms cs-webhostingdotnet.mojohost.com [64.59.83.2]
17: 31ms [64.71.225.26]
18: 47ms [64.187.101.31]

Network Locations
192.168.250.1 => Local Network
96.178.58.1 => Mt Laurel, NJ, USA
68.86.127.141 => Mt Laurel, NJ, USA
68.86.173.25 => Mt Laurel, NJ, USA
68.86.173.21 => Mt Laurel, NJ, USA
68.86.173.17 => Mt Laurel, NJ, USA
68.86.173.13 => Mt Laurel, NJ, USA
68.86.173.9 => Mt Laurel, NJ, USA
68.86.173.5 => Mt Laurel, NJ, USA
68.86.173.1 => Mt Laurel, NJ, USA
68.86.172.129 => Mt Laurel, NJ, USA
68.86.91.113 => Mt Laurel, NJ, USA
68.86.85.213 => Mt Laurel, NJ, USA
68.86.86.65 => Mt Laurel, NJ, USA
75.149.228.50 => Mount Laurel, NJ, USA
64.59.80.1 => Farmington Hills, MI, USA
64.59.83.2 => Farmington Hills, MI, USA
64.71.225.26 => Miami, FL, USA
64.187.101.31 => Wilmington, DE, USA

TraceRoute LogFor
(64.187.101.31)
(FROM Cox - PROBLEMS)

0: 0ms [192.168.1.1]
1: 0ms [10.64.232.1]
2: 0ms [98.172.172.29]
3: 0ms [98.172.172.13]
4: 31ms ashbbbrj02-ae0.0.r2.as.cox.net [68.1.0.221]
5: 62ms cr2-cr1.wdc005.internap.net [66.79.146.202]
6: -1ms [?.?.?.?]
7: -1ms [?.?.?.?]
8: 47ms cr1.mia004.inappnet.cr2.acs007.internap.net [66.79.147.194]
9: 47ms core3.mia003.inappnet-427.cr1.mia003.internap.net [66.79.144.162]
10: 47ms border5.pc1.bbnet1.mia003.pnap.net [69.25.0.13]
11: 47ms webhosting-12.border5.mia003.pnap.net [216.52.162.66]
12: 46ms ibgp.border3.nota.mia.webhosting.net [64.71.226.34]
13: 46ms [64.71.225.26]
14: 46ms [64.187.101.31]

Network Locations
192.168.1.1 => Local Network
10.64.232.1 => Local Network
98.172.172.29 => Atlanta, GA, USA
98.172.172.13 => Atlanta, GA, USA
68.1.0.221 => Atlanta, GA, USA
66.79.146.202 => Atlanta, GA, USA
66.79.147.194 => Atlanta, GA, USA
66.79.144.162 => Atlanta, GA, USA
69.25.0.13 => Atlanta, GA, USA
216.52.162.66 => Atlanta, GA, USA
64.71.226.34 => Miami, FL, USA
64.71.225.26 => Miami, FL, USA
64.187.101.31 => Wilmington, DE, USA

TraceRoute LogFor
(64.187.101.31)
(FROM 2nd COX Location - PRoBLEMS)

0: 0ms [192.168.1.1]
1: 10ms [10.64.232.1]
2: 10ms [98.172.172.29]
3: 10ms [98.172.172.13]
4: 20ms ashbbbrj02-ae0.0.r2.as.cox.net [68.1.0.221]
5: 50ms cr2-cr1.wdc005.internap.net [66.79.146.202]
6: -1ms [?.?.?.?]
7: -1ms [?.?.?.?]
8: 90ms cr1.mia004.inappnet.cr2.acs007.internap.net [66.79.147.194]
9: 50ms core3.mia003.inappnet-427.cr1.mia003.internap.net [66.79.144.162]
10: 50ms border5.pc2.bbnet2.mia003.pnap.net [69.25.0.77]
11: 50ms webhosting-12.border5.mia003.pnap.net [216.52.162.66]
12: 50ms ibgp.border3.nota.mia.webhosting.net [64.71.226.34]
13: 60ms [64.71.225.26]
14: 50ms [64.187.101.31]

Network Locations
192.168.1.1 => Local Network
10.64.232.1 => Local Network
98.172.172.29 => Atlanta, GA, USA
98.172.172.13 => Atlanta, GA, USA
68.1.0.221 => Atlanta, GA, USA
66.79.146.202 => Atlanta, GA, USA
66.79.147.194 => Atlanta, GA, USA
66.79.144.162 => Atlanta, GA, USA
69.25.0.77 => Atlanta, GA, USA
216.52.162.66 => Atlanta, GA, USA
64.71.226.34 => Miami, FL, USA
64.71.225.26 => Miami, FL, USA
64.187.101.31 => Wilmington, DE, USA

TraceRoute LogFor
(64.187.101.31)
(FROM nTelos - GOOD)

0: 16ms [172.20.2.1]
1: 0ms tbgfw01.branchgroup.com [172.20.200.2]
2: 109ms [206.248.228.105]
3: 0ms [10.242.17.130]
4: 16ms cr1-eqix-peer.wdc003.internap.net [206.223.115.129]
5: 187ms cr2-cr1.wdc005.internap.net [66.79.146.202]
6: -1ms [?.?.?.?]
7: -1ms [?.?.?.?]
8: 63ms cr1.mia004.inappnet.cr2.acs007.internap.net [66.79.147.194]
9: 63ms core3.mia003.inappnet-427.cr1.mia003.internap.net [66.79.144.162]
10: 62ms border5.pc1.bbnet1.mia003.pnap.net [69.25.0.13]
11: 62ms webhosting-12.border5.mia003.pnap.net [216.52.162.66]
12: 234ms ibgp.border3.nota.mia.webhosting.net [64.71.226.34]
13: 62ms [64.71.225.26]
14: 47ms [64.187.101.31]

Network Locations
172.20.2.1 => Marina del Rey, CA, USA
172.20.200.2 => Marina del Rey, CA, USA
206.248.228.105 => Waynesboro, VA, USA
10.242.17.130 => Local Network
206.223.115.129 => Foster City, CA, USA
66.79.146.202 => Atlanta, GA, USA
66.79.147.194 => Atlanta, GA, USA
66.79.144.162 => Atlanta, GA, USA
69.25.0.13 => Atlanta, GA, USA
216.52.162.66 => Atlanta, GA, USA
64.71.226.34 => Miami, FL, USA
64.71.225.26 => Miami, FL, USA
64.187.101.31 => Wilmington, DE, USA

******

Thanks for your time.
 
uh yuck. I am not sure we can help with this I've never seen an issue such as this myself, but have you had them try changing FTP moves (active/passive), does it make any difference?
 
Yes... I have gone through all of that with support through the ticket system.

I have personally taken over testing so I know the client is not involved.

The ONLY ISP that it fails on is Cox and I personally still think it is Cox, but was hoping someone would see something.

I am going to go back to Cox because if you compare the nTelos trace route (which has successful transfers) vs the Cox trace route which fails most of the time you can see nTelos actually hits all of the same external links Cox does with one addition so it would make sense to me that Cox could not blame external links.

I am also having the client bring his notebook to me, because I want to be 100% certain that specific PC doesn't fail from one of the other links.. meaning it is PC related. He swears he has taken it to coffee shops and other places with no problems, but I want to test it enough times to be sure it wasn't one of the lucky successful uploads.
 
Hey what do the question marks represent in the trace route?

While the trace route is running (in VisualWhois software) it does seem to pause there before going on. My guess is it is due to some sort of security implementation at that hop.

TraceRoute LogFor
(64.187.101.31)
(FROM Cox - PROBLEMS)

0: 0ms [192.168.1.1]
1: 0ms [10.64.232.1]
2: 0ms [98.172.172.29]
3: 0ms [98.172.172.13]
4: 31ms ashbbbrj02-ae0.0.r2.as.cox.net [68.1.0.221]
5: 62ms cr2-cr1.wdc005.internap.net [66.79.146.202]
6: -1ms [?.?.?.?]
7: -1ms [?.?.?.?]
8: 47ms cr1.mia004.inappnet.cr2.acs007.internap.net [66.79.147.194]
9: 47ms core3.mia003.inappnet-427.cr1.mia003.internap.net [66.79.144.162]
10: 47ms border5.pc1.bbnet1.mia003.pnap.net [69.25.0.13]
11: 47ms webhosting-12.border5.mia003.pnap.net [216.52.162.66]
12: 46ms ibgp.border3.nota.mia.webhosting.net [64.71.226.34]
13: 46ms [64.71.225.26]
14: 46ms [64.187.101.31]

This problem is killing us.

I just went to the client and tried to FTP to multiple other servers at jodo (my servers) and they all start out at about 200kbps and within about 10 to 15 seconds it drops to 0 and then a minute later you get a timeout.

Again only from Cox from his physical location (and neighbor).

Again Cox is saying it is nothing on their end. They have replaced modem, replaced router (and removed it from equation), replaced physical wiring and connections. Short of replacing downline equipment they have done everything you would expect or want them to do.

Now either there is a problem in cox network or there is one further downline.

When the user browses to his website it loads up fine and he doesn't notice any performance issues.

And remember this had been working fine over a year and then just started failing like this. I have scanned his computers and done my manual checks with special tools and both his computers are clean... plus the computers work from another ISP.

What the heck should I be looking at???
 
I think I'd be as frustrated as you with this right now. I simply don't know what else to offer you as a suggestion.

One thing to check quickly, and it certianly isn't the best means to upload, but can you try and see if it only affects FTP protocol? For example, try uploading in webshell?

This does sound like something I experienced on AT&T trying to download CentOS via bittorrent(legally mind you) but it would go from 1.2mb or two, down to 0kb in about 5 minutes, AND take the rest of the net connection with it till I stopped bittorrent for 10-15 minutes it would all come back. I never had proof of this, but I think it was ATT QoS throttling; my laptop would download at many megabit per second when at the datacenter on the same bittorrent, but at home it would die after a 3-5 minute period. I needed to download the latest CentOS so irregularly that it wasn't a big deal and I'd do it from the datacenter(it was all for work anyway).
 
Stephen said:
I think I'd be as frustrated as you with this right now. I simply don't know what else to offer you as a suggestion.

One thing to check quickly, and it certianly isn't the best means to upload, but can you try and see if it only affects FTP protocol? For example, try uploading in webshell?
Click to expand...

Hey Steve,

Great idea... I'll give that a try and see if happens in webshell. At least that will change up the equation a bit.

When I get a chance to do that I'll reply back.

Greg
 
Hey Steve,

WebShell gave the same problems... it worked once out of 5 times. It would ultimately timeout the upload. Download works fine... but uploads don't... again from this one ISP.

Greg
 
gsaunders said:
Hey Steve,

WebShell gave the same problems... it worked once out of 5 times. It would ultimately timeout the upload. Download works fine... but uploads don't... again from this one ISP.

Greg
Click to expand...

I Think that rules out QoS throttling, there is no way they'd do that intentionally to HTTP uploads.

Now for what else could it be? ?( Not sure at this time.
 
You must log in or register to reply here.
Back
Top