Wget on NMT giving wrong address?? [delete /tmp/dns_cache - bug in wget]

[lordy]

I've seen this behaviour before - it clears with a reboot:

wget on my NMT sometimes keeps resolving an address incorrectly:

{{{
EDIT
Ahhh.. maybe I should pay more attention to OVersight users:

http://www.networkedmediatank.com/showth...?tid=21169

Note that wget is also used by Gaya backend so this could be quite a serious network affecting issue.
I have a WRT54G running Tomato. I'll disable DNS Relay and keep an eye on it... }}}

From a telnet session:

nslookup OK

Code:

# nslookup www.bing.com
Server:    192.168.88.1
Address 1: 192.168.88.1 unknown

Name:      www.bing.com
Address 1: 78.33.33.88 78-33-33-88.static.enta.net
Address 2: 78.33.33.80 78-33-33-80.static.enta.net

ping ok

Code:

# ping www.bing.com
PING www.bing.com (78.33.33.88): 56 data bytes
64 bytes from 78.33.33.88: seq=0 ttl=56 time=20.499 ms
64 bytes from 78.33.33.88: seq=1 ttl=57 time=16.589 ms
64 bytes from 78.33.33.88: seq=2 ttl=57 time=17.036 ms

but now wget gets the complete wrong ip and times out:

Code:

# /bin/wget http://www.bing.com
--03:13:26--  http://www.bing.com/
           => `index_3.html'
Resolving www.bing.com... 86.53.218.136
Connecting to www.bing.com|86.53.218.136|:80

Happens with stock wget and CSI/telnet one.

Also tried:

A reboot will clear. I've also tried the --no-dns-cache option, but this is just an in-memory cache anyway.

I guess its a wget dns bug?

---

FWIW I was on static DHCP with
Primary DNS=my router
Secondary DNS=208.67.220.220

I swapped them around. ie

Primary DNS=208.67.220.220
Secondary DNS=my router

and still got

Code:

# nslookup www.bing.com
Server:    208.67.220.220
Address 1: 208.67.220.220 resolver2.opendns.com

Name:      www.bing.com
Address 1: 78.33.33.88 78-33-33-88.static.enta.net
Address 2: 78.33.33.80 78-33-33-80.static.enta.net
# wget http://www.bing.com
--03:37:24--  http://www.bing.com/
           => `index_3.html'
Resolving www.bing.com... 86.53.218.136
Connecting to www.bing.com|86.53.218.136|:80...

---

Definitely looks like a bug in NMT wget.

The wget I compiled works OK

Code:

/share/Apps/feedtime/bin/wget http://www.bing.com
--2009-11-27 03:42:39--  http://www.bing.com/
Resolving www.bing.com... 78.33.33.80, 78.33.33.88
Connecting to www.bing.com|78.33.33.80|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 27333 (27K) [text/html]
Saving to: `index.html.1'

100%[================================================================================​=================>] 27,333      --.-K/s   in 0.07s

2009-11-27 03:42:40 (377 KB/s) - `index.html.1' saved [27333/27333]

[nuke12]

Interesting. What I don’t get is why it isn’t consistently wrong and why a reboot will temporally fix the problem.

[jasam]

(11-27-2009 12:01 PM)nuke12 Wrote:  Interesting. What I don’t get is why it isn’t consistently wrong and why a reboot will temporally fix the problem.

It looks like transparent proxy being run by your ISP - Telecomplete (isn't it?) as IP address you get in WGET belongs to this ISP.
Transparent proxies are very often used to offload internet uplinks. They intercept tcp calls (wget, telnet) and redirect them to local proxy.....but it does not make any sense to redirect bing.com - which is search engine - and nothing to cache in here
You can tell your ISP to set properly proxy rules to not redirect bing.com

Are you having the same with google.com ?

jasam

[nuke12]

None of that explains why his compiled version of wget works fine?

[jasam]

(11-28-2009 09:23 PM)nuke12 Wrote:  None of that explains why his compiled version of wget works fine?

...I only tried to explain network behavior and reason why you have different IP's. I'm not wget expert but the answer to why compiled wget works ? - can be that some extra parameter is compiled and in default wget it's not. Suggest googling it.

jasam.

[lordy]

FWIW a few corrections and clarifications to my earlier post:

It only happens with NTM /bin/wget not with CSI/busybox/wget
(my mistake I forgot I deleted the busybox/wget symlink)

For me, Changing the DNS settings on the router seems to be ruled out at the moment because
1. my DNS1 is set to OpenDNS ip.
2. HDX IP is static.

.. but I haven't rebooted yet.

Code:

BUSYBOX - GOOD

# /share/.bin/busybox wget www.bing.com
wget: not an http or ftp url: www.bing.com
# /share/.bin/busybox wget http://www.bing.com
Connecting to www.bing.com (80.157.169.187:80)
index.html           100% |*******************************| 27221  --:--:-- ETA

MINE - GOOD

# /share/Apps/feedtime/bin/wget http://www.bing.com
--2009-11-28 21:06:55--  http://www.bing.com/
Resolving www.bing.com... 80.157.169.187, 80.157.169.163
Connecting to www.bing.com|80.157.169.187|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 27221 (27K) [text/html]
Saving to: `index.html.1'

100%[======================================>] 27,221      50.7K/s   in 0.5s

2009-11-28 21:06:57 (50.7 KB/s) - `index.html.1' saved [27221/27221]

NMT - OOPS

# /bin/wget http://www.bing.com
--21:07:11--  http://www.bing.com/
           => `index_1.html'
Resolving www.bing.com... 86.53.218.136
Connecting to www.bing.com|86.53.218.136|:80...

Also I am using a Static/fixed/hardcoded IP for HDX. (initiall allocated via a Reservation I think ,and then changed to static)

So I think the DNS relay solution to other similar issues is not a solution to this one.
Its still happening despite having a fixed IP and hardcoding OpenDNS directly.

I haven't rebooted yet because I wanted to test Oversight whist this was happening.
My NMT firmware is 50-17-090204-15-HDD-403-000

I then checked my A100 ( 00-17-090115-15-POP-402 ) - using DHCP Reservation - and the same thing is happening,
despite the wgets both being slightly different:


b5abc78ab5b64b174284a0d8bc9b94f6 */cygdrive/a/nmt-wget (HDX)
44c63b0059a270afbb11a0b219d2a252 */cygdrive/b/nmt-wget (A100)

both HDX and A100 wgets are v 1.10.2 . mine is 1.11.4

I'll check C200 now....

---

And C200 results are in - epic fail

Code:

root@PCH-C200 [/share] > /bin/wget -S http://www.bing.com
--21:43:33--  http://www.bing.com/
           => `index.html'
Resolving www.bing.com... 86.53.218.80
Connecting to www.bing.com|86.53.218.80|:80...
** time out here***

DHCP reservation with direct assignment of DNS servers.

Code:

root@PCH-C200 [/] > grep eth /tmp/setting.txt
eth_dns_manual=0
eth_dns=208.67.222.222
eth_dns2=208.67.220.220
eth_gateway=192.168.88.1
eth_ipaddr=192.168.88.7
eth_netmask=255.255.255.0

all other devices and os's are working OK with bing.com, And a new version of wget works,
so whatever proxying is going on seems to be OK generally

my hunch is that it is a bug in NMT's wget.

After rebooting the C200 it works OK :

Code:

root@PCH-C200 [/] > wget -S www.bing.com
--22:07:25--  http://www.bing.com/
           => `index.html'
Resolving www.bing.com... 213.248.112.176, 213.248.112.184
Connecting to www.bing.com|213.248.112.176|:80... connected.
HTTP request sent, awaiting response...
  HTTP/1.0 200 OK

OK maybe its the way wget talks to opendns?
I'll try different
a different nameserver maybe?

---

FWIW - now the C200 is working again I looked at the HDX/A100

I changed the router primary DNS to 4.2.2.1 & rebooted the router - and the HDX and A100 still no change.
I then added 4.2.2.1 to /etc/resolve.conf - no change (nslookup picked up the change but wget still failed)

Thats enough for now!

[lordy]

OK I just rebooted everything, and problems have gone away, but then I decided to do a diff on the strings in /bin/wget and those in the one I compiled.

Amongst all the fluff the following lines stood out :

Code:

/tmp/dns_cache
/tmp/prog

Darn - if I could have waited a bit longer before rebooting. I might have had a nice workaround for this...

---

OK got a workaround .. delete /tmp/dns_cache (surprise surprise!)

(whose bright idea to cache DNS records and not manage stale/changed entries ? Or even change wget code. )

For anyone who cares .. read on ...

Here is my experiment to verify. We are going to
1.make testing.com point to bing.com
2. do a wget
3. make testing.com point to google.com
4. do a wget
5. results ?

This is bing 195.59.150.64

Code:

# nslookup www.bing.com
Server:    208.67.220.220
Address 1: 208.67.220.220 resolver2.opendns.com
Name:      www.bing.com
Address 1: 195.59.150.64
Address 2: 195.59.150.80

This is google 216.239.59.103

Code:

# nslookup www.google.com
Server:    208.67.220.220
Address 1: 208.67.220.220 resolver2.opendns.com

Name:      www.google.com
Address 1: 216.239.59.103 gv-in-f103.1e100.net
Address 2: 216.239.59.104 gv-in-f104.1e100.net
Address 3: 216.239.59.147 gv-in-f147.1e100.net
Address 4: 216.239.59.99 gv-in-f99.1e100.net

Make testing.com = bing.

Code:

# echo '195.59.150.64 testing.com' >> /etc/hosts

Do a wget on testing.com - it looks at bings ip

Code:

# wget --spider testing.com
--23:02:04--  http://testing.com/
           => `index.html'
Resolving testing.com... 195.59.150.64, c33b:9640:48ee:ee7f::
Connecting to testing.com|195.59.150.64|:80... connected.
HTTP request sent, awaiting response... 400 Bad Request
23:02:04 ERROR 400: Bad Request.

Now remove it and make testing.com = google.com

Code:

# sed -i '/testing.com/ d' /etc/hosts
# echo '216.239.59.103 testing.com' >> /etc/hosts

Do another wget on testing.com - it still things it is bing

Code:

# wget --spider testing.com
--23:02:49--  http://testing.com/
           => `index.html'
Resolving testing.com... 195.59.150.64
Connecting to testing.com|195.59.150.64|:80... connected.
HTTP request sent, awaiting response... 400 Bad Request
23:02:49 ERROR 400: Bad Request.

Completely remove testing.com from the host file.

Code:

# sed -i '/testing.com/ d' /etc/hosts

Now ideally it should use the real testing.com on the internet. - but it still uses bing.
C

Code:

# wget --spider testing.com
--23:03:15--  http://testing.com/
           => `index.html'
Resolving testing.com... 195.59.150.64
Connecting to testing.com|195.59.150.64|:80... connected.
HTTP request sent, awaiting response... 400 Bad Request
23:03:15 ERROR 400: Bad Request.

Double check that testing.com really is the real one on the internet.

Code:

# ping testing.com
PING testing.com (216.34.131.135): 56 data bytes
64 bytes from 216.34.131.135: seq=0 ttl=29 time=171.432 ms
64 bytes from 216.34.131.135: seq=1 ttl=31 time=169.903 ms

--- testing.com ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 169.903/170.667/171.432 ms

Double double check...

Code:

# wget --spider testing.com
--23:03:28--  http://testing.com/
           => `index.html'
Resolving testing.com... 195.59.150.64
Connecting to testing.com|195.59.150.64|:80... connected.
HTTP request sent, awaiting response... 400 Bad Request
23:03:28 ERROR 400: Bad Request.

OK remove the /tmp/dns_cache file...

Code:

# rm /tmp/dns_cache

And as if by magic....

Code:

# wget --spider testing.com
--23:03:38--  http://testing.com/
           => `index.html'
Resolving testing.com... 216.34.131.135
Connecting to testing.com|216.34.131.135|:80... connected.
HTTP request sent, awaiting response... 302 Moved Temporarily
Location: /?d9fb4e60 [following]
--23:03:39--  http://testing.com/?d9fb4e60
           => `index.html?d9fb4e60'
Connecting to testing.com|216.34.131.135|:80... connected.
HTTP request sent, awaiting response... 302 Moved Temporarily
Location: / [following]
--23:03:39--  http://testing.com/
           => `index.html'
Connecting to testing.com|216.34.131.135|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
200 OK

[Ger Teunis]

Very nice find Lordy.
This bug was highly visible on the CSI's appinit as well. I've moved the download from NMT to PC because of this bug (users reporting download errors because of a dns change).

I'll add some (periodical) flushing to appinit as well.
Good job.

---

ps> most (all?) routers at users homes also do a good dns caching. What about if we create empty cache files and mark them readonly for the nmt and root user? Perhaps we can prevent the NMT from using these files all together.
DNS cache from the router is only unmeasurable slower especially on the nmt platform.

[lordy]

It is a strange 'feature'.

In addition to DNS lookups to most routers being quite quick,
wget already has a per-process in-memory dns cache built in.

So the only time this /tmp/dns_cache should be used is when a particular wget process does its very first lookup on a particular domain. This is after the overhead of starting the wget process. (which is not a part of busybox so an expensive process to start). A dns lookup should be fairly insignificant compared to all that.

Also the NMT cpu is so slow (relatively) that the expense of lookups and inserts into /tmp/dns_cache is probably more than just asking another more capable device on the LAN.

I'd be surprised if there were any real world gains, Esp as wget is quite a mature and heavily used bit of code, I'd expect people to think twice, before hacking about with it. QED!

---

Here is a quick fix that can be applied once at bootup. Link /tmp/dns_cache to /dev/null

Code:

if [ ! /dev/null -ef /tmp/dns_cache ] ; then rm -f /tmp/dns_cache && ln -s /dev/null /tmp/dns_cache ; fi

[lordy]

Ah I think the plot thickens:

It looks like with certain network configurations (routers? DNS servers?) , DNS requests issued by the NMT do not complete quickly (eg in 10 seconds rather than a few milliseconds).

This would normally cause wget to become extremely inefficient, and it might explain why the wget binary in NMT firmware was altered to do its own dns caching (which is only cleared on reboot).

It was added as a workaround for a DNS request bug?

So either the wget dns cache was added to fix a DNS delay bug OR the wget dns cache causes the DNS delay bug. One way to find out would be to find a configuration with the dns delay bug, and see if other things are affected besides /bin/wget