weatherunderground not updating

[zwolfpack]

You should be able to verify that your name server lookup (DNS) is working by comparing the results from working & non-working machines.

Open a Windows cmd window (Start->Run-> cmd) and enter the following command:

nslookup www.wunderground.com

This is what I get:

C:\Users\Me>nslookup www.wunderground.com
Server:  UnKnown
Address:  192.168.1.1

Non-authoritative answer:
Name:    e12930.g.akamaiedge.net
Address:  23.210.206.170
Aliases:  www.wunderground.com
          wunderground.com.edgekey.net

You might get different results from your location, but you want results between two computers on the same network to match.

If not, first check if the server addresses match. If not, you likely have a static IP address assigned and have different nameservers specified for the two machines. (If you are using DHCP, the nameserver address is provided by the router and you don't have to worry about this).

Another thing to check is the hosts file
C:\Windows\System32\Drivers\etc\hosts

Generally I'd expect this to be empty unless you are doing something fancy.

[langenet]

Yup.... went through all of that.

[langenet]

Michael,

Have you had a chance to review this?

Robert

[langenet]

Michael,

I've replaced the files noted from this thread as per your recommendation, but don't see any additional logging going on.

Robert

[Michael McSharry]

No change means no errors detected in the HTTP request.
I have added "Accept-Encoding: identity;q=1.0, *;q=0" to the HTTP header as I have seen that mentioned on the WU site. Give it try. The Evaluate Download should be all you need to try to know if it works or not.

[langenet]

Looks like it's showing errors...

Jun-25 12:24:49 PM         mcsSprinklers    Download3 http://www.wunderground.com/weatherstation/WXCurrentObXML.asp?format=XML&ID=IONTARIO493&day=25&year=2017&month=6&Cache=fmiI775 Error=The underlying connection was closed: An unexpected error occurred on a send.   

 Jun-25 12:24:49 PM         mcsSprinklers    Download4 http://www.wunderground.com/weatherstation/WXCurrentObXML.asp?format=XML&ID=IONOTTAW25&day=25&year=2017&month=6&Cache=fmiI14 Error=The underlying connection was closed: An unexpected error occurred on a send.   

 Jun-25 12:24:49 PM         mcsSprinklers    Download2 http://www.wunderground.com/weatherstation/WXCurrentObXML.asp?format=XML&ID=IONTARIO608&day=25&year=2017&month=6&Cache=fmiI302 Error=The underlying connection was closed: An unexpected error occurred on a send.   

 Jun-25 12:24:30 PM         mcsSprinklers    Download4 http://www.wunderground.com/weatherstation/WXCurrentObXML.asp?format=XML&ID=IONOTTAW25&day=25&year=2017&month=6&Cache=fmiI290 Error=The underlying connection was closed: An unexpected error occurred on a send.   

 Jun-25 12:24:30 PM         mcsSprinklers    Download3 http://www.wunderground.com/weatherstation/WXCurrentObXML.asp?format=XML&ID=IONTARIO493&day=25&year=2017&month=6&Cache=fmiI580 Error=The underlying connection was closed: An unexpected error occurred on a send.   

 Jun-25 12:24:30 PM         mcsSprinklers    Download2 http://www.wunderground.com/weatherstation/WXCurrentObXML.asp?format=XML&ID=IONTARIO608&day=25&year=2017&month=6&Cache=fmiI533 Error=The underlying connection was closed: An unexpected error occurred on a send.   

 Jun-25 12:24:02 PM         mcsSprinklers    Download4 http://www.wunderground.com/weatherstation/WXCurrentObXML.asp?format=XML&ID=IONOTTAW25&day=25&year=2017&month=6&Cache=fmiI580 Error=The underlying connection was closed: An unexpected error occurred on a send.   

 Jun-25 12:24:02 PM         mcsSprinklers    Download3 http://www.wunderground.com/weatherstation/WXCurrentObXML.asp?format=XML&ID=IONTARIO493&day=25&year=2017&month=6&Cache=fmiI533 Error=The underlying connection was closed: An unexpected error occurred on a send.   

 Jun-25 12:24:02 PM         mcsSprinklers    Download2 http://www.wunderground.com/weatherstation/WXCurrentObXML.asp?format=XML&ID=IONTARIO608&day=25&year=2017&month=6&Cache=fmiI706 Error=The underlying connection was closed: An unexpected error occurred on a send.   

 Jun-25 12:23:56 PM         mcsSprinklers    Download4 http://www.wunderground.com/weatherstation/WXCurrentObXML.asp?format=XML&ID=IONOTTAW25&day=25&year=2017&month=6&Cache=fmiI580 Error=The underlying connection was closed: An unexpected error occurred on a send.   

 Jun-25 12:23:56 PM         mcsSprinklers    Download3 http://www.wunderground.com/weatherstation/WXCurrentObXML.asp?format=XML&ID=IONTARIO493&day=25&year=2017&month=6&Cache=fmiI533 Error=The underlying connection was closed: An unexpected error occurred on a send.   

 Jun-25 12:23:56 PM         mcsSprinklers    Download2 http://www.wunderground.com/weatherstation/WXCurrentObXML.asp?format=XML&ID=IONTARIO608&day=25&year=2017&month=6&Cache=fmiI706 Error=The underlying connection was closed: An unexpected error occurred on a send.   

 Jun-25 12:23:01 PM         Plug-In    Finished initializing plug-in MCSSPRINKLERSP   

 Jun-25 12:22:58 PM         mcsSprinklers    Download2 http://www.wunderground.com/weatherstation/WXCurrentObXML.asp?format=XML&ID=IONTARIO608&day=25&year=2017&month=6&Cache=fmiI706 Error=The underlying connection was closed: An unexpected error occurred on a send.   

 Jun-25 12:22:53 PM         mcsSprinklers    Version 2.17.0.10 | V2.17.0.10/2.17.0.10 (Professional) Copyright 2004-2017 mcsSolutions Registered with Homeseer   

 Jun-25 12:22:53 PM         Info    Plugin MCSSPRINKLERSP has connected. IP:127.0.0.1:3934   

 Jun-25 12:22:45 PM         Info    Plugin MCSSPRINKLERSP with instance: has disconnected   

 Jun-25 12:22:44 PM         Plug-In    Shutting down Plug-In: MCSSPRINKLERSP   

 Jun-25 12:22:10 PM         Plug-In    Finished initializing plug-in MCSSPRINKLERSP 

[Michael McSharry]

It seems to indicate that the connection was closed prematurely. I am operating blind as to why and why is it only when the request is made on some computers. The code segment is below where header information has been added recently. The actual transactions occurs in one .NET line "oWebClient.DownloadData(sURL)". What the WU server is expecting and what actually gets delivered in the HTTP transaction are both not visible to me.

            Using oWebClient As New WebClient()
                Dim sURL As String = URL2
                oWebClient.Headers.Add("User-Agent: Other")
                oWebClient.Headers.Add("Accept-Encoding: identity;q=1.0, *;q=0")
                Dim myDatabuffer As Byte() = oWebClient.DownloadData(sURL)
                rtsINET_Data(index) = Encoding.ASCII.GetString(myDatabuffer)
                myDatabuffer = Nothing
                oWebClient.Dispose()
            End Using

[langenet]

Was the header information added in your code?
I would pretty much guarantee that you could reproduce this using a WinXP box making the calls using I.E.

Robert

[Michael McSharry]

I captured the HTTP traffic using Wireshark for mcsSprinklers download and for Firefox. The request packets in each case are below. Of interest are the HTTP protocol where secure protocol upgrade and connection keep alive are specified. Compression is also accepted. Through evaluation I determined that data was not returned to mcsSprinklers if compression or protocol upgrade were selected. Connection keep alive allowed data to be returned. The user agent and the other Accept parameters were also included. Download 2,3,&4.txt all have different combinations of the parameters. See if you get something from any of the three. Based upon your error message about connection being closed, the keep alive has the best hope.

mcsSprinklers

Frame 125: 256 bytes on wire (2048 bits), 256 bytes captured (2048 bits) on interface 0
Ethernet II, Src: AsrockIn_37:b2:36 (d0:50:99:37:b2:36), Dst: AsustekC_a2:ee:68 (40:16:7e:a2:ee:68)
Internet Protocol Version 4, Src: 192.168.0.7, Dst: 23.60.74.115
    0100 .... = Version: 4
    .... 0101 = Header Length: 20 bytes (5)
    Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
    Total Length: 242
    Identification: 0x43bf (17343)
    Flags: 0x02 (Don't Fragment)
    Fragment offset: 0
    Time to live: 128
    Protocol: TCP (6)
    Header checksum: 0x93e8 [validation disabled]
    [Header checksum status: Unverified]
    Source: 192.168.0.7
    Destination: 23.60.74.115
    [Source GeoIP: Unknown]
    [Destination GeoIP: Unknown]
Transmission Control Protocol, Src Port: 50725, Dst Port: 80, Seq: 1, Ack: 1, Len: 202
    Source Port: 50725
    Destination Port: 80
    [Stream index: 13]
    [TCP Segment Len: 202]
    Sequence number: 1    (relative sequence number)
    [Next sequence number: 203    (relative sequence number)]
    Acknowledgment number: 1    (relative ack number)
    Header Length: 20 bytes
    Flags: 0x018 (PSH, ACK)
    Window size value: 49717
    [Calculated window size: 198868]
    [Window size scaling factor: 4]
    Checksum: 0x4a20 [unverified]
    [Checksum Status: Unverified]
    Urgent pointer: 0
    [SEQ/ACK analysis]
Hypertext Transfer Protocol
    GET /weatherstation/WXCurrentObXML.asp?format=XML&ID=IONTARIO493&day=27&year=2017&month=6&Cache=fmiI14 HTTP/1.1\r\n
    Accept-Encoding: identity;q=1.0, *;q=0\r\n
    User-Agent: Other\r\n
    Host: www.wunderground.com\r\n
    \r\n
    [Full request URI: http://www.wunderground.com/weatherstation/WXCurrentObXML.asp?format=XML&ID=IONTARIO493&day=27&year=2017&month=6&Cache=fmiI14]
    [HTTP request 1/1]
    [Response in frame: 129]

Firefox

Frame 439: 705 bytes on wire (5640 bits), 705 bytes captured (5640 bits) on interface 0
Ethernet II, Src: AsrockIn_37:b2:36 (d0:50:99:37:b2:36), Dst: AsustekC_a2:ee:68 (40:16:7e:a2:ee:68)
Internet Protocol Version 4, Src: 192.168.0.7, Dst: 23.60.74.115
    0100 .... = Version: 4
    .... 0101 = Header Length: 20 bytes (5)
    Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
    Total Length: 691
    Identification: 0x1f87 (8071)
    Flags: 0x02 (Don't Fragment)
    Fragment offset: 0
    Time to live: 128
    Protocol: TCP (6)
    Header checksum: 0xb65f [validation disabled]
    [Header checksum status: Unverified]
    Source: 192.168.0.7
    Destination: 23.60.74.115
    [Source GeoIP: Unknown]
    [Destination GeoIP: Unknown]
Transmission Control Protocol, Src Port: 49976, Dst Port: 80, Seq: 1, Ack: 1, Len: 651
    Source Port: 49976
    Destination Port: 80
    [Stream index: 35]
    [TCP Segment Len: 651]
    Sequence number: 1    (relative sequence number)
    [Next sequence number: 652    (relative sequence number)]
    Acknowledgment number: 1    (relative ack number)
    Header Length: 20 bytes
    Flags: 0x018 (PSH, ACK)
    Window size value: 49717
    [Calculated window size: 198868]
    [Window size scaling factor: 4]
    Checksum: 0x1d4c [unverified]
    [Checksum Status: Unverified]
    Urgent pointer: 0
    [SEQ/ACK analysis]
Hypertext Transfer Protocol
    GET /weatherstation/WXCurrentObXML.asp?format=XML&ID=IONTARIO493&day=27&year=2017&month=6&Cache=fmiI580 HTTP/1.1\r\n
    Host: www.wunderground.com\r\n
    User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:54.0) Gecko/20100101 Firefox/54.0\r\n
    Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\n
    Accept-Language: en-US,en;q=0.5\r\n
    Accept-Encoding: gzip, deflate\r\n
    Cookie: ASC=1498583689:5; DT=1498582938:1159:ip-10-226-237-178; Prefs=FAVS:1|WXSN:1|PWSOBS:1|WPHO:1|PHOT:1|RADC:0|RADALL:0|HIST0:NULL|GIFT:1|PHOTOTHUMBS:50|; akacd_WU_NextGen=3676036408~rv=27~id=4f235cb5f4451466595bdc28a2c64fee\r\n
    Connection: keep-alive\r\n
    Upgrade-Insecure-Requests: 1\r\n
    \r\n
    [Full request URI: http://www.wunderground.com/weatherstation/WXCurrentObXML.asp?format=XML&ID=IONTARIO493&day=27&year=2017&month=6&Cache=fmiI580]
    [HTTP request 1/1]
    [Response in frame: 446]

[langenet]

Michael,

Thanks for trying, but it looks like it doesn't work either...

Jun-27 7:45:02 PM         mcsSprinklers    Download4 http://www.wunderground.com/weatherstation/WXCurrentObXML.asp?format=XML&ID=IONOTTAW25&day=27&year=2017&month=6&Cache=fmiI580 Error=The underlying connection was closed: An unexpected error occurred on a send.   

 Jun-27 7:45:02 PM         mcsSprinklers    Download3 http://www.wunderground.com/weatherstation/WXCurrentObXML.asp?format=XML&ID=IONTARIO493&day=27&year=2017&month=6&Cache=fmiI533 Error=The underlying connection was closed: An unexpected error occurred on a send.   

 Jun-27 7:45:02 PM         mcsSprinklers    Download2 http://www.wunderground.com/weatherstation/WXCurrentObXML.asp?format=XML&ID=IONTARIO608&day=27&year=2017&month=6&Cache=fmiI706 Error=The underlying connection was closed: An unexpected error occurred on a send.   

 Jun-27 7:44:29 PM         mcsSprinklers    Download4 http://www.wunderground.com/weatherstation/WXCurrentObXML.asp?format=XML&ID=IONOTTAW25&day=27&year=2017&month=6&Cache=fmiI14 Error=The underlying connection was closed: An unexpected error occurred on a send. 

Robert

[Michael McSharry]

While I would like to find a solution, I do not know what else I can do. It is not something that WU has imposed because current generation OSs are able to communicate. It points to something associated with the older OSs. It may be some exploit that is being protected at the WU server side that was available in earlier OSs, but now closed in the ones under their lifetime support period.

Don't know if there is any WU support available that could answer the question as to why your XP/IE6 setup cannot download the weather data.

[langenet]

I was wondering if it had anything to do with the User-Agent: Other.
I was going to test with USER_AGENT = "Mozilla/5.0"... Though, I'm not really sure if that would make a difference.

Robert

[Michael McSharry]

Download3 and 4 had it. Download2 had Other and keep-alive. I tried to cover all bases to find some combination that would work.

[langenet]

Ok... Thanks for trying. Time to upgrade the OS I guess...


Robert

[Michael McSharry]

I took another step to evaluate with wireshark on an XP vs. W7 machine. Initially looked and surprised to see the XP request is being made to port 443 rather than 80 as was the case for W7 so the transaction was attempting a secure handshake. Note of differences are below.

Frame
Length 62 vs. 66
Coloring Rule Name TCP SYN/FIN vs. HTTP
Coloring Rule String tcp.flags & 02 || tcp.flags.fin == 1 vs. http || tcp.port == 80 || http2
IP
Total Length 48 vs. 52
Identification 0x3d92 vs. 0x511c
TCP
Destination Port 443 vs. 80 .... XP is giving https transaction request
Window Size 65,535 vs. 8,192
Header Length 28 vs. 32

The standard used by .NET 2 and apparently through .NET 3.5 is for TLS protocol for secure transactions. I suspect that WU upgraded their server to TLS 1.1 protocol so XP / .NET 2.0 using TLS failed and the WU server closed the connection.

I tried to force TLS 1.1 or 1.2 and made some progress with .NET 3.5 where the initial transaction starts and able to get the GET requests to WU via port 80. Subsequently, however XP/.NET 3.5 again tried to handshake with https transaction and the server responded with protocol that was not understood by XP/.NET 3.5. Since .NET 4.0 is expected to use TLS 1.1 it may work better under .NET 4.0 / HS3. Give the attached a try.

I feel that I understand the problem now with a change in the WU server in the security protocol.