January 22, 2007 :: Email issues on Lightning

Jason · January 22, 2007, 01:17:34 PM

Greetings,

I've received quite a few emails in the past several days regarding inbound email issues on Lightning. It appears that all of these issues are for accounts on Lightning only at this point so I'm going to assume that there may be a larger issue on this specific server.

I wanted to start a thread here to provide a centralized area for questions/answers since this appears to be something that may be more global than individualized.

I will keep you posted on what we're doing here.

If you've emailed me already on this, I am using the information you provided already in this investigation. If you have further details, feel free to post them here, although I would recommend against posting personal email addresses.

Thanks,
Jason

slack · January 22, 2007, 01:28:49 PM

Jason, thanks for looking into this... a frustrating problem.

I have one friend who got a non-deliverable return receipt on a message and he will later forward the non-deliverable return receipt (it is on his home computer) -- so maybe that will help track it down.

Jason · January 22, 2007, 01:34:33 PM

Thanks. That will be great.

We'll go through everything piece by piece until we can narrow this down fully.

Jason · January 22, 2007, 07:36:30 PM

A "mini" update for those of you watching this thread.

We have done extensive searches on the email addresses that have been provided with issues. We don't see log entries for them. What this means is that those addresses are not being blocked by the spam blacklists we have in place. Rather, this means those emails are not even reaching the server.

Several people have forwarded me error (bounceback) email messages which have been very helpful. Those messages are mostly timeouts where the senders' servers were timing out when trying to connect to Lightning. This is leading me to something that may be beyond our server itself.

I have opened a ticket with the datacenter describing to them everything that's been occurring. I will keep you posted as we go from there.

We rebuilt Exim (the mailserver software) last week on the server after some of these issues were first reported so I'm starting to lean towards a network configuration that the datacenter may need to correct.

I'll keep this thread updated with everything I know as soon as I get it.

Thank you for your patience,
Jason

Jason · January 23, 2007, 08:49:30 AM

The datacenter has made an Exim configuration change on this server. Can everyone impacted see if their issues are corrected and let me know?

Thanks,
Jason

slack · January 23, 2007, 03:19:14 PM

Still a no go here Jason....

Jason · January 23, 2007, 03:26:42 PM

Quote from: slack on January 23, 2007, 03:19:14 PM
Still a no go here Jason....

Those users can't email you still? Same errors on their end with timeouts?

slack · January 23, 2007, 04:26:55 PM

Right, they still cannot email me -- I never got the "undeliverable message" copy for my sender last night, so I don't know if it is a time out issue or not.... but others report that they do not get anything back e.g. messages are black holed.

Jason · January 23, 2007, 05:34:21 PM

Quote from: slack on January 23, 2007, 04:26:55 PM
Right, they still cannot email me -- I never got the "undeliverable message" copy for my sender last night, so I don't know if it is a time out issue or not.... but others report that they do not get anything back e.g. messages are black holed.

Ok, thank you. I will escalate.

slack · January 24, 2007, 12:35:11 AM

Here is the return receipt:

----- Original Message -----
From: "Mail Delivery Subsystem" <MAILER-DAEMON@ms-smtp-02.texas.rr.com>
To: <jward001@elp.rr.com>
Sent: Sunday, January 21, 2007 3:26 PM
Subject: Returned mail: see transcript for details

> The original message was received at Fri, 19 Jan 2007 14:44:01 -0600 (CST)
> from rrcs-24-227-183-185.sw.biz.rr.com [24.227.183.185]
>
> ----- The following addresses had permanent fatal errors -----
> <slack@bloozbox.com>
> (reason: )
>
> ----- Transcript of session follows -----
> 451 4.4.1 reply: read error from bloozbox.com.
> <slack@bloozbox.com>... Deferred: Connection timed out with bloozbox.com.
> Message could not be delivered for 2 days
> Message will be deleted from queue
>

Reporting-MTA: dns; ms-smtp-02.texas.rr.com
Arrival-Date: Fri, 19 Jan 2007 14:44:01 -0600 (CST)

Final-Recipient: RFC822; slack@bloozbox.com
Action: failed
Status: 4.4.7
Remote-MTA: DNS; bloozbox.com
Diagnostic-Code: SMTP;
Last-Attempt-Date: Sun, 21 Jan 2007 16:26:24 -0600 (CST)

Jason · January 24, 2007, 11:33:06 AM

Thank you. This has been added to the ticket with the dc.

Just an update. This is a top priority of mine right now even though it seems like this has been ongoing for longer than I'd prefer.

I'm looking at a few different paths but am waiting for the dc's response on this email issue before I decide the next steps.

Thank you all for your patience. Feel free to continue to forward me errors and examples as you hear of them.

slack · January 24, 2007, 12:58:42 PM

Thanks Jason -- here is another one -- looks about the same:

----- Original Message -----
From: "Mail Delivery Subsystem" <MAILER-DAEMON@ms-smtp-02.texas.rr.com>
To: <jward001@elp.rr.com>
Sent: Wednesday, January 24, 2007 9:11 AM
Subject: Returned mail: see transcript for details

> The original message was received at Mon, 22 Jan 2007 09:50:16 -0600 (CST)
> from rrcs-24-227-183-185.sw.biz.rr.com [24.227.183.185]
>
> ----- The following addresses had permanent fatal errors -----
> <slack@bloozbox.com>
> (reason: )
>
> ----- Transcript of session follows -----
> 451 4.4.1 reply: read error from bloozbox.com.
> <slack@bloozbox.com>... Deferred: Connection timed out with bloozbox.com.
> Message could not be delivered for 2 days
> Message will be deleted from queue
>

--------------------------------------------------------------------------------

> Return-Path: <jward001@elp.rr.com>
> Received: from HMIS001 (rrcs-24-227-183-185.sw.biz.rr.com
> [24.227.183.185])
> by ms-smtp-02.texas.rr.com (8.13.6/8.13.6) with ESMTP id l0MFoChc023872
> for <slack@bloozbox.com>; Mon, 22 Jan 2007 09:50:16 -0600 (CST)
> Message-Id: <200701221550.l0MFoChc023872@ms-smtp-02.texas.rr.com>
> From: "Jeff Ward" <jward001@elp.rr.com>
> To: "'John Dodson'" <slack@bloozbox.com>
> Subject: FW: Band update
> Date: Mon, 22 Jan 2007 08:50:03 -0700
> MIME-Version: 1.0
> Content-Type: text/plain;
> charset="US-ASCII"
> Content-Transfer-Encoding: 7bit
> X-Mailer: Microsoft Office Outlook, Build 11.0.6353
> Thread-Index: Acc8CYgnOxXa30PcTrOpRz+nkA8AJQAAInmQAIy0n+A=
> X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.3028
> X-Virus-Scanned: Symantec AntiVirus Scan Engine
>

Jason · January 25, 2007, 08:24:34 AM

Ok, let's try this again.

Can I get some feedback on whether you're seeing an improvement in this mail situation?

Another adjustment was made by the dc overnight.

Jason · January 26, 2007, 08:02:28 AM

Quote from: Jason on January 25, 2007, 08:24:34 AM
Ok, let's try this again.

Can I get some feedback on whether you're seeing an improvement in this mail situation?

Another adjustment was made by the dc overnight.

Can someone let me know what you're seeing now?

I've reopened this ticket because I'm getting some errors on Dnsreport.com with timeouts so I'm thinking this is still an issue.

On a side note, I will be making an announcement this afternoon/evening about a change for Lightning that's going to resolve this regardless. So whether or not the datacenter fixes this right now (which I've given them an ultimatum this morning), I have a plan B in progress already. Details to follow on this forum (and via email to everyone) later today.

Jason · January 27, 2007, 04:30:05 PM

Is anyone able to test again to see how it looks now? We've done some big changes today.

- - -

Also, I sent out an "All Client Email" last night that mentions replacing this server. Please read here for details if you haven't yet:

http://www.charlottezweb.com/forums/index.php?topic=623.0

Regards,
Jason