5.8 showing as sent, but not always arriving at destination

Ok, thre is a Hostgator “cases-support”: issue:

Case :: 34226747 - HostGator-HostGator-Email-Send/Receive Mail Issues
… Now on their system.

It may be worth pushing that forward from your end using that reference.

I wrote this to them—

Private Bin Link

For Hostgator technical, can this be fixed please? It stops us sending mail …

A quote from the K9 Email app developer “njeyaakili” on
5.8 showing as sent, but not always arriving at destination - #18 by njeyaakili

Also refering to this technical discussion …

Change HELO greeting · Issue #4428 · k9mail/k-9 · GitHub

"Do what you want in terms of filtering on smtp/25, but on the submission/587 & 465 ports, which are (intended to be) authenticated, filtering on things like the EHLO is foolish and counter productive. In addition to privacy issues, the IPnumber or name that a mobile device or desktop machine presents has a low probability of resolving correctly (forward and reverse) so should (as is the standard practice) be ignored in an authenticated context. Presenting a fixed value is a reasonable approach.

"That hostgator is silently dumping this mail is not standard practice and makes matters even worse. [If for some reason you don’t want to accept mail from your authenticated customer, tell them.]

1 Like

Looking around the web it seemed that some other hosting services are doing the same sorts of messy things.

I doubt that we can clean them all up.

Someone here had suggested adding a permission to K9 and sending the IP-domain-etc for the current connection with the email, even though of course it is not going to resolve backwards always, but it would still be valid going forward?

And would it clear us out of this mess overall?

My message to Hostgator got a lengthy reply from a Tier Administrator which I answered in this way (I’m waiting for another response) …

Thank you for patiently explaining that.

Yes you are right, the K9 developers have wanted to provide some sort of client privacy protection as you have shared.
I know from DroidScript, that if I query my phone 127.0.0.1 is one of the valid IP addresses it could show for it.

For their part as you will have seen, the K9 developers state that a duly authenticated user should not have to worry about all this, asking why would a logged in authenticated user be filtered as potential spam?

As a User I can not understand why such mail would die unnotified? If it is effectively (from the User’s persoective) being “rejected”, wouldn’t it be helpful for it to be returned to “sender” with an appropriate message added to the top of it, rather than just going into a black hole?
I lost several mission critical email moments, from sending mail that successfully to me was nonetheless showing in the Sent folder, but never got to destination, and was effectively (from my perspective) rejected without notification by the Hostgator servers.

I could in such circumstances understand unauthorised mail just being sent to black holes, but authenticated mail?

If authenticated - but apparently viewed as spam mail (possibly from stolen credentials) were rejected and bounced back with a message would that not warn the genuine User that something was up?
And if mail attempting to be sent actually were from a genuine User, would this not warn them that their genuine mail was failing, and why, so that they could then follow up as necessary?

Rather than unknowingly to them (people like me) their mail show as sent, but not be so?

Are there any suggestions for what can be done to resolve this going forward please?

Kindest regards,
Paul

1 Like

Here’s the reply I received …

Hello Paul,

Thank you for contacting HostGator. I would like to address your concerns separately.

“For their part as you will have seen, the K9 developers state that a duly authenticated user should not have to worry about all this, asking why would a logged in authenticated user be filtered as potential spam?”

The developers are only putting forth their opinion on how a mail server should operate, from their perspective as a developer (and based on how they think email should operate). Simply because a user is authenticated does not, in any way, mean they are not going to send spam. Whether or not a user is authenticated is not a viable method for determining their intent post-authentication, since spam being sent from compromised email accounts has been an issue for decades. We reject tens of thousands of spam emails everyday that are sent from authenticated email clients that have been compromised. I understand their reasoning for using a spoofed IP address during the EHLO transaction, although I believe other methods could be used in an obfuscation effort.

However, for the purpose of what is occurring in this case, the email server itself has no way of determining what you (or anyone else) will be doing after they login, so we must implement at least some spam-fighting techniques at every step of the email transaction process. Many mail providers implement IP checks during the transactional process of sending emails, that way if an IP address may be in a blacklist, the IP can be blocked from ever being able to send an email in the first place, stopping the problem before it starts.

“If authenticated - but apparently viewed as spam mail (possibly from stolen credentials) were rejected and bounced back with a message would that not warn the genuine User that something was up?”

CPanel already has this functionality built-in by default and will clearly send end users notifications if their email accounts hit certain outgoing email thresholds and still, the vast majority of users ignore these notifications entirely, despite the fact that these notifications not only tell the user something wrong is going on with their account, it includes the entire email that was sent but then blocked, as well.

We also have other measures in place that bounce verified spam back to end users, explaining why the emails were flagged as spam, however many users ignore those as well. Even when emails are bounced back from email recipients with rejections that make it very clear that spam was/is sent, we have many, many end users that completely ignore those notifications as well.

Over the last 15 years that I have worked at HostGator, I’ve seen VPS and dedicated servers where users have sent tens of millions of spam emails over the course of months – complete with literally tens thousands of bounce backs being sent back to the originating server – and users ignore those notifications entirely, and then they get mad at HostGator for allowing their emails to be blocked and/or to allow this spam to ever occur in the first place when we (HostGator) didn’t know it was occurring in the first place.

The issue is rarely with whether or not an end user received some kind of notification that their email was blocked, for any reason, the issue we frequently encounter is whether or not the end user knows what the notification is for and what to do (or not do) with that information.

“Are there any suggestions for what can be done to resolve this going forward please?”

Unfortunately, unless you decide to switch email applications away from K9, or utilize another mail provider, this issue will remain unresolved on our network, simply due to the choices the developers of K9 are making, in their fight for user privacy. While RFC5321 states that a literal address local can be substituted in place of the user’s own IP/FQDN during the EHLO process, spoofing that information to reflect 127.0.0.1 is a purposeful choice the developers are making, rather than finding other options for generating the IP/FQDN, such as creating a hash of random data as an FQDN which is based on both random data and the user’s existing hostname. Perhaps they have tried these things in the past, I fully admit that I do not know, but I do know that other mobile email clients do not have this issue

The generally used approach to dealing with stolen credentials is in the area of rate-limiting and the like. If an outbound delivered message is reported back as spam the MSP can then deal with the specific user. Most don’t do simplistic spam filtering, though I suspect do take action on messages that rate high on spam scoring.

Their approach of dumping these (and who knows what others):

1mAtIQ-002zkr-2s => /dev/null joe@dah.dah.com

is really not the way to treat a customer. If their problem is that customers can be confused by notifications they then need to work on that, not not notify the customer.

Find a MSP that actually knows what it’s doing, both technically and from a place of reasonable customer expectations.

Are there any suggestions for what can be done to resolve this going forward please?

So, what can we do about this. I just now found out that all the emails I’ve been sending via my hostgator account since the update never made it to their destinations, and that’s been over 50 emails. I don’t have time to screw around with this crap or do debug traces. Is there a solution coming soon? I love k9 mail and have been using it for ages, but if I can’t trust it to work, what choice do I have but to find another client.

Thanks,
-m

The choices have been spelled out already. Find a new email client or find a new email provider.

I just want to point out again that your provider accepted emails for delivery and then dropped them without telling you (or your email client) about it. If I were you, I would definitely switch email providers.

1 Like

I think you need to find a new MSA. If Hostgator is dumping authenticated submissions - “1mAtIQ-002zkr-2s => /dev/null” - because they don’t like the IPnumber on the EHLO without bothering to tell the client (or returning it to the authenticated user), who knows what other excuses they may have for not delivering your mail. This isn’t really a mail client issue. I simply wouldn’t trust them.

Hi Mike,

Hostgator is not the only mail server effected.

There are two different things happening here at once.

K9 (version 5.6) was working fine in this, but the developers made some “improvements” in preparing their entirely new version of K9 (5.8) which have broken some things.
In this instance, one of their “improvements” is that in real-world terms, K9 can now strangely look more like a spammer’s utility to a mail server than is at all necessary in the circumstances.

The K9 developers have even been given advice by Hostgator on how they could better handle the issue.

I have had lengthy dialogues with Hostgator even in writing.
I also would like to see Hostgator and other mail services give better messaging in the circumstances, but understand that such a response method could also flood a User’s account where they had lost their authentication details and password to cyber crime—if Hostgator and other such mail services just too simply followed through on what’s been demanded of them here.

The effect on Users must be paramount here.

It is also I believe the prime duty of developers of apps like K9 5.8.

Hoping finally for a better new day in all this.

Paul

Switching providers for me is not an option. I’ve been using them for a decade and have too many hosted websites there. Unless I can find a separate outbound email provider that will work with this update I’m stuck looking for a new mail client, unfortunately.

Just think what would happen if Thunderbird had this problem. Luckily they don’t. If they did you know it would get fixed pretty damn fast.

It’s fine for a mailserver to reject e-mail it believes is spam, it’s not OK to accept it and then silently drop it.

2 Likes

This exact issue brought me to this forum. Silently burning mail without notice or warning is nuts. I have to resend two weeks of messages through a non-HostGator account. There’s already people mad at me thinking I’ve been ignoring them.

Static side-loaded K9 5.6 (no updates would be available) still works fine with Hostgator, and other email services as well.

K9 5.8 broke what was working fine.

Till it’s fixed in K9 5.8* we’re using:

  1. Export all your settings from K9

  2. Then import into…

OpenMail
https://play.google.com/store/apps/details?id=com.deependhulla.opemail

Hopefully that guy doesn’t break it all as well!

Kind regards,
Paul

Thanks! I have already tried FairMail, but that doesn’t seem to work either. Still can’t believe HostGator would just vanish messages without kickbacks or some kind of warning.

There are good reasons related to spamming techniques that complicate all this—not right to openly discuss here; OpenMail is very similar to K9, was forked from an earlier version of K9, and so far works very well for us with Hostgator, and most likely will work for other like Hosting services as well.

If Hostgator, or like services, notified all such emails that fall into the unnecessary changes now made in K9 5.8, and someone’s credentials were stolen through cyber crime, or whatever, you would have your inbox flooded with notifications.

So Hostgator, and like service provides, are trying to deal with a real world problem and protect you, not expecting such a change in a previously successful app as has just happened in K9.

Why change K9 to cause this problem is more the question!
Hostgator have suggested to K9 what they can legitimately do, and they don’t want to.

So now everyone has painted themselves into silly corners and nothing gets fixed.

OpenMail is the same as K-9 5.3 and won’t be updated either.

With Fairmail, the default that they put out on the EHLO is “dummy.faircode.eu” which is a CNAME for cloudfront addresses. If you check the “use local IP address…” box it puts out “127.0.0.1”, which is what K9 now uses (and hostgator will send to /dev/null). It is slightly more robust than K9 in that it lets the user override these and enter something else. It appears to never put out the client’s real ipnumber/hostname, which is the privacy issue that K9 is addressing too.

Based on the responses above from hostgator it appears that they don’t trust their authenticated users from the get-go. How a site handles MTA-MTA spam is one thing, how they handle (authenticated) MUA-MSP mail should be something different. With the approach they seem to take, it would be hard to feel comfortable that any given message that they accepted from your MUA would actually be delivered out, or that you’d be notified if it isn’t.

There are ways to deal with stolen credentials or bogus accounts - free mail services like gmail and hotmail have to deal with this as part of their basic approach to business - but always treating all accounts as suspect isn’t the standard method. With hostgator, I’m guessing that getting past the “EHLO 127.0.0.1” issue is only the start of their obstacle course, so good luck - regardless the client you use.

1 Like

The idea that the inbox would be “flooded with notifications” seems ridiculous. A legitimate user would detect a problem with their client right away. In the case of a hacked account the provider could block the account after a handful of failed send attempts and not “flood” the inbox with notifications. If your credentials have been stolen, I think you can live with cleaning up a couple of messages after access to your account has been restored. I’d definitely prefer that over my provider silently dropping messages and not notifying me that there’s a high chance my account has been compromised.

In this particular case they know they’re not going to deliver the email even before the email client uploads the message to the outgoing mail server (EHLO is the first command a client sends to the outgoing mail server). They could simply not accept the outgoing message and return an error code. The email client would display that error to the user and there would be no “inbox flooded with notifications”.

Let’s look at that.

While RFC5321 states that a literal address local can be substituted in place of the user’s own IP/FQDN during the EHLO process, spoofing that information to reflect 127.0.0.1 is a purposeful choice the developers are making, rather than finding other options for generating the IP/FQDN, such as creating a hash of random data as an FQDN which is based on both random data and the user’s existing hostname.

So their suggestion is instead of “spoofing that information to reflect 127.0.0.1” we should just make something up that looks like a hostname. Maybe that would satisfy their sophisticated spam fighting algorithm. But in the past using hostnames has resulted in a lot more error reports from users than using 127.0.0.1 has since we changed it.

2 Likes

What about to create an option to allow to customize it for specific accounts - not a global settung. Default is current behavior.
And this option should clearly state that this should only be changed in case of troubles with default setting.

1 Like