5.8 showing as sent, but not always arriving at destination

Hi,

Ok so that is confirmed now.

Looking around this forum, I believe that some other earlier versions of K9 had the same problems.

I’ve obtained access to the relevant mail logs for the period of time I had problems with K9 5.8*, and will post what ever might be relevant once I have downloaded them and had time to sort them out.

Paul

Thanks for that I’ll process the mail logs through later on, first, incase you can spot something relevant before I try another .apk as it may be inconclusive or indeterminate overall, and I don’t want my mail potentially mucked up at the moment please.

Ok, here are a couple of samples …

IP addresses / domains obstrifucated = dah.dah.

Un-sucessful from K9 5.8*

2021-08-03 07:13:22 1mAtIQ-002zkr-2s <= paul@dah.dah.info H=dah.dah..dyn.cust.vf.net.nz ([127.0.0.1]) [dah.dah.]:41276 I=[dah.dah.]:465 P=esmtpsa X=TLS1.2:ECDHE-RSA-AES128-GCM-SHA256:128 CV=no A=dovecot_plain:paul@dah.dah.info S=1200 id=75B12B67-8517-41D3-91F4-26D8A6418A5E@dah.dah.info T="Test from K9" from <paul@dah.dah.info> for joe@dah.dah.com record@dah.dah.com
2021-08-03 07:13:22 1mAtIQ-002zkr-2s => /dev/null <joe@dah.dah.com> F=<paul@dah.dah.info> R=fightspamHG T=**bypassed** S=0
2021-08-03 07:13:22 1mAtIQ-002zkr-2s => record <record@dah.dah.com> F=<paul@dah.dah.info> R=virtual_user T=dovecot_virtual_delivery_no_batch S=1360 C="250 2.0.0 <record@dah.dah.com> kAAeJOIyCWHwzgoA2xhqkA Saved"
2021-08-03 07:13:22 1mAtIQ-002zkr-2s Completed

Sucessful from K9 5.6*

2021-08-03 09:13:17 1mAvAS-004581-KE <= paul@dah.dah.info H=dah.dah..dyn.cust.vf.net.nz [dah.dah.]:42626 I=[dah.dah.]:465 P=esmtpsa X=TLS1.2:ECDHE-RSA-AES128-GCM-SHA256:128 CV=no A=dovecot_plain:paul@dah.dah.info S=1550 id=8A0E2D93-B82E-494A-90D1-E0429B94496A@dah.dah.info T="Trial from old K9 version 5.6 on phone" from <paul@dah.dah.info> for joe@dah.dah.com michael.too@dah.dah.info record@dah.dah.com
2021-08-03 09:13:17 1mAvAS-004581-KE Sender identification U=pauloinf D=dah.dah.info S=paul@dah.dah.info
2021-08-03 09:13:17 1mAvAS-004581-KE SMTP connection outbound 1627999997 1mAvAS-004581-KE dah.dah.info joe@dah.dah.com
2021-08-03 09:13:17 1mAvAS-004581-KE Sender identification U=pauloinf D=dah.dah.info S=paul@dah.dah.info
2021-08-03 09:13:17 1mAvAS-004581-KE Sender identification U=pauloinf D=dah.dah.info S=paul@dah.dah.info
2021-08-03 09:13:17 1mAvAS-004581-KE => michael.too <michael.too@dah.dah.info> F=<paul@dah.dah.info> R=virtual_user T=dovecot_virtual_delivery S=1725 C="250 2.0.0 <michael.too@dah.dah.info> SHfJCP1OCWEWyA4A2xhqkA Saved"
2021-08-03 09:13:17 1mAvAS-004581-KE => record <record@dah.dah.com> F=<paul@dah.dah.info> R=virtual_user T=dovecot_virtual_delivery_no_batch S=1719 C="250 2.0.0 <record@dah.dah.com> KHplDP1OCWEWyA4A2xhqkA Saved"
2021-08-03 09:13:18 1mAvAS-004581-KE => joe@dah.dah.com F=<paul@dah.dah.info> R=send_to_gateway T=remote_smtp S=2311 H=cm.websitewelcome.com [100.42.49.18] I=[dah.dah.] X=TLS1.2:AES256-GCM-SHA384:256 CV=yes DN="/CN=*.websitewelcome.com" C="250 2.0.0 AvATmpE8uMGeEAvAUm0JlB mail accepted for delivery"
2021-08-03 09:13:18 1mAvAS-004581-KE Completed

If you think that .apk you mentioned earlier has this covered then I will try it out for you.

Kindest regards,
Paul

Hi cketti,

Please test the APK linked in this comment, it might fix the issue

Ok on spec we gave that a go.

The apk froze my phone on first install attempt.

After a reboot, installed ok.

Unfortunately it did not fix the problem.

The two gmail accounts I sent to, did not receive their emails.

The troublesome Hostgator account, nor another that was working on 5.8 previously, didn’t get theirs at all.

I reverted back again to K9 5.6 for now, and successfully got through to both Hostgator accounts and the two gmail.com accounts using 5.6 again.

Hopefully those log entries I supplied can help?

Thanks,
paul

I’m guessing it’s a (stupid) spam filter not liking the message. What’s “fightspamHG”?

I found this (Change Helo/Ehlo address | Odoo):

I talked to hostgator about this. Apparently the Helo / Ehlo record of 127.0.0.1 is being blocked by their outgoing spam filters.

Checking the argument of the EHLO command seems to be an effective spam fighting measure when using SMTP. However, trying to do anything with this value when using authenticated SUBMIT (this is what a client like K-9 Mail is using to send messages via the user’s email provider) is quite pointless and bound to cause problems. Many mobile clients have no way of getting their (public) host name and/or IP address because they are behind NAT gateways. The SUBMIT server is in a much better position to do so (it sees the mobile device’s public IP address).

K-9 Mail 5.8xx is using the EHLO command with a “domain” value of 127.0.0.1. That seems to be the least bad option that works more often than it does not. I don’t want to make this value configurable because I consider SUBMIT servers that refuse to accept this value broken.

You seem to have the following options:
a) get your provider to fix their server
b) find another provider
c) use a different email client that works around your email provider’s broken rules

I’m not sure what “fightspamHG”, K9 5.6* doesn’t seem to trip it on Hostgator, while 5.8* does for some reason.

Seems to similarly affect inwards Gmail as well.

Thing is this could be going on and some people won’t realise it yet.

A simple web search suggests “fightspamHG” is a common cPanel setting, so it could appear on a number of different web host solutions.

Are there any new code page settings in the new K9 5.8* send mail at all (even a bad mix?), or is it all just utf8 by default?

P.S.

Your mentioning spam makes me wonder, is there anything different in the way K9 5.8* as client, is identifying itself in the headers to mail host servers, that now differs from K9 5.6* ?

Our postings seemed to cross when I replied from the phone :slight_smile:

It looks llike a simillar problem to what happened under K9 5.7* ?
A forum thread here on that pointed me to the K9 5.6* apk

What was done to fix that, or was it something different?

Cant you do it all as was working for K9 5.6? And which still works in K9 5.6* today?

Hostgator is an enormous market share …

What K-9 Mail 5.600 did wasn’t great either. See Always send IP instead of hostname in HELO/EHLO by Valodim · Pull Request #3798 · k9mail/k-9 · GitHub

Perhaps using something in the 172.16.0.0/12 range would elicit a less foolish response from poorly configured/managed servers.

Is this for mail dropped directly on gmail (as the configured “outgoing mail server”) or addressed to someone at gmail but dropped on your hostgator server?

Seems to similarly affect inwards Gmail as well.

I’m not seeing this with gmail.

“Inwards” for Gmail, in the sense that it’s being sent to it from K9 5.8* via Hostgator server sending it.

It is fickle sometimes in that mail can get through to some people, perhaps under Reseller accounts on Hostgator.

This is just flying a kite, one man I have mostly been able to send to under 5.8* is under our Reseller account, and 127.0.0.1 might actually make sense then, if that helps as a diagnostic?

I won’t be available to answer for a while – after midnight here now :slight_smile:

1 Like

It appears that this is due to hostgator foolishly filtering based on the EHLO on the (authenticated) submission port. Providers do need to have things in place in case of stolen credentials, but dumping mail because of:

EHLO [127.0.0.1]

isn’t anywhere in the range of “best current practices” when handling authenticated submission.

One last from me for the night …

Is this the same issue as this one refered to earlier here and on the tracker?
https://github.com/k9mail/k-9/issues/4428

Yes, that’s a discussion of this issue. I (for what it’s worth) think that cketti’s position is correct. Do what you want in terms of filtering on smtp/25, but on the submission/587 & 465 ports, which are (intended to be) authenticated, filtering on things like the EHLO is foolish and counter productive. In addition to privacy issues, the IPnumber or name that a mobile device or desktop machine presents has a low probability of resolving correctly (forward and reverse) so should (as is the standard practice) be ignored in an authenticated context. Presenting a fixed value is a reasonable approach.

That hostgator is silently dumping this mail is not standard practice and makes matters even worse. [If for some reason you don’t want to accept mail from your authenticated customer, tell them.]

1 Like

Ok, thre is a Hostgator “cases-support”: issue:

Case :: 34226747 - HostGator-HostGator-Email-Send/Receive Mail Issues
… Now on their system.

It may be worth pushing that forward from your end using that reference.

I wrote this to them—

Private Bin Link

For Hostgator technical, can this be fixed please? It stops us sending mail …

A quote from the K9 Email app developer “njeyaakili” on
5.8 showing as sent, but not always arriving at destination - #18 by njeyaakili

Also refering to this technical discussion …

Change HELO greeting · Issue #4428 · k9mail/k-9 · GitHub

"Do what you want in terms of filtering on smtp/25, but on the submission/587 & 465 ports, which are (intended to be) authenticated, filtering on things like the EHLO is foolish and counter productive. In addition to privacy issues, the IPnumber or name that a mobile device or desktop machine presents has a low probability of resolving correctly (forward and reverse) so should (as is the standard practice) be ignored in an authenticated context. Presenting a fixed value is a reasonable approach.

"That hostgator is silently dumping this mail is not standard practice and makes matters even worse. [If for some reason you don’t want to accept mail from your authenticated customer, tell them.]

1 Like

Looking around the web it seemed that some other hosting services are doing the same sorts of messy things.

I doubt that we can clean them all up.

Someone here had suggested adding a permission to K9 and sending the IP-domain-etc for the current connection with the email, even though of course it is not going to resolve backwards always, but it would still be valid going forward?

And would it clear us out of this mess overall?

My message to Hostgator got a lengthy reply from a Tier Administrator which I answered in this way (I’m waiting for another response) …

Thank you for patiently explaining that.

Yes you are right, the K9 developers have wanted to provide some sort of client privacy protection as you have shared.
I know from DroidScript, that if I query my phone 127.0.0.1 is one of the valid IP addresses it could show for it.

For their part as you will have seen, the K9 developers state that a duly authenticated user should not have to worry about all this, asking why would a logged in authenticated user be filtered as potential spam?

As a User I can not understand why such mail would die unnotified? If it is effectively (from the User’s persoective) being “rejected”, wouldn’t it be helpful for it to be returned to “sender” with an appropriate message added to the top of it, rather than just going into a black hole?
I lost several mission critical email moments, from sending mail that successfully to me was nonetheless showing in the Sent folder, but never got to destination, and was effectively (from my perspective) rejected without notification by the Hostgator servers.

I could in such circumstances understand unauthorised mail just being sent to black holes, but authenticated mail?

If authenticated - but apparently viewed as spam mail (possibly from stolen credentials) were rejected and bounced back with a message would that not warn the genuine User that something was up?
And if mail attempting to be sent actually were from a genuine User, would this not warn them that their genuine mail was failing, and why, so that they could then follow up as necessary?

Rather than unknowingly to them (people like me) their mail show as sent, but not be so?

Are there any suggestions for what can be done to resolve this going forward please?

Kindest regards,
Paul

1 Like

Here’s the reply I received …

Hello Paul,

Thank you for contacting HostGator. I would like to address your concerns separately.

“For their part as you will have seen, the K9 developers state that a duly authenticated user should not have to worry about all this, asking why would a logged in authenticated user be filtered as potential spam?”

The developers are only putting forth their opinion on how a mail server should operate, from their perspective as a developer (and based on how they think email should operate). Simply because a user is authenticated does not, in any way, mean they are not going to send spam. Whether or not a user is authenticated is not a viable method for determining their intent post-authentication, since spam being sent from compromised email accounts has been an issue for decades. We reject tens of thousands of spam emails everyday that are sent from authenticated email clients that have been compromised. I understand their reasoning for using a spoofed IP address during the EHLO transaction, although I believe other methods could be used in an obfuscation effort.

However, for the purpose of what is occurring in this case, the email server itself has no way of determining what you (or anyone else) will be doing after they login, so we must implement at least some spam-fighting techniques at every step of the email transaction process. Many mail providers implement IP checks during the transactional process of sending emails, that way if an IP address may be in a blacklist, the IP can be blocked from ever being able to send an email in the first place, stopping the problem before it starts.

“If authenticated - but apparently viewed as spam mail (possibly from stolen credentials) were rejected and bounced back with a message would that not warn the genuine User that something was up?”

CPanel already has this functionality built-in by default and will clearly send end users notifications if their email accounts hit certain outgoing email thresholds and still, the vast majority of users ignore these notifications entirely, despite the fact that these notifications not only tell the user something wrong is going on with their account, it includes the entire email that was sent but then blocked, as well.

We also have other measures in place that bounce verified spam back to end users, explaining why the emails were flagged as spam, however many users ignore those as well. Even when emails are bounced back from email recipients with rejections that make it very clear that spam was/is sent, we have many, many end users that completely ignore those notifications as well.

Over the last 15 years that I have worked at HostGator, I’ve seen VPS and dedicated servers where users have sent tens of millions of spam emails over the course of months – complete with literally tens thousands of bounce backs being sent back to the originating server – and users ignore those notifications entirely, and then they get mad at HostGator for allowing their emails to be blocked and/or to allow this spam to ever occur in the first place when we (HostGator) didn’t know it was occurring in the first place.

The issue is rarely with whether or not an end user received some kind of notification that their email was blocked, for any reason, the issue we frequently encounter is whether or not the end user knows what the notification is for and what to do (or not do) with that information.

“Are there any suggestions for what can be done to resolve this going forward please?”

Unfortunately, unless you decide to switch email applications away from K9, or utilize another mail provider, this issue will remain unresolved on our network, simply due to the choices the developers of K9 are making, in their fight for user privacy. While RFC5321 states that a literal address local can be substituted in place of the user’s own IP/FQDN during the EHLO process, spoofing that information to reflect 127.0.0.1 is a purposeful choice the developers are making, rather than finding other options for generating the IP/FQDN, such as creating a hash of random data as an FQDN which is based on both random data and the user’s existing hostname. Perhaps they have tried these things in the past, I fully admit that I do not know, but I do know that other mobile email clients do not have this issue