Enable UTF8=ACCEPT if the IMAP server supports that.

[arnt]

With this tiny patch, Roundcube supports using arabic email addresses flawlessly. (Hardly anyone else does, but someone has to be first.)

Note that it's not necessary to check for hasCapability("UTF8=ACCEPT"), RFC5161 permits clients to request enabling whatever they want, even things the server does not support.

[arnt]

Uhm. I should note that I haven't tested the managesieve plugin etc., only sent/received/printed email.

[alecpl]

I'm not sure it's that simple, but first of all you should check UTF8=ACCEPT capability, not ENABLE.

Maybe there's another fix to your issues. As I understand the extension it is a convenience not a requirement. Could you provide imap debug log for what happens if you don't use UTF8=ACCEPT, and what happens when you do? And describe the issues in more detail? It could be in a separate ticket.

[arnt]

Hi,

thanks for your quick response.

ENABLE is the right thing to check. There are three cases:

1. ENABLE is not advertised. In this case, sending ENABLE is not legal and would trigger a BAD from the server.
2. ENABLE is advertised, but neither UTF8=ACCEPT nor UTF8=ONLY are. In this case sending ENABLE UTF8=ACCEPT is legal, see RFC 5161 section 3.1, and has no effect.
3. ENABLE and either UTF8=ACCEPT or UTF8=ONLY are advertised. In this case sending ENABLE UTF8=ACCEPT is legal and enables UTF8=ACCEPT, see the second paragraph RFC 6855 section 3.

Sending ENABLE is also the right thing to do, since without it the IMAP server cannot legally send non-ASCII addresses in e.g. a BODYSTRUCTURE response. With Roundcube, sending the ENABLE command is necessary to avoid seeing sender addresses of the form احمد <[email protected]>, see RFC 6858 section 2.1 for some other examples. This means that Roundcube requires the ENABLE in order to reply. Would you like some screenshots illustrating this in practice? Or protocol traces?

I don't mind adding the test, but it's not necessary. And, uhm, please don't tell me that I've misunderstood these RFCs. ;)

It's possible that Roundcube requires more. I mentioned I haven't tested the managesieve plugin yet, I suspect that'll need something. I'll be using it in March and April, so I'll know, and may send a patch for that.

FYI, this is only the second-smallest diff I've made. Gnus was the smallest, it required only two lines of code to comply with RFC6855. (These small diffs aren't simply an accident, BTW. RFC6855 was written with that goal in mind, to leverage the work people have done to accommodate just-send-8.)

[arnt]

I've made a copy of some protocol logs from a production server. I'll dig out example responses with and without ENABLE. It'll take a little time, I have some urgent problems I need to fix with other software on the same installation.

[arnt]

Come to think of it, I should probably elaborate on what UTF8=ACCEPT enables. There are quite different things, and people conflate them.

1. Suppose your email address is [email protected], and you want the ability to receive and read email from people such as عائشة@هتا.البحري. In that case, you need MTA support, IMAP server support and the current version of Roundcube, no patches necessary.
2. Suppose you want the ability to reply to mail from عائشة@هتا.البحري. In that case you also need this PR, and you probably should read/write Arabic ;)
3. Suppose you want Roundcube to host that arabic email address. In that case you probably need even more, depending on the server configuration. I'll get to that when I've thought more about it.

I sent this PR without waiting, because it solves 2.

[alecpl]

Ok, I see. On a non-supporting server I was able to reproduce this issue. If I APPEND a message with To: عائشة@xn--mgbf0g.xn--mgbcpq6g header, and then do a FETCH it returns To: [email protected].

That being said, I'm afraid that if we just enable the extension we might break other places, like e.g. folder names which are normally assumed to be in UTF7-IMAP, not UTF8.

[arnt]

Hi,

I'm still on my travels and haven't gotten the log data together. I return home on Friday. Typing this from an airport.

Folder names shouldn't be a risk; UTF8 and mUTF7 are used during transmission. The server has to accept both UTF8 and mUTF7 encodings of the same mailbox name and interpret them to refer to the same mailbox. (Or do you mean that Roundcube stores mUTF7 in its database?)

[arnt]

I'm home for more than a few days, finally.

Now that I have a couple of months experience with Roundcube+EAI in production I've seen some problems. I need to add a couple of things; if you don't mind I'll add them to this PR even though they don't really fit the headline.

[Neustradamus]

@arnt: Any news on your PR?

[arnt]

The problem that needs solving is one that I've been "solving" with a hack. Not good enough to push.

If you generate a message that uses EAI syntax, that message may use EAI syntax in its message-id too, which is fine. But later, that message-id may be copied into other messages, and the UTF8 can cause problems there.

IMO the message-id must always use an ASCII domain, something which is perhaps easier to hack than to do properly. I'll try a proper solution again and push again.

[arnt]

Oh, duh. A conditional $domain_part = idn_to_ascii… in gen_message_id is clean and sufficient.

Next then: unit tests. I'm afraid I didn't write any, just ran the code in production and saw that it worked. Do you have a README describing the testing environment phpunit needs for roundcube, or a script that creates a suitable environment?

[arnt]

I wrote a couple of tests now, and made gen_message_id safe against the problem I've seen.

This has been used in production for a while; it seems safe to merge AFAICT.

[arnt]

I'm onsite with actual users this week, and hearing about more issues. Fixes to come, in this PR or separately, not sure yet.

[jb4z]

@arnt This sounds very interesting and I'm looking forward to have it. Any news on this?