Built-in TLS Decryption

[emanuele-f]

At the current status, PCAPdroid requires an external proxy in order to decrypt TLS packets. On the other hand, a built-in solution would provide the following advantages:

• No need to have a PC to decrypt TLS
• PCAPdroid could show the decrypted payload directly into the app. This is related to Add basic payload information #38.
• PCAPdroid could produce a decrypted decryptable PCAP

Requirements

• Should support TLS 1.3
• Should not impact the main PCAPdroid development. TLS decryption is a secondary topic in PCAPdroid, so this feature should be supported by an active third party
• Should comply to the F-Droid policy (e.g. do not bundle any binary)
• The solution should be light, primary in terms of device resources (cpu, memory) and possibly in terms of dependencies and APK size

Candidates

• LittleProxy-mitm/LittleProxy is an HTTP proxy. It fits the purpose of a ready-to-use library, however depends on netty, which is big dependency.
• nitmproxy is a self contained SOCKS/HTTP proxy also using netty. SOCKS offers the possibility to decrypt non-HTTPS traffic.

Other approaches:

• NetBare: unmaintained, requires heavy modifications to only extract the proxy code, does not support TLS 1.3, it has the advantage of using java.nio for networking, which fits the minimal dependencies requirement
• any native C/C++ library could be integrated

User Experience

Users will have an option to enable the built-in TLS decryption by a toggle. The old mitmproxy option will be replaced with a more generic ability to use socks5 proxy, which is independent from the TLS decryption. Once enabled, the TLS decryption will happen under the hoods; android proxy settings will be untouched. It will have the following effects:

• In the connections view, the TLS connections data will be replaced by decrypted protocol data, e.g. HTTPS will be decrypted and the HTTP request/reply will be shown
• The PCAP will also contain the decrypted payload data in place of the original TLS data The traffic could be dumped in the PCAPNG file format, which contains a "Decryption Secrets Block" to store the decryption keys.

In essence TLS will be decapsulated and its inner data shown.

Roadmap

The built-in decryption is now implemented via the PCAPdroid-mitm addon.

• Implement socks5 client in PCAPdroid (Can you add a function to transfer all TCP data to the remote mitmproxy (or other proxy)? #44)
• Build the mitm proxy addon
• Verify that embedded mitm proxy properly proxies the packets received by the socks5 interface
• Show the decrypted connection in PCAPdroid
• Report rejected certificates warnings from mitmproxy in the app
• There are some TLS connections which are not decrypted, possibly fix/report to the user why this happens
• Provide certificate installation via Intent on Android < 11
• Harden the proxy via the SOCKS5 authentication
• Fix export of SSLKEYLOG
• Improve matching of HTTP request/replies in the UI (e.g. highlight row when clicking "jump" icon, or sort connections)
• Documentation, updates docs on methods to bypass untrusted user certs (on Android 7, e.g. magisk trust user certs, bromite chrome://flags), pinning (LSposed module sslunpinning) and Certificate transparency (avoid using a system cert or use magisk hide) - see the docs
• Suggest users to disable Autostart or similar software which can prevent the mitm addon execution

[emanuele-f]

With the purpose of minimizing the dependencies, here is a quick overview of the LittleProxy-mitm depedencies (max depth 3):

- koh-osug/LittleProxy-mitm
  - bcpkix-jdk15to18
  - bcprov-jdk15to18
  - conscrypt-openjdk-uber
  - commons-io
  - slf4j-log4j12
  - mrog/LittleProxy
    - guava
    - commons-lang3
    - netty-all
    - barchart-udt-bundle (UDT transport protocol)
    - slf4j-api

@koh-osug can you please clarify the need for the additional dependencies of LittleProxy-mitm over LittleProxy? E.g. jdk15to18 and conscrypt. Regarding the project structure, I think it's better to keep LittleProxy-mitm in a separate code base, unless it requires a tight integration with PCAPdroid.

[koh-osug]

• I haved removed the commons-io dependency.
• conscrypt-openjdk-uber must be excluded when using the library under Android, it is only required in Java-SE, Android is already using this internally as crypto provider.
• bcpkix-jdk15to18 and bcprov-jdk15to18 are needed for creating the MITM certificates.
• Under Android the barchart-udt-bundle dependency must be excluded, it is optional and not supported and not needed. I have patched the latest version of littleproxy to make it optional under Android with a merge request and it was already merged.
• I guess, I cannot influence the LittleProxy maintainer to remove guava or commons-lang3, these 2 libraries are convenient, but usually only a subset of the functions are needed.

I will check with my Android test project how much is left after using the Android shrinker. Usually almost all from the utilitiy libraries and major parts of bouncy castle should get removed resulting in just a few HD floppy disks.

[emanuele-f]

Great! I think LittleProxy should be run as a thread in PCAPdroid and communicate with it via the socks5 interface (will I will take care of the socks5 migration asap). This will allow PCAPdroid to carry on the TLS decryption asynchronously while processing the network packets in parallel.

I've added a roadmap for this feature in the issue description above, we can work on a branch until it is complete. You can proceed with the integration. Once the embedded LittleProxy works correctly, I will take care of the final step to show the decrypted data in PCAPdroid/in the PCAP. For this I will need a call API in LittleProxy-mitm to read the decrypted data back in PCAPdroid. The API should expose: the decrypted data byte[], the direction of the data (from/to the internet) and the connection identifier (the 5 tuple, or at least proto, dst_ip, src_port, dst_port)

[koh-osug]

I have checked the sizes after shrinking:

• BouncyCastle is just using 261 kb
• guava was reduced to 146 kb
• netty is 812 kb
• LittleProxy-mitm itself is 71 kb

[koh-osug]

Filtering the HTTP request seems to be possible with LittleProxy. I hope the filter is the one seeing the plain text. How will you map the plain text to the original requests? I can record the plain text, but somehow then later you have to know which plain text corresponds to which encrypted text.

[emanuele-f]

I have checked the sizes after shrinking

1.3 MB is a good result, I think we can settle with this. Great work!

Filtering the HTTP request seems to be possible with LittleProxy. I hope the filter is the one seeing the plain text. How will you map the plain text to the original requests? I can record the plain text, but somehow then later you have to know which plain text corresponds to which encrypted text.

The idea is the following: when LittleProxy is started I will register a callback (via a java interface). Whenever the proxy decrypts a packet, it will call the callback with the decrypted data and the connection information (original 5 tuple). With the 5 tuple I can retrieve the original connection and set the plain text data.

In b029895 I've implemented the SOCKS5 proxy client. This replaces the old "tunnel" mode. You should now be able to redirect the TCP packets to LittleProxy by using SOCKS5.

Edit: another step is required in order to make this work with an embedded LittleProxy. Since LittleProxy will make connections on his own, we must exclude these connections from the VPN, otherwise they will go back into the VPNService in a loop. To do this, the VpnService protect api must be called on every socket open by LittleProxy. In order to keep LittleProxy clean, I'd suggest to do this by registering a callback at LittleProxy startup via a java interface. Of course I'm assuming that LittleProxy provides an api to extract the raw socket from netty.

[koh-osug]

I will have a look into this. My first concern is if LittleProxy is actually supporting SOCKS5. I'm not familiar with the protocol, but what I see is that it needs a HTTP CONNECT call to trigger the proxy.

[koh-osug]

I'm investigating this one as alternative: https://github.com/chhsiao90/nitmproxy
It supports socks and can also be used for SMTP / IMAP with TLS then.

[koh-osug]

I had an issue with nitmproxy because the addresses received from SOCKS have been IP addresses, no hostnames anymore. I added a handler to extract the Server Name Indication and this looks good now. I created a fork and check its stability. LittleProxy might be maintained, but it is focusing too much on HTTP, nitmproxy supports more protocols and has the MITM feature is included. I have tested in under Linux and by connecting from Android to it.

[emanuele-f]

Have you tested it with TLS 1.3 and HTTP2? A SOCKS proxy opens up more possibilities so I also encourage this choice.

[koh-osug]

I have tested the page "https://tools.keycdn.com/http2-test" with FF usign the HTTP2 protocol. The web page is working.
I used the web page https://resources.duda.co/tls-1-3 to check for TLS 1.3. This was also working.
Maybe some unit test could be added, too.

[emanuele-f]

Great! Please provide a pull request/branch to review when ready! If you want a more interactive technical discussion you can reach me on the telegram group

[koh-osug]

Sure, will do. I have an issue with a specific domain. I'm in contact with the developer and I'm analyzing this. It is not a problem in a PC environment, but the same code is not working on Android. Most likely the Android SSLEngine stack is behaving differently. But the website seems to use a strange combination of TLS protocols, I could not detect this with any other web page.

[koh-osug]

The Android internal SSL Engine seems to have some issues with TLS 1.3 and ALPN. I have patched netty to be able to use the BouncyCastle TLS implementation which is working for the above mentioned issue. I will wait until this gets accepted to have it available as published dependency.

[koh-osug]

The nitmproxy can now transparently detect HTTP/HTTPS and supports also at least Android 8. I have also some older phones which I can check. A new API was added to plugin loggers I will add an AIDL interface around this so that PCAPdroid can call this.

[emanuele-f]

The android emulator is a convenient tool to test stuff on different SDK versions and devices. I see that the API is HTTPS oriented, does it provide a way to handle non HTTPS traffic? Also I think it could be better to run the proxy as a thread in PCAPdroid, to save IPC complexity and overhead.

[kaoh]

OK, not having it as separate app makes it simpler. Non HTTP is still on the TO DO list.

[qkmaosjtu]

Hi!
I am very excited to see that "Built-in TLS Decryption "!
In addition to supporting TLS1.3, have you considered supporting the quic protocol? It will be very common in the future.

[koh-osug]

Whatever is supported by conscrypt using the Java SSL engine can be supported. If this is not on the roadmap of both support will only be possible with external SOCKS based proxy tools supporting quic.

[qkmaosjtu]

@emanuele-f Hi emanuele! Any update on this topic?

[koh-osug]

The necessary upstream project nitmproxy is now supporting non HTTP data, I'm currently testing it. If it works, a data logger must be still added, but I then we are almost there.

[emanuele-f]

@koh-osug any update on this? Can I help with the tests?

[kaoh]

Sorry, taking longer than expected. I tested SMTP, IMAP and HTTPS. Looks good. You can test the latest version from it from the command line, too: https://github.com/chhsiao90/nitmproxy
But it was necessary to include a dependency to Conscrypt. Android is using it internally, but only later Android versions are supporting ALPN which is necessary for HTTP/2. May idea to use BouncyCastle does not work, because BouncyCastle does not supporting TLS renegotiation which is used by some pages. This means that if all binaries of Conscrypt are included 8 MB are added. If different versions are created, e.g. supporting x86, armv7, armv8 as separate packages only 2,x MBs should be needed in addition.
The loggers might not yet be good enough for the purpose of PCACdroid, see HTTPListener and FowradListener, used for non HTTP. The request, the response, the data are logged with different callbacks, it might be problematic to assemble it again, I assuem to need to source, destination IP and port together with the data in one chunk.

[emanuele-f]

PCAPdroid has minSdkVersion 21, on which android version the built-in Conscrypt provides ALPN support (links appreciated)? The loggers callbacks can be adapted at any time (e.g. via a fork), that's not a problem.

Ps. Here is the link to for BouncyCastle missing TLS renegotiation support: bcgit/bc-java#593

[kaoh]

The APLN support requires a backport from a later Java 9 version to Android (OpenJDK 9+181). The important method is getApplicationProtocol. It is still missing in Android 9 but exists in Android 10, i.e. all version below Android 10 have to use a bundled Conscrypt version.
Maybe it would be possible to remove support for HTTP/2 from nitmproxy for Android versions < 10, I have not yet tested it, because the project is focusing on Desktop Java so far.

[emanuele-f]

Thanks for the clarification. 2 MB overhead is ok, but 8 seems too much to me. Anyway, please note that if BouncyCastle is integrated into PCAPdroid, it must meet the f-droid policy, so it is either built as a submodule or it is downloaded from a trusted repository.

[kaoh]

I have raised a pull request in nitmproxy: chhsiao90/nitmproxy#73 to provide the connection context with the log handler. source/dest IP + port + transmitted data would be sufficient, I assume?

[kaoh]

The nitmproxy maintainer seems not to handle this. So, I started on my own, I think I'm on the right track now, actually it seems simple, let's see how it works.

[emanuele-f]

I'm looking again at this problem as this feature is something users are demanding. The Java libraries evaluated so far are not actively developed and in my opinion they are not mature enough to carry on this task.

The only tool which I trust can do the job is mitmproxy. In #123 I've implemented a plugin which can pass the HTTP and websockets decrypted data to PCAPdroid. My idea to implement the built-in decryption is to bundle mitmproxy with its dependencies so that PCAPdroid can start/stop it at will and receive the decrypted data. This may either be implemented directly in PCAPdroid or into a separate app. The only obstacle to this is python. It seems like building python for Android is not easy and most projects on this matter are now abandoned.

The most promising one seems to be chaquopy, although closed source. I've start experimenting with it but faced dependency issues. Another candidate is python-for-android.

In short, I'm willing to implement a very basic app with mitmproxy which can be controlled by PCAPdroid. I'm opting for a separate apk right now both for convenience and because in my tests bundling python with mitmproxy requires about 10 MB of overhead per ABI (so x4) in the apk size.

[emanuele-f]

With 6f42015 the built-in decryption is now integrated. The decryption requires the installation of the PCAPdroid-mitm addon as a separate apk. I've decided to separate it from the PCAPdroid app because of the size (the addon apk is about ~36 MB) and because it relies on the third-party chaquopy gradle plugin, which is closed source and it has build-time dependencies which complicate the build.

This is just an initial integration and there are a lot of things to improve.

[kaoh]

These are good news. Is the chaquopy approach raising any security concerns? I.e. because it is closed source, is it safe to used it without any risk of hidden code?

[emanuele-f]

Is the chaquopy approach raising any security concerns? I.e. because it is closed source, is it safe to used it without any risk of hidden code?

These concerns are legit for any closed source software. chaquopy is quite a complicate plugin so it's not easy to review it, but given that it widely known, it has a public github page for issues and a clear monetization plan, I think we can trust it. One drawback though is that the license code cannot be make public, so people trying to build the PCAPdroid-mitm module from source will run in "demo" mode.

In short, it does its job and it does it well, although it's not open source. Unless other equivalent foss solutions are implemented, this is what we have now and I'm very satisfied with the result.

[kenduq]

So for now, is there any option to produce decryptable or decrypted pcap file for analysis in wireshark?
I tried to install PCAPdroid-mitm app on my emulator but emulator can not see the app. Install was via adb install .apk

[joksas]

Thank you for this useful package!

I have a question which, I guess, is similar to the one by @kenduq. How does one decrypt the PCAP file with the help of the new plugin? I installed the plugin but I couldn't see any changes in the app itself. Are decryption keys saved separately somewhere?

[emanuele-f]

I tried to install PCAPdroid-mitm app on my emulator but emulator can not see the app

This is not released yet. You either need to build PCAPdroid from source (dev branch) or install a pre-built debug apk. Here is a pre-built apk for the latest commit. Then you should enable the "TLS decryption (preview)" option in the PCAPdroid settings.

How does one decrypt the PCAP file with the help of the new plugin?

This is still a work in progress, at the moment the app shows the decrypted HTTP request data (see this screenshot). I'm implementing the ability to show the full HTTP request/response in the app. See below for decryption keys.

Are decryption keys saved separately somewhere

I've just added (in 12727bd) the ability to export the decryption keys, to be used in Wireshark. Be sure to install the latest debug apk (link above), then you can export them (the capture must be running):

This link explains how to load them in Wireshark. A more practical way to get a decryptable pcap will be available with #185

[kenduq]

Tried to export SSL keylog file but the exported .txt file is empty.
Option of showing the full HTTP request/response in the app works fine.

[emanuele-f]

Tried to export SSL keylog file but the exported .txt file is empty.

The keylog must be exported after the HTTPS connection is established.

The ability to inspect the full HTTP payload from PCAPdroid is now implemented, check out #107 (comment) .

A lock icon now reports the state of the decryption.

• gray closed: decryption is in progress
• gray open: the connection is not encrypted
• green open: the connection was decrypted
• orange closed: decryption not supported (e.g. QUIC or encrypted protocol like telegram/whatsapp)
• red closed: the decryption failed (e.g. due to a TLS error)

[kevin0t]

In latest debug version i can see the app crashing and also hanging when "Full payload" toggle is turned on.

[emanuele-f]

In latest debug version i can see the app crashing and also hanging when "Full payload" toggle is turned on.

Should be fixed in fe26ecc. New debug apk.

[dmnsea]

Consider also creating whitelist of apps that may be captured.

I guess then the whitelist structure may be like that:

package.name, confirmed_version

I don't know what's more reliable for version detection - versionCode or versionName, just because probably average user won't be able to extract versionCode, but versionName finally just a string, which anybody can fill with anything.

i.e. when loading list of target apps you can mark them:

• everything with targetSdk less than android 7 will be "green" or somehow marked as "decryptable"
• apps which are detected in whitelist and with versionName/versionCode lower/equal to value from whitelist also become "green"
• apps which are detected from whitelist by package name, but with version newer than confirmed in whitelist become "orange", or somehow "may be decryptable"
• others will be "red" or somehow "unconfirmed"

The idea is that amount of directly decryptable apps will be small and finally they won't update each day. So there will be added software confirmed by community, so I'm not sure that this list will be updated very often. I guess also it's worth to make minimal in-app update of this list to once per week or even less (but with ability to manually update).

Also one more notice is to avoid adding to list ROM-specific apps, i.e. something with package com.android.browser and etc. Just because depending on ROM it may be completelly different apps

[besendorf]

Today I tested the beta TLS decryption with a patched Signal apk and it worked in PCAPdroid. I could see decrypted TLS packets. But in wireshark via the http server and also after saving the pcap in pcapdroid and opening this file afterwards in wireshark I could only see encrypted TLS packets.
Is saving or serving decrypted packages via http not implemented yet?
Is there currently a way to extract the decrypted pcap?

Edit: I noticed that I can export the SSLKEYLOG. However that crashes the app with the debug release. If I see correctly this should be fixed with 6854284
Could you provide a new debug release?

[emanuele-f]

Could you provide a new debug release?

Try this one. The keylog file is now exported when the capture is stopped. If you still experience the crash, please report the stack trace

[besendorf]

Could you provide a new debug release?

Try this one. The keylog file is now exported when the capture is stopped. If you still experience the crash, please report the stack trace

Thank you very much and also for the fast response. Worked like a charm.
Might I suggest including the sslkeylog in the pcap that can be saved in the app and also the one streamed via http/udp?
https://wiki.wireshark.org/TLS#embedding-decryption-secrets-in-a-pcapng-file

[emanuele-f]

Thank you very much and also for the fast response. Worked like a charm. Might I suggest including the sslkeylog in the pcap that can be saved in the app and also the one streamed via http/udp?

This will be implemented in #185

[kasnder]

This is a great addition to PCAPdroid, and I look forward to using it for my research.

Might it be possible to add an option to drop UDP traffic on port 443? This usually prevents apps from using QUIC, and instead makes these apps fallback to TCP.

There might be some complications with some Private DNS solutions -- in this case, one might just let the user know that this must be disabled?

[emanuele-f]

Might it be possible to add an option to drop UDP traffic on port 443? This usually prevents apps from using QUIC, and instead makes these apps fallback to TCP

I read that this fallback behavior is implemented by chrome. Is this part of the standard or it only applies to some apps? Please provide more info in a separate issue and also possibly an example on how it helps with decryption

There might be some complications with some Private DNS solutions

Private DNS should user TLS on port 443. Do any of them use UDP on port 443?

[kasnder]

Might it be possible to add an option to drop UDP traffic on port 443? This usually prevents apps from using QUIC, and instead makes these apps fallback to TCP

I read that this fallback behavior is implemented by chrome. Is this part of the standard or it only applies to some apps? Please provide more info in a separate issue and also possibly an example on how it helps with decryption

Not sure if this is part of a standard. I guess some firewall configurations don't allow UDP on port 443, and thus a fallback is usually needed.

From my personal experience analysing thousands of Android apps, blocking UDP on port 443 works pretty well.

I've opened a new issue #213.

There might be some complications with some Private DNS solutions

Private DNS should user TLS on port 443. Do any of them use UDP on port 443?

Actually, no. I just checked that. Just wasn't sure!

EDIT: This has been added to PCAPdroid now.