Advertisement frame with bad parsing ?

[zian31]

Hi !

This time I think I've found bug : I'm testing with a Xiaomi LYWSD03MMC (this time with an "atc1441 modified version").

Lots of Advertisement frames are missing and often, no one arrive after the first one, which is not the good one (bytes missing in the end).

I've done screenshots of your soft with in the same time WireShark in background.
At the same second : 19:08:01, Wireshark trace 2 Advertisement frames, and your soft only one, I think a truncated one :
Wireshark end frame (show next screenshot) => A4 C1 38 90 EA 05 00 D6 40 44 0B 0D C3 CE
ToolBLEx end frame (show next screenshot) => A4 C1 38 90 EA 05 (8 bytes missing)

Sometimes, ToolBLEx shows the good frame, but if the truncated frame appears, no more Advertisement frames occurs after (but Advertisement frames continue with WireShark).

I think it's a kind of parsing problem, which also block the next treatment (Maybe there is a link with my last Screenshot : another Advertisement frame arriving very close in time ?)

[emericg]

I think you are right, it's probably a parsing bug that I already know about, but forget to fix 🤔
But that bug stopping the advertisement messages, I'm not sure. It's probably just that the software won't show two identical messages?

I'll have a look tomorrow (and thanks for the Wireshark trace)

[zian31]

Hi!

I see stopping of the advertisement messages on many devices now with ToolBLEx but not with WireShark in background.

I think in order to fix that bug, you'll need WireShark on an OS Win10, so here is what to do to install it easily :

• Install lastest Wireshark (v4) it will work with only Npcap case checked during installation (no need to check all others cases)
  (my older Wireshark v2 was not working for this Bluetooth tracking)
• Install Microsoft "Bluetooth Virtual Sniffer (btvs.exe)" here : https://download.microsoft.com/download/e/e/e/eeed3cd5-bdbd-47db-9b8e-ca9d2df2cd29/BluetoothTestPlatformPack-1.14.0.msi (Infos here : https://learn.microsoft.com/fr-fr/windows-hardware/drivers/bluetooth/testing-btp-tools-btvs)
  It will be installed here : C:\BTP\v1.14.0\x86
• Win+R => PowerShell (admin or not)
• cd C:\BTP\v1.14.0\x86
• .\btvs -Mode Wireshark
• Wireshark opens automatically with the good connexion [email protected]:24352 (bluetooth is sent to this local address and port with BTP)
• Let the window BTP openned

[emericg]

Ok so I've looked into this issue and I didn't find anything yet. I also have a LYWSD03MMC with a custom firmware, and testing with Linux and macOS I didn't see a parsing problem. I thought it could be an old issue but turns out it was actually fixed.

I don't have a Windows to test that today, but I'll look into it if I can.
Does that problem happens a lot? And right away, or it can take a while? On the latest version only or you had this before?
Maybe it's not a parsing problem, but just that the Bluetooth stop working, and you can see advertisement data cut mid-air. Can you check if it always happens at the same time/position (which would point to a parsing problem) or if it's a bit more random (BLE interruption?).

I have a similar problem with my Linux install as I previously mentioned, but looking deeper into it, I can see that the application probably isn't the problem. When I loose Bluetooth after a couple seconds of parsing, I can see that BLE traffic completely stop with Wireshark too. And it does not appear to happen mid-frame, it's just that no more scan/advertisement events are happening. If I'm already connected to a device, then the connection continue to work as intended and I can read/write to characteristics as long as I want.

[zian31]

With you new online reports, I think will be able to do more tests to try to understand, expecially with the LYWSD03MMC custom ATC because it's working with advertising frames with data inside (a rare case for BLE devices).

Moreover, I will be more precise with the descricption of my tests done next time.
I will also try with a raspberry with Linux.

[emericg]

By the way, just to be sure, the "scanning timeout" setting is set to 0 (unlimited) or to a high enough value, so it's not what's stopping the scanning in your case right?

[zian31]

Yes, scanning timeout is set to 0 for permanent scanning.

Here are my last results with a screenshot at the end :

• Using LYWSD03MMC with ATC custom inside (with 10s between each advertising frames containing temp, humi, etc...)
• Wireshark with an ADDR filter to see only LYWSD03MMC ATC frames : bthci_evt.bd_addr == A4:C1:38:90:EA:05
• Only 2 types of advertising frames are received with Wireshark :

(Length = 27) Sub Event = 0x02 Event type = 0x04 (Scan response)
0x 043E1802010400 05EA9038C1A4 0C0B09 4154435F393045413035 DA
05EA9038C1A4 = MAC reversed
0C0B09 = ?
4154435F393045413035 = ASCII of "ATC_90EA05"
DA = RSSI

(Length = 32) Sub Event = 0x02 Event type = x00 (Connectable Undirect Advertising)
0x 043E1D02010000 05EA9038C1A4 111016 1A18 A4C13890EA05 00DE403F0AE2BF E0
05EA9038C1A4 = MAC reversed
111016 = ?
1A18 = UUID
A4C13890EA05 = MAC
00DE = 222 -> 22.2 = Temperature in int16
40 = 64 -> 64% = Humidity in percent
3F = 63 -> 63% = Battery in percent
Batt = 0x0AE2 = 2786 -> 2.786V = Battery in mV uint16_t
BF = frame packet counter
E0 = RSSI

One good new information : frame packet counter change only if values change ! So it was not a bug in ToolBLEx (sorry :) )

But the main bug is remaining : with ToolBLEx, only one advertising frame is present (with partial datas) :
only 6 bytes A4C13890EA05 = MAC

I think parsing is bloquing somewhere (maybe in the "ignoring same frames" code part ?)

[emericg]

It's weird, I just tested on Windows and nothing seems wrong either. I can get 30m of advertisement from the LYWSD03MMC...

[zian31]

You see the LYWSD03MMC ATC custom with lots of advertisement frames with temp & humi & batt ? Or only the original Xiaomi LYWSD03MMC ? Because the original Xiaomi don't send advertisement frames with temp & humi & batt, just classic advertisement frames.

So I will continue tests with other BLE devices to try to understand with more informations.

[zian31]

I've understood :)

I've decided to use a device with programmable datas in advertising frames (I can change UUID, etc...) : the original breakboard HM-10 (BLE 4.0)

Conclusion : the parsing of ToolBLEx adverstising frames is uncomplete compared with the Wireshark parsing, and I think it could be a huge work to do it due to many BLE rules :'(
That's why with LYWSD03MMC custom ATC it was with uncomplete datas (and maybe blocking ?)

Here is my test screenshot :

For example here :

• 0x4D48 is not an UUID (wireshark know it, but not ToolBLEx)
• 0xB000 is an UUID, but next datas are not recognised with ToolBLEx and well recognised with Wireshark

I think the solution for ToolBLEx is to try parsing less with these advertising frames (more RAW data than parsed data) to avoid bad parsing if it's possible (i'm not a BLE specialist, is it possible) ?

[emericg]

0x4D48 is not an UUID (wireshark know it, but not ToolBLEx)

Manufacturer data don't have UUID, they have 32 bits "manufacturer ID".
Service data have either 32, 64 or 128 bits UUIDs, you can see the short form of the 32b UUID (0xb000) in the advertisement panel or the long form (full uuid) in the log panel. Making this more explicit is on the TODO list.

0xB000 is an UUID, but next datas are not recognised with ToolBLEx and well recognised with Wireshark

Incomplete data as you call it is a problem with your Window install, or maybe the relation between your Window and the software, I don't really know at the moment. I certainly don't see it on my computer, which makes it harder to fix...

ToolBLEx doesn't parse data, it just prints them in a pretty format. What is logged is the actual size of the data received and the content of the data is just the block of data in hexadecimal format, printed almost as soon as it's received.
So I'm confident that in your case the app did get handed a 0 byte block of data, and there isn't much to be done with that.

The applications receive blocks of data from the OS (applications can't do Bluetooth directly). What travels on the air is not what an application will get, the "network transmission" part of the packet will be handled by the OS (that's what you see on the left panel of Wireshark) and the actual payload forwarded to the apps. For instance, a device will advertise its name and data in separate packets, that's how the BLE transmission works, not duplicating the name in each advertisement packet. Yet applications always received "assembled" packets with a device name and data from the OS BLE stack (actually on macOS depending on the SDK version used it's sometime not true, but it remains to be seen if it's a bug or if it is intended).

The device in your screenshot seems to be a beacon, that's usually what we see with nameless devices (on Linux they are presented with the MAC in their names, on macOS usually empty names, on Windows "Bluetooth + MAC") so maybe there is just no data in the packet? But in your screenshot we can see the name in the packet payload. So again in this case (the example from the screenshot not the problem described in this issue altogether) I'm really tempted to blame the OS?

[zian31]

Thx, interesting,

Is it possible to add a button with very simple RAW log (by file to save as your other log) in hexa of every advertising frames received (with no treatment) in your ToolBLEx with also a timestamp at the begining of each line ?
So that I can see the exact difference between Wireshark and what is received with these libraries Qt.

For example an ASCII file like this :
15:43:28.550 > 02 40 20 1B 00 17 ...
15:43:29.750 > 02 41 E0 1B 00 17 FF FF ...
etc...

Maybe it's not possible with these Qt libraries ? So elsewhere a file log af ASCII text advertising datas lines reveived from Qt functions ?

EDIT : I've just tested "LYWSD03MMC custom ATC" and "HM-10", here are the screenshots with the good datas (so Android is working well because datas read are as expected, it can be a Win10 problem, but we have to be sure :) )