Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate firmware to stream from PCM to OPUS #317

Closed
josancamon19 opened this issue Jun 20, 2024 · 3 comments
Closed

Migrate firmware to stream from PCM to OPUS #317

josancamon19 opened this issue Jun 20, 2024 · 3 comments
Assignees
Labels
flutter flutter work task

Comments

@josancamon19
Copy link
Contributor

Is your feature request related to a problem? Please describe.
Better audio quality, less bandwitdh.

Describe the solution you'd like
Firmware should stream audio bytes using Opus codec, and the mobile app should receive those bytes properly modifying respectively on transcript.dart to handle that, do the .

Describe alternatives you've considered
uLaw vs Opus, I don't know which one is better, from what I've been suggested Opus, but lmk if you think differently.

@diegoasua
Copy link

The other way. PCM to Opus. Can't change title

@josancamon19 josancamon19 changed the title Migrate firmware to stream from Opus to PCM ($300) Migrate firmware to stream from PCM to OPUS ($300) Jun 20, 2024
@josancamon19 josancamon19 added the flutter flutter work label Jun 21, 2024
@ebariaux
Copy link
Contributor

Last update on my current status I posted on Discord around that topic:

  • I have µ-Law encoding working with Friend and my own app
  • I created a PR for the change on firmware side ONLY (Configure firmware to send audio as µ-law 16kHz #356), which is straight forward
  • This bring 16kHz (meaning frequency range up to 8kHz) with the same bandwidth
  • I did not update the Friend app and do not intend to (I'm not a Flutter/Dart dev and not intersted in trying besides trivial changes). With a quick look I see that some work in wav_bytes.dart would be required (e.g. WAV header that hardcodes PCM) and certainly other places (also depending on what you want to send to the SST service)
  • I did publish my native iOS code in https://github.com/nelcea/PAL. If you look into https://github.com/nelcea/PAL/blob/main/iOS/PALApp/Codecs.swift, in the µLawCodec class you'll see the simple logic I use to decode the signal to PCM using a static lookup table. This might be an inspiration for whomever wants to have a go at the change in the Friend app
  • I'm still work on the Opus codec but getting some errors during the decoding on iOS. I need to go deeper into the original opus lib doc/code.
  • The size gain I saw between Opus over µ-law is quite small (10 %, unless the encoding is wrong). So my current thinking is that given I believe the audio quality would be only slightly better with Opus but the processing load for encoding/decoding is greater, I would go for µ-law at this stage.
  • As a side note, it seems the device could go slightly higher to a bit more than 20kHz sampling frequency, given us an additional 2kHz for the frequency range

@ebariaux
Copy link
Contributor

And in case somebody wants to test with Opus, changes required in the firmware are:

  • Enabled CODEC_OPUS in config.h
  • Add CONFIG_CODEC_OPUS=y in prj.conf

@josancamon19 josancamon19 self-assigned this Jun 27, 2024
@josancamon19 josancamon19 changed the title Migrate firmware to stream from PCM to OPUS ($300) Migrate firmware to stream from PCM to OPUS Jun 27, 2024
@github-project-automation github-project-automation bot moved this from Backlog to Done in omi TODO / bounties Jun 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
flutter flutter work task
Projects
Status: Done
Development

No branches or pull requests

3 participants