Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Decoding non-string-able-keyed maps #232

Open
fillest opened this issue Jul 25, 2018 · 4 comments
Open

Decoding non-string-able-keyed maps #232

fillest opened this issue Jul 25, 2018 · 4 comments

Comments

@fillest
Copy link

fillest commented Jul 25, 2018

Hello.
We have some data that is currently encoded with msgpack/msgpack-python and decoded with vmihailenco/msgpack. I'm trying to switch the decoding to msgp. I've added duplicate decoding code and reflect.DeepEqual the results and diffing them when it fails.
Where I'm getting with vmihailenco/msgpack:

 TopicIDs: (map[int]bool) (len=1) {
  (int) 360: (bool) true
 },

with msgp I'm getting TopicIDs: (map[int]bool) <nil> silently. The docs say map must have str or bin keys, is that right? It also says "You can still manually decode arbitrary maps with the primitives built into the library", but how? So:

  1. such decoding silently loses the data, it should at least return an error
  2. how to manually decode such arbitrary maps?
@erikdubbelboer
Copy link
Contributor

What happens when you use TopicIDs map[string]bool in your struct?

There is not really a reason for msgp to not allow other types as keys seeing as encoding/json supports this now as well.

The online editor on https://msgpack.org/ doesn't seem to allow int keys either.

You can manually decode it using an extension. This means msgp will pass a raw []byte to your extension code which then has to determine by itself how to decode this. You can use functions like ReadMapHeaderBytes, ReadMapKeyZC, ReadBoolBytes and ReadIntBytes to parse the data.

@Dieterbe
Copy link

Has anyone figured out how to actually do this (marshalling or unmarshalling a map with non-string keys). The wiki page for custom extensions seems geared towards single, custom types. Not sure how to set the struct tags on a field that is a composite type (which in my case is a type alias for the map composed of my custom type as key and integers as values). Are we supposed to write the custom extension for the map as a whole, or only for the key type? (I did it for the key type but it's not getting picked up by the library when running go generate)

@fillest
Copy link
Author

fillest commented Sep 25, 2019

I was going to try to (back then), but never-ending tasks with higher priority don't ever end :) And in the meantime, our CPU bottlenecks shifted a lot, so I have no idea when I'll get to it..

Dieterbe added a commit to grafana/metrictank that referenced this issue Sep 25, 2019
such as SeriesSource map[SeriesSourceProperties]uint32
but we need to pass it through in our http requests...
i haven't gotten this to work yet.
tinylib/msgp#232 (comment)
not sure yet how i will solve this
@Dieterbe
Copy link

BTW I agree

You can still manually decode arbitrary maps with the primitives built into the library.

this could use some expanding. which primitives? custom shims? extensions? on the map level or the key level? can we get an example for both encoding and decoding? thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants