How to make Mongo.find return a document as a list of tuples to preserve order

[rhcarvalho]

BSON documents have fields in a specific order. As much as I understand, when I use Mongo.find or Mongo.find_one I get back an enumerable (cursor) which generates maps or a single map. And by getting a map the order of the keys is lost.

The README says about data representation:

This driver chose to accept both maps and lists of key-value pairs when encoding but will only decode documents to lists. This has the side-effect that it's impossible to discern empty arrays from empty documents.

Maybe I don't understand the above well, but I expected to get [{key, value}, ...] for a single document.

How could I preserve order as stored in the database? Thanks!

[zookzook]

Why do you expect to get an ordered list of key-value pairs? The documents are represented as maps. You send and receive the documents as maps. Why do you need the order of keys? In this case, you can just use a list with key-values embedded in a document like this:

%{
 key_values: [
 %{k: "key-1", v: "value-1"},
 %{k: "key-2", v: "value-2"},
 %{k: "key-3", v: "value-3"},
 ]
}

But to be honest I don't understand the requirements.

[rhcarvalho]

Hi, Michael!

Thanks for the reply! Let me try sharing a bit more context.

I'm using Livebook to access data written by a legacy system written in Go that uses MongoDB. I'm doing live data exploration and generating some reports with Elixir/Livebook.

The data is written to Mongo such that documents have a certain field order, same as declared in code (Go structs). That order makes debugging and exploration easier, both because the fields are somewhat grouped semantically and because I can look at query output and the source code and the order is the same.

As an example, when I query a collection in the Mongo Shell, this is what I get:

> db.Page.findOne()                                                                                                                               
{                                                                                                                                                 
        "_id" : ObjectId("..."),                                                                                             
        "model" : {                                                                                                                               
                "_created" : ISODate("2021-09-29T13:04:25.496Z"),                                                                                 
                "_modified" : ISODate("2021-09-29T13:04:25.496Z"),                                                                                                                                                      
        },                                                                                                                                        
        "tenancy" : "...",                                                                                                   
        "name" : "...",                                                                                                                        
        "slug" : "...",                                                                                                                        
        "components" : [                                                                                                                          
                {...},
                ...
         ],
        "settings" : {                                                   
                "language" : "...",                                    
                "showCustomFields" : true                                                                                                         
        },          
        ...
}

Note the document keys have some non-alphabetical order.

When I run the same query on a Livebook notebook, I get a map with keys sorted alphabetically (not sure which layer is doing this sorting, probably Livebook when rendering the output, since Elixir maps have no key order):

> Mongo.find_one(mongo, "Page", %{})
%{
  "_id" => #BSON.ObjectId<...>,
  "components" => [
    %{
      ... keys sorted alphabetically ...
      },
      ...
  ],
  "model" => %{
    "_created" => ~U[2021-09-29 13:04:25.496Z],
    "_modified" => ~U[2021-09-29 13:04:25.496Z]
  },
  "name" => "...",
  "settings" => %{"language" => "...", "showCustomFields" => true},
  "slug" => "...",
  "tenancy" => "..."
}

Since maps have no order, I'd like to get the output as lists of tuples with the same order as stored in MongoDB, if possible, like:

[
  {"_id", ...},
  {"model", [{...}, ...]},
  {"tenancy", "..."},
  {"name", "..."},
  ...
]

I know I can use lists for input (useful for defining sort order in queries, for example), but now I'm looking to control the output of queries.

Does that make sense?

[zookzook]

Hey Rodolfo,

I understand now. Currently, it is not supported. In the encoder, the list of attributes is converted into a map like this:

  defp doc_fields("", acc) do
    acc |> Enum.reverse() |> Enum.into(%{})
  end

You see, the driver has the order of keys, but unfortunately, it gets lost by converting it into a map. I can imagine, that I can add an option like preserve_order: :order to turn the code into:

 defp doc_fields("", acc) do
    acc
    |> Map.new()
    |> Map.put(:order, Enum.map(acc, fn {key, _value} -> key end))
  end

In this case, each document contains an attribute :order that includes a list of keys in the original order. I cannot change the result of the encoder because BSON also contains metadata that uses the driver by pattern matching. If I change the result of the encoder from a map into a list of key-value items then the driver would work anymore, because the the pattern matching would work anymore.

The new extension :preserve_order would increase the memory footprint and CPU usage.

Since this requirement seems to be very rare, I need to add some code to make sure that the performance of the encoder is not affected without this option.

Still, a good opportunity to take another look at the decoder and encoder for some improvements. 😀

[rhcarvalho]

If I change the result of the encoder from a map into a list of key-value items then the driver would work anymore, because the the pattern matching would work anymore.

Indeed, I think I haven't thought about it from that perspective... preserving order by working with lists of tuples would make pattern matching on the documents much less convenient than using maps (though, on second thought, it would still be possible to convert from list to a map before pattern matching).

The preserve_order option sounds interesting. I suspect I'd have to wrap my query results somehow to go from maps to a list of tuples (recursively). Not ergonomic but doable!

I am equally happy if you decide not to add more options and complicate the library if no one else is looking for what I asked 😄

[zookzook]

I provided a PR #211 The solution is, that you define a decoder module that preserves the order. Feel free to test it.

[rhcarvalho]

😍 amazing, thank you! I'll try it during the week on my next queries and give feedback.