[RFE] add support for Reader/Writer for chaining

[ignatenkobrain]

My use-case is pretty simple, I have file which contains base64-encoded, zlib-compressed JSON ;)

[ignatenkobrain]

I hope internally this could do something partially decoding base64 (since you can decode it by chunks), you don't need full string to be available.

[marshallpierce]

Are you picturing something like a Base64BufReader that wraps an underlying BufRead?

[ignatenkobrain]

yes, something like that. Unfortunately, I'm still in the middle of understanding how to chain FlateReader from flate2, so I can't come up with syntax how I would like to see it.

P.S. it's my first experience with chaining readers in Rust

[ignatenkobrain]

Also I'm not sure whether it should be BufRead or just Read

[ignatenkobrain]

impl EntitlementCertificate {
    pub fn from_file(file: &str) -> Self {
        let path = Path::new(file);

        let rdr = BufReader::new(File::open(path).unwrap());
        let b64iter = rdr.lines()
            .skip_while(|l| l.as_ref().unwrap() != ENTITLEMENT_DATA_HEADER)
            .skip(1)
            .take_while(|l| l.as_ref().unwrap() != ENTITLEMENT_DATA_FOOTER);
        let mut b64rdr = IterRead::new(b64iter);

        let mut data = String::new();
        b64rdr.read_to_string(&mut data).unwrap();
        let zlib_encoded = base64::decode(&data).unwrap();

        let decompressor = flate2::read::ZlibDecoder::new(&*zlib_encoded);

        serde_json::from_reader(decompressor).unwrap()
    }
}

that's what I've got so far in my project, as you can see, I have reading file, decompressing and deserializing working on top of Read.

[brianm]

I have 80% of an implementation of this internally at work. Basically is a Read and Write implementation loosely based on the golang implementation of the streaming encoder and decoder.

I'd be happy to extract the code and contribute it if there is interest.

(80% == it is not allocation free on read(..) right now, and doesn't handle the case of unpadded base64 because config.pad is not exposed (which would not be a problem if the code moved here)).

[marshallpierce]

Sure, I'd be happy to take a look at that, if you feel like making a PR (even a rough one). I haven't forgotten about this feature; just got several things ahead of it on the to-do list. :)

BTW, thanks for JDBI!

[brianm]

Okay, got rid of the allocation at the expense of having to do extra reads sometimes (return less than full buffer), and I think need for access to config.pad.

PR incoming.

[marshallpierce]

@brianm well, finally shipped encoding... decoding next? If you think your decoder is a good starting point I'm happy to run with it, or I can start from scratch.

[dignifiedquire]

@marshallpierce I have a working version of a streaming decoder here: https://github.com/dignifiedquire/pgp/blob/master/src/base64_decoder.rs if you want something as a starting point. It is modeled after the go implementation, but has some additional logic I needed, to be able to handle input ending for my use case.

It also is not very well tuned yet, in terms of how the buffers are used.

[marshallpierce]

Thanks; I'll look at it when I have some time!

[nwalfield]

I've implemented a streaming decoder. Please see this MR: #106

[ggriffiniii]

In case anyone else stumbles here waiting for this feature. It's been implemented in the radix64 crate. radix64::io has both an EncodeWriter and DecodeReader. https://docs.rs/radix64