Skip to content

Creates a readable stream of ZIP file with no compression out of given files, streams or buffers

License

Notifications You must be signed in to change notification settings

tinovyatkin/zip-store-stream

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

88 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

zip-store-stream codecov Maintainability Rating

Highly optimized Node.JS library to create in-memory ZIP archive (as Readable stream and without compression) from given strings, Buffers or streams. Storing without compression is fast and in many cases is enough, if you just want to bundle some files together (as our use case for Apple Wallet .pkpass files, which consists mostly of already compressed PNG files)

Motivation

There are tons of ZIP creating libraries on NPM, however, none of them is optimized for speed, memory and asynchronism point of view and I was needed a way to generate ZIP archives (Apple Wallet Passes) at scale of about 50000 RPS.

This library:

  • Generates ZIP archive in memory and returns stream to be piped into a file, HTTPS response, etc.
  • Works with strings, buffers or readable streams as file sources.
  • Starts pushing data as soon as possible to decrease response latency.
  • Pushes in small chunks, file by file, to improve message loop flowing.
  • Uses super-fast @node-rs/crc32 Rust-powered library for CRC32 calculation.
  • Uses native Node.JS Buffer and efficient multi-bytes writing functions (writeUInt16LE / writeUInt32LE) for archive structure building.

Written in TypeScript, 100% test coverage.

import { ZipStoreStream } from 'zip-store-stream';

const FILENAME = 'test-mixed.zip';

const zip = new ZipStoreStream([
  { path: 'string.txt', data: 'Hello world!' },
  { path: 'buffer.bin', data: randomBytes(10000) },
  {
    path: 'stream.ts',
    data: createReadStream(__filename, { encoding: 'utf8' }),
  },
]);
await pipeline(zip, createWriteStream(FILENAME));
unzip -v test-mixed.zip

    Length   Method    Size  Cmpr    Date    Time   CRC-32   Name
--------  ------  ------- ---- ---------- -----     -------- ----
        12  Stored       12   0% 00-00-1980 00:00 1b851995  string.txt
    10000   Stored    10000   0% 00-00-1980 00:00 983c6c5a  buffer.bin
    3472    Stored     3472   0% 00-00-1980 00:00 f21cdbc5  stream.ts
--------          -------  ---                               -------
    13484            13484   0%                            3 files

Author and License

MIT licensed by Konstantin Vyatkin [email protected]

About

Creates a readable stream of ZIP file with no compression out of given files, streams or buffers

Topics

Resources

License

Stars

Watchers

Forks

Packages

No packages published