panic: runtime error: slice bounds out of range [:-1797448000]

[thorfour]

When playing around with compaction of parquet files I ran into what appears to be another panic caused by an integer overflow?

panic: runtime error: slice bounds out of range [:-1797448000]

goroutine 9400 [running]:
github.com/segmentio/parquet-go.(*bufferPool).get(0x1080a4d20, 0xffffffff94dd1ec0)
	/Users/thor/go/pkg/mod/github.com/segmentio/[email protected]/buffer.go:350 +0x224
github.com/segmentio/parquet-go.(*Column).decompress(0x1400bce29c0, {0x14077a80000, 0x40c1999, 0x40c1999}, 0x15fff290?)
	/Users/thor/go/pkg/mod/github.com/segmentio/[email protected]/column.go:496 +0x3c
github.com/segmentio/parquet-go.(*Column).decodeDictionary(0x1400bce29c0, {0x0?}, 0x0?, 0xdc80cf0?)
	/Users/thor/go/pkg/mod/github.com/segmentio/[email protected]/column.go:744 +0xb8
github.com/segmentio/parquet-go.(*filePages).readDictionaryPage(0x14000e6b540, 0x14015fff230?, 0x14015fff230?)
	/Users/thor/go/pkg/mod/github.com/segmentio/[email protected]/file.go:584 +0x3c
github.com/segmentio/parquet-go.(*filePages).ReadPage(0x14000e6b540)
	/Users/thor/go/pkg/mod/github.com/segmentio/[email protected]/file.go:522 +0x120
github.com/polarsignals/frostdb/dynparquet.(*remappedPages).ReadPage(0x14000f2a930)
	/Users/thor/go/pkg/mod/github.com/polarsignals/[email protected]/dynparquet/schema.go:1011 +0x30
github.com/segmentio/parquet-go.readPages({0x106c62630, 0x14000f2a930}, 0x140596d1560, 0x1400cddd420, 0x140596d0840)
	/Users/thor/go/pkg/mod/github.com/segmentio/[email protected]/page.go:219 +0x90
created by github.com/segmentio/parquet-go.(*asyncPages).init
	/Users/thor/go/pkg/mod/github.com/segmentio/[email protected]/page.go:161 +0x18c

[achille-roussel]

Hello @thorfour, thanks for reporting the issue.

Would you happen to know if the dictionary page in your file is larger than 2 GiB?

[thorfour]

I don't believe it was, I'll have to try and repro to double check

[sdressler]

@achille-roussel I am running into a similar issue. How can I check the dictionary page size?