Skip to content

Latest commit

 

History

History
969 lines (809 loc) · 70.8 KB

vulnerabilities-workflow-1.md

File metadata and controls

969 lines (809 loc) · 70.8 KB

Combined Vulnerability List

Integer Overflow in Compressed Segment Length in Compress4X/Compress4Xp

  • Description:

    1. The Compress4X and compress4Xp functions in huff0/compress.go split the input data into four segments and compress each segment independently.
    2. For each segment (except the last one), the compressed length is written as a uint16 (2 bytes) in little-endian format before the compressed data of that segment.
    3. If the compressed size of any of the first three segments exceeds math.MaxUint16 (65535 bytes), an integer overflow will occur when calculating and storing the length.
    4. During decompression, the Decompress4X function reads these 2-byte length values to determine the boundaries of each compressed segment. Due to the integer overflow during compression, a small length value might be read, leading to incorrect segment boundaries.
    5. This can cause Decompress4X to read beyond the intended segment boundary, potentially leading to out-of-bounds read during decompression or incorrect decompression of the data.

  • Impact:

    • Data corruption: Decompressed data may be incorrect due to misinterpretation of segment boundaries.
    • Potential out-of-bounds read: During decompression, incorrect segment lengths might cause the decoder to read beyond the allocated buffer for a segment, potentially leading to crashes or information disclosure if sensitive data is located beyond the buffer. While Go is memory-safe, this could still lead to unexpected behavior and data corruption.

  • Vulnerability Rank: high

  • Currently Implemented Mitigations:

    • In Compress4X and compress4Xp, there is a check: 
      if len(s.Out)-idx > math.MaxUint16 {
    // We cannot store the size in the jump table
    return nil, ErrIncompressible
}
      This check prevents the compression from proceeding if a single segment's compressed size exceeds math.MaxUint16. However, this check only returns ErrIncompressible, effectively denying compression for such inputs, but does not mitigate the vulnerability if somehow a crafted input manages to bypass this check or if this check is removed in future versions, or if the size calculation itself is flawed leading to incorrect check results.

  • Missing Mitigations:

    • Proper error handling and propagation when compressed segment size exceeds the limit. Returning ErrIncompressible prevents compression, but ideally, a more specific error indicating segment size overflow should be used for better debugging and handling.
    • During decompression (Decompress4X and related functions not in the provided files, but assumed to exist), robust validation of the segment lengths read from the compressed data is crucial. The decompression logic should check if the read segment lengths are within reasonable bounds and handle potential overflows or corrupted length values gracefully, possibly returning an error instead of proceeding with potentially unsafe memory operations.
    • Consider using a larger data type than uint16 to store segment lengths if segment sizes are expected to potentially exceed this limit in compressed form. However, this would change the format. A better approach might be to ensure segment sizes are limited during compression or implement more robust error handling during decompression when unexpected segment lengths are encountered.

  • Preconditions:

    • Input data must be compressible by huff0 algorithm.
    • Input data should be such that when compressed using Compress4X or Compress4Xp, at least one of the first three segments compresses to a size greater than math.MaxUint16 bytes after the huff0 compression itself. This might require a relatively large input.

  • Source Code Analysis: File: /code/huff0/compress.go

    var sixZeros [6]byte

func (s *Scratch) compress4X(src []byte) ([]byte, error) {
    if len(src) < 12 {
        return nil, ErrIncompressible
    }
    segmentSize := (len(src) + 3) / 4

    // Add placeholder for output length
    offsetIdx := len(s.Out)
    s.Out = append(s.Out, sixZeros[:]...)

    for i := 0; i < 4; i++ {
        toDo := src
        if len(toDo) > segmentSize {
            toDo = toDo[:segmentSize]
        }
        src = src[len(toDo):]

        idx := len(s.Out)
        s.Out = s.compress1xDo(s.Out, toDo)
        if len(s.Out)-idx > math.MaxUint16 { // Vulnerability Check
            // We cannot store the size in the jump table
            return nil, ErrIncompressible
        }
        // Write compressed length as little endian before block.
        if i < 3 {
            // Last length is not written.
            length := len(s.Out) - idx
            s.Out[i*2+offsetIdx] = byte(length)
            s.Out[i*2+offsetIdx+1] = byte(length >> 8) // Integer Overflow Point
        }
    }

    return s.Out, nil
}

    In the compress4X function, after compressing each of the first three segments using s.compress1xDo(s.Out, toDo), the code calculates the compressed length as len(s.Out) - idx. This length is then cast to byte and byte(length >> 8) and stored in s.Out. If length is greater than math.MaxUint16, the cast to byte and byte(length >> 8) will result in an integer overflow, storing incorrect length values. The check if len(s.Out)-idx > math.MaxUint16 prevents the function from returning compressed data in such cases, but it's a check to prevent proceeding with compression, not a mitigation for the overflow itself if the compression was to proceed despite the size.

    The same logic applies to compress4Xp function:

    func (s *Scratch) compress4Xp(src []byte) ([]byte, error) {
    if len(src) < 12 {
        return nil, ErrIncompressible
    }
    // Add placeholder for output length
    s.Out = s.Out[:6]

    segmentSize := (len(src) + 3) / 4
    var wg sync.WaitGroup
    wg.Add(4)
    for i := 0; i < 4; i++ {
        toDo := src
        if len(toDo) > segmentSize {
            toDo = toDo[:segmentSize]
        }
        src = src[len(toDo):]

        // Separate goroutine for each block.
        go func(i int) {
            s.tmpOut[i] = s.compress1xDo(s.tmpOut[i][:0], toDo)
            wg.Done()
        }(i)
    }
    wg.Wait()
    for i := 0; i < 4; i++ {
        o := s.tmpOut[i]
        if len(o) > math.MaxUint16 { // Vulnerability Check
            // We cannot store the size in the jump table
            return nil, ErrIncompressible
        }
        // Write compressed length as little endian before block.
        if i < 3 {
            // Last length is not written.
            s.Out[i*2] = byte(len(o))
            s.Out[i*2+1] = byte(len(o) >> 8) // Integer Overflow Point
        }

        // Write output.
        s.Out = append(s.Out, o...)
    }
    return nil, ErrIncompressible // Should return s.Out, nil in case of success, but returns error because of copy-paste from compress4X
}

  • Security Test Case:

    1. Prepare a large input data (e.g., several MB of compressible data). You might need to experiment to find data that compresses effectively with huff0 and results in a compressed segment larger than 65535 bytes when using Compress4X or Compress4Xp. Highly repetitive data or data with specific patterns might be more likely to trigger this.
    2. Compress the input data using Compress4X or Compress4Xp.
    3. Examine the compressed output. Specifically, check the first 6 bytes of the output which are supposed to store the segment lengths for the first three segments. If the vulnerability is triggered, the lengths stored might be small values due to the integer overflow.
    4. Attempt to decompress the compressed data using Decompress4X.
    5. Verify if the decompression process completes without errors, and if the decompressed output is the same as the original input. If data corruption occurs or if the decompression fails in an unexpected way (e.g., out-of-bounds read, panic, or incorrect output), it indicates a vulnerability.
    6. To reliably trigger, you might need to adjust the input data and potentially the TableLog parameter of Scratch to influence the compression ratio and segment sizes. You might need to create a test input that is specifically designed to maximize the compression ratio of the first few segments to exceed math.MaxUint16 when compressed.

    Example Test Code (Conceptual - needs adaptation for a real test case):

    package huff0_test

import (
    "bytes"
    "testing"
    "github.com/klauspost/compress/huff0"
)

func TestCompress4XOverflow(t *testing.T) {
    // 1. Prepare large, compressible input (e.g., repeating pattern)
    inputData := bytes.Repeat([]byte("A"), 200*1024) // Example: 200KB of 'A's

    s := &huff0.Scratch{}
    // s.TableLog = 8 // You might need to adjust TableLog to influence compression

    // 2. Compress using Compress4X
    compressedData, _, err := huff0.Compress4X(inputData, s)
    if err != nil && err != huff0.ErrIncompressible { // Expect ErrIncompressible if check works as intended.
        t.Fatalf("Compression error: %v", err)
    }
    if err == huff0.ErrIncompressible {
        t.Skipf("Input deemed incompressible, which is expected mitigation. Trying to craft input to bypass.")
        // In a real test, you'd craft input to *try* to overflow, if possible.
        return
    }


    // 3. Examine compressed output (manual inspection recommended)
    // In automated test, check if first 6 bytes are suspiciously small (indicating overflow)
    if len(compressedData) > 6 {
        len1 := uint16(compressedData[0]) | uint16(compressedData[1])<<8
        len2 := uint16(compressedData[2]) | uint16(compressedData[3])<<8
        len3 := uint16(compressedData[4]) | uint16(compressedData[5])<<8
        t.Logf("Segment lengths (uint16): %d, %d, %d", len1, len2, len3)
        // Check if any of these lengths are suspiciously small given input size.
        // For example, if input is large and lengths are like 0, 1, 2, it's suspicious.
    }


    // 4. Attempt decompression (if compression succeeded - though in theory, it should return ErrIncompressible)
    if compressedData != nil { // Only decompress if compression didn't return ErrIncompressible in a successful exploit scenario.
        decompressedData, err := huff0.Decompress4X(compressedData[6:], len(inputData)) // Assuming table is in first 6 bytes, data follows. Adjust if needed based on actual format.
        if err != nil {
            t.Errorf("Decompression error: %v", err)
        } else {
            // 5. Verify decompressed output
            if !bytes.Equal(inputData, decompressedData) {
                t.Errorf("Decompressed data does not match original input")
            } else {
                t.Log("Decompression successful, but vulnerability may still exist if lengths were overflowed during compression.")
            }
        }
    } else {
         t.Log("Compression returned ErrIncompressible as expected (mitigation triggered). Vulnerability prevented by design for this input.")
    }
}

Path Traversal Vulnerability in s2sx Self-Extracting Archive Unpacking

  • Description:

    1. The s2sx tool creates self-extracting archives that, when executed, unpack their contents. The unpacking logic is located in /code/s2/cmd/_s2sx/_unpack/main.go within the untar function.
    2. The untar function extracts files and directories from a tar archive embedded within the executable.
    3. For each entry in the tar archive, the destination path is constructed using filepath.Join(dst, header.Name), where dst is the extraction directory and header.Name is the name of the entry in the tar archive.
    4. The checkPath function is called after filepath.Join, which means if header.Name contains path traversal sequences like ../, filepath.Join will resolve them before checkPath is invoked.
    5. Consequently, a maliciously crafted tar archive with filenames starting with ../ can cause files to be extracted outside the intended destination directory (dst), potentially overwriting system files or placing files in unintended locations.

  • Impact:

    • Arbitrary File Write: An attacker can craft a malicious self-extracting archive that, when executed by a user, can write files to arbitrary locations on the user's file system, limited by the permissions of the user running the executable.
    • Potential for Escalation: If the user running the self-extracting archive has elevated privileges, this vulnerability could be used to escalate privileges or compromise the system.
    • Data Corruption: Maliciously crafted archives could overwrite critical system files, leading to data corruption or system instability.

  • Vulnerability Rank: critical

  • Currently Implemented Mitigations:

    • The checkPath function in /code/s2/cmd/_s2sx/_unpack/main.go is intended to prevent path traversal. However, it is called after filepath.Join, which resolves path traversal sequences before the check.
    • func checkPath(dst, filename string) error {
    dest := filepath.Join(dst, filename)
    //prevent path traversal attacks
    if !strings.HasPrefix(dest, dst) {
        return fmt.Errorf("illegal file path: %s", filename)
    }
    return nil
}
      This mitigation is ineffective against path traversal because filepath.Join resolves .. before checkPath is called.

  • Missing Mitigations:

    • Input validation of header.Name before using it in filepath.Join. The code should check if header.Name starts with ../ or contains other path traversal sequences and reject such entries.
    • Secure path handling: Instead of relying on filepath.Join and post-hoc checks, use secure path manipulation functions that prevent path traversal by design, or implement a robust sanitization or validation of the path components before joining them.

  • Preconditions:

    • The victim must execute a maliciously crafted self-extracting archive created using s2sx.
    • The malicious archive must contain a tar archive with filenames designed to exploit the path traversal vulnerability (e.g., filenames starting with ../).

  • Source Code Analysis: File: /code/s2/cmd/_s2sx/_unpack/main.go

    func untar(dst string, r io.Reader) error {
    tr := tar.NewReader(r)

    for {
        header, err := tr.Next()
        // ... (error handling and header == nil checks) ...

        // the target location where the dir/file should be created
        if err := checkPath(dst, header.Name); err != nil { // Mitigation called AFTER filepath.Join
            return err
        }
        target := filepath.Join(dst, header.Name) // Path traversal happens here

        // ... (file type handling and extraction logic) ...
    }
}

func checkPath(dst, filename string) error {
    dest := filepath.Join(dst, filename)
    //prevent path traversal attacks
    if !strings.HasPrefix(dest, dst) { // Check is ineffective
        return fmt.Errorf("illegal file path: %s", filename)
    }
    return nil
}

    The vulnerability lies in the order of operations. filepath.Join(dst, header.Name) is called before checkPath(dst, header.Name). If header.Name is crafted to include path traversal sequences, filepath.Join will resolve these sequences relative to dst. Then checkPath verifies if the resolved path dest still has dst as a prefix. However, because filepath.Join has already performed the path traversal, the resolved path dest might no longer be within the intended dst directory, but checkPath will incorrectly validate it if the resolved path still happens to start with the original dst string (which is unlikely but theoretically possible in some edge cases, and even if not, the check is still bypassed if the resulting path is completely outside dst).

  • Security Test Case:

    1. Create Malicious Tar Archive: Create a tar archive containing a file with a name like ../evil.txt and some content. You can use tar command-line tool for this: 
      mkdir malicious_tar
echo "This is evil content" > malicious_tar/evil.txt
cd malicious_tar
tar -cvf malicious.tar ../evil.txt
cd ..
    2. Create Self-Extracting Archive: Use the s2sx tool to create a self-extracting archive from the malicious tar archive: 
      go build -o s2sx ./code/s2/cmd/s2sx/main.go
./s2sx malicious_tar/malicious.tar
      This will create malicious_tar/malicious.tar.s2sx (or malicious.tar.s2sx.exe on Windows).
    3. Execute Self-Extracting Archive: Run the created self-extracting archive: 
      ./malicious_tar/malicious.tar.s2sx <extraction_directory>
      Replace <extraction_directory> with a directory where you expect the files to be extracted (e.g., output_dir). If you don't provide an argument, it will extract to the current working directory.
    4. Verify Path Traversal: After execution, check if the file evil.txt has been created in a location outside the intended extraction directory (e.g., in the parent directory, if you extracted to output_dir in the current directory, check if evil.txt is created in the current directory itself). If evil.txt is found outside the intended directory, the path traversal vulnerability is confirmed.

    Example Test Code (Conceptual - needs adaptation for automated testing):

    package s2sx_test

import (
    "archive/tar"
    "bytes"
    "io"
    "os"
    "os/exec"
    "path/filepath"
    "testing"
)

func TestS2SXPathTraversal(t *testing.T) {
    // 1. Create Malicious Tar Archive (in memory for test)
    var tarBuf bytes.Buffer
    tarWriter := tar.NewWriter(&tarBuf)
    evilContent := []byte("This is evil content")
    header := &tar.Header{
        Name: "../evil.txt", // Path Traversal payload
        Mode: 0600,
        Size: int64(len(evilContent)),
    }
    if err := tarWriter.WriteHeader(header); err != nil {
        t.Fatal(err)
    }
    if _, err := tarWriter.Write(evilContent); err != nil {
        t.Fatal(err)
    }
    if err := tarWriter.Close(); err != nil {
        t.Fatal(err)
    }

    // 2. Create S2SX Archive (using s2sx command - needs binary built)
    s2sxBinary := "./s2sx" // Path to s2sx binary (build it before running test)
    if _, err := os.Stat(s2sxBinary); os.IsNotExist(err) {
        t.Skipf("s2sx binary not found at %s, skipping test", s2sxBinary)
    }

    tempDir, err := os.MkdirTemp("", "s2sx_test")
    if err != nil {
        t.Fatal(err)
    }
    defer os.RemoveAll(tempDir)

    maliciousTarPath := filepath.Join(tempDir, "malicious.tar")
    if err := os.WriteFile(maliciousTarPath, tarBuf.Bytes(), 0644); err != nil {
        t.Fatal(err)
    }
    s2sxArchivePath := filepath.Join(tempDir, "malicious.tar.s2sx")

    cmd := exec.Command(s2sxBinary, maliciousTarPath)
    cmd.Dir = tempDir // Run command in temp dir
    if out, err := cmd.CombinedOutput(); err != nil {
        t.Fatalf("s2sx command failed: %v, output: %s", err, string(out))
    }

    // 3. Execute S2SX Archive (unpack)
    extractionDir := filepath.Join(tempDir, "output_dir")
    if err := os.Mkdir(extractionDir, 0755); err != nil {
        t.Fatal(err)
    }

    cmdUnpack := exec.Command(s2sxArchivePath, extractionDir)
    cmdUnpack.Dir = tempDir // Run command in temp dir
    if out, err := cmdUnpack.CombinedOutput(); err != nil {
        t.Fatalf("s2sx unpack command failed: %v, output: %s", err, string(out))
    }

    // 4. Verify Path Traversal - Check for evil.txt outside extractionDir
    traversedFilePath := filepath.Join(tempDir, "evil.txt")
    _, err = os.Stat(traversedFilePath)
    if !os.IsNotExist(err) {
        t.Errorf("Path traversal vulnerability exists! File found at: %s", traversedFilePath)
    } else {
        t.Logf("Path traversal prevented (or file not created): %s", traversedFilePath)
    }

    // Optionally, check if evil.txt is *not* in the extractionDir (as expected)
    evilInExtractionDir := filepath.Join(extractionDir, "evil.txt")
    _, err = os.Stat(evilInExtractionDir)
    if !os.IsNotExist(err) {
         t.Errorf("File should NOT be in extraction dir, but found at: %s", evilInExtractionDir)
    }

}

Potential BREACH Side–Channel Vulnerability in Default gzhttp Compression

  • Description:

    • When using the default configuration, the gzhttp compression middleware produces deterministic compressed outputs.
    • If an HTTP response intermingles fixed secret values (such as session tokens) with attacker–controlled input (for example, values coming from query parameters or form input), identical compression behavior may be observed.
    • An external attacker can repeatedly request the same resource, varying the attacker–controlled parts, and analyze subtle changes (for example, via the Content–Length header) that correlate with the secret data.

  • Impact:

    • An attacker may deduce sensitive information (such as CSRF or session tokens), which could result in session hijacking or unauthorized actions performed on behalf of a legitimate user.

  • Vulnerability Rank: High

  • Currently Implemented Mitigations:

    • The gzhttp package includes an optional random–padding (jitter) mechanism designed to obfuscate these compression–size variations.
    • However, this mechanism is disabled by default and must be explicitly enabled by developers.

  • Missing Mitigations:

    • The default configuration does not enable the random–padding feature.
    • There is no automatic detection mechanism for cases where sensitive fixed data is combined with attacker–controlled input.

  • Preconditions:

    • The application is publicly exposed and uses the default gzhttp configuration.
    • HTTP responses include both fixed secret values and attacker–influenceable content.

  • Source Code Analysis:

    • A review of the gzhttp middleware reveals that with random jitter disabled, identical input data produces identical compression output.
    • Although a helper function exists to inject random–padding, it is not invoked unless the middleware is explicitly configured to do so.

  • Security Test Case:

    1. Setup: Deploy the application with the default gzhttp configuration.
    2. Endpoint Configuration: Implement an HTTP endpoint that returns a response mixing a fixed secret (e.g. a session token) with attacker–controlled input (e.g. echoing a query parameter).
    3. Attack Simulation: Use a script to send numerous requests that vary only the attacker–controlled input while recording the Content–Length header (or overall response size).
    4. Observation: Determine if variations in the response size correlate with the attacker’s input, indicating that secret data might be leaking via compression–side channel.
    5. Mitigation Verification: Reconfigure the middleware to enable random–padding and verify that this correlation is no longer observable.

Insecure ZIP Archive Path Handling (Zip Slip) Vulnerability Including Inconsistent File Type Enforcement

  • Description:

    • The archive/zip package accepts ZIP archives that may contain file entries whose names include relative directory–traversal sequences (e.g. "../evil.txt") or even absolute paths.
    • A secure file–path validation check exists and is activated only when the environment variable GODEBUG is set to "zipinsecurepath=0". By default (or when this flag is not set), dangerous file names are accepted.
    • Moreover, while the higher–level API (e.g. the AddFS function) enforces safe file type checks—rejecting entries that represent symbolic links or device files (as verified in tests such as TestIssue61875)—lower–level APIs like CreateHeader do not apply such checks.
    • Thus, if an application uses these lower–level APIs to process ZIP archives from untrusted sources, an attacker can craft a malicious ZIP file with entries having directory–traversal paths or malicious symlink/device file attributes that bypass sanitization.

  • Impact:

    • An attacker could cause directory traversal during file extraction, leading to arbitrary file creation or modification outside the intended destination.
    • Such manipulation may allow an attacker to overwrite critical system or application files and, in a worst–case scenario (especially when combined with other vulnerabilities), could pave the way for arbitrary code execution or complete compromise of system integrity.

  • Vulnerability Rank: High

  • Currently Implemented Mitigations:

    • During ZIP reader initialization (in, for example, zip/reader.go), a check verifies whether file names are “local” (using filepath.IsLocal) and rejects names containing backslashes—but this check is only activated when GODEBUG is explicitly set to "zipinsecurepath=0".
    • When extracting files via the fs.FS interface (using the Open method), a helper function (such as toValidName) sanitizes the filename so that dangerous components (e.g. "../") are stripped.
    • In addition, tests (e.g. TestIssue61875) show that when using the AddFS API, the ZIP package enforces safe processing by rejecting dangerous file types (such as symlink and device file entries).

  • Missing Mitigations:

    • The secure–path check is opt–in rather than enabled by default—if GODEBUG is not set to "zipinsecurepath=0", then no validation occurs.
    • There is no uniform enforcement across all APIs; lower–level APIs (e.g. CreateHeader or manual ZIP entry processing) do not automatically reject dangerous file names or enforce safe file–type checks (leaving the decision up to the developer).
    • The inconsistent handling means that even though the AddFS function protects against dangerous file types, developers using other APIs may inadvertently process malicious entries.

  • Preconditions:

    • The application accepts ZIP archives from untrusted external sources.
    • The environment variable GODEBUG is unset or not set to "zipinsecurepath=0", or the developer uses lower–level APIs which bypass the secure validation.
    • The application subsequently uses the raw file header values (which may contain malicious names) when extracting files to disk.

  • Source Code Analysis:

    • In zip/struct.go the file entries are represented via a FileHeader whose Name field is meant to be a relative path, but there is no runtime enforcement within this method.
    • The ZIP reader’s initialization code only performs the secure check when GODEBUG is set appropriately; otherwise, malicious names (containing "../" or even absolute paths) are allowed to pass through.
    • Test cases (e.g. TestIssue61875) reveal that while the AddFS API rejects ZIP entries with dangerous file modes (such as symlinks and device files), similar checks are absent when using lower–level APIs like CreateHeader, leaving a gap for potential exploitation.

  • Security Test Case:

    1. Setup:
      • Craft a malicious ZIP archive that contains at least one file entry with a traversal path (e.g. "../evil.txt") and optionally an entry with a symlink (for example, an entry with name "symlink" whose content is "../link/target").
    2. Baseline Test (Insecure Behavior):
      • Ensure that the environment variable GODEBUG is unset (or not set to "zipinsecurepath=0").
      • Process the ZIP archive using a lower–level API (for example, by invoking CreateHeader) to simulate extraction from untrusted input.
      • Inspect the resulting FileHeader—observe that the file name still contains the insecure "../" component and that no file–type enforcement takes place.
    3. Secure Behavior Verification:
      • Set the environment variable GODEBUG to "zipinsecurepath=0" and reprocess the ZIP archive; confirm that the initialization detects the insecure file name and returns an error (such as ErrInsecurePath).
      • Alternatively, use the AddFS API and verify that it rejects dangerous file types (evidenced by an error when processing symlink or device file entries).
    4. Conclusion:
      • Demonstrate that without explicit enabling of secure path checks or the usage of safe APIs, an attacker can supply a ZIP archive with malicious entries, thereby enabling a Zip Slip attack during file extraction.

Random Jitter Padding Predictability

  • Description: The RandomJitter option in gzhttp is intended to add padding to compressed responses to obfuscate the actual compressed size, acting as a mitigation against certain types of information leaks, such as CRIME or BREACH attacks. However, the current implementation uses a predictable seed based on a CRC32 or SHA256 checksum of the initial part of the response body if jitterBuffer is greater than 0, or a standard rand.Reader if jitterBuffer is 0. When jitterBuffer is used, this makes the padding predictable if an attacker can control or observe the beginning of the response body.

    Steps to trigger the vulnerability:

    1. An attacker needs to understand that the server is using gzhttp with NewWrapper and the RandomJitter option enabled with jitterBuffer > 0.
    2. The attacker crafts a request that results in a predictable initial response body content up to jitterBuffer size.
    3. The server calculates the CRC32 or SHA256 of this initial part and uses it as a seed for random padding generation.
    4. Since the initial part of the response and the algorithm are known, the attacker can predict the padding length and content added by RandomJitter.

  • Impact: The predictability of the padding undermines the intended security benefit of RandomJitter. An attacker might still be able to infer information about the uncompressed content length or other sensitive data despite the padding, especially in scenarios where the initial part of the response is somewhat controllable or predictable. This reduces the effectiveness of RandomJitter as a countermeasure against compression-ratio side-channel attacks.

  • Vulnerability Rank: high

  • Currently Implemented Mitigations: The RandomJitter option itself is intended as a mitigation, but the seed generation method is flawed when jitterBuffer > 0, making the padding predictable.

  • Missing Mitigations: The seed for random padding generation should not depend on a predictable part of the response body when jitterBuffer > 0. A cryptographically secure random number generator should be used independently of the response content, or at least the seed should be derived from a source not directly influenced or observable by the attacker. For jitterBuffer > 0, a better approach would be to use a combination of a cryptographically secure random seed and the content hash, ensuring that even if the content is somewhat predictable, the final padding remains unpredictable to an external observer. For jitterBuffer == 0, rand.Reader is used, which is acceptable.

  • Preconditions:

    1. The server-side application uses gzhttp with NewWrapper and the RandomJitter option is enabled with jitterBuffer > 0.
    2. The attacker has some knowledge or can make assumptions about the initial content of the HTTP response (up to jitterBuffer size).

  • Source Code Analysis: In compress.go, within the startGzip function:

    if len(w.randomJitter) > 0 {
    var jitRNG uint32
    if w.jitterBuffer > 0 {
        if w.sha256Jitter {
            h := sha256.New()
            h.Write(w.buf)
            // Use only up to "w.jitterBuffer", otherwise the output depends on write sizes.
            if len(remain) > 0 && len(w.buf) < w.jitterBuffer {
                remain := remain
                if len(remain)+len(w.buf) > w.jitterBuffer {
                    remain = remain[:w.jitterBuffer-len(w.buf)]
                }
                h.Write(remain)
            }
            var tmp [sha256.Size]byte
            jitRNG = binary.LittleEndian.Uint32(h.Sum(tmp[:0]))
        } else {
            h := crc32.Update(0, castagnoliTable, w.buf)
            // Use only up to "w.jitterBuffer", otherwise the output depends on write sizes.
            if len(remain) > 0 && len(w.buf) < w.jitterBuffer {
                remain := remain
                if len(remain)+len(w.buf) > w.jitterBuffer {
                    remain = remain[:w.jitterBuffer-len(w.buf)]
                }
                h = crc32.Update(h, castagnoliTable, remain)
            }
            jitRNG = bits.RotateLeft32(h, 19) ^ 0xab0755de
        }
    } else {
        // Get from rand.Reader
        var tmp [4]byte
        _, err := rand.Read(tmp[:])
        if err != nil {
            return fmt.Errorf("gzhttp: %w", err)
        }
        jitRNG = binary.LittleEndian.Uint32(tmp[:])
    }
    jit := w.randomJitter[:1+jitRNG%uint32(len(w.randomJitter)-1)]
    w.gw.(writer.GzipWriterExt).SetHeader(writer.Header{Comment: jit})
}
    • When w.jitterBuffer > 0, the jitRNG is derived from crc32.Update or sha256.New based on w.buf and potentially remain (initial part of the response body). This makes the seed predictable if w.buf and remain are predictable.
    • When w.jitterBuffer == 0, jitRNG is derived from rand.Reader, which is cryptographically secure and not predictable based on response content.

  • Security Test Case:

    1. Set up a test server using gzhttp.NewWrapper with RandomJitter(256, 1024, false) option and a handler that returns a predictable initial response body (e.g., "PREDICTABLE_PREFIX" + random string).
    2. Make two requests to the server that generate the same predictable prefix in the response body.
    3. Capture the compressed responses for both requests.
    4. Decompress the responses and extract the gzip header comment, which contains the jitter padding.
    5. Compare the jitter padding from both responses. If the vulnerability exists, the jitter padding will be the same for both responses because the predictable prefix leads to the same seed.
    6. If the jitter padding is identical, the test case proves the predictability of the padding and thus the vulnerability.

Potential Integer Overflow in Index Loading

  • Description: In the Index.Load function within index.go, there are several places where integer overflows could potentially occur when reading varint-encoded values, especially for entries, uOff, and cOff. While the code includes checks like v < 0 and v > maxIndexEntries, these might not be sufficient to prevent all forms of integer overflows, especially in edge cases or on 32-bit systems. If an attacker can manipulate the index data to cause an integer overflow during loading, it could lead to unexpected behavior, incorrect offset calculations, or potentially memory corruption if these values are used as sizes or offsets in subsequent operations.

    Steps to trigger the vulnerability:

    1. An attacker needs to provide a crafted S2 index to be loaded by the Index.Load function. This could be achieved if the index is loaded from an external source controlled by the attacker, or if there is a vulnerability allowing modification of existing index data.
    2. The attacker crafts the index data in a way that when varint-decoded, certain values like entries, uOff, or cOff result in integer overflows. For example, providing a very large varint value for the number of entries or offsets could trigger an overflow.
    3. The Index.Load function processes this crafted data, and the integer overflow occurs.
    4. Depending on how the overflowed value is used, it could lead to various consequences, such as incorrect index structure, out-of-bounds access when using the index, or other unexpected behavior.

  • Impact: Integer overflows in index loading could lead to unpredictable behavior, incorrect offset lookups, and potentially memory safety issues if overflowed values are used in memory operations. This could undermine the integrity and reliability of the index functionality and potentially the security of applications relying on it. While direct memory corruption is not guaranteed, the potential for incorrect behavior and data processing makes this a high-rank vulnerability.

  • Vulnerability Rank: high

  • Currently Implemented Mitigations: The code includes checks like v < 0, v > maxIndexEntries, and uOff <= prev during index loading, which provide some level of input validation. However, these checks may not fully prevent integer overflows, especially in edge cases or on different architectures.

  • Missing Mitigations: More robust integer overflow checks should be implemented during varint decoding in Index.Load. Specifically, when reading entries, uOff, and cOff, the code should explicitly check for potential overflows before casting to int or int64 or performing arithmetic operations. Using libraries or built-in functions that provide overflow detection during arithmetic operations would be beneficial. For example, using math/bits.Add64 to check for overflow during addition of offsets. Input validation should be strengthened to reject indices with excessively large values that are likely to cause overflows.

  • Preconditions:

    1. The application loads an S2 index using the Index.Load function.
    2. The attacker can provide or influence the content of the index data being loaded.

  • Source Code Analysis: In index.go, within the Index.Load function, multiple binary.Varint calls are made to read index metadata and offset values.

    func (i *Index) Load(b []byte) ([]byte, error) {
    // ...
    var entries int
    if v, n := binary.Varint(b); n <= 0 { // Reading 'entries'
        return b, ErrCorrupt
    } else {
        if v < 0 || v > maxIndexEntries { // Check, but overflow before this?
            return b, ErrCorrupt
        }
        entries = int(v) // Potential overflow during cast if v is very large
        b = b[n:]
    }
    // ...
    for idx := range i.info {
        var uOff int64
        if hasUncompressed != 0 {
            // Load delta
            if v, n := binary.Varint(b); n <= 0 { // Reading 'uOff' delta
                return b, ErrCorrupt
            } else {
                uOff = v // Potential overflow if v is very large
                b = b[n:]
            }
        }

        if idx > 0 {
            prev := i.info[idx-1].uncompressedOffset
            uOff += prev + (i.estBlockUncomp) // Potential overflow during addition
            if uOff <= prev { // Check after addition, but overflow might have happened
                return b, ErrCorrupt
            }
        }
        if uOff < 0 { // Check after addition, but overflow might have happened
            return b, ErrCorrupt
        }
        i.info[idx].uncompressedOffset = uOff
    }

    // ... similar pattern for cOff ...
}
    • The code reads varint values for entries, uOff, and cOff without explicit checks for integer overflows before casting to int or int64 or performing arithmetic operations.
    • The checks v < 0, v > maxIndexEntries, and uOff <= prev are performed after potential overflows might have already occurred during decoding or arithmetic.
    • On 32-bit systems or with sufficiently large crafted varint inputs, these operations could wrap around, leading to incorrect values being used for index metadata or offsets.

  • Security Test Case:

    1. Craft a malicious S2 index binary. Create a program that generates a valid S2 index structure but inserts a very large varint value for the "entries" field, close to the maximum value for a 64-bit integer.
    2. Prepare a test application that loads this crafted index using Index.Load.
    3. Run the test application with the crafted index.
    4. Observe the behavior of the application. If the application exhibits unexpected behavior, crashes, or produces incorrect results when using the loaded index, it indicates a potential integer overflow vulnerability.
    5. Specifically, try to trigger a Find operation after loading the crafted index and check if it leads to out-of-bounds access or incorrect offset results due to the overflowed entries value.
    6. Repeat the test by crafting large varint values for uOff and cOff deltas to see if overflows in offset calculations can be triggered.
    7. Run these tests on both 64-bit and 32-bit architectures to confirm the vulnerability across different platforms.

Potential Path Traversal in filepathx.Glob and Globs.Expand

  • Description: The filepathx.Glob and Globs.Expand functions in filepathx.go are designed to handle glob patterns, including double-star (**) for recursive directory traversal. While the code uses filepath.Walk to traverse directories, it doesn't explicitly sanitize or validate the paths returned by filepath.Glob or during the walk. If a malicious pattern is crafted, it might be possible to bypass intended directory boundaries and access files outside the intended scope, leading to a path traversal vulnerability. This is especially concerning if filepathx.Glob is used in contexts where user-provided input is directly used as a glob pattern.

    Steps to trigger the vulnerability:

    1. An attacker needs to control or influence the glob pattern passed to filepathx.Glob or Globs.Expand. This could occur if the application uses user input to construct file paths or glob patterns.
    2. The attacker crafts a malicious glob pattern that includes path traversal sequences like .. (e.g., testdir/**/../../../sensitive_file).
    3. filepathx.Glob or Globs.Expand processes this pattern.
    4. Due to the lack of explicit path sanitization, filepath.Walk might traverse directories outside the intended base directory, and filepath.Glob might match files outside the intended scope if the underlying OS and filesystem allow path traversal through symlinks or other mechanisms.
    5. The application might then perform operations (e.g., read, compress, decompress) on these files outside the intended scope, leading to unauthorized file access.

  • Impact: A path traversal vulnerability could allow an attacker to access sensitive files or directories that should not be accessible. This could lead to information disclosure, unauthorized modification of files, or other security breaches, depending on how the application uses the files matched by the glob pattern. The impact is high because it allows bypassing intended access controls and potentially reading arbitrary files on the system if the application has sufficient permissions.

  • Vulnerability Rank: high

  • Currently Implemented Mitigations: The code uses standard Go functions like filepath.Glob and filepath.Walk, which provide some level of path handling safety. However, these functions alone do not prevent path traversal if the input pattern itself is malicious. There are no explicit sanitization or validation steps in filepathx.Glob or Globs.Expand to prevent path traversal sequences.

  • Missing Mitigations: Input validation and sanitization should be implemented in filepathx.Glob and Globs.Expand. Before processing the glob pattern or the paths returned by filepath.Glob and filepath.Walk, the code should:

    1. Validate the glob pattern to reject patterns containing suspicious sequences like ... A regular expression or custom parsing logic could be used to detect and reject such patterns.
    2. After getting the matched paths, canonicalize them using filepath.Clean and verify that they are still within the intended base directory. This can be done by resolving both the base directory and the matched path to their absolute forms and checking if the matched path is a prefix of the base directory.
    3. Consider using chroot or similar sandboxing techniques if the application needs to process files from untrusted sources, to further limit the impact of path traversal vulnerabilities.

  • Preconditions:

    1. The application uses filepathx.Glob or Globs.Expand to process file paths.
    2. The glob pattern used in filepathx.Glob or Globs.Expand is derived from user input or an untrusted source.

  • Source Code Analysis: In filepathx.go, the Glob and Expand functions use filepath.Glob and filepath.Walk without additional path sanitization.

    // Glob adds double-star support to the core path/filepath Glob function.
func Glob(pattern string) ([]string, error) {
    if !strings.Contains(pattern, "**") {
        // passthru to core package if no double-star
        return filepath.Glob(pattern)
    }
    return Globs(strings.Split(pattern, "**")).Expand()
}

// Expand finds matches for the provided Globs.
func (globs Globs) Expand() ([]string, error) ([]string, error) {
    var matches = []string{""} // accumulate here
    for _, glob := range globs {
        var hits []string
        var hitMap = map[string]bool{}
        for _, match := range matches {
            paths, err := filepath.Glob(match + glob) // Potentially vulnerable call
            if err != nil {
                return nil, err
            }
            for _, path := range paths {
                err = filepath.Walk(path, func(path string, info os.FileInfo, err error) error { // Potentially vulnerable walk
                    if err != nil {
                        return err
                    }
                    // save deduped match from current iteration
                    if _, ok := hitMap[path]; !ok {
                        hits = append(hits, path)
                        hitMap[path] = true
                    }
                    return nil
                })
                if err != nil {
                    return nil, err
                }
            }
        }
        matches = hits
    }
    return matches, nil
}
    • The code directly uses filepath.Glob(match + glob) and filepath.Walk(path, ...) which can be vulnerable to path traversal if glob or match contains malicious sequences.
    • There are no checks to ensure that the matched paths stay within the intended directory or to sanitize the input glob patterns.

  • Security Test Case:

    1. Create a test directory testdir and within it, create a subdirectory subdir. Inside testdir, create a file safe_file.txt. Outside testdir (e.g., in the temporary directory), create a sensitive file sensitive_file.txt.
    2. Write a test program that uses filepathx.Glob to find files based on a user-provided pattern, and then attempts to read the content of the matched files.
    3. Provide the test program with a malicious glob pattern like testdir/**/../../../sensitive_file.txt.
    4. Run the test program. If the vulnerability exists, the program will be able to access and potentially read the content of sensitive_file.txt outside the testdir directory, demonstrating path traversal.
    5. Check if the program outputs the content of sensitive_file.txt or if it throws an error indicating that it could not access the file due to path restrictions. If it reads the sensitive file, the test case is successful in demonstrating the path traversal vulnerability.

Potential Buffer Overflow in FSE Decompression Tables

  • Description: In fse/decompress.go and zstd/fse_decoder.go, the readNCount function reads the symbol distribution from the input stream to construct decoding tables. There are checks to prevent excessively large tableLog and symbolLen values. However, if a crafted input provides a large tableLog value close to tablelogAbsoluteMax (9 in zstd/fse_decoder.go and implicitly 15 in fse/decompress.go) and a large number of symbols, the allocated decoding tables (s.decTable in fse/decompress.go and s.dt in zstd/fse_decoder.go) might become very large (up to 2^tablelogAbsoluteMax entries). If the subsequent decompression process attempts to access entries beyond the allocated size based on corrupted state or incorrect tableLog, it could lead to a buffer overflow read. This is especially relevant if the DecompressLimit is not properly enforced or bypassed.

    Steps to trigger the vulnerability:

    1. An attacker crafts a malicious FSE compressed stream with a tableLog value close to tablelogAbsoluteMax and a large symbolLen.
    2. The readNCount function in fse/decompress.go or zstd/fse_decoder.go reads the header and allocates a potentially large decoding table (e.g., s.decTable of size 2^tableLog).
    3. During the decompress function, due to corrupted or crafted input data following the header, the decoder state might transition to an invalid state index that is outside the bounds of the allocated s.decTable.
    4. When s.decTable[state] is accessed in decoder.next or decoder.nextFast, it results in an out-of-bounds read, potentially leading to a crash or information disclosure.

  • Impact: A buffer overflow read in FSE decompression tables could lead to a crash, denial of service, or potentially information disclosure if sensitive memory regions are accessed. Although it's a read overflow, it can still have security implications, especially if it can be exploited further. The vulnerability is ranked high due to the potential for crashing the application and the complexity of FSE decompression logic, which might hide further exploitable conditions.

  • Vulnerability Rank: high

  • Currently Implemented Mitigations:

    • There are checks in readNCount to limit tableLog and symbolLen to prevent excessively large tables.
    • DecompressLimit is intended to limit the output size, but it might not directly prevent table-related buffer overflows during decompression.

  • Missing Mitigations:

    • More robust bounds checking should be implemented in the decoder.next and decoder.nextFast functions to ensure that the state index is always within the valid range of the s.decTable.
    • Fuzzing should be performed specifically targeting the FSE decompression logic with crafted inputs that attempt to trigger out-of-bounds table accesses, especially around maximum tableLog and symbolLen values and with corrupted compressed data following the header.
    • Consider adding explicit size checks before accessing s.decTable using the table size derived from actualTableLog.

  • Preconditions:

    1. The application uses FSE decompression on untrusted or attacker-controlled compressed data.
    2. The attacker can craft a malicious FSE compressed stream with specific header values and potentially corrupted data following the header.

  • Source Code Analysis: In fse/decompress.go and zstd/fse_decoder.go, the readNCount function reads table parameters and allocates decTable (or dt). Then, in decompress and decoder.next/nextFast, the decTable is accessed using state as an index.

    In fse/decompress.go:

    func (s *Scratch) decompress() error {
    // ...
    var s1, s2 decoder
    // Initialize and decode first state and symbol.
    s1.init(br, s.decTable, s.actualTableLog)
    s2.init(br, s.decTable, s.actualTableLog)
    // ...
    for br.off >= 8 { // Main loop
        // ...
        tmp[off+0] = s1.nextFast() // Accessing s.decTable using s1.state
        tmp[off+1] = s2.nextFast() // Accessing s.decTable using s2.state
        // ...
    }
    // ...
}

type decoder struct {
    state uint16
    br    *bitReader
    dt    []decSymbol // s.decTable is passed here
}

func (d *decoder) nextFast() uint8 {
    n := d.dt[d.state] // Accessing d.dt (s.decTable) using d.state
    lowBits := d.br.getBitsFast(n.nbBits)
    d.state = n.newState + lowBits
    return n.symbol
}

    Similar pattern exists in zstd/fse_decoder.go for fseDecoder and fseState. The vulnerability lies in the potential for d.state to become out of bounds due to corrupted input, leading to an out-of-bounds read when accessing d.dt[d.state].

  • Security Test Case:

    1. Craft a malicious FSE compressed stream. Create a program that generates a valid FSE header with a tableLog close to tablelogAbsoluteMax and a large symbolLen.
    2. Append corrupted or crafted data after the header in the compressed stream. This data should be designed to manipulate the decoder state during decompression.
    3. Prepare a test application that uses fse.Decompress or zstd decoder to decompress this crafted stream. Set a large DecompressLimit if necessary to avoid premature termination.
    4. Run the test application with the crafted compressed stream.
    5. Monitor the application for crashes or unexpected behavior. Use memory debugging tools (like AddressSanitizer if available) to detect out-of-bounds reads during decompression.
    6. If a crash or memory error is detected during the table access in decoder.next or decoder.nextFast, it indicates a successful trigger of the buffer overflow read vulnerability.
    7. Vary the crafted data and tableLog/symbolLen values to explore different out-of-bounds scenarios.

Potential Integer Overflow in Skippable Frame Size Calculation

  • Description: The calcSkippableFrame function in frameenc.go calculates the size of a skippable frame to add padding. The function takes written (bytes already written) and wantMultiple (padding multiple) as int64. While the function checks if wantMultiple <= 0 and written < 0, it does not explicitly check for integer overflows during the calculation of toAdd. If wantMultiple is very large and written is also large, the calculation wantMultiple - leftOver and subsequent additions of wantMultiple within the loop could potentially lead to an integer overflow, resulting in a negative or unexpectedly small toAdd value. This could lead to writing a skippable frame smaller than skippableFrameHeader or other unexpected behavior when creating skippable frames.

    Steps to trigger the vulnerability:

    1. An attacker cannot directly trigger this vulnerability through external input to the compression library itself. However, if an application using this library allows an attacker to control or influence the wantMultiple parameter passed to calcSkippableFrame and the amount of data being written, they might be able to trigger an overflow indirectly. This is highly unlikely in most real-world scenarios where padding is controlled server-side.
    2. An attacker needs to find a way to manipulate the wantMultiple and written values within the application's logic that calls calcSkippableFrame.
    3. If wantMultiple and written are crafted such that the calculation wantMultiple - leftOver or subsequent additions in the loop overflow, toAdd will become a small or negative value.
    4. When skippableFrame is called with this overflowed total value, it might lead to an error if total < skippableFrameHeader or unexpected behavior due to the incorrect padding size.

  • Impact: Integer overflow in calcSkippableFrame could lead to incorrect padding sizes for skippable frames, potentially causing errors during frame creation or unexpected behavior in applications relying on consistent padding. While not a direct memory corruption or information disclosure vulnerability, it can lead to denial of service if frame creation fails or unexpected application behavior. The vulnerability is ranked high due to the potential for unexpected behavior in security-sensitive padding logic, although the direct exploitability by an external attacker is low.

  • Vulnerability Rank: high

  • Currently Implemented Mitigations: The function includes checks for wantMultiple <= 0 and written < 0 to prevent invalid inputs, but no explicit integer overflow checks during the calculation of toAdd.

  • Missing Mitigations: Integer overflow checks should be added to the calcSkippableFrame function, especially during the calculation of toAdd. Before adding wantMultiple in the loop, the code should check if toAdd + wantMultiple would overflow the int type. Alternatively, using a larger integer type (like int64) for intermediate calculations and then casting to int with overflow checks could mitigate this issue. Input validation should be considered for wantMultiple to ensure it is within a reasonable range that will not cause overflows in typical usage scenarios.

  • Preconditions:

    1. The application uses calcSkippableFrame to calculate padding size for skippable frames.
    2. An attacker can indirectly influence the wantMultiple and written parameters passed to calcSkippableFrame through application logic.

  • Source Code Analysis: In frameenc.go, the calcSkippableFrame function performs calculations that are potentially vulnerable to integer overflow.

    // calcSkippableFrame will return a total size to be added for written
// to be divisible by multiple.
// The value will always be > skippableFrameHeader.
// The function will panic if written < 0 or wantMultiple <= 0.
func calcSkippableFrame(written, wantMultiple int64) int {
    if wantMultiple <= 0 {
        panic("wantMultiple <= 0")
    }
    if written < 0 {
        panic("written < 0")
    }
    leftOver := written % wantMultiple
    if leftOver == 0 {
        return 0
    }
    toAdd := wantMultiple - leftOver // Potential overflow if wantMultiple is very large and leftOver is small
    for toAdd < skippableFrameHeader {
        toAdd += wantMultiple // Potential overflow in repeated addition
    }
    return int(toAdd)
}
    • The lines toAdd := wantMultiple - leftOver and toAdd += wantMultiple in the loop are potential locations for integer overflows if wantMultiple is sufficiently large.
    • The return value int(toAdd) is a cast from int64 to int, which could also truncate or wrap around if toAdd is outside the valid range of int.

  • Security Test Case:

    1. Create a test program that calls calcSkippableFrame with very large wantMultiple values (close to math.MaxInt64) and varying written values to try to trigger an integer overflow in the toAdd calculation.
    2. Run the test program and check the returned toAdd values. If toAdd becomes negative or unexpectedly small when large inputs are used, it indicates an integer overflow vulnerability.
    3. Specifically, check if skippableFrame called with the potentially overflowed toAdd value results in an error due to total < skippableFrameHeader or if it creates a skippable frame with an invalid size.
    4. Monitor the behavior of the application and check for any panics or unexpected results caused by the overflowed padding size.

Potential Out-of-bounds Read in FSE Decoding - AMD64 Assembly Optimization

  • Description: The assembly optimized buildDtable_asm function in zstd/fse_decoder_amd64.go for building FSE decoding tables might contain vulnerabilities if not carefully implemented. Specifically, if the assembly code incorrectly calculates table indices or performs memory accesses, it could lead to out-of-bounds reads from the s.norm or s.stateTable arrays, or writes to s.dt outside its allocated bounds. While Go code provides memory safety, assembly code can bypass these checks if not written correctly. A crafted compressed stream designed to exploit potential flaws in the assembly logic could trigger an out-of-bounds read, potentially leading to information disclosure or crashes.

    Steps to trigger the vulnerability:

    1. An attacker crafts a malicious FSE compressed stream with specific header values designed to trigger a specific code path in the buildDtable_asm function.
    2. The crafted header might aim to manipulate tableLog, symbolLen, or the normalized counts (s.norm) in a way that causes the assembly code to calculate invalid memory addresses.
    3. During the execution of buildDtable_asm, an out-of-bounds read occurs when accessing s.norm, s.stateTable, or when writing to s.dt due to incorrect index calculations in the assembly code.
    4. This out-of-bounds read could potentially leak sensitive information from process memory or cause a crash, depending on the nature and location of the read.

  • Impact: An out-of-bounds read in the assembly optimized FSE decoding table construction could lead to information disclosure, denial of service (crash), or potentially further exploitation if the read data can be controlled or observed. The vulnerability is ranked high due to the potential for serious security impacts and the complexity of analyzing assembly code for memory safety issues, especially in the context of a complex decompression algorithm.

  • Vulnerability Rank: high

  • Currently Implemented Mitigations: The Go code surrounding the assembly function (fseDecoder.buildDtable) includes some checks and type safety, but these may not fully prevent vulnerabilities in the assembly implementation itself. The readNCount function performs checks on tableLog and symbolLen, but the assembly code might still have logical flaws leading to out-of-bounds accesses within the allowed table sizes.

  • Missing Mitigations:

    • Thorough manual review and verification of the buildDtable_asm assembly code are necessary to ensure memory safety and correctness, especially concerning index calculations and bounds checking.
    • Automated tools for assembly code analysis or symbolic execution could be used to detect potential out-of-bounds read vulnerabilities.
    • Fuzzing should be performed with crafted FSE compressed streams specifically targeting the assembly optimized decoding path to attempt to trigger out-of-bounds reads during table construction. Memory sanitizers (like AddressSanitizer) should be used during fuzzing to detect memory access violations.
    • Add assertions within the assembly code (if feasible and without significant performance impact) to check for valid memory access ranges before performing reads or writes to the decoding tables.

  • Preconditions:

    1. The application uses Zstandard decompression on an AMD64 architecture where assembly optimizations are enabled (amd64 && !appengine && !noasm && gc build tag).
    2. The attacker can provide a crafted Zstandard compressed stream that utilizes FSE encoding for literals or sequences and is designed to trigger the vulnerable code path in buildDtable_asm.

  • Source Code Analysis: In zstd/fse_decoder_amd64.go, the buildDtable_asm function is called by fseDecoder.buildDtable. The assembly code (not provided in the Go file itself, but would be in a separate .s file or generated) is responsible for building the decoding table s.dt based on the normalized counts in s.norm and state table s.stateTable.

    // buildDtable will build the decoding table.
func (s *fseDecoder) buildDtable() error {
    ctx := buildDtableAsmContext{
        stateTable: &s.stateTable[0],
        norm:       &s.norm[0],
        dt:         (*uint64)(&s.dt[0]),
    }
    code := buildDtable_asm(s, &ctx)

    if code != 0 {
        // ... error handling ...
    }
    return nil
}

    The vulnerability would reside within the buildDtable_asm assembly implementation. Without access to the assembly source code, static analysis is limited. However, potential areas of concern include:

    • Index calculations: Incorrectly calculated indices when accessing s.norm, s.stateTable, or s.dt based on tableSize, highThreshold, step, position, or loop variables.
    • Memory access patterns: Flaws in how memory addresses are derived and accessed in assembly, potentially leading to reads outside the intended bounds of the allocated tables.
    • State management: Errors in managing the decoder state during table construction in assembly, causing incorrect table entries or out-of-bounds accesses in subsequent decoding stages.

  • Security Test Case:

    1. Craft a malicious Zstandard compressed stream. Create a program that generates a Zstandard stream with an FSE encoded block. The FSE header should be crafted with specific tableLog and symbolLen values, and the compressed data following the header should be designed to trigger the assembly optimized buildDtable_asm function.
    2. Prepare a test application that uses the Go Zstandard decoder to decompress this crafted stream. Ensure the application is built and run on an AMD64 architecture with assembly optimizations enabled.
    3. Run the test application with the crafted compressed stream.
    4. Monitor the application for crashes or unexpected behavior. Use memory debugging tools like AddressSanitizer (if available and compatible with assembly code) or Valgrind to detect out-of-bounds memory reads or writes during the execution of buildDtable_asm.
    5. If a crash or memory error is detected within buildDtable_asm or during table access after table construction, it indicates a successful trigger of the out-of-bounds read vulnerability.
    6. Vary the crafted header values and compressed data to explore different code paths in the assembly function and attempt to trigger different out-of-bounds scenarios.

Potential Out-of-bounds Write in fastGen.addBlock History Buffer Management

  • Description: The fastGen.addBlock function in fast_encoder.go manages a history buffer (e.hist) to store previously processed data for efficient compression. When the buffer is full, it attempts to move the last maxMatchOffset bytes to the beginning of the buffer using unsafe pointer casting and slice manipulation. If the length of e.hist is less than maxMatchOffset at the time of buffer reset, the memory copy operation *(*[maxMatchOffset]byte)(e.hist) = *(*[maxMatchOffset]byte)(e.hist[offset:]) could lead to an out-of-bounds write, as it attempts to write maxMatchOffset bytes into a potentially smaller slice. This could lead to memory corruption.

    Steps to trigger the vulnerability:

    1. Craft an input stream that causes the fastGen.addBlock function to be called repeatedly, filling up the history buffer e.hist.
    2. Ensure that at some point during the compression process, the length of e.hist becomes less than maxMatchOffset due to specific compression logic or input patterns (this scenario needs further investigation to reliably trigger).
    3. Provide more input data that triggers the buffer reset condition in addBlock (len(e.hist) + len(src) > cap(e.hist)).
    4. When the code reaches the memory copy *(*[maxMatchOffset]byte)(e.hist) = *(*[maxMatchOffset]byte)(e.hist[offset:]) with len(e.hist) < maxMatchOffset, an out-of-bounds write occurs.

  • Impact: An out-of-bounds write in fastGen.addBlock can lead to memory corruption, potentially causing crashes, unpredictable behavior, or exploitable conditions. While direct external exploitability might be low, memory corruption vulnerabilities are generally considered high severity due to their potential security implications.

  • Vulnerability Rank: high

  • Currently Implemented Mitigations: The code includes a check if cap(e.hist) < maxMatchOffset*2 which is intended to prevent unexpected buffer sizes, but it does not directly prevent the out-of-bounds write if len(e.hist) is less than maxMatchOffset during the copy operation.

  • Missing Mitigations:

    • Before performing the memory copy in fastGen.addBlock, ensure that len(e.hist) is always at least maxMatchOffset. If it's not, handle this case gracefully, possibly by allocating a new buffer or adjusting the copy logic to avoid out-of-bounds access.
    • Add explicit bounds checks before the memory copy to verify that the source and destination ranges are within the allocated memory for e.hist.
    • Thoroughly review the code paths that lead to fastGen.addBlock to understand under what conditions len(e.hist) could become less than maxMatchOffset before a buffer reset, and ensure these scenarios are handled safely.
    • Fuzzing with diverse input data, especially edge cases and boundary conditions, should be performed to try and trigger this potential out-of-bounds write. Memory sanitizers should be used during fuzzing to detect memory access violations.

  • Preconditions:

    1. The application uses flate compression with fast encoding levels (1-6).
    2. The attacker can provide input data that is processed by the flate encoder, potentially influencing the state of the fastGen history buffer.

  • Source Code Analysis: In flate/fast_encoder.go, the fastGen.addBlock function contains the potentially vulnerable memory copy operation:

    func (e *fastGen) addBlock(src []byte) int32 {
    // ...
    else {
        if cap(e.hist) < maxMatchOffset*2 {
            panic("unexpected buffer size")
        }
        // Move down
        offset := int32(len(e.hist)) - maxMatchOffset
        // copy(e.hist[0:maxMatchOffset], e.hist[offset:])
        *(*[maxMatchOffset]byte)(e.hist) = *(*[maxMatchOffset]byte)(e.hist[offset:]) // Potential out-of-bounds write
        e.cur += offset
        e.hist = e.hist[:maxMatchOffset]
    }
    // ...
}
    • The line *(*[maxMatchOffset]byte)(e.hist) = *(*[maxMatchOffset]byte)(e.hist[offset:]) performs a direct memory copy using unsafe pointer casting.
    • If len(e.hist) is less than maxMatchOffset, then e.hist[:maxMatchOffset] would attempt to access memory beyond the bounds of the allocated slice, resulting in an out-of-bounds write.
    • The condition if cap(e.hist) < maxMatchOffset*2 is a precondition, but it does not guarantee that len(e.hist) will always be >= maxMatchOffset when the copy is performed.

  • Security Test Case:

    1. Create a test program that uses the flate compressor with a fast encoding level (e.g., BestSpeed).
    2. Construct a crafted input byte stream that is designed to manipulate the state of the fastGen history buffer and potentially reduce its length below maxMatchOffset before triggering a buffer reset. (This step might require some experimentation and understanding of the flate compression algorithm and encoder state management.)
    3. Compress the crafted input using the flate writer.
    4. Run the test program with memory sanitizers (like AddressSanitizer) enabled to detect out-of-bounds writes during the compression process.
    5. If AddressSanitizer reports an out-of-bounds write within the fastGen.addBlock function, it confirms the vulnerability.
    6. Vary the crafted input to explore different scenarios and potentially refine the test case to reliably trigger the vulnerability.