[Support]: Frigate unable to handle zfs dataset quota #11266

hollanbm · 2024-05-06T14:56:30Z

hollanbm
May 6, 2024

Describe the problem you are having

I have a 4TB zpool, with a dataset (frigate/data) with a quota of 512GB. Frigate is unable to continue recording, or function normally when the dataset has reached it's quota.

When the dataset reaches it's quota, the webui is unable to display images for the camera display, and I'm also unable to watch cameras live. The recordings do seem to continue to save.

If I delete the data inside the dataset, functionality resumes, but I was under the impression, frigate would delete old files, if the storage ran out. I suspect hitting quota limit, is not the same thing as running out of disk space???

Version

ghcr.io/blakeblackshear/frigate@sha256:2068baac8d8b9a29a97483d0ab603b024fe606ec7d555b70da5a024fbeab12e3

Frigate config file

# yaml-language-server: $schema=http://10.254.1.77:5000/api/config/schema.json

mqtt:
  enabled: false

detectors:
  tensorrt:
    type: tensorrt
    device: 0 #This is the default, select the first GPU

model:
  path: /config/model_cache/tensorrt/yolov7x-320.trt
  input_tensor: nchw
  input_pixel_format: rgb
  width: 320
  height: 320

# NOTE: Can be overridden at the camera level
objects:
  filters:
    person:
      # minimum score for the object to initiate tracking (default: shown below)
      min_score: 0.60
      # minimum decimal percentage for tracked object's computed score to be considered a true positive (default: shown below)
      threshold: 0.75
    car:
      min_score: 0.60
      threshold: 0.75
    dog:
      min_score: 0.60
      threshold: 0.75
    bicycle:
      min_score: 0.60
      threshold: 0.75
    motorcycle:
      min_score: 0.60
      threshold: 0.75
    package:
      min_score: 0.60
      threshold: 0.75
    ups:
      min_score: 0.60
      threshold: 0.75
    fedex:
      min_score: 0.60
      threshold: 0.75
    amazon:
      min_score: 0.60
      threshold: 0.75

record:
  enabled: true
  retain:
    days: 14
    mode: motion
  events:
    retain:
      default: 30
      mode: active_objects

snapshots:
  enabled: true
  bounding_box: true
  timestamp: false
  crop: false
  quality: 100
  retain:
    default: 30

detect:
  enabled: true
  fps: 5

audio:
  enabled: false
  min_volume: 500
  listen:
  - bark

ffmpeg:
  hwaccel_args: preset-nvidia-h264
  output_args:
    record: preset-record-ubiquiti #preset-record-generic-audio-copy

live:
  height: 1080
  quality: 1

birdseye:
  quality: 1

go2rtc:
  streams:
    back_driveway:
    - rtspx://10.254.15.2:7441/2HmyMRarR9xUXYB7
    back_driveway_med:
    - rtspx://10.254.15.2:7441/iWjfzQBqzMyTNE3W
    back_porch:
    - rtspx://10.254.15.2:7441/W6744vv3WmVYboJv
    back_porch_med:
    - rtspx://10.254.15.2:7441/RXH2vWlIMRWrUkWd
    back_yard:
    - rtspx://10.254.15.2:7441/SGZRCKsW6D4i0jWd
    back_yard_med:
    - rtsps://10.254.15.2:7441/Vv1PTR75u1cNngIJ
    bear_roof:
    - rtspx://10.254.15.2:7441/NsycYa8Vy93sN9Mb
    bear_roof_med:
    - rtspx://10.254.15.2:7441/IaFIPXGG4lLUFq4o
    front_porch:
    - rtspx://10.254.15.2:7441/X68oQnxtEaGPpHyd
    front_porch_med:
    - rtspx://10.254.15.2:7441/66OBUm2m2Ntulraz
    front_yard:
    - rtspx://10.254.15.2:7441/PoSOR67kIdCMXG4Y
    front_yard_med:
    - rtspx://10.254.15.2:7441/p54u2PVlyxHoBo7c
    garage_driveway:
    - rtspx://10.254.15.2:7441/lwbS2B4Q5Tj1aLwX
    garage_driveway_med:
    - rtspx://10.254.15.2:7441/Nj0Eh1nO8OY9CLtI
    side_yard:
    - rtspx://10.254.15.2:7441/zjLRxxvTghHyKHfz
    side_yard_med:
    - rtspx://10.254.15.2:7441/LRbMhmUXq2llA0pt
cameras:
  front_yard: # g5 pr # g3 bullet
    ui:
      order: 1
    detect:
      width: 1280
      height: 720
    live:
      stream_name: front_yard
    ffmpeg:
      inputs:
      - path: rtsp://127.0.0.1:8554/front_yard
        input_args: preset-rtsp-restream   #preset-rtsp-restream-low-latency
        roles:
        - record
      - path: rtsp://127.0.0.1:8554/front_yard_med
        input_args: preset-rtsp-restream   #preset-rtsp-restream-low-latency
        roles:
        - detect
    objects:
      track:
      - bicycle
      - car
      - dog
      - motorcycle
      - person
      - package
      - ups
      - fedex
      - amazon
    motion:
      mask:
      - 169,0,551,0,588,107,508,191,278,199,170,108
      - 1036,510,1139,563,1226,366,1188,343
      - 1280,229,1242,344,1212,321,1252,199
      - 1101,285,1146,309,1161,255,1119,248
      - 917,173,1007,216,1035,177,949,137
      - 688,156,700,91,787,102,784,161
      - 84,149,197,102,283,0,0,0,0,204
      - 1027,720,1078,652,1050,638,989,600,958,571,941,532,923,492,935,453,900,426,838,417,809,430,788,450,770,499,757,556,741,614,688,675,619,720
    zones:
      grass:
        objects:
        - dog
        - person
        coordinates: 
          502,199,0,269,30,720,615,720,735,619,768,563,771,536,781,505,800,457,852,433,918,476,1155,312,1073,253,959,204,843,170
      street:
        objects:
        - amazon
        - bicycle
        - car
        - fedex
        - motorcycle
        - ups
        coordinates: 
          1001,117,951,95,926,84,858,86,759,89,710,84,565,84,463,90,354,99,239,115,109,127,0,126,0,199
      driveway:
        objects:
        - amazon
        - bicycle
        - car
        - fedex
        - motorcycle
        - ups
        coordinates:
          1242,356,1256,325,824,135,740,137
      sidewalk:
        objects:
          - package
          - person
        coordinates:
          0,263,76,229,131,220,192,210,288,194,411,184,578,166,712,163,812,158,852,174
      front_sidewalk:
        objects:
          - package
          - person
        coordinates: 
          1034,491,1115,424,1220,340,1177,306,938,440,921,476,937,533,962,561,1077,638,1136,559
  front_porch: # g5 flex
    ui:
      order: 2
    detect:
      width: 1280
      height: 720
    live:
      stream_name: front_porch
    ffmpeg:
      inputs:
      - path: rtsp://127.0.0.1:8554/front_porch
        input_args: preset-rtsp-restream   #preset-rtsp-restream-low-latency
        roles:
        - record
      - path: rtsp://127.0.0.1:8554/front_porch_med
        input_args: preset-rtsp-restream   #preset-rtsp-restream-low-latency
        roles:
        - detect
    objects:
      track:
      - bicycle
      - car
      - dog
      - motorcycle
      - person
      - face
      - package
    motion:
      mask:
      - 1280,333,1280,143,1108,62,1057,245,861,396,727,491,593,627,726,720,1074,720,1177,508
      - 203,720,156,661,166,632,187,600,211,559,263,510,321,474,405,482,415,528,357,635,370,720
      - 1024,140,1067,149,1074,102,1031,81
      - 757,48,800,27,921,29,917,75
      - 497,69,490,83,570,77,576,53
  back_driveway: # g3 bullet
    ui:
      order: 3
    detect:
      width: 1024
      height: 576
    live:
      stream_name: back_driveway
    ffmpeg:
      inputs:
      - path: rtsp://127.0.0.1:8554/back_driveway
        input_args: preset-rtsp-restream   #preset-rtsp-restream-low-latency
        roles:
        - record
      - path: rtsp://127.0.0.1:8554/back_driveway_med
        input_args: preset-rtsp-restream   #preset-rtsp-restream-low-latency
        roles:
        - detect
    objects:
      track:
      - dog
      - person
      - bicycle
  garage_driveway: # g4 bullet
    ui:
      order: 4
    detect:
      width: 1280
      height: 720
    live:
      stream_name: garage_driveway
    ffmpeg:
      inputs:
      - path: rtsp://127.0.0.1:8554/garage_driveway
        input_args: preset-rtsp-restream   #preset-rtsp-restream-low-latency
        roles:
        - record
      - path: rtsp://127.0.0.1:8554/garage_driveway_med
        input_args: preset-rtsp-restream   #preset-rtsp-restream-low-latency
        roles:
        - detect
        - audio
    objects:
      track:
      - bicycle
      - dog
      - person
  back_porch: # g5 flex
    ui:
      order: 5
    detect:
      width: 1280
      height: 720
    live:
      stream_name: back_porch
    ffmpeg:
      inputs:
      - path: rtsp://127.0.0.1:8554/back_porch
        input_args: preset-rtsp-restream   #preset-rtsp-restream-low-latency
        roles:
        - record
      - path: rtsp://127.0.0.1:8554/back_porch_med
        input_args: preset-rtsp-restream   #preset-rtsp-restream-low-latency
        roles:
        - detect
        - audio
    objects:
      track:
      - dog
      - person
      - face
    motion:
      mask:
      - 973,201,789,135,724,98,793,42,1022,0,1233,96,1199,244
  back_yard: # g5 pr # g3 bullet
    ui:
      order: 6
    detect:
      width: 1280
      height: 720
    live:
      stream_name: back_yard
    ffmpeg:
      inputs:
      - path: rtsp://127.0.0.1:8554/back_yard
        input_args: preset-rtsp-restream   #preset-rtsp-restream-low-latency
        roles:
        - record
      - path: rtsp://127.0.0.1:8554/back_yard_med
        input_args: preset-rtsp-restream   #preset-rtsp-restream-low-latency
        roles:
        - detect
        - audio
    objects:
      track:
      - dog
      - person
    motion:
      mask:
      - 1032,158,1016,94,1072,63,1280,129,1280,294
  side_yard: # g3 bullet
    ui:
      order: 7
    detect:
      width: 1024
      height: 576
    live:
      stream_name: side_yard
    ffmpeg:
      inputs:
      - path: rtsp://127.0.0.1:8554/side_yard
        input_args: preset-rtsp-restream   #preset-rtsp-restream-low-latency
        roles:
        - record
      - path: rtsp://127.0.0.1:8554/side_yard_med
        input_args: preset-rtsp-restream   #preset-rtsp-restream-low-latency
        roles:
        - detect
    objects:
      track:
      - bicycle
      - dog
      - person
    motion:
      mask:
      - 694,301,764,196,824,147,961,212,842,395
      - 0,0,0,21,189,24,190,0
  bear_roof: # g3 bullet
    ui:
      order: 8
    detect:
      width: 1024
      height: 576
    live:
      stream_name: bear_roof
    ffmpeg:
      inputs:
      - path: rtsp://127.0.0.1:8554/bear_roof
        input_args: preset-rtsp-restream   #preset-rtsp-restream-low-latency
        roles:
        - record
      - path: rtsp://127.0.0.1:8554/bear_roof_med
        input_args: preset-rtsp-restream   #preset-rtsp-restream-low-latency
        roles:
        - detect
    objects:
      track:
      - person

Relevant log output

frigate  | 2024-05-06 14:36:20.251295253  [INFO] Starting go2rtc healthcheck service...
frigate  | 2024-05-06 14:36:29.213997486  [2024-05-06 14:36:29] frigate.record.maintainer      ERROR   : Error occurred when attempting to maintain recording cache
frigate  | 2024-05-06 14:36:29.214146427  [2024-05-06 14:36:29] frigate.record.maintainer      ERROR   : [Errno 122] Disk quota exceeded: '/media/frigate/recordings/2024-05-06'
frigate  | 2024-05-06 14:36:33.742134396  [2024-05-06 14:36:33] frigate.record.maintainer      ERROR   : Error occurred when attempting to maintain recording cache
frigate  | 2024-05-06 14:36:33.742296262  [2024-05-06 14:36:33] frigate.record.maintainer      ERROR   : [Errno 122] Disk quota exceeded: '/media/frigate/recordings/2024-05-06'
frigate  | 2024-05-06 14:36:39.138197980  [2024-05-06 14:36:39] frigate.record.maintainer      ERROR   : Error occurred when attempting to maintain recording cache
frigate  | 2024-05-06 14:36:39.138319730  [2024-05-06 14:36:39] frigate.record.maintainer      ERROR   : [Errno 122] Disk quota exceeded: '/media/frigate/recordings/2024-05-06'
frigate  | 2024-05-06 14:36:43.704327041  [2024-05-06 14:36:43] frigate.record.maintainer      ERROR   : Error occurred when attempting to maintain recording cache
frigate  | 2024-05-06 14:36:43.704614930  [2024-05-06 14:36:43] frigate.record.maintainer      ERROR   : [Errno 122] Disk quota exceeded: '/media/frigate/recordings/2024-05-06'
frigate  | 2024-05-06 14:36:48.969768612  [2024-05-06 14:36:48] frigate.record.maintainer      ERROR   : Error occurred when attempting to maintain recording cache
frigate  | 2024-05-06 14:36:48.969936526  [2024-05-06 14:36:48] frigate.record.maintainer      ERROR   : [Errno 122] Disk quota exceeded: '/media/frigate/recordings/2024-05-06'
frigate  | 2024-05-06 14:36:53.706294918  [2024-05-06 14:36:53] frigate.record.maintainer      ERROR   : Error occurred when attempting to maintain recording cache
frigate  | 2024-05-06 14:36:53.706463127  [2024-05-06 14:36:53] frigate.record.maintainer      ERROR   : [Errno 122] Disk quota exceeded: '/media/frigate/recordings/2024-05-06'
frigate  | 2024-05-06 14:36:59.139358950  [2024-05-06 14:36:59] frigate.record.maintainer      ERROR   : Error occurred when attempting to maintain recording cache
frigate  | 2024-05-06 14:36:59.139447634  [2024-05-06 14:36:59] frigate.record.maintainer      ERROR   : [Errno 122] Disk quota exceeded: '/media/frigate/recordings/2024-05-06'
frigate  | 2024-05-06 14:37:03.544154244  [2024-05-06 14:37:03] frigate.record.maintainer      ERROR   : Error occurred when attempting to maintain recording cache
frigate  | 2024-05-06 14:37:03.544294713  [2024-05-06 14:37:03] frigate.record.maintainer      ERROR   : [Errno 122] Disk quota exceeded: '/media/frigate/recordings/2024-05-06'
frigate  | 2024-05-06 14:37:09.143988812  [2024-05-06 14:37:09] frigate.record.maintainer      ERROR   : Error occurred when attempting to maintain recording cache
frigate  | 2024-05-06 14:37:09.144168119  [2024-05-06 14:37:09] frigate.record.maintainer      ERROR   : [Errno 122] Disk quota exceeded: '/media/frigate/recordings/2024-05-06'
frigate  | 2024-05-06 14:37:13.545666406  [2024-05-06 14:37:13] frigate.record.maintainer      ERROR   : Error occurred when attempting to maintain recording cache
frigate  | 2024-05-06 14:37:13.545792552  [2024-05-06 14:37:13] frigate.record.maintainer      ERROR   : [Errno 122] Disk quota exceeded: '/media/frigate/recordings/2024-05-06'
frigate  | 2024-05-06 14:37:14.663580032  Exception in thread detected_frames_processor:
frigate  | 2024-05-06 14:37:14.663604617  Traceback (most recent call last):
frigate  | 2024-05-06 14:37:14.663612622    File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner
frigate  | 2024-05-06 14:37:14.668097971      self.run()
frigate  | 2024-05-06 14:37:14.668113199    File "/opt/frigate/frigate/object_processing.py", line 1075, in run
frigate  | 2024-05-06 14:37:14.672464802      camera_state.update(
frigate  | 2024-05-06 14:37:14.672467959    File "/opt/frigate/frigate/object_processing.py", line 672, in update
frigate  | 2024-05-06 14:37:14.672781831      c(self.name, removed_obj, frame_time)
frigate  | 2024-05-06 14:37:14.672793422    File "/opt/frigate/frigate/object_processing.py", line 848, in end
frigate  | 2024-05-06 14:37:14.673133711      with open(
frigate  | 2024-05-06 14:37:14.673200068  OSError: [Errno 122] Disk quota exceeded: '/media/frigate/clips/front_yard-1715006224.78061-605ask.jpg'
frigate  | 2024-05-06 14:38:28.538857084  [2024-05-06 14:38:28] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for back_yard. Keeping the 6 most recent segments out of 7 and discarding the rest...
frigate  | 2024-05-06 14:38:28.540484891  [2024-05-06 14:38:28] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for front_porch. Keeping the 6 most recent segments out of 7 and discarding the rest...
frigate  | 2024-05-06 14:38:28.540900142  [2024-05-06 14:38:28] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for back_porch. Keeping the 6 most recent segments out of 7 and discarding the rest...
frigate  | 2024-05-06 14:38:28.541679260  [2024-05-06 14:38:28] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for side_yard. Keeping the 6 most recent segments out of 7 and discarding the rest...
frigate  | 2024-05-06 14:38:28.542007403  [2024-05-06 14:38:28] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for front_yard. Keeping the 6 most recent segments out of 7 and discarding the rest...
frigate  | 2024-05-06 14:38:28.543807982  [2024-05-06 14:38:28] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for bear_roof. Keeping the 6 most recent segments out of 7 and discarding the rest...
frigate  | 2024-05-06 14:38:28.544154472  [2024-05-06 14:38:28] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for garage_driveway. Keeping the 6 most recent segments out of 7 and discarding the rest...
frigate  | 2024-05-06 14:38:28.544465268  [2024-05-06 14:38:28] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for back_driveway. Keeping the 6 most recent segments out of 7 and discarding the rest...
frigate  | 2024-05-06 14:38:38.534167610  [2024-05-06 14:38:38] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for back_yard. Keeping the 6 most recent segments out of 7 and discarding the rest...
frigate  | 2024-05-06 14:38:38.535632632  [2024-05-06 14:38:38] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for back_porch. Keeping the 6 most recent segments out of 7 and discarding the rest...
frigate  | 2024-05-06 14:38:38.536037168  [2024-05-06 14:38:38] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for front_porch. Keeping the 6 most recent segments out of 7 and discarding the rest...
frigate  | 2024-05-06 14:38:38.536577923  [2024-05-06 14:38:38] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for side_yard. Keeping the 6 most recent segments out of 7 and discarding the rest...
frigate  | 2024-05-06 14:38:38.536969720  [2024-05-06 14:38:38] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for bear_roof. Keeping the 6 most recent segments out of 7 and discarding the rest...
frigate  | 2024-05-06 14:38:38.537333265  [2024-05-06 14:38:38] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for back_driveway. Keeping the 6 most recent segments out of 7 and discarding the rest...
frigate  | 2024-05-06 14:38:38.537602957  [2024-05-06 14:38:38] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for garage_driveway. Keeping the 6 most recent segments out of 7 and discarding the rest...
frigate  | 2024-05-06 14:38:38.537884845  [2024-05-06 14:38:38] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for front_yard. Keeping the 6 most recent segments out of 7 and discarding the rest...
frigate  | 2024-05-06 14:38:48.531817509  [2024-05-06 14:38:48] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for back_yard. Keeping the 6 most recent segments out of 7 and discarding the rest...
frigate  | 2024-05-06 14:38:48.532800001  [2024-05-06 14:38:48] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for back_porch. Keeping the 6 most recent segments out of 7 and discarding the rest...
frigate  | 2024-05-06 14:38:48.533216840  [2024-05-06 14:38:48] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for front_porch. Keeping the 6 most recent segments out of 7 and discarding the rest...
frigate  | 2024-05-06 14:38:48.533681480  [2024-05-06 14:38:48] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for side_yard. Keeping the 6 most recent segments out of 7 and discarding the rest...
frigate  | 2024-05-06 14:38:48.533909125  [2024-05-06 14:38:48] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for back_driveway. Keeping the 6 most recent segments out of 7 and discarding the rest...
frigate  | 2024-05-06 14:38:48.534154758  [2024-05-06 14:38:48] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for bear_roof. Keeping the 6 most recent segments out of 7 and discarding the rest...
frigate  | 2024-05-06 14:38:48.534468488  [2024-05-06 14:38:48] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for garage_driveway. Keeping the 6 most recent segments out of 7 and discarding the rest...
frigate  | 2024-05-06 14:38:53.532633517  [2024-05-06 14:38:53] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for front_yard. Keeping the 6 most recent segments out of 7 and discarding the rest...
frigate  | 2024-05-06 14:38:58.531495833  [2024-05-06 14:38:58] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for back_yard. Keeping the 6 most recent segments out of 7 and discarding the rest...
frigate  | 2024-05-06 14:38:58.532598903  [2024-05-06 14:38:58] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for front_porch. Keeping the 6 most recent segments out of 7 and discarding the rest...
frigate  | 2024-05-06 14:38:58.532970860  [2024-05-06 14:38:58] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for back_porch. Keeping the 6 most recent segments out of 7 and discarding the rest...
frigate  | 2024-05-06 14:38:58.533741687  [2024-05-06 14:38:58] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for side_yard. Keeping the 6 most recent segments out of 7 and discarding the rest...
frigate  | 2024-05-06 14:38:58.533937738  [2024-05-06 14:38:58] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for back_driveway. Keeping the 6 most recent segments out of 7 and discarding the rest...
frigate  | 2024-05-06 14:38:58.534089754  [2024-05-06 14:38:58] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for bear_roof. Keeping the 6 most recent segments out of 7 and discarding the rest...
frigate  | 2024-05-06 14:38:58.534351825  [2024-05-06 14:38:58] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for garage_driveway. Keeping the 6 most recent segments out of 7 and discarding the rest...
frigate  | 2024-05-06 14:39:03.533124152  [2024-05-06 14:39:03] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for front_yard. Keeping the 6 most recent segments out of 7 and discarding the rest...
frigate  | 2024-05-06 14:39:08.533410697  [2024-05-06 14:39:08] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for back_yard. Keeping the 6 most recent segments out of 7 and discarding the rest...
frigate  | 2024-05-06 14:39:08.534428195  [2024-05-06 14:39:08] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for front_porch. Keeping the 6 most recent segments out of 7 and discarding the rest...
frigate  | 2024-05-06 14:39:08.534982748  [2024-05-06 14:39:08] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for back_porch. Keeping the 6 most recent segments out of 7 and discarding the rest...
frigate  | 2024-05-06 14:39:08.535139838  [2024-05-06 14:39:08] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for side_yard. Keeping the 6 most recent segments out of 7 and discarding the rest...
frigate  | 2024-05-06 14:39:08.535319598  [2024-05-06 14:39:08] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for bear_roof. Keeping the 6 most recent segments out of 7 and discarding the rest...
frigate  | 2024-05-06 14:39:08.535652301  [2024-05-06 14:39:08] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for back_driveway. Keeping the 6 most recent segments out of 7 and discarding the rest...
frigate  | 2024-05-06 14:39:08.535857288  [2024-05-06 14:39:08] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for garage_driveway. Keeping the 6 most recent segments out of 7 and discarding the rest...
frigate  | 2024-05-06 14:39:13.533003625  [2024-05-06 14:39:13] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for front_yard. Keeping the 6 most recent segments out of 7 and discarding the rest...
frigate  | 2024-05-06 14:39:18.535151683  [2024-05-06 14:39:18] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for back_yard. Keeping the 6 most recent segments out of 7 and discarding the rest...
frigate  | 2024-05-06 14:39:18.536395549  [2024-05-06 14:39:18] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for front_porch. Keeping the 6 most recent segments out of 7 and discarding the rest...
frigate  | 2024-05-06 14:39:18.536540911  [2024-05-06 14:39:18] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for side_yard. Keeping the 6 most recent segments out of 7 and discarding the rest...
frigate  | 2024-05-06 14:39:18.536764659  [2024-05-06 14:39:18] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for back_porch. Keeping the 6 most recent segments out of 7 and discarding the rest...
frigate  | 2024-05-06 14:39:18.537296035  [2024-05-06 14:39:18] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for back_driveway. Keeping the 6 most recent segments out of 7 and discarding the rest...
frigate  | 2024-05-06 14:39:18.537488538  [2024-05-06 14:39:18] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for bear_roof. Keeping the 6 most recent segments out of 7 and discarding the rest...
frigate  | 2024-05-06 14:39:18.537833508  [2024-05-06 14:39:18] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for garage_driveway. Keeping the 6 most recent segments out of 7 and discarding the rest...

FFprobe output from your camera

N/A

Frigate stats

N/A

Operating system

Debian

Install method

Docker Compose

Coral version

Other

Network connection

Wired

Camera make and model

N/A

Any other information that may be helpful

No response

reefland · 2024-06-03T18:00:11Z

reefland
Jun 3, 2024

Have you learned anything else about this? I was having a problem with Frigate not cleaning up old recordings as well, and put a 50GB ZFS disk quota on the NFS share to prevent Frigate from thinking it can consume all storage. Instead it just fails at 50GB disk space used as it thinks there is 20TB+ more storage it can use.

Same error's you reported:

2024-06-03 13:57:46.039382400  [2024-06-03 09:57:46] frigate.record.maintainer      ERROR   : Error occurred when attempting to maintain recording cache
2024-06-03 13:57:46.039450070  [2024-06-03 09:57:46] frigate.record.maintainer      ERROR   : [Errno 122] Disk quota exceeded: '/media/frigate/recordings/2024-06-03'
2024-06-03 13:57:50.937830303  [2024-06-03 09:57:50] frigate.record.maintainer      ERROR   : Error occurred when attempting to maintain recording cache
2024-06-03 13:57:50.937990492  [2024-06-03 09:57:50] frigate.record.maintainer      ERROR   : [Errno 122] Disk quota exceeded: '/media/frigate/recordings/2024-06-03'

And within a few seconds...

2024-06-03 13:58:45.929151144  [2024-06-03 09:58:45] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for kitchen. Keeping the 6 most recent segments out of 7 and discarding the rest...
2024-06-03 13:58:45.929359025  [2024-06-03 09:58:45] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for office. Keeping the 6 most recent segments out of 7 and discarding the rest...

Frigate is left in a state where it is unable to record and seems unable to cleanup old recordings. I had to manually purge out old directories to get free space within the disk quota.

        record:
          enabled: true
          retain:
            days: 7
            mode: active_objects
          events:
            pre_capture: 3
            post_capture: 3
            retain:
              default: 14
              mode: active_objects
              objects:
                dog: 3

For now I changed record.retain.mode to active_objects instead of motion to reduce the amount of motion being written to disk. Hopefully that will help.

5 replies

hollanbm Jun 18, 2024
Author

Have you learned anything else about this? I was having a problem with Frigate not cleaning up old recordings as well, and put a 50GB ZFS disk quota on the NFS share to prevent Frigate from thinking it can consume all storage. Instead it just fails at 50GB disk space used as it thinks there is 20TB+ more storage it can use.

I never was able to find a fix. In the end I just deleted the dataset contents and kept recording

simone7121 Dec 2, 2024

I've got the same problem too, running in proxmox, doesn't purge the space automatically

mircoboschi Jan 26, 2025

Same problem here; cleanup on zfs with quota set is not working.
I suggest @hollanbm to open a bug using "Create issue from discussion" button.

hawkeye217 Jan 26, 2025
Collaborator

"Create issue from discussion" button.

Issues on this repo are for feature requests, so this support discussion serves as the issue report and support request.

mircoboschi Jan 26, 2025

didn't know that, sorry.
By the way, i have same problem: storage on zfs dataset with quota.
I can also provide my conf, but i'm confident nothing there is relevant to this problem.
Are you using snapshot? i'm using Sanoid.

NickM-27 · 2025-01-26T23:50:35Z

NickM-27
Jan 26, 2025
Collaborator Sponsor

Looks like this issue got missed in general. For starters we would need to see debug logs for frigate.record.cleanup along with frigate config, frigate logs, etc

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Support]: Frigate unable to handle zfs dataset quota #11266

{{title}}

Replies: 2 comments 5 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

Select a reply

[Support]: Frigate unable to handle zfs dataset quota #11266

hollanbm May 6, 2024

Describe the problem you are having

Version

Frigate config file

Relevant log output

FFprobe output from your camera

Frigate stats

Operating system

Install method

Coral version

Network connection

Camera make and model

Any other information that may be helpful

Replies: 2 comments · 5 replies

reefland Jun 3, 2024

hollanbm Jun 18, 2024 Author

simone7121 Dec 2, 2024

mircoboschi Jan 26, 2025

hawkeye217 Jan 26, 2025 Collaborator

mircoboschi Jan 26, 2025

NickM-27 Jan 26, 2025 Collaborator Sponsor

hollanbm
May 6, 2024

Replies: 2 comments 5 replies

reefland
Jun 3, 2024

hollanbm Jun 18, 2024
Author

hawkeye217 Jan 26, 2025
Collaborator

NickM-27
Jan 26, 2025
Collaborator Sponsor