Proper use of \Jcupitt\Vips\Image class

[bluesoulx]

Shoud class \Jcupitt\Vips\Image instance need to close or destroy?
I found some php-fpm process was not terminated after use this class/extension...

My Code as follow:

$image = Vips\Image::newFromBuffer($imgBlobStr);
$image = $image->crop(0, $y, $width, $height)->resize($ratio);
$buffer = $image->jpegsave_buffer();
$image = null;

header('Content-Type: image/jpeg');
echo $buffer;

Please correct me if I am wrong. Thanks

[jcupitt]

Hello @bluesoulx,

PHP will free resources for you automatically, you don't need to do anything.

libvips does keep a cache of recently used images. You can turn this cache off with:

Vips\Config::CacheSetMax(0);

https://jcupitt.github.io/php-vips/docs/classes/Jcupitt.Vips.Config.html#method_cacheSetMax

Though I would leave it on.

[bluesoulx]

Hi @jcupitt

Thank you for quick response.
I will keep watch this and report back.

[mzur]

Hi, I think I have a related issue. I'm running Vips in a Laravel daemon queue worker. These workers run constantly and do not exit and reboot PHP. Although I have yet to observe any memory issues (since it is not explicitly freed), I recently found that Vips seems not to automatically close the file handles. The code in my worker boils down to:

Vips\Image::newFromFile($path, ['access' => 'sequential'])
   ->crop()
   ->resize()
   ->writeToFile();

When this is run and I check lsof, I see the processed image files still open by PHP until I completely restart the worker. I'll try to dig deeper into the Vips codebase as I'm sure that some equivalent of fclose can/should be called in writeToFile.

Anyway @jcupitt thank you for this great library and your amazing support!

[jcupitt]

Hi @mzur, I made a test program:

#!/usr/bin/env php
<?php

require __DIR__ . '/vendor/autoload.php';

use Jcupitt\Vips;

# vips_cache_set_max(0);

for ($i = 0; $i < 10000; $i++) { 
    echo "loop " . $i . " ... \n";

    $image = Vips\Image::newFromFile("x/" . $i . ".jpg",
        ['access' => 'sequential']);
    $image = $image->crop(10, 10, 100, 100);
    $image = $image->resize(0.1);
    $image->writeToFile("x/tn_" . $i . ".jpg");
}

Then ran it like this:

$ mkdir x
$ for i in {0..10000}; do cp ~/pics/small.jpg x/$i.jpg; done
$ ./soak2.php 

If I watch top as it runs, RES (resident memory) is stable at around 70MB. If I look at the fds for that process, it's stable at around 70.

$ ps aux | grep php
john      6590 68.1  0.8 595952 64296 pts/4    T    11:23   0:21 php ./soak2.php
$ ls /proc/6590/fd | wc
     59      59     168

[jcupitt]

Off-topic, but could you crop after the resize? You should find it's faster (5x faster?) to do:

$image = Vips\Image::thumbnail("somefile.jpg", 100);
$image = $image->crop(x, y, w, h);
$image->writeToFile("out.jpg");

Because thumbnail can exploit shrink-on-load for some file formats (at least JPG, WebP, SVG, PDF). It knows about transparency too, so you'll get better quality on PNGs.

[mzur]

I can simulate it with the PHP REPL:

ubuntu@host-192-168-11-9:~$ php -a
Interactive mode enabled

php > require __DIR__ . '/vendor/autoload.php';
php > $c = function ($path, $target) {Jcupitt\Vips\Image::newFromFile($path, ['access' => 'sequential'])->writeToFile($target);};
php > $c('image.jpg', 'out.jpg');
php > $c('image.jpg', 'out.jpg');
php > $c('image.jpg', 'out.jpg');

Now I have three open files until I exit the REPL:

ubuntu@host-192-168-11-9:~$ sudo lsof | grep image
php       20615                ubuntu    4r      REG              253,1    37075     272789 /home/ubuntu/image.jpg
php       20615                ubuntu    5r      REG              253,1    37075     272789 /home/ubuntu/image.jpg
php       20615                ubuntu    6r      REG              253,1    37075     272789 /home/ubuntu/image.jpg

I noticed that if I don't use sequential access, there is only ever one open file no matter how often I call the callback.

Regarding your OT: What about your approach if I need to handle huge TIFFs (several GB with px dimensions in the tens of thousands)? Shouldn't it be faster to crop first and then resize in this case?

Edit: The cropped region is usually very small compared to the original image, especially for the huge TIFFs mentioned above.

[jcupitt]

You're right, for TIFF crop first is probably better. Are you using tiled TIFFs? They will help a lot.

I tried this:

php > for($i = 1; $i < 10000; $i++){$c('/home/john/pics/k2.jpg', 'out.jpg');}

lsof goes up and down a bit, depending on the PHP GC and the libvips cache:

john@kiwi:~$ sudo lsof | grep k2.jpg | wc
    306    2810   35496
john@kiwi:~$ sudo lsof | grep k2.jpg | wc
    826    8010   95816
john@kiwi:~$ sudo lsof | grep k2.jpg | wc
    250    2250   29000

But I don't think there's a leak. If you want to keep the number of open files down, you could make the cache smaller. If I add:

php > Jcupitt\Vips\Config::cacheSetMax(0);

Then lsof stays under 10.

[mzur]

Ah I see, the number of open files always goes back to 250 for me, too. I just thought it would increase without end but I didn't try that often 😄 Thanks for the clarification!

Actually I don't know if the TIFFs are tiled or not. Can you perhaps point me to a command how to check if they are? I tried identify -verbose (no luck) and looked for some equivalent Vips command but couldn't find anything.

[jcupitt]

To tile a TIFF, use:

$ vips copy k2.jpg x.tif[tile]
$ tiffinfo x.tif 
TIFF Directory at offset 0x900008 (9437192)
  Image Width: 1450 Image Length: 2048
  Tile Width: 128 Tile Length: 128
  Resolution: 28.346, 28.346 pixels/cm
  Bits/Sample: 8
  Sample Format: unsigned integer
  Compression Scheme: None
  Photometric Interpretation: RGB color
  Orientation: row 0 top, col 0 lhs
  Samples/Pixel: 3
  Planar Configuration: single image plane

ie. look for Tile Width and Tile Length in tiffinfo.

You can JPEG-compress as well, if your images are OK with that.

$ vips copy k2.jpg x.tif[tile,compression=jpeg]

I wrote a stackoverflow answer about tiled TIFF:

https://stackoverflow.com/a/48964573/894763

[mzur]

Awesome, thank you very much!

@bluesoulx Sorry for hijacking your issue.

[bluesoulx]

@mzur It's ok. I also learned a lot from your question.

@jcupitt
I follow the suggestion

Vips\Config::CacheSetMax(0);

Memory usage was really stable.

And I also use another issue suggestion from php-vips-ext Trying to understand memory usage with php-fpm

It's really work, Thank you

I will keep monitor one more day.

[jcupitt]

I'd only turn off the cache if you really have to. It's best left alone, I think. You could maybe turn it down to 100.

[bluesoulx]

@jcupitt

I understand. I will make some test for this value.

Thank you.

[bluesoulx]

@jcupitt

After days for watching this, it's just stable now.

I think my use case is relative rough.

I use php-vips on my server for realtime crop + resize with some traffic.

Now I use 64G memory for this service and

Vips\Config::CacheSetMax(100);

I think it ok now. Thanks for your help.