php-fpm 8.4.4 high cpu usage

[jostreff]

Description

The following code:

Resulted in this output:

But I expected this output instead:

PHP Version

PHP 8.4.4

Operating System

FreeBSD 14.2-STABLE stable/14-n270410-720d25547745 GENERIC amd64

[devnexen]

Hi. We might need more context ; e.g. pool configuration, what kind of web application could it be, was cpu usage significantly lower with previous php releases .. etc ...

[jostreff]

php.ini:
[opcache]
opcache.enable=1
opcache.enable_cli=1
opcache.interned_strings_buffer=128
opcache.max_accelerated_files=20000
opcache.save_comments=1
opcache.jit=1235
opcache.revalidate_freq=0
opcache.validate_timestamps=1
opcache.max_accelerated_files=10000
opcache.memory_consumption=192
opcache.max_wasted_percentage=10
opcache.interned_strings_buffer=16
opcache.fast_shutdown=1
opcache.jit_buffer_size=500000000

and php-fpm pool:

pm.status_path = /fpm-status
slowlog = log/$pool.log.slow
security.limit_extensions = .php .php3 .php4 .php5 .php7 .php8
php_admin_value[upload_max_filesize] = 5M
php_admin_value[error_reporting] = E_ALL & ~E_NOTICE & ~E_WARNING & ~E_STRICT & ~E_DEPRECATED
php_admin_value[mysqli.allow_local_infile] = 1

pm = dynamic
pm.max_children = 16
pm.start_servers = 2
pm.min_spare_servers = 2
pm.max_spare_servers = 5
pm.max_requests = 1000
pm.process_idle_timeout = 60s
rlimit_files = 131072
rlimit_core = unlimited
catch_workers_output = yes

[devnexen]

Thanks.

was cpu usage significantly lower with previous php releases

The reason I ask, often this kind of report comes from major migrations e.g. from 8.2 to 8.4 when behavior differences could be witnessed.

[jostreff]

i'm experiencing issues since 8.3 that was the reason to try 8.4 and it performs like that since that moment. Only way to lower cpu usage is just restart service via service php_fpm restart.

[jostreff]

My big issue is that I don't know ideas how to debug which php code mades this high php-fpm cpu usage. Typically when there is problem I see more then 8 php-fpm processes consuming 100% of cpu. Good news is that machine has 12 cpu cores ...

[jostreff]

[devnexen]

Eventually, are fpm logs giving any clue ?

[jostreff]

[14-Feb-2025 21:12:03] NOTICE: ready to handle connections
[14-Feb-2025 21:49:11] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 0 idle, and 10 total children
[14-Feb-2025 21:49:12] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 16 children, there are 0 idle, and 12 total children
[14-Feb-2025 21:52:49] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 0 idle, and 10 total children
[14-Feb-2025 21:52:50] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 16 children, there are 0 idle, and 12 total children
[14-Feb-2025 21:52:51] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 14 total children
[14-Feb-2025 21:52:52] WARNING: [pool www] server reached pm.max_children setting (16), consider raising it
[14-Feb-2025 21:53:00] WARNING: [pool www] server reached pm.max_children setting (16), consider raising it```

since last restart

[devnexen]

I see... did you try to adjust the pool settings ? e.g. pm.start_servers, pm.max_spare_servers, pm.max_children ... as advised in the logs.

[jostreff]

Changed pool settings to:
pm = ondemand
pm.max_children = 16
pm.start_servers = 48 ( 4x12core )
pm.min_spare_servers = 24 ( 2x12core )
pm.max_spare_servers = 48 ( =pm.start_servers )
pm.max_requests = 200
pm.process_idle_timeout = 10s

will wait to see...

[jostreff]

almost identical situation after reboot of the machine.

[devnexen]

Would it be possible to profile your web app ? e.g. pmcstat is able to do it on a cpu cycles level pcmstat -O php-sample.dat -P cycles -T -p <php process ID> might give further clues.

[bukka]

I would guess it won't be related to FPM but the actual execution. Profiling might help with that.

Try to also comment out pm.max_requests = 200 or set it very high which can lead to too many unnecessary restarts ...

[github-actions]

No feedback was provided. The issue is being suspended because we assume that you are no longer experiencing the problem. If this is not the case and you are able to provide the information that was requested earlier, please do so. Thank you.

[grisha2217]

Same problem on production server
I compared 2 configs

/usr/bin/php8.3 -i > php83.txt
/usr/bin/php8.4 -i > php84.txt
diff php83.txt php84.txt | less

OUTPUT:
Version & Build Information
PHP Version:
8.3: 8.3.17
8.4: 8.4.4
Build Date:
8.3: Mar 10 2025 15:24:50
8.4: Mar 9 2025 17:05:03
Configuration Files & Directories
php.ini Location:
PHP 8.3 uses: /etc/php/8.3/cli/php.ini
PHP 8.4 uses: /etc/php/8.4/cli/php.ini
Additional .ini Files Parsed:
Almost the same list except PHP 8.4 includes a new file (25-memcached.ini) that isn’t in PHP 8.3.
Extension Directory:
8.3: /usr/lib/php/20230831
8.4: /usr/lib/php/20240924
Core API & Engine Changes
PHP API / Extension Build:
Updated from API20230831 to API20240924.
Zend Engine:
8.3: Zend Engine v4.3.17 with OPcache v8.3.17
8.4: Zend Engine v4.4.4 with OPcache v8.4.4
Output Formatting Changes
Syntax Highlighting:
In PHP 8.3, highlight colors are output using tags.
In PHP 8.4, these are output using tags.
Specific Directive & Extension Differences
FFI Setting:
ffi.enable changed from preload in 8.3 to On in 8.4.
libmagic:
Version changed slightly from 543 to 545.
IMAP Extension:
PHP 8.4 now shows an “IMAP extension Version => 1.0.3” entry.
Memcached Extension:
PHP 8.4 includes a detailed block showing memcached settings and support (e.g. serializer, compression settings).
MySQL Native Driver (mysqlnd):
Updated from version 8.3.17 to 8.4.4 in several places.
PostgreSQL Settings:
pgsql.auto_reset_persistent changed from On to Off.
Session Settings:
session.sid_bits_per_character changes from 5 to 4.
session.sid_length changes from 26 to 32.
OPcache JIT Buffer:
opcache.jit_buffer_size has been reduced from 128M to 64M.
Miscellaneous
Zip Version:
Updated from 1.22.3 to 1.22.4.
CLI Binary Reference:
The underscore variable ($SERVER['']) now reflects the PHP 8.4 binary (/usr/bin/php8.4) instead of PHP 8.3.
Timing Values:
Slight differences in REQUEST_TIME_FLOAT and REQUEST_TIME reflecting the moment the CLI started.

[devnexen]

Same problem on production server I compared 2 configs

/usr/bin/php8.3 -i > php83.txt /usr/bin/php8.4 -i > php84.txt diff php83.txt php84.txt | less

OUTPUT: Version & Build Information PHP Version: 8.3: 8.3.17 8.4: 8.4.4 Build Date: 8.3: Mar 10 2025 15:24:50 8.4: Mar 9 2025 17:05:03 Configuration Files & Directories php.ini Location: PHP 8.3 uses: /etc/php/8.3/cli/php.ini PHP 8.4 uses: /etc/php/8.4/cli/php.ini Additional .ini Files Parsed: Almost the same list except PHP 8.4 includes a new file (25-memcached.ini) that isn’t in PHP 8.3. Extension Directory: 8.3: /usr/lib/php/20230831 8.4: /usr/lib/php/20240924 Core API & Engine Changes PHP API / Extension Build: Updated from API20230831 to API20240924. Zend Engine: 8.3: Zend Engine v4.3.17 with OPcache v8.3.17 8.4: Zend Engine v4.4.4 with OPcache v8.4.4 Output Formatting Changes Syntax Highlighting: In PHP 8.3, highlight colors are output using tags. In PHP 8.4, these are output using tags. Specific Directive & Extension Differences FFI Setting: ffi.enable changed from preload in 8.3 to On in 8.4. libmagic: Version changed slightly from 543 to 545. IMAP Extension: PHP 8.4 now shows an “IMAP extension Version => 1.0.3” entry. Memcached Extension: PHP 8.4 includes a detailed block showing memcached settings and support (e.g. serializer, compression settings). MySQL Native Driver (mysqlnd): Updated from version 8.3.17 to 8.4.4 in several places. PostgreSQL Settings: pgsql.auto_reset_persistent changed from On to Off. Session Settings: session.sid_bits_per_character changes from 5 to 4. session.sid_length changes from 26 to 32. OPcache JIT Buffer: opcache.jit_buffer_size has been reduced from 128M to 64M. Miscellaneous Zip Version: Updated from 1.22.3 to 1.22.4. CLI Binary Reference: The underscore variable ($SERVER['']) now reflects the PHP 8.4 binary (/usr/bin/php8.4) instead of PHP 8.3. Timing Values: Slight differences in REQUEST_TIME_FLOAT and REQUEST_TIME reflecting the moment the CLI started.

Hi @grisha2217, not entirely sure it is "kind of the same" here.
Is disabling memcached, for example, for a while and restesting a possibility ?

[Nations112]

Same problem after update from 8.3...
I have a PHP pool for each project. With this problem, all pools are using the processor. Restarting PHP helps.