Memory leaks when using module

[pbirkants]

Hello!

Using this module for a high-traffic web application, I've observed that the server gradually runs out of memory due to ever-increasing httpd worker process memory usage.

I've reproduced the issue with a very basic configuration, please see attached Dockerfile, it builds the module, enables JWT token auth, starts httpd and runs ab to generate requests.

Sample output of build:

 ---> Running in 99174bd2f0d0
AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 172.17.0.5. Set the 'ServerName' directive globally to suppress this message
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  3.5  0.0   2392   752 ?        Ss   16:51   0:00 /bin/sh -c httpd && sleep 3 && ps aux && ab -q -n 1000000 -c 50 http://localhost/ >/dev/null && ps aux
root           7  0.0  0.0  11832  4016 ?        Ss   16:51   0:00 httpd
daemon         9  0.0  0.0 2002932 12252 ?       Sl   16:51   0:00 httpd
daemon        10  0.0  0.0 2002932 12252 ?       Sl   16:51   0:00 httpd
daemon        11  0.0  0.0 2002932 12252 ?       Sl   16:51   0:00 httpd
root          93  0.0  0.0   7644  2800 ?        R    16:51   0:00 ps aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  0.4  0.0   2392   752 ?        Ss   16:51   0:00 /bin/sh -c httpd && sleep 3 && ps aux && ab -q -n 1000000 -c 50 http://localhost/ >/dev/null && ps aux
root           7  0.0  0.0  11832  4016 ?        Ss   16:51   0:00 httpd
daemon         9 60.0  1.0 3067060 542372 ?      Sl   16:51   0:18 httpd
daemon        10  113  2.1 4144172 1082132 ?     Sl   16:51   0:34 httpd
daemon        11  108  2.1 4069540 1044720 ?     Sl   16:51   0:32 httpd
daemon        95  165  2.7 4735308 1378620 ?     Sl   16:51   0:42 httpd
root         123  0.0  0.0   7644  2724 ?        R    16:51   0:00 ps aux

As you can see, after 1M requests, RSS column adds up to almost 4GiB.
If Require valid-user is commented out, this does not happen.

[AnthonyDeroche]

Hello,
Thanks for the investigation.
I will have a look on this asap.

Maybe you can downgrade the module version or the libjwt version to check if it still occurs.

[GRRedWings]

Was there ever any resolution to this? In looking at using this module I'd be fearful of a known memory leak

[AnthonyDeroche]

I am not able to reproduce the leak with last version of everything :

• Apache 2.4.46
• libjwt 1.12.1
• mod_authnz_jwt 1.2.0

USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  1.3  0.0   2384   696 ?        Ss   09:50   0:00 /bin/sh -c httpd && sleep 3 && ps aux && ab -q -n 1000000 -c 50 http://localhost/ >/dev/null && ps aux
root           8  0.0  0.0  11832  4152 ?        Ss   09:50   0:00 httpd
daemon        10  0.0  0.0 2002932 4200 ?        Sl   09:50   0:00 httpd
daemon        11  0.0  0.0 2002932 4176 ?        Sl   09:50   0:00 httpd
daemon        12  0.0  0.0 2002932 4192 ?        Sl   09:50   0:00 httpd
root          94  0.0  0.0   7636  2740 ?        R    09:50   0:00 ps aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  0.0  0.0   2384   696 ?        Ss   09:50   0:00 /bin/sh -c httpd && sleep 3 && ps aux && ab -q -n 1000000 -c 50 http://localhost/ >/dev/null && ps aux
root           8  0.0  0.0  11832  4152 ?        Ss   09:50   0:00 httpd
daemon        10  9.6  0.0 2004084 5580 ?        Sl   09:50   0:09 httpd
daemon        11 20.9  0.0 2004028 5268 ?        Sl   09:50   0:21 httpd
daemon        12 54.5  0.0 2004028 5404 ?        Sl   09:50   0:56 httpd
daemon        96  139  0.0 2004028 5196 ?        Sl   09:50   2:17 httpd
root         124  0.0  0.0   7636  2760 ?        R    09:52   0:00 ps aux

Dockerfile I used

FROM debian:buster-slim as build

WORKDIR /build

RUN apt-get update && \
  apt-get install -y ca-certificates make automake git g++ libtool pkg-config autoconf libssl-dev check libjansson-dev libz-dev procps apache2 apache2-dev
  
ARG LIBJWT_VERSION=1.12.1
ARG MOD_AUTHNZ_JWT_VERSION=1.2.0

RUN git clone https://github.com/benmcollins/libjwt.git && \
    cd libjwt && \
    git checkout tags/v$LIBJWT_VERSION && \
    autoreconf -i && \
    ./configure && \
    make && \
    make install

RUN git clone https://github.com/AnthonyDeroche/mod_authnz_jwt.git && \
    cd mod_authnz_jwt && \
    git checkout tags/v$MOD_AUTHNZ_JWT_VERSION && \
    autoreconf -ivf && \
    PKG_CONFIG_PATH=/usr/local ./configure && \
    make && \
    make install
    
FROM httpd:2.4

COPY --from=build /usr/local/lib/libjwt.so /usr/lib/x86_64-linux-gnu/libjwt.so.1
COPY --from=build /usr/lib/apache2/modules/mod_authnz_jwt.so /usr/local/apache2/modules/mod_authnz_jwt.so

RUN echo "LoadModule auth_jwt_module modules/mod_authnz_jwt.so" >> /usr/local/apache2/conf/httpd.conf

RUN echo "AuthJWTSignatureAlgorithm HS512\n \
AuthJWTSignatureSharedSecret test\n \
AuthJWTIss test\n \
<Location />\n \
        AuthType jwt\n \ 
        AuthName \"test\"\n \
        Require valid-user\n \
</Location>\n \
" >>/usr/local/apache2/conf/httpd.conf

RUN apt-get update && apt-get install procps -y

RUN httpd && sleep 3 && ps aux && ab -q -n 1000000 -c 50 http://localhost/ >/dev/null && ps aux

[jbloggz]

I found a memory leak the the create_token() function (see pull request #55). I doubt this is the same leak, but thought it worth mentioning