Proposed Update to support OpenSSL 3.x and earlier versions #529

MWASoftware · 2024-04-01T14:04:15Z

This pull request includes work performed by @mezen in developing a version of Indy supporting OpenSSL 1.1.1. This version is intended to overcome two criticisms of @mezen's update: The first is that the proposal for 1.1.1 did not use the existing Indy OpenSSL classes. The second is that Openssl 3.x was not yet supported.

This pull request has taken @mezen's new openssl loader, and intermediate code OpenSSL header files, added Pascal versions of additional OpenSSL header files (e.g. for DES support) and a new code processor (a sed script) to process the intermediate code into usable unit files. With a few mods to IdSSLOpenSSL, the result supports OpenSSL 1.0.2, 1.1.1 and 3.x and probably works with earlier versions - but not tested.

The IdSSLOpenSSL components are unchanged except for the addition of two useful properties:

to report the TLS protocol in use (ideally always TLS 1.3), and
a new feature to automatically use the Windows crypto API so that the Windows Root Certificate store can be used for validation on Windows hosts (the default - but can be disabled at compile time).

There are also some low level changes to handle the change to opaque data structures (e.g. SSL_CTX) in 1.1.1 and 3.x.

Note that the IdSSLOpenSSLHeaders.pas and IdSSLOpenSSLHeaders_static.pas units have been removed and have been replaced by the header files in Lib/Protocols/opensslHdrs. The new header files are intended to have a one-to-one correspondence with the OpenSSL header (.h) files.

Three link models are supported.

Dynamic Library Load (the default and the approach used in previous versions and supports OpenSSL 1.0.2, 1.1.1 and 3.x).
static linkage to a shared (.so or .dll) library (OpenSSL 3.x only)
static linkage to a static library (FPC only with gcc compiled OpenSSL 3.x).

This version has been tested on both Windows and Linux with both Lazarus/FPC
(3.2.2) and Delphi (Berlin 32 bit only). There are two test programs used for
validation:

Https Get from a public website with certification validation
Https Get from an embedded TIdHttpServer using a private PKI and password protected keys.

Hopefully the result will be found useful.

Please read README.OpenSSL for further information on the update.

NOTES FOR USING THE TEST PROGRAMS

The directory Test/OpenSSL contains the two test programs together with a
README giving more details on their use. They are intended to run "standalone".
That is without having to integrate Indy into the IDE. There are variants for
both Delphi and Lazarus/fpc.

Note: when testing under Windows, if there are no default OpenSSL dlls in the
search path then you will either need to copy libssl and libcrypto into the
project folder or add a command line parameter providing the location of the
OpenSSL dlls.

NOTES FOR LAZARUS/FPC

Please try out the test programs before following the instructions below.

Before trying out the update with your own programs, you should run the script:

getindy4lazfpc.sh (Linux)
getindy4lazfpc.bat (Windows)

These scripts are intended to extract the files relevant to Lazarus/FPC to
another directory and to place the Lazarus package files in the destination
directory itself. By default the scripts extract the files to:

$HOME/Indy (Linux)
"%homedrive%%homepath%"\Indy (Windows)

To override the default, provide the target directory as the only command line
argument for the script. There is no need for the target directory to exist
before running the script.

After running the script, you should read the file README.lazarus-fpc for
further guidance.

Note that you must ensure that any existing Lazarus project that you test this
update with, includes the indyprotocols run time package in the requirements
list.

NOTES FOR DELPHI USERS

Integration with the Delphi IDE is tbd and needs someone to take this on. That is there is a need to update the Delphi package definitions, but otherwise, the update should work with Delphi.

… problem on Linux where -L option meant that the OpenSSL lib directory given on the command line was ignored.

…were introduced in 3.2 rather than 3.0

…r static linking.

baka0815 · 2024-04-25T11:55:56Z

Lib/Protocols/IdSSLOpenSSLLoader.pas

+begin
+  inherited;
+  FFailed := TStringList.Create();
+  FLibraryLoaded := TIdThreadSafeBoolean.Create;


You need to initialize FFailedToLoad to False in the constructor. Otherwise it might be accidentally initialized as True and TOpenSSLLoader.Load will fail (and not even try) to load the DLLs.

@baka0815 The memory for a class object is initialized with zero bytes before the constructor is called. Thus, by default all numeric members are 0, booleans are False, strings are empty, etc.

Ok, looks like a testing mistake on my end.

The TOpenSSLLoader is only created once and after the loading failed once it won't try to load again. So you need to restart the program if the OpenSSL 3.x DLLs were not present when you first tried to load them

However I'm a fan of explicit initializations, but it then worked for me without it. So this thread can be closed.

I am not really that happy with the whole strategy for handling load failures. The existing strategy of refusing to load the SSL libraries as soon as any function fails to load follows on from the legacy version. However, it does cause a lot of problems given that different builds select different compile time options - especially with legacy functions such as MD2. I also inherited the one shot load approach.

I would really like to change the whole approach to one where only when certain core functions (such as reading version information) will cause a failure to load. Otherwise, a run time exception is called when and if a function that failed to load is invoked. That way missing functions that are not critical for the calling app do not cause a problem and an explicit error message can focus attention on the problematic function.

This is a relatively simply change to bring in as the update is to the header file generator script. Any views?

There are other areas of Indy (ie, WinSock, ZLib) that dynamically load using per-function stubs, raising load failures only at the invocation site. I didn't write the OpenSSL support code, but I think at the time it was probably not a good idea for OpenSSL to use such stubs because of the sheer number of functions being loaded, that's probably why a one-time load strategy was used instead. I'm not against switching to stubs, but I can imagine that would severely bloat the loading unit.

I have created a new branch to my proposed update. This is called "NewLibLoadStrategy". This changes the function loading strategy such if a function cannot be loaded, the call is set to a call to an exception handler unless an appropriate backwards or forwards compatibility function exists. When the call is set to an exception handler, the function name is added to the FailedToLoad list but no exception is generated. it is up to the user to check this list if they need to test if a required (optional) function is present. If they do not then when it is called a run time exception is generated given the name of the missing function.

I have made this a separate branch so that they two load strategies can be compared for userability/bloat, etc.

baka0815 · 2024-04-25T12:50:27Z

I'm creating my server with the following options:

  LIOHandleSSL.SSLOptions.SSLVersions := [sslvTLSv1, sslvTLSv1_1, sslvTLSv1_2];
  LIOHandleSSL.SSLOptions.Method := sslvTLSv1_2;
  LIOHandleSSL.SSLOptions.Mode := sslmServer;

However the connection via Firefox is still established as TLS v1.3 (TLS_AES_1128_GCM_SHA256, 128-Bit-Key, TLS 1.3).

Also cURL can connect via --tlsv1.1, --tlsv1.2 and --tlsv1.3, even if I only allow TLS 1.0 or 1.1.

  LIOHandleSSL.SSLOptions.SSLVersions := [sslvTLSv1, sslvTLSv1_1];
  LIOHandleSSL.SSLOptions.Method := sslvTLSv1_1;
  LIOHandleSSL.SSLOptions.Mode := sslmServer;

baka0815 · 2024-04-26T08:55:08Z

Loading the legacy OpenSSL 1.0.2 DLLs fails for me with the following function not being present: EVP_md2

Is this expected or should this be added to LegacyAllowFailed?

baka0815 · 2024-04-26T08:58:33Z

Lib/Protocols/IdSSLOpenSSLLoader.pas

+      {$IFDEF DEBUG}
+      if not Result then
+      begin
+        writeln;


This gives me E/A error 105 if I'm running my program as a GUI application.
You might want to use OutputDebugMessage ({$IFDEF Windows}) or check for {$IFDEF CONSOLE}.

Point taken - all my testing was on console mode apps or Linux...

MWASoftware · 2024-04-26T10:51:24Z

I'm creating my server with the following options:
  LIOHandleSSL.SSLOptions.SSLVersions := [sslvTLSv1, sslvTLSv1_1, sslvTLSv1_2];
  LIOHandleSSL.SSLOptions.Method := sslvTLSv1_2;
  LIOHandleSSL.SSLOptions.Mode := sslmServer;
However the connection via Firefox is still established as TLS v1.3 (TLS_AES_1128_GCM_SHA256, 128-Bit-Key, TLS 1.3).

Also cURL can connect via --tlsv1.1, --tlsv1.2 and --tlsv1.3, even if I only allow TLS 1.0 or 1.1.
  LIOHandleSSL.SSLOptions.SSLVersions := [sslvTLSv1, sslvTLSv1_1];
  LIOHandleSSL.SSLOptions.Method := sslvTLSv1_1;
  LIOHandleSSL.SSLOptions.Mode := sslmServer;

In the README.OpenSSL.txt I have noted a typo when I said that "SSLOptions.Mode" is ignored. I meant "SLLOptions.Method" is ignored. "SSLOptions.Versions" is now the only means of control over which version is supported. The README does state that this determines in the minimum version only. This was intentional. On thinking about it, perhaps it should also determined the maximum supported. I will update.

… maximum versions of the TLS Protocol negotiated for OpenSSLVersions 1.1.0 and later

MWASoftware · 2024-04-26T12:41:46Z

In the README.OpenSSL.txt I have noted a typo when I said that "SSLOptions.Mode" is ignored. I meant "SLLOptions.Method" is ignored. "SSLOptions.Versions" is now the only means of control over which version is supported. The README does state that this determines in the minimum version only. This was intentional. On thinking about it, perhaps it should also determined the maximum supported. I will update.

Both the README and the code have been updated (and tested), and pushed to github. As the update log says "SSLOptions.SSLVersions can now be used to select both the minimum and maximum versions of the TLS Protocol negotiated for OpenSSLVersions 1.1.0 and later. OpenSSL will always try to negotiate the highest version acceptable to both ends of the session."

baka0815 · 2024-04-26T13:22:16Z

Thanks for the change @MWASoftware and the work you're putting into this! I will take a look at the changes later on.

Loading the legacy OpenSSL 1.0.2 DLLs fails for me with the following function not being present: EVP_md2

Is this expected or should this be added to LegacyAllowFailed?

Any comments regarding this?
I'm using the pre-compiled OpenSSL library from https://indy.fulgan.com/SSL/ (1.0.2u)

MWASoftware · 2024-05-01T12:22:51Z

Thanks for the change @MWASoftware and the work you're putting into this! I will take a look at the changes later on.

Loading the legacy OpenSSL 1.0.2 DLLs fails for me with the following function not being present: EVP_md2
Is this expected or should this be added to LegacyAllowFailed?

Any comments regarding this? I'm using the pre-compiled OpenSSL library from https://indy.fulgan.com/SSL/ (1.0.2u)

See NewLibLoadStrategy branch and response to earlier comment.

sviklvil · 2024-06-26T10:58:00Z

Lib/Protocols/IdSSLOpenSSL.pas

-      SSL_CTX_set_min_proto_version(fContext,SSL3_VERSION);
+    if SSLVersions <> [] then
+    begin
+      for v := sslvSSLv3 to high(TIdSSLVersions) do


In Delphi (XE7, 11) compilation fails with "E2008 Incompatible types".
High/Low functions does not work with set types.

sviklvil · 2024-06-26T10:58:27Z

Lib/Protocols/IdSSLOpenSSL.pas

+          break;
+        end;
+     end;
+      for v := high(TIdSSLVersions) downto sslvSSLv3 do


In Delphi (XE7, 11) compilation fails with "E2008 Incompatible types".
High/Low functions does not work with set types.

I have updated the code that defines the "high value" as a constant in order to work around this problem. I don't have Delphi XE7 available, but do have Delphi 2010 still running on an old Windows XP VM. The code compiled and runs OK after the update - so hopefully should also run under XE7. I have updated both branches (old and new lib loaders).

…d Delphi versions

rlebeau · 2024-08-05T03:05:51Z

@MWASoftware (and anyone else who is interested in contributing to this effort) - there is a new joint initiative with Embarcadero and some other 3rd party devs to get Indy updated with OpenSSL. Please contact me privately if you are interested in joining in.

https://www.indyproject.org/2024/08/05/ongoing-work-in-indy-for-openssl-updates/

MWASoftware · 2024-08-05T15:18:30Z

@MWASoftware (and anyone else who is interested in contributing to this effort) - there is a new joint initiative with Embarcadero and some other 3rd party devs to get Indy updated with OpenSSL. Please contact me privately if you are interested in joining in.

https://www.indyproject.org/2024/08/05/ongoing-work-in-indy-for-openssl-updates/

Remy,
Did you get the message I sent you via the contact page on indyproject.org? The response I got on message submission was empty.

rlebeau · 2024-08-05T18:14:35Z

Remy, Did you get the message I sent you via the contact page on indyproject.org?

I saw your comment on the recent blog post. I replied to you privately.

MWASoftware added 25 commits March 19, 2024 08:45

merge

228cd80

merge

326ef02

OpenSSL HDr Gen

cd8add8

tests added

9ee16dd

Top level files

a50c745

Top level files

e1f3b24

testing

96f8d10

Delphi Mods

8ee4bc4

FPC extract test

7284f84

windows bat file

445eb28

README for update added

15c20f7

text file folding

1c715a1

2024/3/21 Updated Test/OpenSSL/openssl-client/TestClient.pas to avoid…

1777290

… problem on Linux where -L option meant that the OpenSSL lib directory given on the command line was ignored.

Addition of FIPS Support

ef8ae85

Addition of FIPS Support

250fa5a

Added support for Delphi delayed loading

c1ac1e8

NTLMv2 now uses new OpenSSL Headers

ca76d07

NTLMv2 now uses new OpenSSL Headers

087c507

Remove reference to old SSL headers from IdDSNRegister

9395d7a

Changelog updated

349ec10

Fixed missing units in fpcmake build

033b580

Fixed missing units in fpcmake build

07039f4

Supported Delphi Delayed Loading Reversed out

2663608

Update IdOpenSSLHeaders_provider to correct markup of functions that …

cc647dd

…were introduced in 3.2 rather than 3.0

Update README.OpenSSL.txt to note that OpenSSL 3.2 is now required fo…

21c3416

…r static linking.

This comment was marked as outdated.

Sign in to view

MWASoftware added 2 commits April 15, 2024 10:09

Fix for Issue #2 and restored static link with OpenSSL 3.0 or later

9a8f20c

Issue 3 Fixes

02eabf7

baka0815 reviewed Apr 25, 2024

View reviewed changes

baka0815 reviewed Apr 26, 2024

View reviewed changes

SSLOptions.SSLVersions can now be used to select both the minimum and…

47cc615

… maximum versions of the TLS Protocol negotiated for OpenSSLVersions 1.1.0 and later

HMAC header markup corrected to avoid lib load failure for 1.0.2

075e2f5

sviklvil reviewed Jun 26, 2024

View reviewed changes

Change to priority order of ssl libraries and fix to compile under ol…

e50fdd0

…d Delphi versions

rlebeau added this to the Indy 10.7 - Maintenance Release milestone Dec 12, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Proposed Update to support OpenSSL 3.x and earlier versions #529

Proposed Update to support OpenSSL 3.x and earlier versions #529

MWASoftware commented Apr 1, 2024

This comment was marked as outdated.

baka0815 Apr 25, 2024

rlebeau Apr 25, 2024 •

edited

Loading

This comment was marked as outdated.

baka0815 Apr 26, 2024

MWASoftware Apr 26, 2024

rlebeau Apr 26, 2024

MWASoftware May 1, 2024

baka0815 commented Apr 25, 2024 •

edited

Loading

baka0815 commented Apr 26, 2024

baka0815 Apr 26, 2024 •

edited

Loading

MWASoftware Apr 26, 2024

MWASoftware commented Apr 26, 2024

MWASoftware commented Apr 26, 2024

baka0815 commented Apr 26, 2024 •

edited

Loading

MWASoftware commented May 1, 2024

sviklvil Jun 26, 2024

sviklvil Jun 26, 2024

MWASoftware Jun 26, 2024

rlebeau commented Aug 5, 2024

MWASoftware commented Aug 5, 2024

rlebeau commented Aug 5, 2024 •

edited

Loading

Proposed Update to support OpenSSL 3.x and earlier versions #529

Are you sure you want to change the base?

Proposed Update to support OpenSSL 3.x and earlier versions #529

Conversation

MWASoftware commented Apr 1, 2024

NOTES FOR USING THE TEST PROGRAMS

NOTES FOR LAZARUS/FPC

NOTES FOR DELPHI USERS

This comment was marked as outdated.

Choose a reason for hiding this comment

rlebeau Apr 25, 2024 • edited Loading

Choose a reason for hiding this comment

This comment was marked as outdated.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

baka0815 commented Apr 25, 2024 • edited Loading

baka0815 commented Apr 26, 2024

baka0815 Apr 26, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MWASoftware commented Apr 26, 2024

MWASoftware commented Apr 26, 2024

baka0815 commented Apr 26, 2024 • edited Loading

MWASoftware commented May 1, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rlebeau commented Aug 5, 2024

MWASoftware commented Aug 5, 2024

rlebeau commented Aug 5, 2024 • edited Loading

rlebeau Apr 25, 2024 •

edited

Loading

baka0815 commented Apr 25, 2024 •

edited

Loading

baka0815 Apr 26, 2024 •

edited

Loading

baka0815 commented Apr 26, 2024 •

edited

Loading

rlebeau commented Aug 5, 2024 •

edited

Loading