Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CertificateRequest.CreateSelfSigned fails on macOS Sequoia #106775

Closed
1 task done
Annny-Cap-Daniel opened this issue Aug 19, 2024 · 31 comments · Fixed by #106973
Closed
1 task done

CertificateRequest.CreateSelfSigned fails on macOS Sequoia #106775

Annny-Cap-Daniel opened this issue Aug 19, 2024 · 31 comments · Fixed by #106973
Assignees
Labels
area-System.Security in-pr There is an active PR which will close this issue when it is merged os-macos-sequoia (macOS15)
Milestone

Comments

@Annny-Cap-Daniel
Copy link

Annny-Cap-Daniel commented Aug 19, 2024

Update: repro app here

Is there an existing issue for this?

  • I have searched the existing issues

Describe the bug

the command dotnet dev-certs https fails with the error:
There was an error creating the HTTPS developer certificate.

Exception:

An error has occurred generating the certificate: Interop+AppleCrypto+AppleCommonCryptoCryptographicException: The specified item is no longer valid. It may have been deleted from the keychain.
   at Interop.AppleCrypto.X509CopyWithPrivateKey(SafeSecCertificateHandle certHandle, SafeSecKeyRefHandle privateKeyHandle, SafeKeychainHandle targetKeychain)
   at System.Security.Cryptography.X509Certificates.AppleCertificatePal.CopyWithPrivateKey(SafeSecKeyRefHandle privateKey)
   at System.Security.Cryptography.X509Certificates.AppleCertificatePal.CopyWithPrivateKey(RSA privateKey)
   at System.Security.Cryptography.X509Certificates.RSACertificateExtensions.CopyWithPrivateKey(X509Certificate2 certificate, RSA privateKey)
   at System.Security.Cryptography.X509Certificates.CertificateRequest.CreateSelfSigned(DateTimeOffset notBefore, DateTimeOffset notAfter)
   at Microsoft.AspNetCore.Certificates.Generation.CertificateManager.CreateSelfSignedCertificate(X500DistinguishedName subject, IEnumerable`1 extensions, DateTimeOffset notBefore, DateTimeOffset notAfter)
   at Microsoft.AspNetCore.Certificates.Generation.CertificateManager.CreateAspNetCoreHttpsDevelopmentCertificate(DateTimeOffset notBefore, DateTimeOffset notAfter)
   at Microsoft.AspNetCore.Certificates.Generation.CertificateManager.EnsureAspNetCoreHttpsDevelopmentCertificate(DateTimeOffset notBefore, DateTimeOffset notAfter, String path, Boolean trust, Boolean includePrivateKey, String password, CertificateKeyExportFormat keyExportFormat, Boolean isInteractive).

What I already tried:

  • dotnet dev-certs https --clean
  • installing the lastest dotnet version
  • uninstalling all old dotnet versions
  • reinstalling dotnet completely
  • restarting the Terminal, reboot, etc.

Nothing changed the behaviour and i still get the same error.

Expected Behavior

the certificate should be created.

Steps To Reproduce

No response

Exceptions (if any)

No response

.NET Version

8.0.401

Anything else?

dotnet --info
.NET SDK:
 Version:           8.0.401
 Commit:            811edcc344
 Workload version:  8.0.400-manifests.56cd0383
 MSBuild version:   17.11.4+37eb419ad

Runtime Environment:
 OS Name:     Mac OS X
 OS Version:  15.0
 OS Platform: Darwin
 RID:         osx-arm64
 Base Path:   /usr/local/share/dotnet/sdk/8.0.401/

.NET workloads installed:
Configured to use loose manifests when installing new manifests.
 [wasm-tools]
   Installation Source: SDK 8.0.400
   Manifest Version:    8.0.8/8.0.100
   Manifest Path:       /usr/local/share/dotnet/sdk-manifests/8.0.100/microsoft.net.workload.mono.toolchain.current/8.0.8/WorkloadManifest.json
   Install Type:        FileBased

 [wasi-experimental]
   Installation Source: SDK 8.0.400
   Manifest Version:    8.0.8/8.0.100
   Manifest Path:       /usr/local/share/dotnet/sdk-manifests/8.0.100/microsoft.net.workload.mono.toolchain.current/8.0.8/WorkloadManifest.json
   Install Type:        FileBased

 [ios]
   Installation Source: SDK 8.0.400
   Manifest Version:    17.5.8020/8.0.100
   Manifest Path:       /usr/local/share/dotnet/sdk-manifests/8.0.100/microsoft.net.sdk.ios/17.5.8020/WorkloadManifest.json
   Install Type:        FileBased

 [maccatalyst]
   Installation Source: SDK 8.0.400
   Manifest Version:    17.5.8020/8.0.100
   Manifest Path:       /usr/local/share/dotnet/sdk-manifests/8.0.100/microsoft.net.sdk.maccatalyst/17.5.8020/WorkloadManifest.json
   Install Type:        FileBased


Host:
  Version:      8.0.8
  Architecture: arm64
  Commit:       08338fcaa5

.NET SDKs installed:
  8.0.401 [/usr/local/share/dotnet/sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 8.0.8 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 8.0.8 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]

Other architectures found:
  x64   [/usr/local/share/dotnet/x64]
    registered at [/etc/dotnet/install_location_x64]

Environment variables:
  Not set

global.json file:
  Not found

Learn more:
  https://aka.ms/dotnet/info

Download .NET:
  https://aka.ms/dotnet/download
@amcasey
Copy link
Member

amcasey commented Aug 20, 2024

@Annny-Cap-Daniel Can you please try again with the --verbose flag? It won't fix the problem, but it should provide more information about what's going wrong. Please run dotnet dev-certs https --clean --verbose and then dotnet dev-certs https --verbose.

@Annny-Cap-Daniel
Copy link
Author

Thanks for your prompt reply!

dotnet dev-certs https --clean --verbose outputs:

Cleaning HTTPS development certificates from the machine. This operation might require elevated privileges. If that is the case, a prompt for credentials will be displayed.
[1] Listing certificates from CurrentUser\My
[2] Found certificates: no certificates
[6] Finished listing certificates.
[8] Filtered certificates: no certificates
[9] Excluded certificates: no certificates
HTTPS development certificates successfully removed from the machine.

and dotnet dev-certs https --verbose outputs:

[1] Listing certificates from CurrentUser\My
[2] Found certificates: no certificates
[3] Checking certificates validity
[4] Valid certificates: no certificates
[5] Invalid certificates: no certificates
[6] Finished listing certificates.
[1] Listing certificates from CurrentUser\My
[2] Found certificates: no certificates
[3] Checking certificates validity
[4] Valid certificates: no certificates
[5] Invalid certificates: no certificates
[6] Finished listing certificates.
[1] Listing certificates from LocalMachine\My
[2] Found certificates: no certificates
[3] Checking certificates validity
[4] Valid certificates: no certificates
[5] Invalid certificates: no certificates
[6] Finished listing certificates.
[8] Filtered certificates: no certificates
[9] Excluded certificates: no certificates
[16] No valid certificates found.
[17] Generating HTTPS development certificate.
[19] An error has occurred generating the certificate: Interop+AppleCrypto+AppleCommonCryptoCryptographicException: The specified item is no longer valid. It may have been deleted from the keychain.
   at Interop.AppleCrypto.X509CopyWithPrivateKey(SafeSecCertificateHandle certHandle, SafeSecKeyRefHandle privateKeyHandle, SafeKeychainHandle targetKeychain)
   at System.Security.Cryptography.X509Certificates.AppleCertificatePal.CopyWithPrivateKey(SafeSecKeyRefHandle privateKey)
   at System.Security.Cryptography.X509Certificates.AppleCertificatePal.CopyWithPrivateKey(RSA privateKey)
   at System.Security.Cryptography.X509Certificates.RSACertificateExtensions.CopyWithPrivateKey(X509Certificate2 certificate, RSA privateKey)
   at System.Security.Cryptography.X509Certificates.CertificateRequest.CreateSelfSigned(DateTimeOffset notBefore, DateTimeOffset notAfter)
   at Microsoft.AspNetCore.Certificates.Generation.CertificateManager.CreateSelfSignedCertificate(X500DistinguishedName subject, IEnumerable`1 extensions, DateTimeOffset notBefore, DateTimeOffset notAfter)
   at Microsoft.AspNetCore.Certificates.Generation.CertificateManager.CreateAspNetCoreHttpsDevelopmentCertificate(DateTimeOffset notBefore, DateTimeOffset notAfter)
   at Microsoft.AspNetCore.Certificates.Generation.CertificateManager.EnsureAspNetCoreHttpsDevelopmentCertificate(DateTimeOffset notBefore, DateTimeOffset notAfter, String path, Boolean trust, Boolean includePrivateKey, String password, CertificateKeyExportFormat keyExportFormat, Boolean isInteractive).
There was an error creating the HTTPS developer certificate.

@sh0knah
Copy link

sh0knah commented Aug 21, 2024

I'm getting the same thing. Same exact output.

@amcasey
Copy link
Member

amcasey commented Aug 21, 2024

@sh0knah Are you also on Sequoia? I think we had some other cert problems when 14.4.4 came out, so it could be a system API change that we need to react to.

@amcasey
Copy link
Member

amcasey commented Aug 21, 2024

@vcsjones Could this be another change on Apple's end?

@amcasey
Copy link
Member

amcasey commented Aug 21, 2024

Possibly relevant: dotnet/aspnetcore#19590 (comment)

@sh0knah
Copy link

sh0knah commented Aug 21, 2024

I am on Sequoia. Beta 7.

@amcasey
Copy link
Member

amcasey commented Aug 21, 2024

What happens if you run this console app?

using System.Security.Cryptography;
using System.Security.Cryptography.X509Certificates;

var subject = new X500DistinguishedName("CN=localhost");

var sanBuilder = new SubjectAlternativeNameBuilder();
sanBuilder.AddDnsName("localhost");

var keyUsage = new X509KeyUsageExtension(X509KeyUsageFlags.KeyEncipherment | X509KeyUsageFlags.DigitalSignature, critical: true);
var enhancedKeyUsage = new X509EnhancedKeyUsageExtension(
    [ new Oid("1.3.6.1.5.5.7.3.1", "Server Authentication") ],
    critical: true);
var basicConstraints = new X509BasicConstraintsExtension(
    certificateAuthority: false,
    hasPathLengthConstraint: false,
    pathLengthConstraint: 0,
    critical: true);

using var rsa = RSA.Create(2048);
var request = new CertificateRequest(subject, rsa, HashAlgorithmName.SHA256, RSASignaturePadding.Pkcs1);
request.CertificateExtensions.Add(basicConstraints);
request.CertificateExtensions.Add(keyUsage);
request.CertificateExtensions.Add(enhancedKeyUsage);
request.CertificateExtensions.Add(sanBuilder.Build());

var notBefore = DateTimeOffset.UtcNow.AddDays(1);
var notAfter = notBefore.AddDays(1);
using var cert = request.CreateSelfSigned(notBefore, notAfter);
Console.WriteLine(cert is not null);

It creates a cert that's similar to the dev cert without all the extra baggage of being a dotnet tool.

@Annny-Cap-Daniel
Copy link
Author

I'm on MacOS Sequoia (15.0 Beta 24A5327a).

Just ran the sample console app and get the same exception as in the dotnet tool:

Interop+AppleCrypto+AppleCommonCryptoCryptographicException: The specified item is no longer valid. It may have been deleted from the keychain.
   at Interop.AppleCrypto.X509CopyWithPrivateKey(SafeSecCertificateHandle certHandle, SafeSecKeyRefHandle privateKeyHandle, SafeKeychainHandle targetKeychain)
   at System.Security.Cryptography.X509Certificates.AppleCertificatePal.CopyWithPrivateKey(SafeSecKeyRefHandle privateKey)
   at System.Security.Cryptography.X509Certificates.AppleCertificatePal.CopyWithPrivateKey(RSA privateKey)
   at System.Security.Cryptography.X509Certificates.RSACertificateExtensions.CopyWithPrivateKey(X509Certificate2 certificate, RSA privateKey)
   at System.Security.Cryptography.X509Certificates.CertificateRequest.CreateSelfSigned(DateTimeOffset notBefore, DateTimeOffset notAfter)
   at Program.<Main>$(String[] args) in /Users/user/Projects/ConsoleApp1/ConsoleApp1/Program.cs:line 28

@amcasey
Copy link
Member

amcasey commented Aug 21, 2024

And what if you change this?

-certificateAuthority: false,
+certificateAuthority: true,

@Annny-Cap-Daniel
Copy link
Author

same exception unfortunately.

@amcasey
Copy link
Member

amcasey commented Aug 21, 2024

same exception unfortunately.

That's actually good news because it means they haven't decided they don't like the slightly unusual shape of our certificate.

@amcasey amcasey changed the title dotnet dev-certs https fails on macOS Sequoia CertificateRequest.CreateSelfSigned fails on macOS Sequoia Aug 21, 2024
@amcasey amcasey transferred this issue from dotnet/aspnetcore Aug 21, 2024
@dotnet-policy-service dotnet-policy-service bot added the untriaged New issue has not been triaged by the area owner label Aug 21, 2024
Copy link
Contributor

Tagging subscribers to this area: @dotnet/area-system-security, @bartonjs, @vcsjones
See info in area-owners.md if you want to be subscribed.

@amcasey
Copy link
Member

amcasey commented Aug 21, 2024

Moving this to dotnet/runtime now that there's a repro not involving aspnetcore.

@bartonjs
Copy link
Member

I don't know when we're going to have a chance to look at this (not "indefinitely far", but probably "at least several days away"). We have a few small fires to deal with for the .NET 9 release on already in-market OSes, and are super-saturated on those tasks.

If anyone from the community at large wants to jump in and help debug what's going on, that'd be appreciated.

@bartonjs bartonjs added the help wanted [up-for-grabs] Good issue for external contributors label Aug 21, 2024
@jeffhandley jeffhandley added this to the 9.0.0 milestone Aug 23, 2024
@jeffhandley jeffhandley removed the untriaged New issue has not been triaged by the area owner label Aug 23, 2024
@oumaima-aarabe
Copy link

first try enabling verbose logging to get more detailed error messages using

export DOTNET_CLI_TELEMETRY_OPTOUT=1
export DOTNET_MULTILEVEL_LOOKUP=0
dotnet dev-certs https --trust --verbose

and several common troubleshooting steps would include to clear existing certificates manually or you use the security command to list and delete any certificates related to localhost or dotnet

security find-certificate -a
security delete-certificate -c "ASP.NET Core HTTPS development certificate"

also did you ensure that your user account has the necessary permissions to access the keychain, tried creating a certificate manually using openssl ?

@sajjadhajavi
Copy link

sajjadhajavi commented Aug 26, 2024

same problem:
Mac OS Sequoia Version 15.0 Beta (24A5327a)

dotnet dev-certs https --trust --verbose

[1] Listing certificates from CurrentUser\My
[2] Found certificates: no certificates
[3] Checking certificates validity
[4] Valid certificates: no certificates
[5] Invalid certificates: no certificates
[6] Finished listing certificates.
Trusting the HTTPS development certificate was requested. If the certificate is not already trusted we will run the following command:
'security add-trusted-cert -p basic -p ssl -k <> <>'
This command might prompt you for your password to install the certificate on the keychain. To undo these changes: 'security remove-trusted-cert <>'

[1] Listing certificates from CurrentUser\My
[2] Found certificates: no certificates
[3] Checking certificates validity
[4] Valid certificates: no certificates
[5] Invalid certificates: no certificates
[6] Finished listing certificates.
[1] Listing certificates from LocalMachine\My
[2] Found certificates: no certificates
[3] Checking certificates validity
[4] Valid certificates: no certificates
[5] Invalid certificates: no certificates
[6] Finished listing certificates.
[8] Filtered certificates: no certificates
[9] Excluded certificates: no certificates
[16] No valid certificates found.
[17] Generating HTTPS development certificate.
[19]
An error has occurred generating the certificate: Interop+AppleCrypto+AppleCommonCryptoCryptographicException: The specified item is no longer valid. It may have been deleted from the keychain.
at Interop.AppleCrypto.X509CopyWithPrivateKey(SafeSecCertificateHandle certHandle, SafeSecKeyRefHandle privateKeyHandle, SafeKeychainHandle targetKeychain)
at System.Security.Cryptography.X509Certificates.AppleCertificatePal.CopyWithPrivateKey(SafeSecKeyRefHandle privateKey)
at System.Security.Cryptography.X509Certificates.AppleCertificatePal.CopyWithPrivateKey(RSA privateKey)
at System.Security.Cryptography.X509Certificates.RSACertificateExtensions.CopyWithPrivateKey(X509Certificate2 certificate, RSA privateKey)
at System.Security.Cryptography.X509Certificates.CertificateRequest.CreateSelfSigned(DateTimeOffset notBefore, DateTimeOffset notAfter)
at Microsoft.AspNetCore.Certificates.Generation.CertificateManager.CreateSelfSignedCertificate(X500DistinguishedName subject, IEnumerable1 extensions, DateTimeOffset notBefore, DateTimeOffset notAfter) at Microsoft.AspNetCore.Certificates.Generation.CertificateManager.CreateAspNetCoreHttpsDevelopmentCertificate(DateTimeOffset notBefore, DateTimeOffset notAfter) at Microsoft.AspNetCore.Certificates.Generation.CertificateManager.EnsureAspNetCoreHttpsDevelopmentCertificate(DateTimeOffset notBefore, DateTimeOffset notAfter, String path, Boolean trust, Boolean includePrivateKey, String password, CertificateKeyExportFormat keyExportFormat, Boolean isInteractive). There was an error creating the HTTPS developer certificate.

@vcsjones
Copy link
Member

This almost certainly looks like another lifetime issue. I should be able to take a look by the end of the week.

@vcsjones

This comment was marked as outdated.

@vcsjones
Copy link
Member

Okay, curiosity took over so I had a peek.

The "good news" is that our unit tests fail and reproduce the issue:

Interop+AppleCrypto+AppleCommonCryptoCryptographicException : The specified item is no longer valid. It may have been deleted from the keychain.
    Stack Trace:
     at Interop.AppleCrypto.X509CopyWithPrivateKey(SafeSecCertificateHandle certHandle, SafeSecKeyRefHandle privateKeyHandle, SafeKeychainHandle targetKeychain) in /Users/vcsjones/Projects/runtime/src/libraries/Common/src/Interop/OSX/System.Security.Cryptography.Native.Apple/Interop.X509.macOS.cs:line 279
     at System.Security.Cryptography.X509Certificates.AppleCertificatePal.CopyWithPrivateKey(SafeSecKeyRefHandle privateKey) in /Users/vcsjones/Projects/runtime/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/X509Certificates/AppleCertificatePal.Keys.macOS.cs:line 199
     at System.Security.Cryptography.X509Certificates.AppleCertificatePal.CopyWithPrivateKey(RSA privateKey) in /Users/vcsjones/Projects/runtime/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/X509Certificates/AppleCertificatePal.Keys.macOS.cs:line 131
     at System.Security.Cryptography.X509Certificates.RSACertificateExtensions.CopyWithPrivateKey(X509Certificate2 certificate, RSA privateKey) in /Users/vcsjones/Projects/runtime/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/X509Certificates/RSACertificateExtensions.cs:line 53
     at System.Security.Cryptography.X509Certificates.CertificateRequest.CreateSelfSigned(DateTimeOffset notBefore, DateTimeOffset notAfter) in /Users/vcsjones/Projects/runtime/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/X509Certificates/CertificateRequest.cs:line 519
     at System.Security.Cryptography.X509Certificates.Tests.CertificateCreation.CertificateRequestChainTests.CreateAndTestChain(AsymmetricAlgorithm rootPrivKey, AsymmetricAlgorithm intermed1PrivKey, AsymmetricAlgorithm intermed2PrivKey, AsymmetricAlgorithm leafPubKey) in /Users/vcsjones/Projects/runtime/src/libraries/System.Security.Cryptography/tests/X509Certificates/CertificateCreation/CertificateRequestChainTests.cs:line 388
     at System.Security.Cryptography.X509Certificates.Tests.CertificateCreation.CertificateRequestChainTests.CreateChain_RSA() in /Users/vcsjones/Projects/runtime/src/libraries/System.Security.Cryptography/tests/X509Certificates/CertificateCreation/CertificateRequestChainTests.cs:line 43

The other "good news" is that this is the only thing that fails on macOS 15.

@vcsjones vcsjones added os-macos-sequoia (macOS15) and removed help wanted [up-for-grabs] Good issue for external contributors labels Aug 26, 2024
@vcsjones vcsjones self-assigned this Aug 26, 2024
@dotnet-policy-service dotnet-policy-service bot added the in-pr There is an active PR which will close this issue when it is merged label Aug 26, 2024
@vcsjones
Copy link
Member

This is a behavioral change for macOS 15. I have a fix put up for it, but unfortunately there are no reasonable work around at the moment.

The core of the issue is that X509Certificate2.CopyWithPrivateKey does not work for RSA/ECDSA when the private key is an ephemeral key that was created or imported in to the internal Algorithm.Create(). On macOS, that is SecurityTransforms.

CertificateRequest uses this API itself, which is why it fails.

@ergunr

This comment was marked as off-topic.

@adityamandaleeka
Copy link
Member

@lewing @vcsjones Just confirming, this is not getting fixed in RC1?

@vcsjones
Copy link
Member

vcsjones commented Sep 5, 2024

@adityamandaleeka the fix is not present in the release/9.0-rc1 branch, so no.

@ZephiroRB
Copy link

ZephiroRB commented Sep 19, 2024

Hey same error MacOs Sequoia final version!

@vcsjones
Copy link
Member

Hey same error MacOs Sequoia final version!

Yep. This will be fixed in the October patch cycle for .NET. See the corresponding announcement for additional information.

@Flight
Copy link

Flight commented Sep 19, 2024

Any update or workaround here? Our team is blocked because of this. Should we just reinstall the old version of Mac OS just to have this command work? dotnet dev-certs https --trust

@vcsjones
Copy link
Member

I am not aware of any work arounds. If downgrading is an option, that seems reasonable. The only other option would be to wait until the October release of .NET.

@JohnGalt1717
Copy link

This shouldn't be closed until there is a release out the door. And someone internal should be making the call to release an emergency patch release .402 that fixes this.

@ryanmendoza
Copy link

This shouldn't be closed until there is a release out the door. And someone internal should be making the call to release an emergency patch release .402 that fixes this.

Agreed. Otherwise pointing to some documented steps on how to manually generate the certification and import it using the dotnet CLI would be better than a "wait until the October update train" hand-waive.

We don't all have the luxury of not upgrading to Apple's latest OS because we're not all just dotnet developers.

@Flight
Copy link

Flight commented Sep 19, 2024

So all the users of the modern Mac OS is blocked from using .net and you just tell to wait for the October release? Guys, it looks like a critical issue and the fix should be deployed in hours, not even days...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area-System.Security in-pr There is an active PR which will close this issue when it is merged os-macos-sequoia (macOS15)
Projects
None yet
Development

Successfully merging a pull request may close this issue.