I had the same issues, thus I was deploying on traditional server and not on Vercel. The 404 error is due to some hardcoded hostname-matching login in the middleware.ts file. The redirect to the login page is due to the session cookie being rejected from the browser because having the wrong domain hardcoded.

With the changes I proposed in #1490 I was able to successfully login on selfhosted instances (haven't tried Vercel deployment though).

Nevertheless, to get a fully complete selfhosted instance other tweaks in the codebase are required, I will try to send other PR when I have time if the project seems receptive to improve the selfhosting experience.

I guess they will be happy to offer commercial support on selfhosting.

Having the same problem.

Something I found is that the custom domain you set in these variables gets blocked (404).

NEXTAUTH_URL=http://localhost:3000
NEXT_PUBLIC_BASE_URL=http://localhost:3000

So weird!