Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug] Error using dbtRunner with profiles_dir set and no $HOME/.dbt directory #11286

Open
2 tasks done
tnk-ysk opened this issue Feb 9, 2025 · 1 comment · May be fixed by #11287
Open
2 tasks done

[Bug] Error using dbtRunner with profiles_dir set and no $HOME/.dbt directory #11286

tnk-ysk opened this issue Feb 9, 2025 · 1 comment · May be fixed by #11287
Labels
bug Something isn't working

Comments

@tnk-ysk
Copy link

tnk-ysk commented Feb 9, 2025 •
edited
Loading

Is this a new bug in dbt-core?

  • I believe this is a new bug in dbt-core
  • I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

Error occurs even if profiles_dir is specified in kwargs on dbt cli when the home/.dbt directory does not exist.
But it worked when I specified --profiles-dir in the args or when I specified it in an environment variable.

I don't think this is the expected result.

Expected Behavior

Even if home/.dbt does not exist, no error will occur if valid profiles_dir is specified.

Steps To Reproduce

  1. Delete home/.dbt.
  2. Run the code below.
DBT_DIR = "/path/to/dbt_dir"
res: dbtRunnerResult = dbt.invoke(
    args=["ls"],
    project_dir=DBT_DIR,
    profiles_dir=DBT_DIR,
)
if res.success == False:
    logger.error(f"Command failed with error: {res.exception}")
  1. Error occured.
Command failed with error: Path '/Users/tnk-ysk/.dbt' does not exist.

Relevant log output



Environment


- OS: macOS 14.4
- Python: 3.11.10
- dbt: 1.9.1


Which database adapter are you using with dbt?


No response


Additional Context


No response

      		
    

      
  





        

            


            

              
            

        

      


      
    








      

    



      

  
            

    

      
    

    


      

          @tnk-ysk
tnk-ysk




            added
      

  bug

  Something isn't working

      

  triage


  labels


      Feb 9, 2025

    

  



  

  

    
  


  

      

    @tnk-ysk
tnk-ysk



    linked a pull request
    
      Feb 9, 2025
    

    that will
    close
    this issue





  


  

      
   Open

  





  
  

      
        
      
      

  
    
    
  


    5 tasks
  





  









      

  
      



  

  @dbeatty10





  


    

      

  

    

        
    
    

  


      
          
  
      
            Copy link

  

      
    

  


  

      

  
    Contributor


      

  


  

    

  





      


        


  
    

      
          
Thanks for opening this issue @tnk-ysk !


I see what you are saying about getting an error when using programmatic invocations to set the profiles_dir but no .dbt directory is present within the $HOME directory.


Note
This only occurs for programmatic invocations specifically; the same error did not happen for me when using the dbt CLI. But I didn't investigate enough to determine why one and not the other.


👉 We'll probably want to gain and understanding why.




See below for reprex showing both.


Reprex


1. Create these files:


dbt_dir/dbt_project.yml 


name: "dbt_project"
version: "1.0.0"
config-version: 2
profile: "duckdb"



runner.py


from dbt.cli.main import dbtRunner, dbtRunnerResult

dbt = dbtRunner()

DBT_DIR = "dbt_dir"
res: dbtRunnerResult = dbt.invoke(
    args=["ls"],
    project_dir=DBT_DIR,
    profiles_dir=DBT_DIR,
)
if res.success == False:
    print(f"Command failed with error: {res.exception}")


Afterwards, the file tree will look like this:


.
├── dbt_dir
│   ├── dbt_project.yml
│   └── profiles.yml
└── runner.py



2. Run these commands:


Rename the user's .dbt home directory (if it exists):


mv $HOME/.dbt $HOME/.dbt.bkup 


dbt ls --project-dir dbt_dir --profiles-dir dbt_dir
python runner.py


Restore the user's .dbt home directory:


mv $HOME/.dbt.bkup $HOME/.dbt


3. Get this output:


$ dbt ls --project-dir dbt_dir --profiles-dir dbt_dir
11:40:45  Running with dbt=1.9.3
11:40:45  Registered adapter: duckdb=1.9.2
11:40:45  Found 424 macros
11:40:45  No nodes selected!

$ python runner.py                                   
Command failed with error: Path '/Users/dbeatty/.dbt' does not exist.


      		
    

  





        


            

            

              
            

        

      


      

            
  

    
  
    
    

  

  



    












      

  
            

    

      
    

    


      

          @dbeatty10
dbeatty10




          removed
  the 

  triage

 label


      Mar 27, 2025

    

  



    

    

      
    

    


      

          @dbeatty10
dbeatty10




        changed the title
[Bug] Error occurs even if profiles_dir is specified on dbt cli when the home/.dbt directory does not exist

[Bug] Error using dbtRunner with profiles_dir set and no $HOME/.dbt directory


      Mar 27, 2025

    

  












  
  

    

      

  





  




  
  

              

  
    Sign up for free
    to join this conversation on GitHub.
    Already have an account?
    Sign in to comment


  



  






  
                  




    

  


      
  

    Assignees
  



        

    No one assigned







      


  


  

    Labels
  



    

      

    bug

  Something isn't working








      

  

    

        

    Projects
  


        




    None yet




  



      


  

    
  

    Milestone
  


      No milestone





    
      
          


  

    

      
        

          
    

    Development
  




            
  
Successfully merging a pull request may close this issue.





    


  
          
      Fix default_profiles_dir

    
      tnk-ysk/dbt-core
    





      
    

  




      

    

  
      

  

    

      2 participants
    

    

        
          @dbeatty10 
        
          @tnk-ysk