Creating Client from explicit credentials requires passing project

[offlinemark]

Thanks for stopping by to let us know something could be better!

Environment details

• OS type and version: mac 10.14.6
• Python version: python --version 3.7.6
• pip version: pip --version 20.0.2
• google-cloud-storage version: pip show google-cloud-storage 1.26.0

Steps to reproduce

Creating a Client using an explicit credentials object fails if the project parameter is not explicitly given. Since the project id should also be available from the credentials object, this should not be required of the caller.

The docs advertise an API where the project is not passed.
https://googleapis.dev/python/google-api-core/latest/auth.html#explicit-credentials

context: I'm trying to use GCS from heroku, where configuration is more easily done via env variables than the filesystem.

thanks!

Code example

import os
import json

from google.cloud import storage
from google.oauth2 import service_account

# env contains contents json keyfile
cred = os.environ['GCP_CRED']

c = json.loads(cred)
cred = service_account.Credentials.from_service_account_info(c)

# client = storage.Client(credentials=cred) # fails
client = storage.Client(project=c['project_id'], credentials=cred) # works

print(list(client.list_buckets()))

output

google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application. For more information, please see https://cloud.google.com/docs/authentication/getting-started

Making sure to follow these steps will guarantee the quickest resolution possible.

Thanks!

[HemangChothani]

@offlinemark I am not able to reproduce, Found two ways to create client with explicit credentials which works for me.

Sample code:

import json
from google.cloud import storage
from google.oauth2 import service_account

# env contains contents json keyfile (path)
cred_env = os.environ['GCP_CRED']

with io.open(cred_env, "r", encoding="utf-8") as json_file:
    data = json.load(json_file)
cred = service_account.Credentials.from_service_account_info(data)
client = storage.Client(credentials=cred) # success

Another Example:

import json
from google.cloud import storage
from google.oauth2 import service_account

# env contains contents json keyfile (path)
cred_env = os.environ['GCP_CRED']

cred = service_account.Credentials.from_service_account_file(cred_env)
client = storage.Client(credentials=cred) # success

[offlinemark]

@HemangChothani I think was unclear; in my code the environment variable contains the contents of the json file. Not a path.

When deploying on Heroku, in order to have the json file on the filesystem, I'd need to check the file into version control. I'd prefer to keep it in memory via environment variables.

[offlinemark]

Seems like the issue I'm seeing is here. If there is no explicit project passed, the environment is the only place that is searched, although it is also plainly available in credentials.project_id.

https://github.com/googleapis/python-cloud-core/blob/557a7cd408d75d69e11790eafa8beb03174cbee9/google/cloud/client.py#L226

[offlinemark]

Also, fwiw none of your sample code works for me, even if the env did have a path (instead of a json strong). :/ You're certain that you don't have a valid GOOGLE_APPLICATION_CREDENTIALS in your env?

[HemangChothani]

@offlinemark Yes my bad , GOOGLE_APPLICATION_CREDENTIALS was set as default in my system, now able to reproduce it. I examined whole code and its necessary to pass project when use explicit credential, I think need to update auth documentation.
After discuss with concerned person I will create issue in related library and update the doc.

[HemangChothani]

Seems like the issue I'm seeing is here. If there is no explicit project passed, the environment is the only place that is searched, although it is also plainly available in credentials.project_id.

Here is the documentation of Credentials class to create explicit credentials .The class has no property like project_id so won't search in credential.

[offlinemark]

that is the base class. here's the actual class that is instantiated. https://google-auth.readthedocs.io/en/latest/reference/google.oauth2.service_account.html#google.oauth2.service_account.Credentials.project_id

…

On Fri, Apr 3, 2020 at 3:31 AM HemangChothani ***@***.***> wrote: Seems like the issue I'm seeing is here. If there is no explicit project passed, the environment is the only place that is searched, although it is also plainly available in credentials.project_id. Here is the documentation of Credentials <https://google-auth.readthedocs.io/en/stable/reference/google.auth.credentials.html#google.auth.credentials.Credentials> class to create explicit credentials .The class has no property like project_id so won't search in credential. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#93 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAS2MGZ4YLA5ZB2UNCGT7CDRKWGELANCNFSM4LYD5MUA> .

[HemangChothani]

@offlinemark Ok, To fulfill this feature we need to pass 'credentials' here and also need to add method in google-auth here to check the project in credential or need to add some condition before this line.

Another point to set the priorities also a bit complex, like what if user pass project_id as keyword argument and also found in credentials and both are different, that might be create dilemma.

To fulfill this, it's necessary to update these google-cloud-core and google-auth libraries, but these two are used by other libraries as well therefore i am not sure it's good and necessary change.

Need to ask responsible persons of those two libraries as it's not related to storage library.

@frankyn Your comment, if any, would be greatly appreciated.Thanks in advance.

[HemangChothani]

@frankyn Could you comment on this?