BigQuery Python Client v0.28.0: num_rows, schema property of a Table returning None and empty list

[pramodvspk]

Hi,

I am using the BigQuery Python client v0.28.0

Using the bigquery client to list all tables in a dataset using method list_dataset_tables, I get an iterator of table references in the dataset. When I try to access the num_rows and schema properties of the Table object it returns None and [].

But when I access the table using client.get_table() I am able to access both the num_rows and schema properties.

I am confused since both the methods return a google.cloud.bigquery.table.Table object but are inconsistent.

for table in bqclient.list_dataset_tables(dataset_ref):
	print(table.num_rows)
	print(table.schema)

Returns None and []

table = bqclient.get_table(table)
	print(table.num_rows)
	print(table.schema)

Returns the number of rows and list of schemaFields

Thanks

[tswast]

Thanks for the report.

It appears as though the BigQuery tables.list method only returns a subset of the fields of a table.

I see a few potentially sane fixes:

1. (most desirable) See if we can get the backend team to expose a full Table resource in list operations (maybe with an optional parameter).
2. (breaking, but maybe our best option) Make list_dataset_tables return a list of table references and discard the few extra table fields that the list call returns.
3. (inefficient) Make a call to get the table resource for each item in the returned list.

I'll send a note to my contacts on the BigQuery backend team about doing (1), but the most likely fix will be (2).

[tswast]

I guess there's another option (4) document that only a subset of fields are exposed in the table response objects from the list request.

I dislike that option the most, as it leaves a lot of room for confusion.

[tswast]

I got confirmation from the backend team that (1) is infeasible. Properties like the schema and total rows of the table take much longer to fetch in the backend. The list API call includes only properties which are fast for the backend to fetch.

I think I'd like to propose another option (5) Introduce a new type containing this subset of properties. Maybe PartialTable or TableListItem which includes only those properties present in the list API response. That way it will be much clearer from the documentation which properties are included.

[tseaver]

I had (naively, I guess) assumed that this issue was the exact reason TableReference even exists.

[tswast]

No, I unfortunately hadn't thought that listing would have some but not all properties. TableReference was added for the very similar problem of where properties such as destination table on the query job only include the table ID.

I'm thinking we really do need to introduce a third type.

[tswast]

Just sent #4427 to add TableListItem. This issue is also present on datasets and will need a similar change for them.