a !f@sdZddlZddlZddlZddlZddlmZddlmZddlmZddl m Z ddl m Z ddl m Z dd lmZdd lmZdd lmZeZGd d d eZGdddeZddZddZddZGdddeZGdddeZdS)zDefine API Datasets.N)_datetime_from_microseconds)_microseconds_from_datetime)_millis_from_datetime)NotFound)Request)make_api_request)RESUMABLE_UPLOAD)Upload)_rows_from_jsonc@s"eZdZdZdddZddZdS) SchemaFieldahDescribe a single field within a table schema. :type name: str :param name: the name of the field :type field_type: str :param field_type: the type of the field (one of 'STRING', 'INTEGER', 'FLOAT', 'BOOLEAN', 'TIMESTAMP' or 'RECORD') :type mode: str :param mode: the type of the field (one of 'NULLABLE', 'REQUIRED', or 'REPEATED') :type description: str :param description: optional description for the field :type fields: list of :class:`SchemaField`, or None :param fields: subfields (requires ``field_type`` of 'RECORD'). NULLABLENcCs"||_||_||_||_||_dSN)name field_typemode descriptionfields)selfrrrrrrV/var/www/html/python-backend/venv/lib/python3.9/site-packages/gcloud/bigquery/table.py__init__9s zSchemaField.__init__cCsD|j|jkoB|j|jkoB|j|jkoB|j|jkoB|j|jkSr )rrlowerrrr)rotherrrr__eq__As    zSchemaField.__eq__)r NN)__name__ __module__ __qualname____doc__rrrrrrr %s  r c@seZdZdZdZdKddZeddZedd Zed d Z ed d Z e j dd Z eddZ eddZ eddZeddZeddZeddZeddZeddZedd Zej d!d Zed"d#Zej d$d#Zed%d&Zej d'd&Zed(d)Zej d*d)Zed+d,Zej d-d,Zejd.d,Zed/d0Zd1d2Zd3d4Zd5d6ZdLd7d8ZdMd9d:Z dNd;d<Z!de"e"e"e"e"e"fd=d>Z#dOd?d@Z$dPdAdBZ%dQdCdDZ&dRdEdFZ'dSdIdJZ(dS)TTableaTables represent a set of rows whose values correspond to a schema. See: https://cloud.google.com/bigquery/docs/reference/v2/tables :type name: str :param name: the name of the table :type dataset: :class:`gcloud.bigquery.dataset.Dataset` :param dataset: The dataset which contains the table. :type schema: list of :class:`SchemaField` :param schema: The table's schema NrcCs||_||_i|_||_dSr )r_dataset _propertiesschema)rrdatasetr!rrrr\szTable.__init__cCs|jjS)zsProject bound to the table. :rtype: str :returns: the project (derived from the dataset). )rprojectrrrrr#csz Table.projectcCs|jjS)zxName of dataset containing the table. :rtype: str :returns: the ID (derived from the dataset). )rrr$rrr dataset_namelszTable.dataset_namecCsd|jj|jfS)zzURL path for the table's APIs. :rtype: str :returns: the path based on project and dataste name. z %s/tables/%s)rpathrr$rrrr&usz Table.pathcCs t|jS)zuTable's schema. :rtype: list of :class:`SchemaField` :returns: fields describing the schema )list_schemar$rrrr!~sz Table.schemacCs(tdd|Dstdt||_dS)aUpdate table's schema :type value: list of :class:`SchemaField` :param value: fields describing the schema :raises: TypeError if 'value' is not a sequence, or ValueError if any item in the sequence is not a SchemaField css|]}t|tVqdSr ) isinstancer ).0fieldrrr zTable.schema..zSchema items must be fieldsN)all ValueErrortupler(rvaluerrrr!s cCs$|jd}|dur td|SdS)zDatetime at which the table was created. :rtype: ``datetime.datetime``, or ``NoneType`` :returns: the creation time (None until set from the server). creationTimeN@@r getr)rZ creation_timerrrcreateds z Table.createdcCs |jdS)zETag for the table resource. :rtype: str, or ``NoneType`` :returns: the ETag (None until set from the server). etagr r6r$rrrr8sz Table.etagcCs$|jd}|dur td|SdS)zDatetime at which the table was last modified. :rtype: ``datetime.datetime``, or ``NoneType`` :returns: the modification time (None until set from the server). lastModifiedTimeNr4r5)rZ modified_timerrrmodifieds zTable.modifiedcCs |jd}|durt|SdS)zThe size of the table in bytes. :rtype: integer, or ``NoneType`` :returns: the byte count (None until set from the server). ZnumBytesNr r6int)rZnum_bytes_as_strrrr num_bytess zTable.num_bytescCs |jd}|durt|SdS)zThe number of rows in the table. :rtype: integer, or ``NoneType`` :returns: the row count (None until set from the server). ZnumRowsNr<)rZnum_rows_as_strrrrnum_rowss zTable.num_rowscCs |jdS)zURL for the table resource. :rtype: str, or ``NoneType`` :returns: the URL (None until set from the server). ZselfLinkr9r$rrr self_linkszTable.self_linkcCs |jdS)zID for the table resource. :rtype: str, or ``NoneType`` :returns: the ID (None until set from the server). idr9r$rrrtable_idszTable.table_idcCs |jdS)zThe type of the table. Possible values are "TABLE" or "VIEW". :rtype: str, or ``NoneType`` :returns: the URL (None until set from the server). typer9r$rrr table_types zTable.table_typecCs |jdS)zDescription of the table. :rtype: str, or ``NoneType`` :returns: The description as set by the user, or None (the default). rr9r$rrrrszTable.descriptioncCs*t|tjs|durtd||jd<dS)zUpdate description of the table. :type value: str, or ``NoneType`` :param value: new description :raises: ValueError for invalid value types. NPass a string, or Nonerr)six string_typesr/r r1rrrrs cCs$|jd}|dur td|SdS)zDatetime at which the table will be removed. :rtype: ``datetime.datetime``, or ``NoneType`` :returns: the expiration time, or None expirationTimeNr4r5)rZexpiration_timerrrexpiress z Table.expirescCs.t|tjs|durtdt||jd<dS)zUpdate datetime at which the table will be removed. :type value: ``datetime.datetime``, or ``NoneType`` :param value: the new expiration time, or None NPass a datetime, or NonerI)r)datetimer/rr r1rrrrJ scCs |jdS)zTitle of the table. :rtype: str, or ``NoneType`` :returns: The name as set by the user, or None (the default). friendlyNamer9r$rrr friendly_nameszTable.friendly_namecCs*t|tjs|durtd||jd<dS)zUpdate title of the table. :type value: str, or ``NoneType`` :param value: new title :raises: ValueError for invalid value types. NrErMrFr1rrrrNs cCs |jdS)zLocation in which the table is hosted. :rtype: str, or ``NoneType`` :returns: The location as set by the user, or None (the default). locationr9r$rrrrO,szTable.locationcCs*t|tjs|durtd||jd<dS)zUpdate location in which the table is hosted. :type value: str, or ``NoneType`` :param value: new location :raises: ValueError for invalid value types. NrErOrFr1rrrrO5s cCs"|jd}|dur|dSdS)zSQL query defining the table as a view. :rtype: str, or ``NoneType`` :returns: The query as set by the user, or None (the default). viewNqueryr9)rrPrrr view_queryBs zTable.view_querycCs&t|tjstdd|i|jd<dS)zUpdate SQL query defining the table as a view. :type value: str :param value: new query :raises: ValueError for invalid value types. z Pass a stringrQrPNrFr1rrrrRMs cCs|jdddS)z.Delete SQL query defining the table as a view.rPN)r popr$rrrrRZscCsBd|vsd|dvrtd|dd}|||d}|||S)aFactory: construct a table given its API representation :type resource: dict :param resource: table resource representation returned from the API :type dataset: :class:`gcloud.bigquery.dataset.Dataset` :param dataset: The dataset containing the table. :rtype: :class:`gcloud.bigquery.table.Table` :returns: Table parsed from ``resource``. tableReferencetableIdzJResource lacks required identity information:["tableReference"]["tableId"])r")KeyError_set_properties)clsresourcer"Z table_nametablerrr from_api_repr_s     zTable.from_api_reprcCs|dur|jj}|S)aCheck client or verify over-ride. :type client: :class:`gcloud.bigquery.client.Client` or ``NoneType`` :param client: the client to use. If not passed, falls back to the ``client`` stored on the current dataset. :rtype: :class:`gcloud.bigquery.client.Client` :returns: The client passed in or the currently bound client. N)rZ_clientrclientrrr_require_clientus zTable._require_clientcCs|j|}|dddi}t||_d|vrDt|d|d<d|vr\t|d|d<d|vrtt|d|d<|j|dS)zUpdate properties from resource in body of ``api_response`` :type api_response: httplib2.Response :param api_response: response returned from an API call r!rrr3r:rIN)r clearcopyrS_parse_schema_resourcer!floatupdate)r api_responsecleanedr!rrrrWs  zTable._set_propertiescCsd|jj|jj|jdi}|jdur.|j|d<|jdurJt|j}||d<|jdur^|j|d<|jdurr|j|d<|jduri}|d<|j|d <n"|j rd t |j i|d <nt d |S) z1Generate a resource for ``create`` or ``update``.rTZ projectIdZ datasetIdrUNrrIrMrOrPrQrr!z$Set either 'view_query' or 'schema'.) rr#rrrJrrNrOrRr(_build_schema_resourcer/)rrYr2rPrrr_build_resources.            zTable._build_resourcecCsB||}d|jj|jjf}|jjd||d}||dS)akAPI call: create the dataset via a PUT request See: https://cloud.google.com/bigquery/docs/reference/v2/tables/insert :type client: :class:`gcloud.bigquery.client.Client` or ``NoneType`` :param client: the client to use. If not passed, falls back to the ``client`` stored on the current dataset. z/projects/%s/datasets/%s/tablesPOSTmethodr&dataN)r^rr#r connection api_requestrhrW)rr]r&rdrrrcreates   z Table.createcCsD||}z|jjd|jddidWnty:YdS0dSdS)aAPI call: test for the existence of the table via a GET request See https://cloud.google.com/bigquery/docs/reference/v2/tables/get :type client: :class:`gcloud.bigquery.client.Client` or ``NoneType`` :param client: the client to use. If not passed, falls back to the ``client`` stored on the current dataset. :rtype: bool :returns: Boolean indicating existence of the table. GETrrArkr& query_paramsFTN)r^rmrnr&rr\rrrexistss    z Table.existscCs*||}|jjd|jd}||dS)amAPI call: refresh table properties via a GET request See https://cloud.google.com/bigquery/docs/reference/v2/tables/get :type client: :class:`gcloud.bigquery.client.Client` or ``NoneType`` :param client: the client to use. If not passed, falls back to the ``client`` stored on the current dataset. rprkr&N)r^rmrnr&rWrr]rdrrrreloads z Table.reloadc Cs||}i}|tur>t|tjs2|dur2tdt||d<|turN||d<|tur^||d<|turn||d<|tur|durd|d<n d|i|d<|tur|durd|d <nd t|i|d <|jjd |j |d } | | dS) a_API call: update individual table properties via a PATCH request See https://cloud.google.com/bigquery/docs/reference/v2/tables/patch :type client: :class:`gcloud.bigquery.client.Client` or ``NoneType`` :param client: the client to use. If not passed, falls back to the ``client`` stored on the current dataset. :type friendly_name: str or ``NoneType`` :param friendly_name: point in time at which the table expires. :type description: str or ``NoneType`` :param description: point in time at which the table expires. :type location: str or ``NoneType`` :param location: point in time at which the table expires. :type expires: :class:`datetime.datetime` or ``NoneType`` :param expires: point in time at which the table expires. :type view_query: str :param view_query: SQL query defining the table as a view :type schema: list of :class:`SchemaField` :param schema: fields describing the schema :raises: ValueError for invalid value types. NrKrIrrMrOrPrQr!rPATCHrj) r^_MARKERr)rLr/rrgrmrnr&rW) rr]rNrrOrJrRr!partialrdrrrpatchs6%      z Table.patchcCs0||}|jjd|j|d}||dS)aoAPI call: update table properties via a PUT request See https://cloud.google.com/bigquery/docs/reference/v2/tables/update :type client: :class:`gcloud.bigquery.client.Client` or ``NoneType`` :param client: the client to use. If not passed, falls back to the ``client`` stored on the current dataset. PUTrjN)r^rmrnr&rhrWrurrrrc7s  z Table.updatecCs ||}|jjd|jddS)alAPI call: delete the table via a DELETE request See: https://cloud.google.com/bigquery/docs/reference/v2/tables/delete :type client: :class:`gcloud.bigquery.client.Client` or ``NoneType`` :param client: the client to use. If not passed, falls back to the ``client`` stored on the current dataset. DELETErtN)r^rmrnr&r\rrrdeleteFs z Table.deletecCs||}i}|dur||d<|dur.||d<|jjdd|j|d}|d}|dur`t|}|d}t|dd |j}|||fS) aAPI call: fetch the table data via a GET request See: https://cloud.google.com/bigquery/docs/reference/v2/tabledata/list .. note:: This method assumes that its instance's ``schema`` attribute is up-to-date with the schema as defined on the back-end: if the two schemas are not identical, the values returned may be incomplete. To ensure that the local copy of the schema is up-to-date, call the table's ``reload`` method. :type max_results: integer or ``NoneType`` :param max_results: maximum number of rows to return. :type page_token: str or ``NoneType`` :param page_token: token representing a cursor into the table's rows. :type client: :class:`gcloud.bigquery.client.Client` or ``NoneType`` :param client: the client to use. If not passed, falls back to the ``client`` stored on the current dataset. :rtype: tuple :returns: ``(row_data, total_rows, page_token)``, where ``row_data`` is a list of tuples, one per result row, containing only the values; ``total_rows`` is a count of the total number of rows in the table; and ``page_token`` is an opaque string which can be used to fetch the next batch of rows (``None`` if no further batches can be fetched). NZ maxResultsZ pageTokenrpz%s/datarqZ totalRowsrowsr)r^rmrnr&r6r=r r()rZ max_resultsZ page_tokenr]paramsresponseZ total_rowsZ rows_datarrr fetch_dataSs   zTable.fetch_datacCs||}g}d|i}t|D]p\} } i} t|j| D]0\} } | jdkr\| dur\t| d} | | | j<q6d| i}|dur|| |d<||q|dur||d<|dur||d<|dur||d <|jj d d |j |d }g}| d dD] }|t |d|ddq|S)aAPI call: insert table data via a POST request See: https://cloud.google.com/bigquery/docs/reference/v2/tabledata/insertAll :type rows: list of tuples :param rows: Row data to be inserted. Each tuple should contain data for each schema field on the current table and in the same order as the schema fields. :type row_ids: list of string :param row_ids: Unique ids, one per row being inserted. If not passed, no de-duplication occurs. :type skip_invalid_rows: boolean or ``NoneType`` :param skip_invalid_rows: skip rows w/ invalid data? :type ignore_unknown_values: boolean or ``NoneType`` :param ignore_unknown_values: ignore columns beyond schema? :type template_suffix: str or ``NoneType`` :param template_suffix: treat ``name`` as a template table and provide a suffix. BigQuery will create the table `` + `` based on the schema of the template table. See: https://cloud.google.com/bigquery/streaming-data-into-bigquery#template-tables :type client: :class:`gcloud.bigquery.client.Client` or ``NoneType`` :param client: the client to use. If not passed, falls back to the ``client`` stored on the current dataset. :rtype: list of mappings :returns: One mapping per row with insert errors: the "index" key identifies the row, and the "errors" key contains a list of the mappings describing one or more problems with the row. r~ TIMESTAMPNgư>jsonZinsertIdZskipInvalidRowsignoreUnknownValuesZtemplateSuffixriz %s/insertAllrjZ insertErrorsrindexerrors)rr) r^ enumeratezipr(rrrappendrmrnr&r6r=)rr~Zrow_idsZskip_invalid_rowsignore_unknown_valuesZtemplate_suffixr]Z rows_inforlrrowZrow_infor+r2inforrerrorrrr insert_datas<,      zTable.insert_dataFc! Cs||}|j}d}|r&|dtjt|dd}|durJ|dkrJtd|}|durzt|drrt| j }ntdd d |j d d }d d |dt |j i|jj|jj|jddii}t||||| | | | | || t|||dd}t}t}|jd}d|jjf}|j||d}t|d|t|d}|||||j}|jd}|j|||d|_|||j|jt kr|j!dd}nt"|j||d}|j#} t$| t%j&s| 'd} |(t)| S)al Upload the contents of this table from a file-like object. The content type of the upload will either be - The value passed in to the function (if any) - ``text/csv``. :type file_obj: file :param file_obj: A file handle opened in binary mode for reading. :type source_format: str :param source_format: one of 'CSV' or 'NEWLINE_DELIMITED_JSON'. job configuration option; see :meth:`gcloud.bigquery.job.LoadJob` :type rewind: boolean :param rewind: If True, seek to the beginning of the file handle before writing the file to Cloud Storage. :type size: int :param size: The number of bytes to read from the file handle. If not provided, we'll try to guess the size using :func:`os.fstat`. (If the file handle is not from the filesystem this won't be possible.) :type num_retries: integer :param num_retries: Number of upload retries. Defaults to 6. :type allow_jagged_rows: boolean :param allow_jagged_rows: job configuration option; see :meth:`gcloud.bigquery.job.LoadJob` :type allow_quoted_newlines: boolean :param allow_quoted_newlines: job configuration option; see :meth:`gcloud.bigquery.job.LoadJob` :type create_disposition: str :param create_disposition: job configuration option; see :meth:`gcloud.bigquery.job.LoadJob` :type encoding: str :param encoding: job configuration option; see :meth:`gcloud.bigquery.job.LoadJob` :type field_delimiter: str :param field_delimiter: job configuration option; see :meth:`gcloud.bigquery.job.LoadJob` :type ignore_unknown_values: boolean :param ignore_unknown_values: job configuration option; see :meth:`gcloud.bigquery.job.LoadJob` :type max_bad_records: integer :param max_bad_records: job configuration option; see :meth:`gcloud.bigquery.job.LoadJob` :type quote_character: str :param quote_character: job configuration option; see :meth:`gcloud.bigquery.job.LoadJob` :type skip_leading_rows: integer :param skip_leading_rows: job configuration option; see :meth:`gcloud.bigquery.job.LoadJob` :type write_disposition: str :param write_disposition: job configuration option; see :meth:`gcloud.bigquery.job.LoadJob` :type client: :class:`gcloud.storage.client.Client` or ``NoneType`` :param client: Optional. The client to use. If not passed, falls back to the ``client`` stored on the current dataset. :rtype: :class:`gcloud.bigquery.jobs.LoadTableFromStorageJob` :returns: the job instance used to load the data (e.g., for querying status) :raises: :class:`ValueError` if ``size`` is not passed in and can not be determined, or if the ``file_obj`` can be detected to be a file opened in text mode. zapplication/octet-streamrrNrbzGCannot upload files opened in text mode: use open(filename, mode='rb')filenozBtotal bytes could not be determined. Please pass an explicit size.zapplication/jsonz gzip, deflate)AcceptzAccept-Encodingz User-Agentz content-type configurationloadrrf)Z sourceFormatr!ZdestinationTableF)Z auto_transferz/uploadz/projects/%s/jobs) api_base_urlr&ri)body)rr&rrT)Z use_chunks)retrieszutf-8)*r^rmseekosSEEK_SETgetattrr/hasattrfstatrst_sizeZ USER_AGENTrgr(rr#r_configure_job_metadatar _UrlBuilder _UploadConfigZ API_BASE_URLZ build_api_urlrrdumpsZconfigure_requestrrurlZinitialize_uploadhttpZstrategyrZ stream_filercontentr)rGrHdecodeZjob_from_resourceloads)!rZfile_objZ source_formatrewindsizeZ num_retriesallow_jagged_rowsallow_quoted_newlinescreate_dispositionencodingfield_delimiterrmax_bad_recordsquote_characterskip_leading_rowswrite_dispositionr]rm content_typer total_bytesheadersmetadatauploadZ url_builderZ upload_configbase_urlr&Z upload_urlrequestrr http_responseZresponse_contentrrrupload_from_files_        zTable.upload_from_file)r)N)N)N)N)N)NNN)NNNNN)FNrNNNNNNNNNNN))rrrrr(rpropertyr#r%r&r!setterr7r8r;r>r?r@rBrDrrJrNrOrRdeleter classmethodr[r^rWrhrorsrvrxrzrcr}rrrrrrrrJs                          !    I  6 Zrc Cs|dd} |dur|| d<|dur,|| d<|dur<|| d<|durL|| d<|dur\|| d<|durl|| d <|dur||| d <|dur|| d <| dur| | d <| dur| | d <dS)z*Helper for :meth:`Table.upload_from_file`.rrNZallowJaggedRowsZallowQuotedNewlinesZcreateDispositionrZfieldDelimiterrZ maxBadRecordsquoteZskipLeadingRowsZwriteDispositionr) rrrrrrrrrrrZ load_configrrrrs* rc Csfd|vr dSg}|dD]H}|d}|d}|dd}|d}t|}|t|||||q|S)a1Parse a resource fragment into a schema field. :type info: mapping :param info: should contain a "fields" key to be parsed :rtype: list of :class:`SchemaField`, or ``NoneType`` :returns: a list of parsed fields, or ``None`` if no "fields" key is present in ``info``. rNrrCrr r)r6rarr )rr!Zr_fieldrrrrZ sub_fieldsrrrras    racCsZg}|D]L}|j|j|jd}|jdur2|j|d<|jdurJt|j|d<||q|S)zGenerate a resource fragment for a schema. :type fields: sequence of :class:`SchemaField` :param fields: schema to be dumped :rtype: mapping :returns: a mapping describing the schema of the supplied fields. )rrCrNrr)rrrrrrgr)rinfosr+rrrrrgs     rgc@s*eZdZdZdgZdZdZdZdZdZ dS)rz/Faux message FBO apitools' 'configure_request'.z*/*NTz+/upload/bigquery/v2/projects/{project}/jobs) rrrracceptmax_sizeZresumable_multipartZresumable_pathZsimple_multipartZ simple_pathrrrrrsrc@seZdZdZddZdS)rz.Faux builder FBO apitools' 'configure_request'cCsi|_d|_dS)N)rrZ_relative_pathr$rrrrsz_UrlBuilder.__init__N)rrrrrrrrrrsr)rrLrrrGZgcloud._helpersrrrZgcloud.exceptionsrZgcloud.streaming.http_wrapperrrZgcloud.streaming.transferrr Zgcloud.bigquery._helpersr objectrxr rrrargrrrrrrs6         %Q-