a !f@sdZddlZddlmZddlmZddlmZddlm Z ddlm Z ddlm Z dd lm Z dd l mZdd l mZGd d d eZddZGdddeZGdddeZGdddeZGdddeZGdddeZGdddeZGdddeZGdddeZGd d!d!eZGd"d#d#eZGd$d%d%eZGd&d'd'eZGd(d)d)eZGd*d+d+eZ Gd,d-d-eZ!Gd.d/d/eZ"Gd0d1d1eZ#Gd2d3d3eZ$dS)4zDefine API Jobs.N)NotFound)_datetime_from_microseconds)Dataset) SchemaField)Table)_build_schema_resource)_parse_schema_resource) _EnumProperty)_TypedPropertyc@s eZdZdZddZddZdS) UDFResourcea4Describe a single user-defined function (UDF) resource. :type udf_type: str :param udf_type: the type of the resource ('inlineCode' or 'resourceUri') :type value: str :param value: the inline code or resource URI See https://cloud.google.com/bigquery/user-defined-functions#api cCs||_||_dSNudf_typevalue)selfrrrT/var/www/html/python-backend/venv/lib/python3.9/site-packages/gcloud/bigquery/job.py__init__)szUDFResource.__init__cCs|j|jko|j|jkSr r )rotherrrr__eq__-s  zUDFResource.__eq__N)__name__ __module__ __qualname____doc__rrrrrrr s r cCs(g}|D]}|j|ji}||q|S)z :type resources: sequence of :class:`UDFResource` :param resources: fields to be appended :rtype: mapping :returns: a mapping describing userDefinedFunctionResources for the query. )rrappend) resourcesZudfsresourceZudfrrr_build_udf_resources3s   rc@s eZdZdZddZddZdS)UDFResourcesPropertyziCustom property type for :class:`QueryJob`. Also used by :class:`~gcloud.bigquery.query.Query`. cCs|dur |St|jS)zDescriptor protocal: accesstorN)list_udf_resources)rinstanceownerrrr__get__GszUDFResourcesProperty.__get__cCs(tdd|Dstdt||_dS)zDescriptor protocal: mutatorcss|]}t|tVqdSr ) isinstancer ).0urrr Oz/UDFResourcesProperty.__set__..zudf items must be UDFResourceN)all ValueErrortupler )rr!rrrr__set__MszUDFResourcesProperty.__set__N)rrrrr#r,rrrrrBsrc@s eZdZdZdZdZeefZdS) Compressionz+Pseudo-enum for ``compression`` properties.GZIPNONEN)rrrrr.r/ALLOWEDrrrrr-Tsr-c@s eZdZdZdZdZeefZdS)CreateDispositionz2Pseudo-enum for ``create_disposition`` properties.CREATE_IF_NEEDED CREATE_NEVERN)rrrrr2r3r0rrrrr1[sr1c@s&eZdZdZdZdZdZeeefZdS)DestinationFormatz2Pseudo-enum for ``destination_format`` properties.CSVNEWLINE_DELIMITED_JSONAVRON)rrrrr5r6r7r0rrrrr4bs r4c@s eZdZdZdZdZeefZdS)Encodingz(Pseudo-enum for ``encoding`` properties.zUTF-8z ISO-8559-1N)rrrrZUTF_8Z ISO_8559_1r0rrrrr8jsr8c@s eZdZdZdZdZeefZdS) QueryPriorityz/Pseudo-enum for ``QueryJob.priority`` property. INTERACTIVEBATCHN)rrrrr:r;r0rrrrr9qsr9c@s&eZdZdZdZdZdZeeefZdS) SourceFormatz-Pseudo-enum for ``source_format`` properties.r5DATASTORE_BACKUPr6N)rrrrr5r=r6r0rrrrr<xs r<c@s&eZdZdZdZdZdZeeefZdS)WriteDispositionz1Pseudo-enum for ``write_disposition`` properties. WRITE_APPENDWRITE_TRUNCATE WRITE_EMPTYN)rrrrr?r@rAr0rrrrr>s r>c@s,eZdZdZddZeddZddZdS) _BaseJobzBase class for jobs. :type client: :class:`gcloud.bigquery.client.Client` :param client: A client which holds credentials and project configuration for the dataset (which requires a project). cCs||_i|_dSr )_client _propertiesrclientrrrrsz_BaseJob.__init__cCs|jjS)zsProject bound to the job. :rtype: string :returns: the project (derived from the client). )rCprojectrrrrrGsz_BaseJob.projectcCs|dur|j}|S)aCheck client or verify over-ride. :type client: :class:`gcloud.bigquery.client.Client` or ``NoneType`` :param client: the client to use. If not passed, falls back to the ``client`` stored on the current dataset. :rtype: :class:`gcloud.bigquery.client.Client` :returns: The client passed in or the currently bound client. N)rCrErrr_require_clients z_BaseJob._require_clientN)rrrrrpropertyrGrIrrrrrBs  rBcseZdZdZfddZeddZeddZedd Zed d Z ed d Z eddZ eddZ eddZ eddZeddZeddZddZddZeddZd)d!d"Zd*d#d$Zd+d%d&Zd,d'd(ZZS)- _AsyncJoba*Base class for asynchronous jobs. :type name: string :param name: the name of the job :type client: :class:`gcloud.bigquery.client.Client` :param client: A client which holds credentials and project configuration for the dataset (which requires a project). cstt||||_dSr )superrKrname)rrMrF __class__rrrsz_AsyncJob.__init__cCs|jS)zhType of job :rtype: string :returns: one of 'load', 'copy', 'extract', 'query' ) _JOB_TYPErHrrrjob_typesz_AsyncJob.job_typecCsd|j|jfS)zwURL path for the job's APIs. :rtype: string :returns: the path based on project and job name. z/projects/%s/jobs/%s)rGrMrHrrrpathsz_AsyncJob.pathcCs |jdS)zETag for the job resource. :rtype: string, or ``NoneType`` :returns: the ETag (None until set from the server). etagrDgetrHrrrrSsz_AsyncJob.etagcCs |jdS)zURL for the job resource. :rtype: string, or ``NoneType`` :returns: the URL (None until set from the server). ZselfLinkrTrHrrr self_linksz_AsyncJob.self_linkcCs |jdS)zE-mail address of user who submitted the job. :rtype: string, or ``NoneType`` :returns: the URL (None until set from the server). user_emailrTrHrrrrWsz_AsyncJob.user_emailcCs6|jd}|dur2|d}|dur2t|dSdS)zDatetime at which the job was created. :rtype: ``datetime.datetime``, or ``NoneType`` :returns: the creation time (None until set from the server). statisticsN creationTime@@rDrUrrrXZmillisrrrcreateds   z_AsyncJob.createdcCs6|jd}|dur2|d}|dur2t|dSdS)zDatetime at which the job was started. :rtype: ``datetime.datetime``, or ``NoneType`` :returns: the start time (None until set from the server). rXN startTimerZr[r\rrrstarteds   z_AsyncJob.startedcCs6|jd}|dur2|d}|dur2t|dSdS)zDatetime at which the job finished. :rtype: ``datetime.datetime``, or ``NoneType`` :returns: the end time (None until set from the server). rXNendTimerZr[r\rrrendeds   z_AsyncJob.endedcCs"|jd}|dur|dSdS)zError information about the job as a whole. :rtype: mapping, or ``NoneType`` :returns: the error information (None until set from the server). statusNZ errorResultrTrrbrrr error_result s z_AsyncJob.error_resultcCs"|jd}|dur|dSdS)zInformation about individual errors generated by the job. :rtype: list of mappings, or ``NoneType`` :returns: the error information (None until set from the server). rbNerrorsrTrcrrrres z_AsyncJob.errorscCs"|jd}|dur|dSdS)zStatus of the job. :rtype: string, or ``NoneType`` :returns: the state (None until set from the server). rbNstaterTrcrrrrf#s z_AsyncJob.statecCsdS)/Helper: handle subclass properties in cleaned.Nr)rcleanedrrr_scrub_local_properties.sz!_AsyncJob._scrub_local_propertiescCs|}|||di}d|vr6t|d|d<d|vrNt|d|d<d|vrft|d|d<|j|j|dS)zUpdate properties from resource in body of ``api_response`` :type api_response: httplib2.Response :param api_response: response returned from an API call rXrYr^r`N)copyrirUfloatrDclearupdate)r api_responserhrXrrr_set_properties2s   z_AsyncJob._set_propertiescCsbd|vsd|dvrtd|dd}d|vs>|j|dvrLtd|j|d|j}||fS)aHelper for :meth:`from_api_repr` :type resource: dict :param resource: resource for the job :rtype: dict :returns: tuple (string, dict), where the first element is the job name and the second contains job-specific configuration. :raises: :class:`KeyError` if the resource has no identifier, or is missing the appropriate configuration. jobReferencejobIdzGResource lacks required identity information: ["jobReference"]["jobId"] configurationz>Resource lacks required configuration: ["configuration"]["%s"])KeyErrorrP)clsrrMconfigrrr_get_resource_configFs    z_AsyncJob._get_resource_configNcCs:||}d|jf}|jjd||d}||dS)aeAPI call: begin the job via a POST request See: https://cloud.google.com/bigquery/docs/reference/v2/jobs/insert :type client: :class:`gcloud.bigquery.client.Client` or ``NoneType`` :param client: the client to use. If not passed, falls back to the ``client`` stored on the current dataset. z/projects/%s/jobsPOST)methodrRdataN)rIrG connection api_request_build_resourcero)rrFrRrnrrrbegin_s   z_AsyncJob.begincCsD||}z|jjd|jddidWnty:YdS0dSdS)aAPI call: test for the existence of the job via a GET request See https://cloud.google.com/bigquery/docs/reference/v2/jobs/get :type client: :class:`gcloud.bigquery.client.Client` or ``NoneType`` :param client: the client to use. If not passed, falls back to the ``client`` stored on the current dataset. :rtype: bool :returns: Boolean indicating existence of the job. GETfieldsid)rxrRZ query_paramsFTN)rIrzr{rRrrErrrexistsos    z_AsyncJob.existscCs*||}|jjd|jd}||dS)aiAPI call: refresh job properties via a GET request See https://cloud.google.com/bigquery/docs/reference/v2/jobs/get :type client: :class:`gcloud.bigquery.client.Client` or ``NoneType`` :param client: the client to use. If not passed, falls back to the ``client`` stored on the current dataset. r~rxrRNrIrzr{rRrorrFrnrrrreloads z_AsyncJob.reloadcCs0||}|jjdd|jfd}||dS)aaAPI call: cancel job via a POST request See https://cloud.google.com/bigquery/docs/reference/v2/jobs/cancel :type client: :class:`gcloud.bigquery.client.Client` or ``NoneType`` :param client: the client to use. If not passed, falls back to the ``client`` stored on the current dataset. rwz %s/cancelrNrrrrrcancels  z_AsyncJob.cancel)N)N)N)N)rrrrrrJrQrRrSrVrWr]r_rardrerfriro classmethodrvr}rrr __classcell__rrrNrrKs@                rKc@s<eZdZdZdZdZdZdZdZdZ dZ dZ dZ dZ dZdS)_LoadConfigurationzjUser-settable configuration options for load jobs. Values which are ``None`` -> server defaults. N)rrrrZ_allow_jagged_rowsZ_allow_quoted_newlines_create_disposition _encoding_field_delimiterZ_ignore_unknown_valuesZ_max_bad_recordsZ_quote_characterZ_skip_leading_rowsZ_source_format_write_dispositionrrrrrsrcseZdZdZdZdZd%fdd ZeddZej d dZed d Z ed d Z eddZ eddZ edeZedeZedZedZedejZedeZedejZedejZedejZedZe dZ!ddZ"dd Z#d!d"Z$e%d#d$Z&Z'S)&LoadTableFromStorageJobaAsynchronous job for loading data into a table from CloudStorage. :type name: string :param name: the name of the job :type destination: :class:`gcloud.bigquery.table.Table` :param destination: Table into which data is to be loaded. :type source_uris: sequence of string :param source_uris: URIs of one or more data files to be loaded, in format ``gs:///``. :type client: :class:`gcloud.bigquery.client.Client` :param client: A client which holds credentials and project configuration for the dataset (which requires a project). :type schema: list of :class:`gcloud.bigquery.table.SchemaField` :param schema: The job's schema Nloadrcs0tt|||||_||_||_t|_dSr )rLrr destination source_urisschemar_configuration)rrMrrrFrrNrrrs z LoadTableFromStorageJob.__init__cCs t|jS)zuTable's schema. :rtype: list of :class:`SchemaField` :returns: fields describing the schema )r_schemarHrrrrszLoadTableFromStorageJob.schemacCs(tdd|Dstdt||_dS)aUpdate table's schema :type value: list of :class:`SchemaField` :param value: fields describing the schema :raises: TypeError if 'value' is not a sequence, or ValueError if any item in the sequence is not a SchemaField css|]}t|tVqdSr )r$r)r%fieldrrrr'r(z1LoadTableFromStorageJob.schema..zSchema items must be fieldsN)r)r*r+r)rrrrrrs cCs(|jd}|dur$t|ddSdS)zCount of bytes loaded from source files. :rtype: integer, or ``NoneType`` :returns: the count (None until set from the server). rXNrZinputFileBytesrDrUintrrXrrrinput_file_bytess z(LoadTableFromStorageJob.input_file_bytescCs(|jd}|dur$t|ddSdS)zCount of source files. :rtype: integer, or ``NoneType`` :returns: the count (None until set from the server). rXNrZ inputFilesrrrrr input_filess z#LoadTableFromStorageJob.input_filescCs(|jd}|dur$t|ddSdS)zCount of bytes saved to destination table. :rtype: integer, or ``NoneType`` :returns: the count (None until set from the server). rXNrZ outputBytesrrrrr output_bytess z$LoadTableFromStorageJob.output_bytescCs(|jd}|dur$t|ddSdS)zCount of rows saved to destination table. :rtype: integer, or ``NoneType`` :returns: the count (None until set from the server). rXNrZ outputRowsrrrrr output_rowss z#LoadTableFromStorageJob.output_rowsallow_jagged_rowsallow_quoted_newlinescreate_dispositionencodingfield_delimiterignore_unknown_valuesmax_bad_recordsquote_characterskip_leading_rows source_formatwrite_dispositioncCs|jdur|j|d<|jdur(|j|d<|jdur<|j|d<|jdurP|j|d<|jdurd|j|d<|jdurx|j|d<|jdur|j|d<|jdur|j|d <|jdur|j|d <|j dur|j |d <|j dur|j |d <dS) >Helper for _build_resource: copy config properties to resourceNZallowJaggedRowsZallowQuotedNewlinescreateDispositionrfieldDelimiterZignoreUnknownValuesZ maxBadRecordsquoteZskipLeadingRowsZ sourceFormatwriteDisposition) rrrrrrrrrrrrrrrrr_populate_config_resourceSs,                     z1LoadTableFromStorageJob._populate_config_resourcecCsr|j|jd|j|j|jj|jj|jjddid}|d|j}||t|jdkrndt |ji|d<|S) &Generate a resource for :meth:`begin`. projectIdrqr datasetIdtableId) sourceUrisdestinationTablerprrrrrrr) rGrMrPrr dataset_namerlenrrrrrrrrrr|ls$  z'LoadTableFromStorageJob._build_resourcecCs|dddi}t||_dS)rgrrrN)poprr)rrhrrrrrisz/LoadTableFromStorageJob._scrub_local_propertiesc Cs\||\}}|d}t|d|}t|d|}|dd}|||||d} | || S)aUFactory: construct a job given its API representation .. note: This method assumes that the project found in the resource matches the client's project. :type resource: dict :param resource: dataset job representation returned from the API :type client: :class:`gcloud.bigquery.client.Client` :param client: Client which holds credentials and project configuration for the dataset. :rtype: :class:`gcloud.bigquery.job.LoadTableFromStorageJob` :returns: Job parsed from ``resource``. rrrrrrF)rvrrrUro) rtrrFrMru dest_configdatasetrZ source_urlsjobrrr from_api_reprs  z%LoadTableFromStorageJob.from_api_repr)r)(rrrrrrPrrJrsetterrrrrr boolrrr1rr8rsix string_typesrr integer_typesrrrr<rr>rrr|rirrrrrrNrrs@            rc@seZdZdZdZdZdS)_CopyConfigurationzjUser-settable configuration options for copy jobs. Values which are ``None`` -> server defaults. N)rrrrrrrrrrrsrcsPeZdZdZdZfddZedZedZ ddZ d d Z e d d Z ZS) CopyJoba<Asynchronous job: copy data into a table from other tables. :type name: string :param name: the name of the job :type destination: :class:`gcloud.bigquery.table.Table` :param destination: Table into which data is to be loaded. :type sources: list of :class:`gcloud.bigquery.table.Table` :param sources: Table into which data is to be loaded. :type client: :class:`gcloud.bigquery.client.Client` :param client: A client which holds credentials and project configuration for the dataset (which requires a project). rjcs*tt|||||_||_t|_dSr )rLrrrsourcesrr)rrMrrrFrNrrrszCopyJob.__init__rrcCs,|jdur|j|d<|jdur(|j|d<dS)rNrr)rrrrrrrs   z!CopyJob._populate_config_resourcecCs`dd|jD}|j|jd|j||jj|jj|jjddid}|d|j}|||S)rcSsg|]}|j|j|jdqS)r)rGrrM)r%tablerrr s z+CopyJob._build_resource..rr) sourceTablesrrrr)rrGrMrPrrr)rZ source_refsrrrrrrr|s$ zCopyJob._build_resourcec Cs||\}}|d}t|d|}t|d|}g}|dD]&} t| d|}|t| d|q>|||||d} | || S)aEFactory: construct a job given its API representation .. note: This method assumes that the project found in the resource matches the client's project. :type resource: dict :param resource: dataset job representation returned from the API :type client: :class:`gcloud.bigquery.client.Client` :param client: Client which holds credentials and project configuration for the dataset. :rtype: :class:`gcloud.bigquery.job.CopyJob` :returns: Job parsed from ``resource``. rrrrr)rvrrrro) rtrrFrMrurrrr source_configrrrrrs  zCopyJob.from_api_repr)rrrrrPrr1rr>rrr|rrrrrrNrrs rc@s eZdZdZdZdZdZdZdS)_ExtractConfigurationzmUser-settable configuration options for extract jobs. Values which are ``None`` -> server defaults. N)rrrr _compressionZ_destination_formatrZ _print_headerrrrrrs rcsfeZdZdZdZfddZedZedZ e de j Z e deZd d Zd d Zed dZZS)ExtractTableToStorageJobaAsynchronous job: extract data from a table into Cloud Storage. :type name: string :param name: the name of the job :type source: :class:`gcloud.bigquery.table.Table` :param source: Table into which data is to be loaded. :type destination_uris: list of string :param destination_uris: URIs describing Cloud Storage blobs into which extracted data will be written, in format ``gs:///``. :type client: :class:`gcloud.bigquery.client.Client` :param client: A client which holds credentials and project configuration for the dataset (which requires a project). extractcs*tt|||||_||_t|_dSr )rLrrsourcedestination_urisrr)rrMrrrFrNrrr9sz!ExtractTableToStorageJob.__init__ compressiondestination_formatr print_headercCsT|jdur|j|d<|jdur(|j|d<|jdur<|j|d<|jdurP|j|d<dS)rNrZdestinationFormatrZ printHeader)rrrrrrrrrSs       z2ExtractTableToStorageJob._populate_config_resourcecCsV|jj|jj|jjd}|j|jd|j||jdid}|d|j}|||S)rrr) sourceTabledestinationUrisrrr)rrGrrMrPrr)rZ source_refrrrrrrr|^s   z(ExtractTableToStorageJob._build_resourcec CsX||\}}|d}t|d|}t|d|}|d}|||||d} | || S)aVFactory: construct a job given its API representation .. note: This method assumes that the project found in the resource matches the client's project. :type resource: dict :param resource: dataset job representation returned from the API :type client: :class:`gcloud.bigquery.client.Client` :param client: Client which holds credentials and project configuration for the dataset. :rtype: :class:`gcloud.bigquery.job.ExtractTableToStorageJob` :returns: Job parsed from ``resource``. rrrrr)rvrrro) rtrrFrMrurrrrrrrrrxs z&ExtractTableToStorageJob.from_api_repr)rrrrrPrr-rr4rr rrrrrrr|rrrrrrNrr%s    rc@s4eZdZdZdZdZdZdZdZdZ dZ dZ dZ dS)_AsyncQueryConfigurationzxUser-settable configuration options for asynchronous query jobs. Values which are ``None`` -> server defaults. N) rrrrZ_allow_large_resultsrZ_default_datasetZ _destinationZ_flatten_resultsZ _priorityZ_use_query_cacheZ_use_legacy_sqlrrrrrrsrcseZdZdZdZdZdfdd ZedeZ e dZ ed e Z ed eZed eZed ZeZed eZedeZedZddZddZddZddZeddZZS)QueryJobaAsynchronous job: query tables. :type name: string :param name: the name of the job :type query: string :param query: SQL query string :type client: :class:`gcloud.bigquery.client.Client` :param client: A client which holds credentials and project configuration for the dataset (which requires a project). :type udf_resources: tuple :param udf_resources: An iterable of :class:`gcloud.bigquery.job.UDFResource` (empty by default) queryZuserDefinedFunctionResourcesrcs*tt|||||_||_t|_dSr )rLrrr udf_resourcesrr)rrMrrFrrNrrrszQueryJob.__init__allow_large_resultsrdefault_datasetrflatten_resultspriorityuse_query_cacheuse_legacy_sqlrcCs&|jdur"|jj|jj|jjdSdS)zCreate a JSON resource for the destination table. Helper for :meth:`_populate_config_resource` and :meth:`_scrub_local_properties` Nr)rrGrrMrHrrr_destination_table_resources  z$QueryJob._destination_table_resourcecCs|jdur|j|d<|jdur(|j|d<|jdurH|jj|jjd|d<|jdurb|}||d<|jdurv|j|d<|jdur|j|d<|j dur|j |d <|j dur|j |d <|j dur|j |d <t |j d krt|j ||j<dS) rNZallowLargeResultsr)rrZdefaultDatasetrZflattenResultsrZ useQueryCacheZ useLegacySqlrr)rrrrGrMrrrrrrrrr r_UDF_KEY)rrrZ table_resrrrrs2                 z"QueryJob._populate_config_resourcecCs<|j|jd|jd|jiid}|d|j}|||S)rrrrrr)rGrMrPrrrrrrr|s  zQueryJob._build_resourcecCsb|dd}|d}|dur.|jdur^|`n0|}||kr^|j|d}||d|_dS)zHelper: handle subclass properties in cleaned. .. note: This method assumes that the project found in the resource matches the client's project. rrrrNrr)rUrrrCrr)rrhrrZ dest_remoteZ dest_localrrrrri,s   z QueryJob._scrub_local_propertiescCs2||\}}|d}||||d}|||S)aFactory: construct a job given its API representation :type resource: dict :param resource: dataset job representation returned from the API :type client: :class:`gcloud.bigquery.client.Client` :param client: Client which holds credentials and project configuration for the dataset. :rtype: :class:`gcloud.bigquery.job.RunAsyncQueryJob` :returns: Job parsed from ``resource``. rr)rvro)rtrrFrMrurrrrrr@s  zQueryJob.from_api_repr)r) rrrrrPrrr rrr1rrrrrrr9rrrrrr>rrrr|rirrrrrrNrrs(       r)%rrZgcloud.exceptionsrZgcloud._helpersrZgcloud.bigquery.datasetrZgcloud.bigquery.tablerrrrZgcloud.bigquery._helpersr r objectr rrr-r1r4r8r9r<r>rBrKrrrrrrrrrrrrs@         #}q h p