a !f;@sdZddlZddlZddlZddlmZddlmZddlmZddlmZddl m Z ddl m Z dd l m Z dd l mZdd l mZdd l mZdd lmZddlmZddlmZddlmZddlmZddlmZddlmZddlmZddlmZejZedj Z!edj"Z#e e fZ$ddZ%ddZ&Gddde'ZddZ(dd Z)d!d"Z*d2d#d$Z+d3d&d'Z,d4d(d)Z-d5d*d+Z.d,d-Z/d.d/Z0d0d1Z1dS)6z-Google Cloud Bigtable HappyBase table module.N)_datetime_from_microseconds)_microseconds_from_datetime) _to_bytes)_total_seconds)GCRuleIntersection) MaxAgeGCRule)MaxVersionsGCRule)_get_column_pairs) _WAL_SENTINELBatch)CellsColumnLimitFilter)ColumnQualifierRegexFilter)FamilyNameRegexFilter)RowFilterChain)RowFilterUnion)RowKeyRegexFilter)TimestampRange)TimestampRangeFilter)Tablez>qcCstdd||dS)aBMake a row dict for a Thrift cell mapping. .. warning:: This method is only provided for HappyBase compatibility, but does not actually work. :type cell_map: dict :param cell_map: Dictionary with ``fam:col`` strings as keys and ``TCell`` instances as values. :type include_timestamp: bool :param include_timestamp: Flag to indicate if cell timestamps should be included with the output. :raises: :class:`NotImplementedError ` always zThe Cloud Bigtable API output is not the same as the output from the Thrift server, so this helper can not be implemented. Called withNNotImplementedError)Zcell_mapinclude_timestampr`/var/www/html/python-backend/venv/lib/python3.9/site-packages/gcloud/bigtable/happybase/table.pymake_row2srcCstdd||dS)a'Make a row dict for sorted Thrift column results from scans. .. warning:: This method is only provided for HappyBase compatibility, but does not actually work. :type sorted_columns: list :param sorted_columns: List of ``TColumn`` instances from Thrift. :type include_timestamp: bool :param include_timestamp: Flag to indicate if cell timestamps should be included with the output. :raises: :class:`NotImplementedError ` always rrNr)sorted_columnsrrrrmake_ordered_rowKsrc@seZdZdZddZddZddZdd Zd$d d Zd%ddZ d&ddZ d'ddZ d e fddZ d d e fddZd d d e fddZddZd(ddZd)d d!Zd*d"d#Zd S)+raRepresentation of Cloud Bigtable table. Used for adding data and :type name: str :param name: The name of the table. :type connection: :class:`Connection <.happybase.connection.Connection>` :param connection: The connection which has access to the table. cCs2||_||_d|_|jdur.t|j|jj|_dS)N)name connection_low_level_table_LowLevelTableZ _instance)selfr r!rrr__init__os zTable.__init__cCs d|jfS)Nz)r r$rrr__repr__yszTable.__repr__cCs4|j}i}t|D]\}}t|j||<q|S)zRetrieve the column families for this table. :rtype: dict :returns: Mapping from column family name to garbage collection rule for a column family. )r"Zlist_column_familiessix iteritems_gc_rule_to_dictgc_rule)r$Zcolumn_family_mapresultZcol_famZ col_fam_objrrrfamilies|s  zTable.familiescCs tddS)aiRetrieve the regions for this table. .. warning:: Cloud Bigtable does not give information about how a table is laid out in memory, so this method does not work. It is provided simply for compatibility. :raises: :class:`NotImplementedError ` always zQThe Cloud Bigtable API does not have a concept of splitting a table into regions.Nrr&rrrregionss z Table.regionsNFcCsPg}|dur|t|td||d}|jj||d}|durDiSt||dS)aRetrieve a single row of data. Returns the latest cells in each column (or all columns if ``columns`` is not specified). If a ``timestamp`` is set, then **latest** becomes **latest** up until ``timestamp``. :type row: str :param row: Row key for the row we are reading from. :type columns: list :param columns: (Optional) Iterable containing column names (as strings). Each column name can be either * an entire column family: ``fam`` or ``fam:`` * a single column: ``fam:col`` :type timestamp: int :param timestamp: (Optional) Timestamp (in milliseconds since the epoch). If specified, only cells returned before the the timestamp will be returned. :type include_timestamp: bool :param include_timestamp: Flag to indicate if cell timestamps should be included with the output. :rtype: dict :returns: Dictionary containing all the latest column values in the row. Nversions timestampfiltersfilter_r)append_columns_filter_helper_filter_chain_helperr"read_row_partial_row_to_dict)r$rowcolumnsr2rr3r5partial_row_datarrrr<sz Table.rowc Cs|sgSg}|dur"|t||t|td||d}|jj|d}|g}|D]4} | |jvrlq\|j| } t| |d} || | fq\|S)aqRetrieve multiple rows of data. All optional arguments behave the same in this method as they do in :meth:`row`. :type rows: list :param rows: Iterable of the row keys for the rows we are reading from. :type columns: list :param columns: (Optional) Iterable containing column names (as strings). Each column name can be either * an entire column family: ``fam`` or ``fam:`` * a single column: ``fam:col`` :type timestamp: int :param timestamp: (Optional) Timestamp (in milliseconds since the epoch). If specified, only cells returned before (or at) the timestamp will be returned. :type include_timestamp: bool :param include_timestamp: Flag to indicate if cell timestamps should be included with the output. :rtype: list :returns: A list of pairs, where the first is the row key and the second is a dictionary with the filtered values returned. Nr/r0r4r6) r7r8_row_keys_filter_helperr9r" read_rowsZ consume_allrowsr;) r$rAr=r2rr3r5partial_rows_datar,row_key curr_row_data curr_row_dictrrrrAs*  z Table.rowsc CsZt|||d}|jj||d}|dur*gS|j}|d\} } || | } t| |dSdS)aRetrieve multiple versions of a single cell from the table. :type row: str :param row: Row key for the row we are reading from. :type column: str :param column: Column we are reading from; of the form ``fam:col``. :type versions: int :param versions: (Optional) The maximum number of cells to return. If not set, returns all cells found. :type timestamp: int :param timestamp: (Optional) Timestamp (in milliseconds since the epoch). If specified, only cells returned before (or at) the timestamp will be returned. :type include_timestamp: bool :param include_timestamp: Flag to indicate if cell timestamps should be included with the output. :rtype: list :returns: List of values in the cell (with timestamps if ``include_timestamp`` is :data:`True`). )columnr1r2r4N:r6)r9r"r:_cellssplit_cells_to_pairs) r$r<rFr1r2rr5r>cellscolumn_family_idcolumn_qualifierZ curr_cellsrrrrKs z Table.cellsckst|||||||\}}} |jj|||| d} | j} z:| t| D]$} | | } t| |d}| |fVqFWq4tyYqYq40q4dS)aCreate a scanner for data in this table. This method returns a generator that can be used for looping over the matching rows. If ``row_prefix`` is specified, only rows with row keys matching the prefix will be returned. If given, ``row_start`` and ``row_stop`` cannot be used. .. note:: Both ``row_start`` and ``row_stop`` can be :data:`None` to specify the start and the end of the table respectively. If both are omitted, a full table scan is done. Note that this usually results in severe performance problems. The keyword argument ``filter`` is also supported (beyond column and row range filters supported here). HappyBase / HBase users will have used this as an HBase filter string. (See the `Thrift docs`_ for more details on those filters.) However, Google Cloud Bigtable doesn't support those filter strings so a :class:`~gcloud.bigtable.row.RowFilter` should be used instead. .. _Thrift docs: http://hbase.apache.org/0.94/book/thrift.html The arguments ``batch_size``, ``scan_batching`` and ``sorted_columns`` are allowed (as keyword arguments) for compatibility with HappyBase. However, they will not be used in any way, and will cause a warning if passed. (The ``batch_size`` determines the number of results to retrieve per request. The HBase scanner defaults to reading one record at a time, so this argument allows HappyBase to increase that number. However, the Cloud Bigtable API uses HTTP/2 streaming so there is no concept of a batched scan. The ``sorted_columns`` flag tells HBase to return columns in order, but Cloud Bigtable doesn't have this feature.) :type row_start: str :param row_start: (Optional) Row key where the scanner should start (includes ``row_start``). If not specified, reads from the first key. If the table does not contain ``row_start``, it will start from the next key after it that **is** contained in the table. :type row_stop: str :param row_stop: (Optional) Row key where the scanner should stop (excludes ``row_stop``). If not specified, reads until the last key. The table does not have to contain ``row_stop``. :type row_prefix: str :param row_prefix: (Optional) Prefix to match row keys. :type columns: list :param columns: (Optional) Iterable containing column names (as strings). Each column name can be either * an entire column family: ``fam`` or ``fam:`` * a single column: ``fam:col`` :type timestamp: int :param timestamp: (Optional) Timestamp (in milliseconds since the epoch). If specified, only cells returned before (or at) the timestamp will be returned. :type include_timestamp: bool :param include_timestamp: Flag to indicate if cell timestamps should be included with the output. :type limit: int :param limit: (Optional) Maximum number of rows to return. :type kwargs: dict :param kwargs: Remaining keyword arguments. Provided for HappyBase compatibility. :raises: If ``limit`` is set but non-positive, or if ``row_prefix`` is used with row start/stop, :class:`TypeError ` if a string ``filter`` is used. )Z start_keyend_keylimitr5r6N) _scan_filter_helperr"r@rAZ consume_nextsortedpopr; StopIteration)r$ row_startrow_stop row_prefixr=r2rrOkwargsZ filter_chainrBZ rows_dictrCrDrErrrscan+s$S    z Table.scancCs>|j||d}|||Wdn1s00YdS)a]Insert data into a row in this table. .. note:: This method will send a request with a single "put" mutation. In many situations, :meth:`batch` is a more appropriate method to manipulate data since it helps combine many mutations into a single request. :type row: str :param row: The row key where the mutation will be "put". :type data: dict :param data: Dictionary containing the data to be inserted. The keys are columns names (of the form ``fam:col``) and the values are strings (bytes) to be stored in those columns. :type timestamp: int :param timestamp: (Optional) Timestamp (in milliseconds since the epoch) that the mutation will be applied at. :type wal: object :param wal: Unused parameter (to be passed to a created batch). Provided for compatibility with HappyBase, but irrelevant for Cloud Bigtable since it does not have a Write Ahead Log. r2walN)batchput)r$r<datar2rZr[rrrr\sz Table.putcCs>|j||d}|||Wdn1s00YdS)aDelete data from a row in this table. This method deletes the entire ``row`` if ``columns`` is not specified. .. note:: This method will send a request with a single delete mutation. In many situations, :meth:`batch` is a more appropriate method to manipulate data since it helps combine many mutations into a single request. :type row: str :param row: The row key where the delete will occur. :type columns: list :param columns: (Optional) Iterable containing column names (as strings). Each column name can be either * an entire column family: ``fam`` or ``fam:`` * a single column: ``fam:col`` :type timestamp: int :param timestamp: (Optional) Timestamp (in milliseconds since the epoch) that the mutation will be applied at. :type wal: object :param wal: Unused parameter (to be passed to a created batch). Provided for compatibility with HappyBase, but irrelevant for Cloud Bigtable since it does not have a Write Ahead Log. rYN)r[delete)r$r<r=r2rZr[rrrr^s!z Table.deletecCst|||||dS)aCreate a new batch operation for this table. This method returns a new :class:`Batch <.happybase.batch.Batch>` instance that can be used for mass data manipulation. :type timestamp: int :param timestamp: (Optional) Timestamp (in milliseconds since the epoch) that all mutations will be applied at. :type batch_size: int :param batch_size: (Optional) The maximum number of mutations to allow to accumulate before committing them. :type transaction: bool :param transaction: Flag indicating if the mutations should be sent transactionally or not. If ``transaction=True`` and an error occurs while a :class:`Batch <.happybase.batch.Batch>` is active, then none of the accumulated mutations will be committed. If ``batch_size`` is set, the mutation can't be transactional. :type wal: object :param wal: Unused parameter (to be passed to the created batch). Provided for compatibility with HappyBase, but irrelevant for Cloud Bigtable since it does not have a Write Ahead Log. :rtype: :class:`Batch ` :returns: A batch bound to this table. )r2 batch_size transactionrZr )r$r2r_r`rZrrrr[s"z Table.batchcCs|j||ddS)aRetrieve the current value of a counter column. This method retrieves the current value of a counter column. If the counter column does not exist, this function initializes it to ``0``. .. note:: Application code should **never** store a counter value directly; use the atomic :meth:`counter_inc` and :meth:`counter_dec` methods for that. :type row: str :param row: Row key for the row we are getting a counter from. :type column: str :param column: Column we are ``get``-ing from; of the form ``fam:col``. :rtype: int :returns: Counter value (after initializing / incrementing by 0). r)value counter_inc)r$r<rFrrr counter_getszTable.counter_getrcCs|||t|idS)a7Set a counter column to a specific value. .. note:: Be careful using this method. It can be useful for setting the initial value of a counter, but it defeats the purpose of using atomic increment and decrement. :type row: str :param row: Row key for the row we are setting a counter in. :type column: str :param column: Column we are setting a value in; of the form ``fam:col``. :type value: int :param value: Value to set the counter to. N)r\ _PACK_I64r$r<rFrarrr counter_setszTable.counter_setr/c Cs|jj|dd}t|tjr&|d}|d\}}|||||}|||}t |dkrjt d|d}|d} t | \} | S)aAtomically increment a counter column. This method atomically increments a counter column in ``row``. If the counter column does not exist, it is automatically initialized to ``0`` before being incremented. :type row: str :param row: Row key for the row we are incrementing a counter in. :type column: str :param column: Column we are incrementing a value in; of the form ``fam:col``. :type value: int :param value: Amount to increment the counter by. (If negative, this is equivalent to decrement.) :rtype: int :returns: Counter value after incrementing. T)r7utf-8rGr/z,Expected server to return one modified cell.r) r"r< isinstancer( binary_typedecoderIZincrement_cell_valuecommitlen ValueError _UNPACK_I64) r$r<rFrarLrMZmodified_cellsZ column_cellsZ column_cellZ bytes_valueZ int_valuerrrrc)s      zTable.counter_inccCs|||| S)aAtomically decrement a counter column. This method atomically decrements a counter column in ``row``. If the counter column does not exist, it is automatically initialized to ``0`` before being decremented. :type row: str :param row: Row key for the row we are decrementing a counter in. :type column: str :param column: Column we are decrementing a value in; of the form ``fam:col``. :type value: int :param value: Amount to decrement the counter by. (If negative, this is equivalent to increment.) :rtype: int :returns: Counter value after decrementing. rbrfrrr counter_decYszTable.counter_dec)NNF)NNF)NNF)NNNNNFN)r)r/)r/)__name__ __module__ __qualname____doc__r%r'r-r.r<rArKrXr r\r^r[rdrgrcrprrrrrcs2    - : , h$ %  0rcCs|}|duri}nt|tr,dt|ji}nt|trBd|ji}nvt|trt|jdkr|j\}}t|t rt|t rt |}t |}| \}| \}||kr||||||i}|S)a'Converts garbage collection rule to dictionary if possible. This is in place to support dictionary values as was done in HappyBase, which has somewhat different garbage collection rule settings for column families. Only does this if the garbage collection rule is: * :class:`gcloud.bigtable.column_family.MaxAgeGCRule` * :class:`gcloud.bigtable.column_family.MaxVersionsGCRule` * Composite :class:`gcloud.bigtable.column_family.GCRuleIntersection` with two rules, one each of type :class:`gcloud.bigtable.column_family.MaxAgeGCRule` and :class:`gcloud.bigtable.column_family.MaxVersionsGCRule` Otherwise, just returns the input without change. :type gc_rule: :data:`NoneType `, :class:`.GarbageCollectionRule` :param gc_rule: A garbage collection rule to convert to a dictionary (if possible). :rtype: dict or :class:`gcloud.bigtable.column_family.GarbageCollectionRule` :returns: The converted garbage collection rule. NZ time_to_liveZ max_versions) rirrZmax_agerZmax_num_versionsrrmrules_SIMPLE_GC_RULESr*keys)r+r,Zrule1Zrule2key1key2rrrr*qs(        r*cCs t||}tt|dddS)aPGets the next character based on a position in a string. :type str_val: str :param str_val: A string containing the character to update. :type index: int :param index: An integer index in ``str_val``. :rtype: str :returns: The next character after the character at ``index`` in ``str_val``. r/latin-1encoding)r( indexbytesrchr)str_valindexZord_valrrr _next_chars rcCsjt|dd}|dkr|St|d}|dkrHt||dkr>qH|d8}q$|dkrTdS|d|t||S) aIncrement and truncate a byte string. Determines shortest string that sorts after the given string when compared using regular string comparison semantics. Modeled after implementation in ``gcloud-golang``. Increments the last byte that is smaller than ``0xFF``, and drops everything after it. If the string only contains ``0xFF`` bytes, ``''`` is returned. :type str_val: str :param str_val: String to increment. :rtype: str :returns: The next string in lexical order after ``str_val``. r{r|r/rN)rrmr(r~r)rrrrr_string_successors   rcCs"|dur dStd|}t|dS)aCreate a timestamp range from an HBase / HappyBase timestamp. HBase uses timestamp as an argument to specify an exclusive end deadline. Cloud Bigtable also uses exclusive end times, so the behavior matches. :type timestamp: int :param timestamp: (Optional) Timestamp (in milliseconds since the epoch). Intended to be used as the end of an HBase time range, which is exclusive. :rtype: :class:`gcloud.bigtable.row.TimestampRange`, :data:`NoneType ` :returns: The timestamp range corresponding to the passed in ``timestamp``. N)end)rr)r2Znext_timestamprrr_convert_to_time_ranges rFcCsBg}|D]4}|r0t|jd}||j|fq||jq|S)aConverts list of cells to HappyBase format. For example:: >>> import datetime >>> from gcloud.bigtable.row_data import Cell >>> cell1 = Cell(b'val1', datetime.datetime.utcnow()) >>> cell2 = Cell(b'val2', datetime.datetime.utcnow()) >>> _cells_to_pairs([cell1, cell2]) [b'val1', b'val2'] >>> _cells_to_pairs([cell1, cell2], include_timestamp=True) [(b'val1', 1456361486255), (b'val2', 1456361491927)] :type cells: list :param cells: List of :class:`gcloud.bigtable.row_data.Cell` returned from a read request. :type include_timestamp: bool :param include_timestamp: Flag to indicate if cell timestamps should be included with the output. :rtype: list :returns: List of values in the cell. If ``include_timestamp=True``, each value will be a pair, with the first part the bytes value in the cell and the second part the number of milliseconds in the timestamp on the cell. r)rr2r7ra)rKrr,cellZ ts_millisrrrrJsrJcCs8i}t|D] \}}t||d}|d||<q|S)aConvert a low-level row data object to a dictionary. Assumes only the latest value in each row is needed. This assumption is due to the fact that this method is used by callers which use a ``CellsColumnLimitFilter(1)`` filter. For example:: >>> import datetime >>> from gcloud.bigtable.row_data import Cell, PartialRowData >>> cell1 = Cell(b'val1', datetime.datetime.utcnow()) >>> cell2 = Cell(b'val2', datetime.datetime.utcnow()) >>> row_data = PartialRowData(b'row-key') >>> _partial_row_to_dict(row_data) {} >>> row_data._cells[u'fam1'] = {b'col1': [cell1], b'col2': [cell2]} >>> _partial_row_to_dict(row_data) {b'fam1:col2': b'val2', b'fam1:col1': b'val1'} >>> _partial_row_to_dict(row_data, include_timestamp=True) {b'fam1:col2': (b'val2', 1456361724480), b'fam1:col1': (b'val1', 1456361721135)} :type partial_row_data: :class:`.row_data.PartialRowData` :param partial_row_data: Row data consumed from a stream. :type include_timestamp: bool :param include_timestamp: Flag to indicate if cell timestamps should be included with the output. :rtype: dict :returns: The row data converted to a dictionary. r6r)r(r)to_dictrJ)r>rr,rFrKZ cell_valsrrrr;s!r;c Cs|dur g}|durVt|tjr*|d}|d\}}t|}t|}|||g|durl|t |t |d}|dur|t |t |} | dkrt dn| dkr|dSt|dSdS) afCreate filter chain to limit a results set. :type column: str :param column: (Optional) The column (``fam:col``) to be selected with the filter. :type versions: int :param versions: (Optional) The maximum number of cells to return. :type timestamp: int :param timestamp: (Optional) Timestamp (in milliseconds since the epoch). If specified, only cells returned before (or at) the timestamp will be matched. :type filters: list :param filters: (Optional) List of existing filters to be extended. :rtype: :class:`RowFilter ` :returns: The chained filter created, or just a single filter if only one was needed. :raises: :class:`ValueError ` if there are no filters to chain. NrhrG)r2rMust have at least one filter.r/r3)rir(rjrkrIrrextendr7r rrrmrnr) rFr1r2r3rLrM fam_filter qual_filterZ time_range num_filtersrrrr9=s(    r9c Cs|dd}g}dD] } | |vr|| || q|rVd|}d|f} t| |rhtd||dur|dkrtd|dur|dus|durtd |}t|}g} t|t j rtd n|dur| ||dur| t |t d|| d }|||fS) z2Helper for :meth:`scan`: build up a filter chain.filterN)r_Z scan_batchingrz, zRThe HappyBase legacy arguments %s were used. These arguments are unused by gcloud.zReceived unexpected argumentsr/zlimit must be positivez8row_prefix cannot be combined with row_start or row_stopzoSpecifying filters as a string is not supported by Cloud Bigtable. Use a gcloud.bigtable.row.RowFilter instead.r0) rRr7join_WARN TypeErrorrxrnrrir( string_typesr8r9) rTrUrVr=r2rOrWr5Z legacy_argsZkw_namemessager3rrrrPos@       rPcCsg}t|D]D\}}t|}|durFt|}t||gd}||q ||q t|}|dkrltdn|dkr||dSt|dSdS)a2Creates a union filter for a list of columns. :type columns: list :param columns: Iterable containing column names (as strings). Each column name can be either * an entire column family: ``fam`` or ``fam:`` * a single column: ``fam:col`` :rtype: :class:`RowFilter ` :returns: The union filter created containing all of the matched columns. :raises: :class:`ValueError ` if there are no filters to union. Nrrrr/)r rrrr7rmrnr)r=r3rLrMrrZcombined_filterrrrrr8s    r8cCsTg}|D]}|t|qt|}|dkr6tdn|dkrF|dSt|dSdS)axCreates a union filter for a list of rows. :type row_keys: list :param row_keys: Iterable containing row keys (as strings). :rtype: :class:`RowFilter ` :returns: The union filter created containing all of the row keys. :raises: :class:`ValueError ` if there are no filters to union. rrr/rN)r7rrmrnr)Zrow_keysr3rCrrrrr?s  r?)N)F)F)NNNN)2rtstructwarningsr(Zgcloud._helpersrrrrZgcloud.bigtable.column_familyrrrZgcloud.bigtable.happybase.batchr r r Zgcloud.bigtable.row_filtersr rrrrrrrZgcloud.bigtable.tablerr#warnrStructpackreunpackrorwrrobjectr*rrrrJr;r9rPr8r?rrrrsZ                     0"  & + 2+#