ó …¾^Yc@sadZddlZddlZddlZddlZddlZddlZddlmZddl m Z m Z ddl m Z d„Ze ddƒd „ƒZd efd „ƒYZeƒjZejd d ejƒZe ddƒd„ƒZe ddƒd„ƒZejr!eZneZe ddƒd„ƒZe ddddƒd„ƒZdS(s"Better tokenizing for coverage.py.iÿÿÿÿN(tenv(titernextt unicode_class(tcontractc csJd }d}d }x1|D])\}}\}}\}} } ||kr|r |jdƒr t} |tjkryt} nG|tjkrÀd|krÀ|jddƒdddkrÀt} qÀn| r t |jdƒdƒd} dd|| f|| d f|fVq n| }|}n||||f|| f| fV|}qWd S( sBReturn all physical tokens, even line continuations. tokenize.generate_tokens() doesn't return a token for the backslash that continues lines. This wrapper provides those tokens so that we can re-create a faithful representation of the original source. Returns the same values as generate_tokens() iÿÿÿÿs\ s iis\iþÿÿÿiŸ†iN( tNonetendswithtTruettokenizetCOMMENTtFalsettokentSTRINGtsplittlen( ttokst last_linet last_linenot last_ttypetttypetttexttslinenotscoltelinenotecoltltexttinject_backslashtccol((s5/tmp/pip-build-hU8Cw8/coverage/coverage/phystokens.pyt phys_tokenss* (  ,   tsourcetunicodeccs¸ttjtjtjtjgƒ}g}d}|jdƒjddƒ}t |ƒ}xLt |ƒD]>\}}\}}\}} }t } xt j d|ƒD]ñ} | dkrÊ|Vg}d}t} n¿| dkrßt} nª||krôt} n•| r*||kr*|jdd||fƒt} ntjj|d ƒjƒd } |tjkrptj|ƒrpd } n|j| | fƒt } d}qžW| rd| }qdqdW|r´|Vnd S( sGenerate a series of lines, one for each line in `source`. Each line is a list of pairs, each pair is a token:: [('key', 'def'), ('ws', ' '), ('nam', 'hello'), ('op', '('), ... ] Each pair has a token class, and the token text. If you concatenate all the token texts, and then join them with newlines, you should have your original `source` back, with two differences: trailing whitespace is not preserved, and a final line with no newline is indistinguishable from a final line with a newline. iis s s( )ttwsu txxitkeyN(tsetR tINDENTtDEDENTtNEWLINERtNLt expandtabstreplacetgenerate_tokensRRtreR R tappendttok_nametgettlowertNAMEtkeywordt iskeyword(Rt ws_tokenstlinetcolttokgenRRt_RRt mark_starttparttmark_endt tok_class((s5/tmp/pip-build-hU8Cw8/coverage/coverage/phystokens.pytsource_token_linesJs<$ .          tCachedTokenizercBs/eZdZd„Zeddƒd„ƒZRS(sXA one-element cache around tokenize.generate_tokens. When reporting, coverage.py tokenizes files twice, once to find the structure of the file, and once to syntax-color it. Tokenizing is expensive, and easily cached. This is a one-element cache so that our twice-in-a-row tokenizing doesn't actually tokenize twice. cCsd|_d|_dS(N(Rt last_textt last_tokens(tself((s5/tmp/pip-build-hU8Cw8/coverage/coverage/phystokens.pyt__init__Šs ttextRcCsO||jkrH||_t|jtƒƒ}ttj|ƒƒ|_n|jS(s*A stand-in for `tokenize.generate_tokens`.(R=Rt splitlinesRtlistRR)R>(R?RAtreadline((s5/tmp/pip-build-hU8Cw8/coverage/coverage/phystokens.pyR)Žs  (t__name__t __module__t__doc__R@RR)(((s5/tmp/pip-build-hU8Cw8/coverage/coverage/phystokens.pyR<s  s#^[ \t]*#.*coding[:=][ \t]*([-\w.]+)tflagstbytescsçt|tƒst‚t|jtƒƒ‰d„‰d}t‰d}‡fd†}‡‡fd†}|ƒ}|jt j ƒršt‰|d}d}n|s¤|S||ƒ}|rº|S|ƒ}|sÍ|S||ƒ}|rã|S|S(sªDetermine the encoding for `source`, according to PEP 263. `source` is a byte string, the text of the program. Returns a string, the name of the encoding. cSsL|d jƒjddƒ}tjd|ƒr2dStjd|ƒrHdS|S(s(Imitates get_normal_name in tokenizer.c.i R6t-s ^utf-8($|-)sutf-8s&^(latin-1|iso-8859-1|iso-latin-1)($|-)s iso-8859-1(R.R(R*tmatch(torig_enctenc((s5/tmp/pip-build-hU8Cw8/coverage/coverage/phystokens.pyt_get_normal_name­s tasciics$y ˆƒSWntk rdSXdS(s Get the next source line, or ''.RN(t StopIteration((RD(s5/tmp/pip-build-hU8Cw8/coverage/coverage/phystokens.pyt read_or_stopÄs  csÌy|jdƒ}Wntk r'dSXtj|ƒ}|sAdSˆ|dƒ}ytj|ƒ}Wn!tk r‡td|ƒ‚nXˆrÈt |d|ƒ}|dkr»tdƒ‚n|d7}n|S( s"Find an encoding cookie in `line`.ROisunknown encoding: tnamesutf-8sencoding problem: utf-8s-sigN( tdecodetUnicodeDecodeErrorRt COOKIE_REtfindalltcodecstlookupt LookupErrort SyntaxErrortgetattr(R3t line_stringtmatchestencodingtcodect codec_name(RNt bom_found(s5/tmp/pip-build-hU8Cw8/coverage/coverage/phystokens.pyt find_cookieËs$    is utf-8-sigN( t isinstanceRItAssertionErrorRRBRR Rt startswithRWtBOM_UTF8(RtdefaultR^RQRbtfirsttsecond((RNRaRDs5/tmp/pip-build-hU8Cw8/coverage/coverage/phystokens.pyt_source_encoding_py2s2        cCs&t|jtƒƒ}tj|ƒdS(sªDetermine the encoding for `source`, according to PEP 263. `source` is a byte string: the text of the program. Returns a string, the name of the encoding. i(RRBRRtdetect_encoding(RRD((s5/tmp/pip-build-hU8Cw8/coverage/coverage/phystokens.pyt_source_encoding_py3üs cCsUt|ƒ}tjr?t|tƒr?|jtjƒdƒ}nt|||ƒ}|S(s»Just like the `compile` builtin, but works on any Unicode string. Python 2's compile() builtin has a stupid restriction: if the source string is Unicode, then it may not have a encoding declaration in it. Why not? Who knows! It also decodes to utf8, and then tries to interpret those utf8 bytes according to the encoding declaration. Why? Who knows! This function neuters the coding declaration, and compiles it. R(( tneuter_encoding_declarationRtPY2RcRtencodetsystgetfilesystemencodingtcompile(Rtfilenametmodetcode((s5/tmp/pip-build-hU8Cw8/coverage/coverage/phystokens.pytcompile_unicodes treturnscCsttj|ƒrp|jtƒ}x=ttdt|ƒƒƒD] }tjd||ƒ||s*       85 _