𝗖𝗨𝗥𝗥𝗘𝗡𝗧 𝗙𝗜𝗟𝗘 : /opt/alt/python27/lib/python2.7/site-packages/html5lib//inputstream.pyo
�
PO�Rc ! @` s� d d l m Z m Z m Z d d l m Z d d l m Z d d l Z d d l Z d d l
m Z m Z m
Z
m Z d d l
m Z m Z d d l m Z d d l m Z y d d
l m Z Wn e k
r� e Z n Xy d d l m Z Wn' e k
rd e f d
� � YZ n Xe g e D] Z e j d � ^ q"� Z e g e
D] Z e j d � ^ qJ� Z e g e D] Z e j d � ^ qr� Z e e d d g � BZ e j d � Z! e" d d d d d d d d d d d d d d d d! d" d# d$ d% d& d' d( d) d* d+ d, d- d. d/ d0 d1 g � Z# e j d2 � Z$ i Z% d3 e f d4 � � YZ& d e( e( d5 � Z) d6 e f d7 � � YZ* d8 e* f d9 � � YZ+ d: e, f d; � � YZ- d< e f d= � � YZ. d> e f d? � � YZ/ d@ � Z0 d S(A i ( t absolute_importt divisiont unicode_literals( t text_type( t http_clientNi ( t EOFt spaceCharacterst asciiLetterst asciiUppercase( t encodingst ReparseException( t utils( t StringIO( t BytesIO( t BufferedIOBaseR c B` s e Z RS( ( t __name__t
__module__( ( ( sE /opt/alt/python27/lib/python2.7/site-packages/html5lib/inputstream.pyR s u asciit >t <u� [- - -�-�-]i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i��
i��
i�� i�� i�� i�� i��
i��
i�� i�� i�� i�� i�� i�� u [ -
-/:-@[-`{-~]t BufferedStreamc B` sM e Z d Z d � Z d � Z d � Z d � Z d � Z d � Z d � Z RS( u� Buffering for streams that do not have buffering of their own
The buffer is implemented as a list of chunks on the assumption that
joining many strings will be slow since it is O(n**2)
c C` s% | | _ g | _ d d g | _ d S( Ni����i ( t streamt buffert position( t selfR ( ( sE /opt/alt/python27/lib/python2.7/site-packages/html5lib/inputstream.pyt __init__6 s c C` sJ d } x, | j | j d D] } | t | � 7} q W| | j d 7} | S( Ni i ( R R t len( R t post chunk( ( sE /opt/alt/python27/lib/python2.7/site-packages/html5lib/inputstream.pyt tell; s
c C` s` | } d } x> t | j | � | k rL | t | j | � 8} | d 7} q W| | g | _ d S( Ni i ( R R R ( R R t offsett i( ( sE /opt/alt/python27/lib/python2.7/site-packages/html5lib/inputstream.pyt seekB s c C` sp | j s | j | � S| j d t | j � k r_ | j d t | j d � k r_ | j | � S| j | � Sd S( Ni i i����( R t _readStreamR R t _readFromBuffer( R t bytes( ( sE /opt/alt/python27/lib/python2.7/site-packages/html5lib/inputstream.pyt readK s
c C` s& t g | j D] } t | � ^ q
� S( N( t sumR R ( R t item( ( sE /opt/alt/python27/lib/python2.7/site-packages/html5lib/inputstream.pyt _bufferedBytesT s c C` sL | j j | � } | j j | � | j d c d 7<t | � | j d <| S( Ni i ( R R# R t appendR R ( R R" t data( ( sE /opt/alt/python27/lib/python2.7/site-packages/html5lib/inputstream.pyR W s
c C` s | } g } | j d } | j d } x� | t | j � k r� | d k r� | j | } | t | � | k r� | } | | | g | _ n/ t | � | } | t | � g | _ | d 7} | j | | | | !� | | 8} d } q) W| r| j | j | � � n d j | � S( Ni i t ( R R R R' R t join( R R" t remainingBytest rvt bufferIndext bufferOffsett bufferedDatat bytesToRead( ( sE /opt/alt/python27/lib/python2.7/site-packages/html5lib/inputstream.pyR! ^ s$
$
(
R R t __doc__R R R R# R& R R! ( ( ( sE /opt/alt/python27/lib/python2.7/site-packages/html5lib/inputstream.pyR / s c C` s� t | t j � r t } n9 t | d � rE t | j d � t � } n t | t � } | r | d k ru t d � � n t | � St
| | | | � Sd S( Nu readi u7 Cannot explicitly set an encoding with a unicode string( t
isinstanceR t HTTPResponset Falset hasattrR# R t Nonet TypeErrort HTMLUnicodeInputStreamt HTMLBinaryInputStream( t sourcet encodingt parseMetat chardett isUnicode( ( sE /opt/alt/python27/lib/python2.7/site-packages/html5lib/inputstream.pyt HTMLInputStreamy s
R8 c B` s} e Z d Z d Z d � Z d � Z d � Z d � Z d � Z d � Z d
d � Z d � Z d
� Z
e d � Z d � Z RS( u� Provides a unicode stream of characters to the HTMLTokenizer.
This class takes care of character encoding and removing or replacing
incorrect byte-sequences and also provides column and line tracking.
i ( c C` s� t d � d k r3 | j | _ t j d � | _ n | j | _ t j d � | _ d g | _ d | _ | j | � | _
| j � d S(
u Initialises the HTMLInputStream.
HTMLInputStream(source, [encoding]) -> Normalized stream from source
for use by html5lib.
source can be either a file-object, local filename or a string.
The optional encoding parameter must be a string that indicates
the encoding. If specified, that encoding will be used,
regardless of any BOM or later declaration (such as in a meta
element)
parseMeta - Look for a <meta> element containing encoding information
u i u [�-�]u0 ([�-�](?![�-�])|(?<![�-�])[�-�])i u utf-8u certainN( u utf-8u certain( R t characterErrorsUCS4t reportCharacterErrorst ret compilet replaceCharactersRegexpt characterErrorsUCS2t newLinest charEncodingt
openStreamt
dataStreamt reset( R R: ( ( sE /opt/alt/python27/lib/python2.7/site-packages/html5lib/inputstream.pyR � s c C` sC d | _ d | _ d | _ g | _ d | _ d | _ d | _ d S( Nu i ( R t chunkSizet chunkOffsett errorst prevNumLinest prevNumColsR6 t _bufferedCharacter( R ( ( sE /opt/alt/python27/lib/python2.7/site-packages/html5lib/inputstream.pyRJ � s c C` s( t | d � r | } n t | � } | S( uv Produces a file object from source.
source can be either a file object, local filename or a string.
u read( R5 R ( R R: R ( ( sE /opt/alt/python27/lib/python2.7/site-packages/html5lib/inputstream.pyRH � s c C` st | j } | j d d | � } | j | } | j d d | � } | d k r\ | j | } n | | d } | | f S( Nu
i i����i ( R t countRN t rfindRO ( R R R t nLinest positionLinet lastLinePost positionColumn( ( sE /opt/alt/python27/lib/python2.7/site-packages/html5lib/inputstream.pyt _position� s
c C` s& | j | j � \ } } | d | f S( u: Returns (line, col) of the current position in the stream.i ( RW RL ( R t linet col( ( sE /opt/alt/python27/lib/python2.7/site-packages/html5lib/inputstream.pyR � s c C` sL | j | j k r% | j � s% t Sn | j } | j | } | d | _ | S( uo Read one character from the stream or queue if available. Return
EOF when EOF is reached.
i ( RL RK t readChunkR R ( R RL t char( ( sE /opt/alt/python27/lib/python2.7/site-packages/html5lib/inputstream.pyR[ � s
c C` sX | d k r | j } n | j | j � \ | _ | _ d | _ d | _ d | _ | j j | � } | j
r� | j
| } d | _
n
| s� t St | � d k r� t
| d � } | d k s� d | k o� d k n r� | d | _
| d } q� n | j | � | j j d | � } | j d d
� } | j d d
� } | | _ t | � | _ t S( Nu i i i����i
i � i�� u �u
u
u
( R6 t _defaultChunkSizeRW RK RN RO R RL RI R# RP R4 R t ordRA RD t subt replacet True( R RK R( t lastv( ( sE /opt/alt/python27/lib/python2.7/site-packages/html5lib/inputstream.pyRZ � s0
(
c C` s: x3 t t t j | � � � D] } | j j d � q Wd S( Nu invalid-codepoint( t rangeR t invalid_unicode_ret findallRM R' ( R R( R ( ( sE /opt/alt/python27/lib/python2.7/site-packages/html5lib/inputstream.pyR@ s "c C` s t } x� t j | � D]� } | r( q n t | j � � } | j � } t j | | | d !� r� t j | | | d !� } | t k r� | j
j d � n t } q | d k r� | d k r� | t
| � d k r� | j
j d � q t } | j
j d � q Wd S( Ni u invalid-codepointi � i�� i ( R4 Rc t finditerR] t groupt startR t isSurrogatePairt surrogatePairToCodepointt non_bmp_invalid_codepointsRM R' R` R ( R R( t skipt matcht codepointR t char_val( ( sE /opt/alt/python27/lib/python2.7/site-packages/html5lib/inputstream.pyRE s c
C` sT y t | | f } Wnq t k
r� d j g | D] } d t | � ^ q1 � } | sc d | } n t j d | � } t | | f <n Xg } x� t r@| j | j | j � } | d k r� | j | j k rPqnB | j � } | | j k r| j
| j | j | !� | | _ Pn | j
| j | j � | j � s� Pq� q� Wd j | � } | S( u� Returns a string of characters from the stream up to but not
including any character in 'characters' or EOF. 'characters' must be
a container that supports the 'in' method and iteration over its
characters.
u u \x%02xu ^%su [%s]+N( t charsUntilRegExt KeyErrorR* R] RB RC R` Rl R RL R6 RK t endR' RZ (
R t
characterst oppositet charst ct regexR, t mRq t r( ( sE /opt/alt/python27/lib/python2.7/site-packages/html5lib/inputstream.pyt
charsUntil5 s.
,
% c C` sS | d k rO | j d k r= | | j | _ | j d 7_ qO | j d 8_ n d S( Ni i ( R6 RL R RK ( R R[ ( ( sE /opt/alt/python27/lib/python2.7/site-packages/html5lib/inputstream.pyt ungetd s N( R R R1 R\ R RJ RH RW R R[ R6 RZ R@ RE R4 Ry Rz ( ( ( sE /opt/alt/python27/lib/python2.7/site-packages/html5lib/inputstream.pyR8 � s ! ( /R9 c B` s\ e Z d Z d e e d � Z d � Z d � Z e e d � Z d � Z d � Z
d � Z RS( u� Provides a unicode stream of characters to the HTMLTokenizer.
This class takes care of character encoding and removing or replacing
incorrect byte-sequences and also provides column and line tracking.
c C` s� | j | � | _ t j | | j � t | � d f | _ d | _ d | _ d | _ | j d d k r� | j
| | � | _ n | j � d S( u Initialises the HTMLInputStream.
HTMLInputStream(source, [encoding]) -> Normalized stream from source
for use by html5lib.
source can be either a file-object, local filename or a string.
The optional encoding parameter must be a string that indicates
the encoding. If specified, that encoding will be used,
regardless of any BOM or later declaration (such as in a meta
element)
parseMeta - Look for a <meta> element containing encoding information
u certaini id u windows-1252i N( RH t rawStreamR8 R t codecNameRG t numBytesMetat numBytesChardett defaultEncodingR6 t detectEncodingRJ ( R R: R; R< R= ( ( sE /opt/alt/python27/lib/python2.7/site-packages/html5lib/inputstream.pyR } s c C` s6 t j | j d � | j d � | _ t j | � d S( Ni u replace( t codecst getreaderRG R{ RI R8 RJ ( R ( ( sE /opt/alt/python27/lib/python2.7/site-packages/html5lib/inputstream.pyRJ � s c C` sU t | d � r | } n t | � } y | j | j � � Wn t | � } n X| S( uv Produces a file object from source.
source can be either a file object, local filename or a string.
u read( R5 R
R R R ( R R: R ( ( sE /opt/alt/python27/lib/python2.7/site-packages/html5lib/inputstream.pyRH � s c
C` s} | j � } d } | d k r9 | r9 | j � } d } n | d k r#| r#d } y� y d d l m } Wn! t k
r� d d l m } n Xg } | � } xF | j s� | j j | j
� } | s� Pn | j | � | j | � q� W| j
� | j d } | j j d � Wq#t k
r q#Xn | d k rAd } | j } n i d d 6} | j � | k rs| | j � } n | | f S( Nu certainu tentativei ( t UniversalDetectoru encodingu windows-1252u
iso-8859-1( t detectBOMR6 t detectEncodingMetat charade.universaldetectorR� t ImportErrort chardet.universaldetectort doneR{ R# R~ R' t feedt closet resultR R t lower(
R R< R= R; t
confidenceR� t bufferst detectorR t encodingSub( ( sE /opt/alt/python27/lib/python2.7/site-packages/html5lib/inputstream.pyR� � s@
c C` s� t | � } | d k r! d } n | d k r1 d S| | j d k r] | j d d f | _ nF | j j d � | j � | d f | _ t d | j d | f � � d S( Nu utf-16u utf-16-beu utf-16-leu utf-8i u certainu Encoding changed from %s to %s( u utf-16u utf-16-beu utf-16-le( R| R6 RG R{ R RJ R
( R t newEncoding( ( sE /opt/alt/python27/lib/python2.7/site-packages/html5lib/inputstream.pyt changeEncoding� s
c C` s� i d t j 6d t j 6d t j 6d t j 6d t j 6} | j j d � } | j | d � } d } | s� | j | � } d } | s� | j | d � } d } q� n | j j | r� | p� d � | S(
u� Attempts to detect at BOM at the start of the stream. If
an encoding can be determined from the BOM return the name of the
encoding otherwise return Noneu utf-8u utf-16-leu utf-16-beu utf-32-leu utf-32-bei i i i (
R� t BOM_UTF8t BOM_UTF16_LEt BOM_UTF16_BEt BOM_UTF32_LEt BOM_UTF32_BER{ R# t getR ( R t bomDictt stringR; R ( ( sE /opt/alt/python27/lib/python2.7/site-packages/html5lib/inputstream.pyR� � s
c C` sV | j j | j � } t | � } | j j d � | j � } | d k rR d } n | S( u9 Report the encoding declared by the meta element
i u utf-16u utf-16-beu utf-16-leu utf-8( u utf-16u utf-16-beu utf-16-le( R{ R# R} t EncodingParserR t getEncoding( R R t parserR; ( ( sE /opt/alt/python27/lib/python2.7/site-packages/html5lib/inputstream.pyR� s N( R R R1 R6 R` R RJ RH R� R� R� R� ( ( ( sE /opt/alt/python27/lib/python2.7/site-packages/html5lib/inputstream.pyR9 u s ( - t
EncodingBytesc B` s� e Z d Z d � Z d � Z d � Z d � Z d � Z d � Z d � Z d � Z
e e
e � Z d � Z
e e
� Z e d
� Z d � Z d � Z d
� Z RS( u� String-like object with an associated position and various extra methods
If the position is ever greater than the string length then an exception is
raisedc C` s t j | | j � � S( N( R" t __new__R� ( R t value( ( sE /opt/alt/python27/lib/python2.7/site-packages/html5lib/inputstream.pyR� + s c C` s
d | _ d S( Ni����( RW ( R R� ( ( sE /opt/alt/python27/lib/python2.7/site-packages/html5lib/inputstream.pyR / s c C` s | S( N( ( R ( ( sE /opt/alt/python27/lib/python2.7/site-packages/html5lib/inputstream.pyt __iter__2 s c C` sS | j d } | _ | t | � k r/ t � n | d k rD t � n | | | d !S( Ni i ( RW R t
StopIterationR7 ( R t p( ( sE /opt/alt/python27/lib/python2.7/site-packages/html5lib/inputstream.pyt __next__5 s c C` s
| j � S( N( R� ( R ( ( sE /opt/alt/python27/lib/python2.7/site-packages/html5lib/inputstream.pyt next= s c C` sY | j } | t | � k r$ t � n | d k r9 t � n | d | _ } | | | d !S( Ni i ( RW R R� R7 ( R R� ( ( sE /opt/alt/python27/lib/python2.7/site-packages/html5lib/inputstream.pyt previousA s c C` s+ | j t | � k r t � n | | _ d S( N( RW R R� ( R R ( ( sE /opt/alt/python27/lib/python2.7/site-packages/html5lib/inputstream.pyt setPositionJ s c C` s< | j t | � k r t � n | j d k r4 | j Sd Sd S( Ni ( RW R R� R6 ( R ( ( sE /opt/alt/python27/lib/python2.7/site-packages/html5lib/inputstream.pyt getPositionO s
c C` s | | j | j d !S( Ni ( R ( R ( ( sE /opt/alt/python27/lib/python2.7/site-packages/html5lib/inputstream.pyt getCurrentByteY s c C` sc | j } xJ | t | � k rU | | | d !} | | k rH | | _ | S| d 7} q W| | _ d S( u Skip past a list of charactersi N( R R RW R6 ( R Rt R� Ru ( ( sE /opt/alt/python27/lib/python2.7/site-packages/html5lib/inputstream.pyRk ^ s c C` sc | j } xJ | t | � k rU | | | d !} | | k rH | | _ | S| d 7} q W| | _ d S( Ni ( R R RW R6 ( R Rt R� Ru ( ( sE /opt/alt/python27/lib/python2.7/site-packages/html5lib/inputstream.pyt skipUntilj s c C` sQ | j } | | | t | � !} | j | � } | rM | j t | � 7_ n | S( u� Look for a sequence of bytes at the start of a string. If the bytes
are found return True and advance the position to the byte after the
match. Otherwise return False and leave the position alone( R R t
startswith( R R" R� R( R, ( ( sE /opt/alt/python27/lib/python2.7/site-packages/html5lib/inputstream.pyt
matchBytesu s c C` sh | | j j | � } | d k r^ | j d k r= d | _ n | j | t | � d 7_ t St � d S( u� Look for the next sequence of bytes matching a given sequence. If
a match is found advance the position to the last byte of the matchi����i i N( R t findRW R R` R� ( R R" t newPosition( ( sE /opt/alt/python27/lib/python2.7/site-packages/html5lib/inputstream.pyt jumpTo� s ( R R R1 R� R R� R� R� R� R� R� t propertyR R� t currentBytet spaceCharactersBytesRk R� R� R� ( ( ( sE /opt/alt/python27/lib/python2.7/site-packages/html5lib/inputstream.pyR� ' s R� c B` s_ e Z d Z d � Z d � Z d � Z d � Z d � Z d � Z d � Z d � Z
d � Z RS(
u? Mini parser for detecting character encoding from meta elementsc C` s t | � | _ d | _ d S( u3 string - the data to work on for encoding detectionN( R� R( R6 R; ( R R( ( ( sE /opt/alt/python27/lib/python2.7/site-packages/html5lib/inputstream.pyR � s c C` s� d | j f d | j f d | j f d | j f d | j f d | j f f } xv | j D]k } t } xR | D]J \ } } | j j | � rk y | � } PWq� t k
r� t } Pq� Xqk qk W| sX PqX qX W| j
S( Ns <!--s <metas </s <!s <?R ( t
handleCommentt
handleMetat handlePossibleEndTagt handleOthert handlePossibleStartTagR( R` R� R� R4 R; ( R t methodDispatcht bytet keepParsingt keyt method( ( sE /opt/alt/python27/lib/python2.7/site-packages/html5lib/inputstream.pyR� � s&
c C` s | j j d � S( u Skip over commentss -->( R( R� ( R ( ( sE /opt/alt/python27/lib/python2.7/site-packages/html5lib/inputstream.pyR� � s c C` sK | j j t k r t St } d } x"t rF| j � } | d k rG t S| d d k r� | d d k } | rC| d k rC| | _ t Sq% | d d k r� | d } t | � } | d k rC| | _ t Sq% | d d k r% t t
| d � � } | j � } | d k rCt | � } | d k r@| r4| | _ t S| } q@qCq% q% Wd S( Ni s
http-equivi s content-typet charsett content( R( R� R� R` R4 R6 t getAttributeR; R| t ContentAttrParserR� t parse( R t hasPragmat pendingEncodingt attrt tentativeEncodingt codect
contentParser( ( sE /opt/alt/python27/lib/python2.7/site-packages/html5lib/inputstream.pyR� � s:
c C` s
| j t � S( N( t handlePossibleTagR4 ( R ( ( sE /opt/alt/python27/lib/python2.7/site-packages/html5lib/inputstream.pyR� � s c C` s t | j � | j t � S( N( R� R( R� R` ( R ( ( sE /opt/alt/python27/lib/python2.7/site-packages/html5lib/inputstream.pyR� � s
c C` s� | j } | j t k r9 | r5 | j � | j � n t S| j t � } | d k ra | j � n+ | j � } x | d k r� | j � } qp Wt S( NR (
R( R� t asciiLettersBytesR� R� R` R� t spacesAngleBracketsR� R6 ( R t endTagR( Ru R� ( ( sE /opt/alt/python27/lib/python2.7/site-packages/html5lib/inputstream.pyR� � s
c C` s | j j d � S( NR ( R( R� ( R ( ( sE /opt/alt/python27/lib/python2.7/site-packages/html5lib/inputstream.pyR� � s c C` s� | j } | j t t d g � B� } | d k r5 d Sg } g } x� t r� | d k r` | r` Pnz | t k r| | j � } Pn^ | d k r� d j | � d f S| t k r� | j | j � � n | d k r� d S| j | � t
| � } qD W| d k r| j � d j | � d f St
| � | j � } | d
k r�| } x� t r�t
| � } | | k r�t
| � d j | � d j | � f S| t k r�| j | j � � q>| j | � q>Wn^ | d k r�d j | � d f S| t k r�| j | j � � n | d k r d S| j | � x} t r�t
| � } | t k rSd j | � d j | � f S| t k ru| j | j � � q| d k r�d S| j | � qWd S( u_ Return a name,value pair for the next attribute in the stream,
if one is found, or Nonet /R t =R) t 't "N( R N( R� R ( R� R� (
R( Rk R� t frozensetR6 R` R* t asciiUppercaseBytesR' R� R� R� R� ( R R( Ru t attrNamet attrValuet quoteChar( ( sE /opt/alt/python27/lib/python2.7/site-packages/html5lib/inputstream.pyR� � sf
( R R R1 R R� R� R� R� R� R� R� R� ( ( ( sE /opt/alt/python27/lib/python2.7/site-packages/html5lib/inputstream.pyR� � s $ R� c B` s e Z d � Z d � Z RS( c C` s
| | _ d S( N( R( ( R R( ( ( sE /opt/alt/python27/lib/python2.7/site-packages/html5lib/inputstream.pyR D s c C` s: y!| j j d � | j j d 7_ | j j � | j j d k sH d S| j j d 7_ | j j � | j j d k r� | j j } | j j d 7_ | j j } | j j | � r� | j | | j j !Sd SnP | j j } y( | j j t � | j | | j j !SWn t k
r | j | SXWn t k
r5d SXd S( NR� i R� R� R� ( R� R� ( R( R� R Rk R� R6 R� R� R� ( R t quoteMarkt oldPosition( ( sE /opt/alt/python27/lib/python2.7/site-packages/html5lib/inputstream.pyR� H s.
( R R R R� ( ( ( sE /opt/alt/python27/lib/python2.7/site-packages/html5lib/inputstream.pyR� C s c C` sp t | t � r: y | j d � } Wq: t k
r6 d SXn | rh t j d | � j � } t j | d � Sd Sd S( u{ Return the python codec name corresponding to an encoding or None if the
string doesn't correspond to a valid encoding.u asciiu N(
R2 R" t decodet UnicodeDecodeErrorR6 t ascii_punctuation_reR^ R� R R� ( R; t
canonicalName( ( sE /opt/alt/python27/lib/python2.7/site-packages/html5lib/inputstream.pyR| j s
(1 t
__future__R R R t sixR t six.movesR R� RB t constantsR R R R R R
R) R t ioR R
R� R t objectR� R% t encodeR� R� R� R� RC Rc t setRj R� Ro R R6 R` R? R8 R9 R" R� R� R� R| ( ( ( sE /opt/alt/python27/lib/python2.7/site-packages/html5lib/inputstream.pyt <module> sL "
(((J��g�'