mò EäFc@sdZdZdkZdkZdkZdkZdkZdkZydk l Z Wn:e j o.ydk l Z Wqe j oqXnXd„Z dfd„ƒYZdfd„ƒYZd eiiifd „ƒYZd eifd „ƒYZd efd„ƒYZdS(st Each XmlEntry object represents a page, as read from an XML source The MediaWikiXmlHandler can be used for the XML given by Special:Export as well as for XML dumps. The XmlDump class reads a pages_current XML dump (like the ones offered on http://download.wikimedia.org/wikipedia/de/) and offers a generator over XmlEntry objects which can be used by other bots. For fastest processing, XmlDump uses the cElementTree library if available (this comes included with Python 2.5, and can be downloaded from http://www.effbot.org/ for earlier versions). If not found, it falls back to the older method using regular expressions. s8$Id: xmlreader.py 3726 2007-06-20 13:28:05Z wikipedian $N(s iterparsecCsš|pdSnd}d}tid|ƒ}|o|idƒ}ntid|ƒ}|o|idƒ}n|djod}d}n||fS(s¾ Parses the characters within a restrictions tag and returns strings representing user groups allowed to edit and to move a page, where None means there are no restrictions. s edit=([^:]*)is move=([^:]*)tsysopN(NN( t restrictionstNoneteditRestrictiontmoveRestrictiontretsearcht editLockMatchtgroupt moveLockMatch(RR RRR((t=/home/cat/public_html/arm/mrobe/wiki/pywikipedia/xmlreader.pytparseRestrictions!s  tXmlEntrycBstZdZd„ZRS(s Represents a page. c Cs[||_||_||_|iƒ|_||_||_||_||_ | |_ dS(N( ttitletselftidttexttusernametstriptipeditt timestampRRt revisionid( RR RRRRRRRR((R t__init__:s       (t__name__t __module__t__doc__R(((R R 6s tXmlHeaderEntrycBstZdZd„ZRS(s# Represents a header entry cCs1d|_d|_d|_d|_h|_dS(Nu(Rtsitenametbaset generatortcaset namespaces(R((R RJs     (RRRR(((R RFs tMediaWikiXmlHandlercBs>tZd„Zd„Zd„Zd„Zd„Zd„ZRS(NcCsGtiiii|ƒt|_t|_d|_ d|_ d|_ dS(Nu( txmltsaxthandlertContentHandlerRRtFalset inRevisionTagtinContributorTagRtheadercallbackRR(R((R RRs     cCs ||_dS(N(tcallbackR(RR)((R t setCallback]scCs ||_dS(N(R(R(RR(((R tsetHeaderCallback`scCs;d|_|djod|_d|_d|_n|djo t|_né|djo t|_ nÏ|djod|_d|_ n¬|djoV|i od|_d|_ q7|iod|_d|_ q7d|_d|_ nI|d jod |_d|_t|_n|d jod |_d|_t|_nñ|d jod |_d|_nÎ|d jod |_d|_n«|d jod |_d|_nˆ|io}|djotƒ|_q7|ddddgjo ||_q7|djo)d|_d|_t|dƒ|_q7ndS(Ntpagetrevisiont contributorRuRtuseridRRtipRR RtsiteinfoRRRRt namespacetkey(RRt destinationtnameRRRtTrueR&R'RR/RRRR%RR RR(RtheaderR2tinttattrst namespaceid(RR5R9((R t startElementcs^                                            c Cs¼|djo t|_nŸ|djot|iƒ\|_|_ns|djo|i}x'|o|ddjo|d }q_Wdi |i dƒƒ}|i dd !|i d d !|i d d !|i dd!|i dd!|i dd!}|i i ƒ|_ t|i |i||i|i||i|i|iƒ }t|_|i|ƒn`|ioU|djo|i|ii|i((R t endElement•s*     V9    cCsÂ|idjo|i|7_nœ|idjo|i|7_ny|idjo|i|7_nV|idjo|i|7_n3|idjo|i|7_n|idjo|i|7_ní|idjo|i |7_ nÊ|i o¿|idjo|i i |7_ q¾|id jo|i i |7_ q¾|id jo|i i|7_q¾|id jo|i i|7_q¾|id jo|i|7_q¾ndS( NRRRRR RRRRRRR2(RR4RtdataRRRR RRR(R7RRRRR2(RR@((R t characters¹s2 (RRRR*R+R;R?RA(((R R Qs    2 $tXmlParserThreadcBs tZdZd„Zd„ZRS(sß This XML parser will run as a single thread. This allows the XmlDump generator to yield pages before the parser has finished reading the entire dump. There surely are more elegant ways to do this. cCs&tii|ƒ||_||_dS(N(t threadingtThreadRRtfilenameR#(RRER#((R RÞs cCstii|i|iƒdS(N(R!R"tparseRRER#(R((R trunãs(RRRRRG(((R RBÖs  tXmlDumpcBs2tZdZd„Zd„Zd„Zd„ZRS(sp Represents an XML dump file. Reads the local file at initialization, parses it, and offers access to the resulting XmlEntries via a generator. NOTE: This used to be done by a SAX parser, but the solution with regular expressions is about 10 to 20 times faster. The cElementTree version is again much, much faster than the regex solution. cCs ||_dS(N(RER(RRE((R RðscCs>dGHdtƒjotidƒ|iƒSn |iƒSdS(s3Return a generator that will yield XmlEntry objectssReading XML dump...t iterparseurNOTE: cElementTree not found. Using slower fallback solution. Consider installing the python-celementtree package.N(tglobalst wikipediatoutputRt regex_parset new_parse(R((R RFós  ccsÜt|iddƒ}d}xº|D]²\}}|djo"|ddjo|d}q"n|djo|djo |}q"n|djoC|i d|jo/|i d |ƒ}|i d |ƒ}|i d |ƒ}|id |ƒ} | i d |ƒ} | i d |ƒ} | id|ƒ}|i d|ƒ}|p|i d|ƒ} | i d|ƒ}t|ƒ\}} td|d|d|pdd| dt|ƒd| d|d| d| ƒ V|iƒq"q"WdS(s/Generator using cElementTree iterparse functionteventststarttendsstart-nsitis{%s}pages {%s}titles{%s}ids{%s}restrictionss {%s}revisions {%s}timestamps{%s}contributors{%s}ips {%s}usernames{%s}textR RRuRRRRRRN(sstartsendsstart-ns(RIRREtcontextRtrootteventtelemturittagtfindtextR tpageidRtfindR-RRR.tipeditorRRR RRR tbooltclear(RRRWR\R.RURZR RRR-RRRRRVRSRT((R RNüsB  !    c cs,tiddddddddd d d d d dddtiƒ} ti|iddti ƒi ƒddƒ}t }d}x›|p“|iƒ} || 7}| djo t}q| idƒoR|iddƒ}|iddƒ}|iddƒ}|iddƒ}| i|ƒ}|pd GH|GHd}q#d}|id!ƒpd}|id"ƒ}t|ƒ\} } |id#ƒo|id#ƒ}t }n|id$ƒ}t}td%|id%ƒd&|id'ƒd!|d#|d(|d)|id)ƒd*| d+| d,|id,ƒƒ VqqWd-S(.s# Generator which reads some lines from the XML dump file, and parses them to create XmlEntry objects. Stops when the end of file is reached. NOTE: This is very slow. It's only a fallback solution for users who haven't installed cElementTree. s \s*s (?P<title>.+?)\s*s(?P\d+?)\s*s8((?P.+?))?\s*s \s*s (?P\d+?)\s*s,(?P.+?)\s*s\s*s*((?P.+?)\s*s2(?P\d+?)|(?P.+?))\s*s\s*s(?P())?\s*s+(?:(?P.+?)\s*)?sR((?P.*?)|)\s*s\s*strtencodingterrorstreplaceuRRu s>t>s<t