j ddlZddlZddlZddlmZmZmZmZmZm Z m Z m Z m Z ddl Z ddlmZmZddlmZmZddlmZmZmZmZddlmZddlmZmZddlmZdd l m!Z!dd l"m#Z#dd l$m%Z%dd l&m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z-dd l.m/Z/ddl0m1Z1m2Z2m3Z3ddl4m5Z5ddl6m7Z7ddl8m9Z9m:Z:ddl;mm?Z?m@Z@ddlAmAZAddlBmCZCmDZDmEZEmFZFmGZGddlHmIZImJZJmKZKmLZLmMZMmNZNmOZOmPZPmQZQgdZRe jSe jTe jUe jVe jWe jXe jYe jZe j[e j\g Z]e j^e j_j`jajbe jce j_j`jdjbiZede jfj/de3deege jhjifdeege egejffdeegefddf dZkde jfj/de3deege jhjifdeege egejffdeegefddf d Zlde3d!e2ddfd"Zmd#e jhjidenfd$Zode1deegefdenfd%Zpd&e/d'eddfd(Zqd)ede3d!e2ddfd*Zrd+e3deege egejffdeegefde egegffd,Zsde3d!e2ddfd-Ztde3deege jhjifdee jhjifd.Zude3deege jhjifde jfj/d/end'eeddf d0Zv d@de3deege jhjifd2e egdeegefd'ed3end/enddfd4Zwde3d5e3d!e2ddfd6Zxde3d!e2deege jhjifd7ee!ee e ffd8e e3ddf d9Zy dAde/d/end;e ee eeegefdfd'e eeegefdfd3ende/fd?ZzdS)BN) AnyCallableDictListOptionalSetTupleTypeUnion) CUSTOM_KEYNUMERIC_DEBUG_HANDLE_KEY) BackendConfigget_native_backend_config)get_fused_module_classesget_pattern_to_dtype_configsget_qat_module_classes-get_root_module_to_quantized_reference_module)_is_activation_post_process)qconfig_equals QConfigAny)QConfigMapping) QuantType)_remove_qconfig) DeQuantStub) _parent_name"activation_is_statically_quantizedget_qparam_dictget_swapped_custom_module_classis_per_channelto_underlying_dtypeweight_is_quantized) GraphModule)ArgumentGraphNode)type_before_parametrizations)quantized_decomposed_lib)convert_eq_obsupdate_obs_for_equalization)ConvertCustomConfigPrepareCustomConfig)_is_observed_module_is_observed_standalone_module)lower_to_fbgemm))_compare_prepare_convert_qconfig_mappings_generate_node_name_to_qconfig&_is_qconfig_supported_by_dtype_configs_update_qconfig_for_fusion_update_qconfig_for_qat) _get_module_is_custom_module_lstm_is_custom_module_mhaassert_and_get_unique_devicecollect_producer_nodescreate_getattr_from_valueget_custom_module_class_keys graph_module_from_producer_nodesnode_arg_is_weight)convertconvert_custom_moduleconvert_standalone_moduleconvert_weighted_modulemodelnodemodulesnode_name_to_scopenode_name_to_qconfigreturnc *|j}|JtjtsJt |\}}|j}t |dr||dStfdtj tj zD} | st|sd| 5j d|dddn #1swxYwYdS|j} d} t |dr|j} d} | t$vr| sd} d}|\}}t)|jrqt-|j}t0jjjj}t0jjjj}|j}|j}tA| }||||||d }nyt0jjj!j}t0jjj"j}tG|}t-|}|j}|j}tA| }|||||d }| 5j d}|g}|$D]h\}}|d vrJt|tFt,fs.tK||||z|z|}|&|S|&|i|'| |tQ|i}|g|d dz}|)|tQ|| ||**tTj+vrktXj+tTvrRtT*j+vri*j+tT<j+tTtX*j+tTtX<|ddddS#1swxYwYdS| r'd} t0jjj!j-}tA| }|t0j.t0j/fvs Jd |j}|j}ta|dt0j1}ta|dt1j2t0j3j4}||||d}tj|} | 5j d}j dg}!|$D]\}}"|!&|"|'d| tQ|!i}#|'dtlj7|#dfi}$|'dtlj7|#d fi}%|d}|d}|d} |$|%||| d }dddn #1swxYwY| 5j d}|g}|$D]6\}}|d vr|}|&|!|&|7|'| |tQ|i}|g|d dz}t0jjj"j-}|)|tQ|| ||**fd}&*tXj+vrj+tX*j+tX<|ddddS#1swxYwYdS| t0j8krt0jjj9j:}'| 5j d}|'d|'|t0j8fi}(|'d|'|(t0j#fi})|)|ddddS#1swxYwYdSdS)auReplace activation_post_process module call node with quantize and dequantize node working with decomposed Tensor Before: ... -> observer_0(x) -> ... After: ... -> torch.ops.quantized_decomposed.quantize_per_tensor(x, ...) -> torch.ops.quantized_decomposed.dequantize_per_tensor() -> ... or quantize_per_channel and dequantize_per_channel Nr>c38K|]}t|VdSN_has_none_qconfig.0nrFs QC:\Projects\Historical\venv\Lib\site-packages\torch/ao/quantization/fx/convert.py zM_replace_observer_with_quantize_dequantize_node_decomposed..yB  !122rF is_dynamicchi}d|jvr&|jdj}|tjkrd|i}|S)Nval out_dtype)metadtypetorchfloat32) dequantize_op input_nodedequantize_op_kwargs dq_out_dtypes rPadd_dequantize_op_kwargsz\_replace_observer_with_quantize_dequantize_node_decomposed..add_dequantize_op_kwargssB! JO # #%?517Lu},,(3\'B$##rS call_function)_scale_ _zero_point__axis_ _quant_min_ _quant_max__dtype_)rbrcrerfrgrbrcr'zVonly uint8 and int8 are supported in reference flow for dynamic quantization right nowqschemeeps)rerf_eps_rgrerfrgc|urn|SrJ)xdequantized_noderCs rPremap_fnzL_replace_observer_with_quantize_dequantize_node_decomposed..remap_fnNs+,99''!;rS);graph isinstancetargetstr_get_module_path_and_prefixhasattrr>alllistargsuserskeys_is_conversion_supportedinserting_beforereplace_all_uses_with erase_noderYrTSUPPORTED_QDTYPEScalculate_qparamsrriintch_axisrZopsquantized_decomposedquantize_per_channeldefaultdequantize_per_channel quant_min quant_maxr quantize_per_tensordequantize_per_tensorfloatitemsr:append create_nodetuplerar rXr tensoruint8int8getattrper_tensor_affinefinfor[rj_QSCHEME_TO_CHOOSE_QPARAMS_OPoperatorgetitemfloat16convert_element_typeno_fuse)+rBrCrDrErFrq module_pathprefixactivation_post_processskip_replacementrYrTr` node_type quantize_opscale zero_pointrr\rrdtype_qparamsr]quantize_op_inputskey value_or_node qparam_nodequantized_node dq_inputsrirjchoose_qparams_opchoose_qparams_op_inputsvaluechoose_qparams_node scale_nodezero_point_noderpdtype_convert_opconvert_fp16_nodeconvert_fp32_noderos+ ` ` @rP:_replace_observer_with_quantize_dequantize_node_decomposedr[sv $ KE     dk3 ' ''' '5  "6K&dk2& 22''t444di4 (9(9#:#::78OPP # #D ) ) # #  & &ty| 4 4 4   T " " " # # # # # # # # # # # # # # #  $ )EJ& 558,7 $$$ !!!:!$ *. 3EEGGz 19 : : 19::G)8MUK .EM 09I/9I(//F *!((! GG )8LTK!I:PXM%LLEZJ/9I/9I(//F *((! G # #D ) )/ #/ #1J", &-mmoo = ="]555"=5#,??6#<ukF&:S&@-##K'--k::::'--m<<<<"..;.@(A(A2N((+=abb+AAI$22i  (( CC     & &'7 8 8 8di'', *0EEE%5%:::8:$)*5NRiO*O, %j12JK   T " " "_/ #/ #/ #/ #/ #/ #/ #/ #/ #/ #/ #/ #/ #/ #/ #/ #/ #/ #` q#$ i4HO %U++%+uz2222 -322,5 +5 19e>UVV-uek%-6P6P6TUU %$   :'B  # #D ) )  1J(, ! ~ $%mmoo 7 7 U)//6666"'"3"3!2E:R4S4SUW## **!14G3KRJ$//!14G3KRO .I .II&E% /(( G)               : # #D ) )( #( #1J", &-mmoo = ="]555#0K&--k::::'--m<<<<"..;.@(A(A2N((+=abb+AAI"I:PWM$22i  (( CC     < < < < < <  & &'7 8 8 8'4944BF),C %&>?   T " " "Q( #( #( #( #( #( #( #( #( #( #( #( #( #( #( #( #( #( #R %-   99NV  # #D ) ) # #1J % 1 1!1J 3NPR!! !& 1 1!14Eu{3SUW!!   & &'8 9 9 9   T " " " # # # # # # # # # # # # # # # # # # s^.6D00D47D4FP::P>P>CXX X"D5]$$](+](1B`>>aac |Jt|jtsJ|j}t ||\}}||j}t fdt |jt |j zD} | st|sd| |5| |jd| |dddn #1swxYwYdS|j} d} t|dr|j} | t"jt"jt"jt"jt"jfvr| sd} d} |\}}t1|jr(t5|j}|||| d}t"j} n0t;|}t5|}||| d}t"j} | |5|jd}|g}|D]L\}}|d vr.tA||||z|z|}|!|7|!|M|"| | tG|i}|$d |f }| || |ddddS#1swxYwYdS| rd} t"j%} t"j&j'j(d v}| |d }| |5|jd}|g}|D]\}}|!||"| | tG|i}|$d |f }| || |ddddS#1swxYwYdS| t"j)krd} d} d| i}| |5|jd}|g}|D]\}}|!||"| | tG|i}|$d |f }| || |ddddS#1swxYwYdSdS)zReplace activation_post_process module call node with quantize and dequantize node Before: ... -> observer_0(x) -> ... After: ... -> torch.quantize_per_tensor(x, ...) -> x.dequantize() -> ... Nc38K|]}t|VdSrJrKrMs rPrQzB_replace_observer_with_quantize_dequantize_node..rRrSrFrTra)rbrcrdrg)rbrcrgrh dequantize)ry)fbgemmx86)rg_reduce_range_ call_methodtorg)*rrrsrtrqrurwrxryrzr{r|r}r~rrYrvrTrZquint8qint8qint32 float8_e5m2 float8_e4m3fnrrrirrrrrrr:rrrrquantize_per_tensor_dynamicbackends quantizedenginer)rBrCrDrErFrqrrrrrYrTrrrrrrr]rrrrrro reduce_rangers ` rP/_replace_observer_with_quantize_dequantize_noderjs     dk3 ' ''' ' KE5  "6K&dk2di4 (9(9#:#::78OPP # #D ) ) # #  & &ty| 4 4 4   T " " " # # # # # # # # # # # # # # #  $ )EJ& 558,7          $ *. 3EEGGz 19 : : 419::G *! G  4KK%LLEZJ"'PUVVG3K # #D ) ) # #1J", &-mmoo = ="]555#<ukF&:S&@-##K'--k::::'--m<<<<"..;.@(A(A2N %00^DU0VV   & &'7 8 8 8   T " " "- # # # # # # # # # # # # # # # # # #. '#$ 7 ~/6:KK #|DD  # #D ) ) # #1J", %mmoo 1 1 U"))%0000"..;.@(A(A2N %00^DU0VV   & &'7 8 8 8   T " " " # # # # # # # # # # # # # # # # # # %-  !  e$  # #D ) ) # #1J", %mmoo 1 1 U#))%0000"..;.@(A(A2N %00^DU0VV   & &'7 8 8 8   T " " " # # # # # # # # # # # # # # # # # # sK6DD  D CK44K8;K8B'O88O<?O<2B'S&&S*-S*rqc|jd}t|ts Jd|||||t ||dS)Nrz@Expecting the for call custom module node to be a Node, but got )ryrrr%r~r_insert_dequantize_node)rCrqcall_custom_module_nodes rP6_replace_observer_or_dequant_stub_with_dequantize_noders#il   ddcJaccdd  6777 T3U;;;;;rSrc~|j}d}t|dr|j}|tvr| p|p|tjkS)NFrT)rYrvrTrrZr)rrYrTs rPr|r| sX # )EJ& 558,7  # # 8Z "  " EM !rSc\t|to|j|vo||jduS)z\Check if a node has a qconfig of None, i.e. user requested to not quantize the node N)rrr%name)rCrFs rPrLrLs: 4 4 I- - 4  +t 3rSobservedbackend_configc|jjD]V}|jdkr|jD]@}|r> )(>x(H(H%(0)I3**&'&((( ) ) )rSargct|tr;|jdkr0|jdkr%|jd}|||dSt|t tfr|D]}t|||dSt|tr*| D]}t|||dStj dt|dS)zIf the arg is a dequantize Node, or a list/tuple/dict of dequantize Node, we'll recursively remove the dequantize Node rrrz6Unsupported node type in recursive remove dequantize: N)rrr%rrsryreplace_input_withrxr"_maybe_recursive_remove_dequantizedictvalueswarningswarntype)rrCrq quantize_node arg_elements rPrr<s#t =!8!8SZ<=W=W  ]33333 C$ ' '   I IK .{D% H H H H I I C   ::<< I IK .{D% H H H H I I  PT#YY P P     rSobs_nodec|jd}t|ts Jd||j|vr||jdund}|r\t |j}|r|dnd}d}|D]0}|jdkr#|jtj j j kr|}n1|r|}d} n|}d} |r|j|vr||j\} } nd} | | fS)aiGiven and observer node, get the `Scope` or the fully qualified name for the submodule containing the observed node, also return a prefix of "_input" when the observed node is an input of a F.linear op, and not the output of another quantized op. TODO: this logic is hacky, we should think about how to remove it or make it more general rz.Expecting observed node to be a Node, but got Nra_input) ryrrr%rrxrzrrsrZnn functionallinear) rrErF observed_nodeis_input_observer_onlyrzfirst_linear_use_or_first_use linear_noderOrr_s rPruruQsIM!$M t  HHG GGHH   !5 5 5 ]/0D88  X^$$49(Cat%   At&&18ux7J7Q+Q+Q   8,7 ))6% &  ) .2D D D+,I,NO QQ   rSc||5|d|f}t|jD]}||ur||| ddddS#1swxYwYdS)z-Inserts dequantize node for `node` in `graph`rN)inserting_afterrrrzr)rCrqdequantize_node user_nodes rPrrs   t $ $DD++L4'BBdj)) D DI//,,T?CCC DDDDDDDDDDDDDDDDDDDsA A--A14A1c|jD]:}|jdkr-|t|j}t |r|cS;dS)z\ If the node is observed, return the observer instance. Otherwise, return None. call_moduleN)rzr{rrtrsr)rCrDmaybe_obs_node maybe_obss rP_maybe_get_observer_for_nodersd*//++!!   - -N$9 : :;I*955 !    4rS is_referencecj|rtjjjj}ntjjjj}|t |j}|jdj }t|j }tt|D]y} | |vrs|| } | jdkr`| jdkrU| j d} || | t| jdkr|j| z|jdj} t| dkr,| ddks Jd t)||j|||} t+|j\}}t-|||| | |t |j<dS)aConverts a observed standalone module to a quantized standalone module by calling the fx convert api, currently using the same `is_reference` flag as parent, but we may changing this behavior in the future (e.g. separating quantization and lowering for standalone module as well) Args: - node: The call_module node of the observed standalone module - modules: named_module of original model - model: original model - is_reference: a flag from parent provided by user to decide if we want to produce a reference model or a fbgemm/qnnpack model - backend_config: backend configuration of the target backend of quantization _observed_graph_module_attrsrrrzCurrently only quantized)rN)rZao quantization quantize_fxconvert_to_reference_fx convert_fxrtrsrX&standalone_module_input_quantized_idxsrxryrangelenrrrzrqr'standalone_module_output_quantized_idxsrrsetattr)rCrDrBrr convert_fnobserved_standalone_modulesm_input_quantized_idxsryidxrrsm_output_quantized_idxsquantized_standalone_module parent_namers rPr@r@s*BX*6N X*6A /6c$+6F6F.G8=&,  ??DSYY00 ) ) )s)Cv&&3:+E+E # ''];;;sy>>Q&&K**3///9>& - #$$q(('*a///1K///( ek222#-*">###%T[11K GK $(CDDD ;GC  rSFobserved_node_names is_decomposedcT|t|j}|j}d} t|} t || rI|j} |}t|j\} } t|| | ||j |v} |t||s| sdSt|}| t|g}t||sdSt|}|sdSd}|}t |t jjjjr |}|d}d|i}t |t jjr|}|}||j||jt3|}t3|}|||dn1t |t jjt jjfr|jD]}}t=||rk|drVtA||}|} | j!t j"kr | |t3| ||<~n{| du}|r:|} tG|}|r| $|| }|r|r|s| |j|t3| tK|}| tM|d}|JdtM||'||}|||d<dSt|j\} } t|| | |dS)aConvert a weighted module to reference quantized module in the model If the QConfig of a QAT module is not set, the module will still be converted to a float module. Args: - node: The call_module node of the observed standalone module - modules: named_module of original model - observed_node_names: names for the set of observed fx node, we can skip this conversion if the node is not observed Nrr) weight_ih weight_hhweightz3No reference quantized module class configured for )(rtrsqconfigrrrweight_fake_quantto_floatrrrrLrgetrr2r!rZrr intrinsic _FusedModule RNNCellBaserrrrupdateLSTMGRU_flat_weights_namesrv startswithrrYrr8rrr& from_float) rCrDrrFrrroriginal_modulerweight_post_processqat_module_classesrr is_observedpattern_to_dtype_configs dtype_configsis_weight_quantized fused_module float_module wq_or_wq_dictweight_post_process_ihweight_post_process_hhweight_qparams_ihweight_qparams_hhwnris_ptqdeviceis_qat)root_module_to_quantized_reference_moduleref_qmodule_cls ref_qmodules rPrArAs#&c$+../O)1G/??/#566=.?)2244(55 T $dO<<<)22K  T#7 8 8    !+<=== y  1 $ $   Y ' ' ' ' ' %$)I)I % $rScustom_module_class_mapping(statically_quantized_custom_module_nodesc2|t|j}t||}|j}t |r||t ||rt|jdkr>t|jdtrt|jddksJ|j\}\} } t|tsJt| tsJt| tsJt|||t|| |t|| |nt||rt|jdksJ|j\} } } t| tsJt| tsJt| tsJt|| |t|| |t|| |nP|jd}t|tsJt|||t||}|J||_t|||}||}t#|j\}}t%||||dS)aConverts an observed custom module to a quantized custom module based on `custom_module_class_mapping` For static quantization, we'll also remove the previous `dequantize` node and attach the observer node for output to the module, the observer for the node will be converted to a dequantize node instead of quantize-dequantize pairs later in the graph. In the end we would have a quantized custom module that has the same interface as a default quantized module in nn.quantized namespace, i.e. quantized input and quantized output. Args: - node: The call_module node of the observed standalone module - graph: The graph containing the node - modules: named_module of original model - custom_module_class_mapping: mapping from observed custom module class to quantized custom module class, used to swap custom modules - statically_quantized_custom_module_nodes: we'll add the custom module node if we find it is statically quantized, this will be used later when converting observers to quant/dequant node pairs, if the observed node is a statically quantized custom module nodes, we'll convert the observer to a dequantize node, this is to keep the interface the same as the default quantized module. TODO: maybe we want to redesign this part to align with reference model design as well, but there has been some discussions around the interface, so we can do it later. r'rN)rtrsrrraddr6rryrrrr%r9r7rr from_observedrr)rCrqrDr:r;observed_custom_modulerrinputshidden0hidden1queryrrrrquantized_custom_module_classquantized_custom_modulerrs rPr?r?s>%S%5%56,T7;;I$,G)'22(U044T::: !$ 0 0& UDI!##ty|U33$ ! %%***+,09 (V'gwfd++ + ++gt,, , ,,gt,, , ,, 8vu M M M 8w N N N 8w N N N N "4 1 1 Uty>>Q&&&& $ E3eT** * **c4(( ( ((eT** * ** 8ue L L L 8sE J J J 8ue L L L L)A,Cc4(( ( (( 8sE J J J&B4&Q&Q #*666=T " :%D ;W%%!<II%T[11K GK $(?@@@@@rSTconvert_custom_configis_standalone_module_remove_qconfig_flagqconfig_mappingc  |t}t|tr0tjdt dtj|}t|tr4tjdt d|rtj|nd}tj |}|t|tsJt|tr0tjdt dtj|}|t}t|s Jd|j d}|j} |j} |j} |j} t|d } |r|j}tj | }|jrt+||t-||t/||t1|||j|| }| D]N\}}||vsJd |d ||/t7|||sJd |d|d||O|} t9|j}|j}|j!t?|| }tA|| |tC||d|jj"D}d}| j#}| j$}tK|}tM|'}tQ|}tS|}tU}tW|jj"D]x}|j,dkr!|} |dz }| |vrt[||j/|j,dkrt]|dkrN|}!|j/d}"t|"tVtLfr"|D]}#ta|"|#|!|jt|"tbtfrd|vrta|"|!|jtjdte|"|j,dkrutg|| }$|$Jti|$rT|j/d}%|%|vrtk||jN|rtm||| | | eto||| | | zt|$tprtk||jts|$rtu|| |||tw|$tU|<|<|vrAtw|$|vrtw|$d|vr8t{|| | | |||Otw|$|vrt}||j| ||z|j?t||j}|st|| | }|rt||C|j Ddd|S)a We will convert an observed model (a module with observer calls) to a reference quantized model, the rule is simple: 1. for each observer module call in the graph, we'll convert it to calls to quantize and dequantize functions based on the observer instance 2. for weighted operations like linear/conv, we need to convert them to reference quantized module, this requires us to know whether the dtype configured for the weight is supported in the backend, this is done in prepare step and the result is stored in observed_node_names, we can decide whether we need to swap the module based on this set Args: * `is_standalone_module`: when this flag is True, it means we are quantizing a submodule that is not inlined in parent module, and will be quantized separately as one unit. * `is_decomposed`: a boolean flag to indicate whether we want to use the quantize operator for decomposed quantized tensor (torch.ops.quantized_decomposed.quantize_per_tensor) or default/standalone quantized tensor (torch.quantize_per_tensor) Returns: a quantized standalone module, whether input/output is quantized is specified by prepare_custom_config, with input_quantized_idxs, output_quantized_idxs, please see docs for :func:`~torch.ao.quantization.prepare_fx` for details NzPassing a convert_custom_config_dict to convert is deprecated and will not be supported in a future version. Please pass in a ConvertCustomConfig instead.r=) stacklevelzPassing a QConfig dictionary to convert is deprecated and will not be supported in a future version. Please pass in a QConfigMapping instead.zPassing a backend_config_dict to prepare is deprecated and will not be supported in a future version. Please pass in a BackendConfig instead.z-incoming model must be produced by prepare_fxrF)remove_duplicatez Expected key z in convert node_name_to_qconfigz Expected k zD to have the same value in prepare and convert QConfigMappings, but z was updated to c2g|]}|jdk |jS) placeholder)rr)rNrCs rP zconvert..js-47m3K3K 3K3K3KrSrrPr'outputz1Unsupported node type for output_quantized_idxs: r)Er+rrrrr FutureWarning from_dictrcopydeepcopyrrr-rXrEprepare_custom_configrrF named_modulesrKr3r4r3r0r1rqrrr;observed_to_quantized_mapping!equalization_node_name_to_qconfigr*r)rrinput_quantized_indexesoutput_quantized_indexesrrr{rrsetrxrrrryrr%rr5rrrrrr.r@r&unionrAr?eliminate_dead_coder"r/rdelete_all_unused_submodulespop)&rBrrHrIrJrKrrobserved_graph_module_attrsrErWrrFrDprepare_qconfig_mapping modules_copyconvert_node_name_to_qconfigkvcustom_module_classesr:weight_eq_obs_dict graph_inputsplaceholder_node_seen_cntinput_quantized_idxsoutput_quantized_idxsr4root_module_classesr$fused_module_classesr;rCcur_placeholder_node_idx return_noderRr modrs& rPr>r>sJ$ 3 5 5'..U  Q      !4 =>S T T/4((    L      :I RN $_ 5 5 5d mO44O  "j.&Q&Q " " Q.$''A  K      '0@@244 u % %VV'VVV %"'*-K"L $6 $9%@$S2M2b5&&&>>??G<2M2]}W-- & - E #O^ D D D"5/:::12I?[[['E <o?Q( ( $ )..00  DAq1111BqBBB211+A.:%a)Ea)HIIP!PPPP.J1.MPPI <8;#8"U"DP9HHug'9:::%000#k/L !"&;&S'<'U 6nEE. I N N P PQQ/??3NCC:=%%,U[&''aa 7m # #'@ $ % * %'+??? (ek::: W ())Q..KYq\F&4-00 0C6s [%+FT4L11 ---6v{EKXXX VV VVW % %dG,,C???*3//;  $ !  $LLLJek%R! #.0 H! #.0 C--$ F%+044 )'5, .c22c:M6N6N6T6T"77e())** 1559MMM4SV<rmrSrPrs OOOOOOOOOOOOOOOOOOOOOO FFFFFFFF GFFFFFDDDDDDDD@@@@@@666666::::::333333! 0000000000CCCCCC211111BBBBBBBBCCCCCCCCMMMMMMMM,,,,,,                          L K L K J L K K    UY;JQ  > W ^! I# 8 I# I##ux& 'I#S%T "223 I# sJ/ I#  I#I#I#I#^J# 8 J# J##ux& 'J#S%T "223 J# sJ/ J#  J#J#J#J#j <  < <  < < < < eho $       *.sJ*?      )K))SW)))), C t E d    *66S%T "2236sJ/6 38_ 6666rD$DuDDDDD   c58?23  eho    =< =< #ux& '=< 8 =< =< ]+ =<  =<=<=<=4S>4?@ Q-c3h=>QQQQQQQQrS