jddlZddlmZddlmZddlmZmZmZm Z m Z m Z ddl Z ddlZddlmcmZddlmZmZddlmZmZmZddlmZddlmZmZdd lmZmZdd l m!Z!dd l"m#Z#m$Z$dd l%m&Z&gd Z'de e(e(fde(de(de(de e(e(ff dZ)de e(e(fde(dee e(e(ffdZ*de(de(dejfdZ+Gddej,Z-Gddej,Z.Gddej,Z/Gdd ej,Z0Gd!d"ej,Z1Gd#d$ej,Z2Gd%d&ej,Z3Gd'd(ej,Z4Gd)d*ej,Z5 d=d,e(d-ee(d.ee(d/e6d0e(d1e(d2e ed3e7d4ede5fd5Z8Gd6d7eZ9eed8e9j:f9dd:d;d2e e9d3e7d4ede5fd<Z;dS)>N) OrderedDict)partial)AnyCallableListOptionalSequenceTuple)nnTensor)register_modelWeights WeightsEnum)_IMAGENET_CATEGORIES)_ovewrite_named_paramhandle_legacy_interface)Conv2dNormActivationSqueezeExcitation)StochasticDepth)ImageClassificationInterpolationMode)_log_api_usage_once)MaxVitMaxVit_T_Weightsmaxvit_t input_size kernel_sizestridepaddingreturnc^|d|z d|zz|zdz|d|z d|zz|zdzfS)Nr)rrrrs JC:\Projects\Historical\venv\Lib\site-packages\torchvision/models/maxvit.py_get_conv_output_shaper&sJ A $q7{ 2v=A A $q7{ 2v=A n_blockscg}t|ddd}t|D])}t|ddd}||*|S)zQUtil function to check that the input size is correct for a MaxVit configuration.r"r#)r&rangeappend)rr(shapesblock_input_shape_s r%_make_block_input_shapesr0 s` F.z1aCC 8__))23DaANN '(((( Mr'heightwidthc$tjtjtj|tj|gd}tj|d}|dddddf|dddddfz }|ddd}|dddddfxx|dz z cc<|dddddfxx|dz z cc<|dddddfxxd|zdz zcc<|dS)Nij)indexingr#r"r)torchstackmeshgridarangeflattenpermute contiguoussum)r1r2coords coords_flatrelative_coordss r%_get_relative_position_indexrB*s@ [f)=)=u|E?R?R(S^bccc d dF-**K!!!!QQQ*- AAAtQQQJ0GGO%--aA66AACCOAAAqqq!G *AAAqqq!G )AAAqqq!GE A -   r " ""r'ceZdZdZ ddedededededed ejfd ed ejfd ed d ffd Z de d e fdZ xZ S)MBConva=MBConv: Mobile Inverted Residual Bottleneck. Args: in_channels (int): Number of input channels. out_channels (int): Number of output channels. expansion_ratio (float): Expansion ratio in the bottleneck. squeeze_ratio (float): Squeeze ratio in the SE Layer. stride (int): Stride of the depthwise convolution. activation_layer (Callable[..., nn.Module]): Activation function. norm_layer (Callable[..., nn.Module]): Normalization function. p_stochastic_dropout (float): Probability of stochastic depth. in_channels out_channelsexpansion_ratio squeeze_ratioractivation_layer. norm_layerp_stochastic_dropoutr Nc (t||dkp||k} | rOtj||dddg} |dkrtjd|dg| z} tj| |_ntj|_t||z} t||z} |rt|d|_ ntj|_ t} ||| d <t|| ddd ||d | d <t| | d|d||| d | d<t| | tj| d<tj| |dd| d<tj| |_dS)Nr#T)rrbiasr"r*rrrrowmodepre_normr)rrrrJrKinplaceconv_a)rrrrJrKgroupsrTconv_b) activationsqueeze_excitation)rFrGrrNconv_c)super__init__r Conv2d AvgPool2d SequentialprojIdentityintrstochastic_depthrrrSiLUlayers)selfrFrGrHrIrrJrKrL should_projr` mid_channels sqz_channels_layers __class__s r%r\zMBConv.__init__Cs  k@[L%@  &Ik2;-S")^,;-$;- ;-;-;-;-;-;-z  F        r'rDcZeZdZdZdedededdffd ZdejfdZd edefd Z xZ S) $RelativePositionalMultiHeadAttentionzRelative Positional Multi-Head Attention. Args: feat_dim (int): Number of input features. head_dim (int): Number of features per head. max_seq_len (int): Maximum sequence length. feat_dimhead_dim max_seq_lenr Nc,t||zdkrtd|d|||z|_||_t t j||_||_ tj ||j|jzdz|_ |dz|_ tj |j|jz||_tjt#jd|jzdz d|jzdz z|jft"j|_|d t-|j|jt"j j|jd dS) Nrz feat_dim: z must be divisible by head_dim: r*gr"r#)dtyperelative_position_index{Gz?std)r[r\ ValueErrorn_headsr|rbmathsqrtsizer}r Linearto_qkv scale_factormerge parameter Parameterr7emptyfloat32relative_position_bias_tableregister_bufferrBinit trunc_normal_)rfr{r|r}rks r%r\z-RelativePositionalMultiHeadAttention.__init__sl  h ! # #^(^^T\^^__ _8+    +..// &i$,*F*JKK $dNYt}t|;XFF ,.L,B,B K!di-!+DI 0ABDLQY^Yf g g g- - ) 68TUYU^`d`i8j8jkkk  ##D$E4#PPPPPr'c |jd}|j||j|jd}|ddd}|dS)Nr6r"rr#)rviewrr}r<r= unsqueeze)rf bias_index relative_biass r%get_relative_positional_biaszARelativePositionalMultiHeadAttention.get_relative_positional_biassv166r:: 9*EJJ4K[]a]moqrr %--aA66AACC &&q)))r'rlc|j\}}}}|j|j}}||}t j|dd\} } } | |||||ddddd} | |||||ddddd} | |||||ddddd} | |jz} t j d| | } | } tj | | zd} t j d | | }|ddddd||||}| |}|S) z Args: x (Tensor): Input tensor with expected layout of [B, G, P, D]. Returns: Tensor: Output tensor with expected layout of [B, G, P, D]. r*r6)dimrr#r"z!B G H I D, B G H J D -> B G H I Jz!B G H I J, B G H J D -> B G H I D)shaperr|rr7chunkreshaper<reinsumrFsoftmaxr)rfrlBGPDHDHqkvqkvdot_prodpos_biasouts r%rpz,RelativePositionalMultiHeadAttention.forwardsxW 1a dm2kk!nn+c1"---1a IIaAq" % % - -aAq! < < IIaAq" % % - -aAq! < < IIaAq" % % - -aAq! < < ! !< CQJJ44669X0b999l>!LLkk!Q1a((00Aq!<<jjoo r') rqrrrsrtrbr\r7r rrprwrxs@r%rzrzsQQQ Q  QQQQQQ8*el**** Fr'rzcTeZdZdZdededdffd ZdejdejfdZxZ S) SwapAxeszPermute the axes of a tensor.abr Ncdt||_||_dSN)r[r\rr)rfrrrks r%r\zSwapAxes.__init__s+ r'rlcFtj||j|j}|Sr)r7swapaxesrrrns r%rpzSwapAxes.forwardsnQ// r') rqrrrsrtrbr\r7r rprwrxs@r%rrs~''##$ %,r'rc8eZdZdZdfd ZdededefdZxZS) WindowPartitionzB Partition the input tensor into non-overlapping windows. r NcHtdSrr[r\rfrks r%r\zWindowPartition.__init__ r'rlpc|j\}}}}|}|||||z|||z|}|dddddd}||||z||zz||z|}|S)z Args: x (Tensor): Input tensor with expected layout of [B, C, H, W]. p (int): Number of partitions. Returns: Tensor: Output tensor with expected layout of [B, H/P, W/P, P*P, C]. rr"rr*r#rrr<)rfrlrrCrWrs r%rpzWindowPartition.forwardsW 1a  IIaAFAqAvq 1 1 IIaAq!Q ' ' IIa!q&Q!V,a!eQ 7 7r'r N rqrrrsrtr\r rbrprwrxs@r%rrspCFr'rc @eZdZdZd fd Zdededededef d ZxZS) WindowDepartitionzo Departition the input tensor of non-overlapping windows into a feature volume of layout [B, C, H, W]. r NcHtdSrrrs r%r\zWindowDepartition.__init__rr'rlr h_partitions w_partitionsc|j\}}}}|} ||} } ||| | | | |}|dddddd}|||| | z| | z}|S)ar Args: x (Tensor): Input tensor with expected layout of [B, (H/P * W/P), P*P, C]. p (int): Number of partitions. h_partitions (int): Number of vertical partitions. w_partitions (int): Number of horizontal partitions. Returns: Tensor: Output tensor with expected layout of [B, C, H, W]. rrr#r*r"rr) rfrlrrrrrPPrrHPWPs r%rpzWindowDepartition.forwards{g 1b! |B IIaRAq ) ) IIaAq!Q ' ' IIaBFBF + +r'rrrxs@r%rrsCs#RXr'rceZdZdZdededededeeefdeded ej fd ed ej fd e d e d e ddffd Z de de fdZ xZS)PartitionAttentionLayera Layer for partitioning the input tensor into non-overlapping windows and applying attention to each window. Args: in_channels (int): Number of input channels. head_dim (int): Dimension of each attention head. partition_size (int): Size of the partitions. partition_type (str): Type of partitioning to use. Can be either "grid" or "window". grid_size (Tuple[int, int]): Size of the grid to partition the input tensor into. mlp_ratio (int): Ratio of the feature size expansion in the MLP layer. activation_layer (Callable[..., nn.Module]): Activation function to use. norm_layer (Callable[..., nn.Module]): Normalization function to use. attention_dropout (float): Dropout probability for the attention layer. mlp_dropout (float): Dropout probability for the MLP layer. p_stochastic_dropout (float): Probability of dropping out a partition. rFr|partition_sizepartition_type grid_size mlp_ratiorJ.rKattention_dropout mlp_dropoutrLr Nc t||z|_||_|d|z|_||_||_|dvrtd|dkr||jc|_|_ n|j|c|_|_ t|_ t|_ |dkrtddntj|_|dkrtddntj|_tj||t)|||dztj| |_tjtj|tj|||z|tj||z|tj| |_t5| d |_dS) Nr)gridwindowz0partition_type must be either 'grid' or 'window'rrr"rPrQ)r[r\rr| n_partitionsrrrrgr partition_oprdepartition_oprr rapartition_swapdepartition_swapr_rzDropout attn_layer LayerNormr mlp_layerrstochastic_dropout) rfrFr|rrrrrJrKrrrLrks r%r\z PartitionAttentionLayer.__init__,s" "h.   %aLN:," !3 3 3OPP P X % %+T->NDFDFF!.NDFDF+--/112@F2J2Jhr2...PRP[P]P]4Bf4L4LR 0 0 0RTR]R_R_- J{ # # 1hXYHY Z Z J( ) )    L % % Ik;#: ; ;     IkI-{ ; ; J{ # #   #22FU"S"S"Sr'rlc|jd|jz|jd|jz}}tj|jd|jzdko|jd|jzdkd|j|j|||j}||}||||z}||| |z}| |}| ||j||}|S)z Args: x (Tensor): Input tensor with expected layout of [B, C, H, W]. Returns: Tensor: Output tensor with expected layout of [B, C, H, W]. rr#z[Grid size must be divisible by partition size. Got grid size of {} and partition size of {}) rrr7_assertformatrrrrrrr)rfrlghgws r%rpzPartitionAttentionLayer.forwardfs."df,dnQ.?46.IB N1  &! + Oq0ADF0Ja0O i p p        a ( (    " " ''(:(:;; ; ''q(9(9:: :  ! !! $ $   462r 2 2r')rqrrrsrtrbstrr rr rvrur\r rprwrxs@r%rrs"8T8T8T  8T  8Tc?8T8T#3 >28TS")^,8T!8T8T$8T !8T8T8T8T8T8TtFr'rceZdZdZdededededededejfd edejfd ed ed ed ededede eefddffd Z de de fdZ xZ S) MaxVitLayera MaxVit layer consisting of a MBConv layer followed by a PartitionAttentionLayer with `window` and a PartitionAttentionLayer with `grid`. Args: in_channels (int): Number of input channels. out_channels (int): Number of output channels. expansion_ratio (float): Expansion ratio in the bottleneck. squeeze_ratio (float): Squeeze ratio in the SE Layer. stride (int): Stride of the depthwise convolution. activation_layer (Callable[..., nn.Module]): Activation function. norm_layer (Callable[..., nn.Module]): Normalization function. head_dim (int): Dimension of the attention heads. mlp_ratio (int): Ratio of the MLP layer. mlp_dropout (float): Dropout probability for the MLP layer. attention_dropout (float): Dropout probability for the attention layer. p_stochastic_dropout (float): Probability of stochastic depth. partition_size (int): Size of the partitions. grid_size (Tuple[int, int]): Size of the input feature grid. rFrGrIrHrrK.rJr|rrrrLrrr Nc ftt}t|||||||| |d<t ||| d|| |t j| | |  |d<t ||| d|| |t j| | |  |d<t j||_dS)N)rFrGrHrIrrJrKrLMBconvr) rFr|rrrrrJrKrrrLwindow_attentionrgrid_attention) r[r\rrDrr rr_re)rfrFrGrIrHrrKrJr|rrrrLrrrerks r%r\zMaxVitLayer.__init__s* )mm"#%+'-!!5    x&=$)#-|/#!5 & & & !"$;$)!-|/#!5 $ $ $  mF++ r'rlc0||}|Sz Args: x (Tensor): Input tensor of shape (B, C, H, W). Returns: Tensor: Output tensor of shape (B, C, H, W). re)rfrls r%rpzMaxVitLayer.forwards KKNNr')rqrrrsrtrbrurr rvr r\r rprwrxs@r%rrs(?,?, ?,  ?,  ?,?,S")^,?,#3 >2?,?,?,?,!?, $!?,$%?,&c?'?,( )?,?,?,?,?,?,BFr'rceZdZdZdedededededejfdedejfd ed ed ed ed ede eefdede eddffd Z de de fdZ xZS) MaxVitBlocka( A MaxVit block consisting of `n_layers` MaxVit layers. Args: in_channels (int): Number of input channels. out_channels (int): Number of output channels. expansion_ratio (float): Expansion ratio in the bottleneck. squeeze_ratio (float): Squeeze ratio in the SE Layer. activation_layer (Callable[..., nn.Module]): Activation function. norm_layer (Callable[..., nn.Module]): Normalization function. head_dim (int): Dimension of the attention heads. mlp_ratio (int): Ratio of the MLP layer. mlp_dropout (float): Dropout probability for the MLP layer. attention_dropout (float): Dropout probability for the attention layer. p_stochastic_dropout (float): Probability of stochastic depth. partition_size (int): Size of the partitions. input_grid_size (Tuple[int, int]): Size of the input feature grid. n_layers (int): Number of layers in the block. p_stochastic (List[float]): List of probabilities for stochastic depth for each layer. rFrGrIrHrK.rJr|rrrrinput_grid_sizen_layers p_stochasticr Nctt|| kstd| d|dt j|_t| ddd|_t|D]H\}}|dkrdnd}|xjt|dkr|n|||||||||| | | |j| gz c_IdS) Nz'p_stochastic must have length n_layers=z, got p_stochastic=.r*r"r#rOr)rFrGrIrHrrKrJr|rrrrrrL) r[r\lenrr ModuleListrer&r enumerater)rfrFrGrIrHrKrJr|rrrrrrridxrrrks r%r\zMaxVitBlock.__init__s, <  H,,sxssdpssstt tmoo /QWXbcddd --  FC((QQF KK/2axx \!-"/$3!)%5%' +&7#1"n)* KKK  r'rlc0|jD] }||}|Srr)rfrllayers r%rpzMaxVitBlock.forward,s)[  EaAAr')rqrrrsrtrbrurr rvr rr\r rprwrxs@r%rrs"*11 1  1  1S")^,1#3 >21111!1 !1"sCx#1&'1(5k)1* +111111f  F        r'rc!eZdZdZdejddddddfdeeefded ed eed eed ed e de e dej fde dej fde de dede de deddf fd Z dedefdZdZxZS)ray Implements MaxVit Transformer from the `MaxViT: Multi-Axis Vision Transformer `_ paper. Args: input_size (Tuple[int, int]): Size of the input image. stem_channels (int): Number of channels in the stem. partition_size (int): Size of the partitions. block_channels (List[int]): Number of channels in each block. block_layers (List[int]): Number of layers in each block. stochastic_depth_prob (float): Probability of stochastic depth. Expands to a list of probabilities for each layer that scales linearly to the specified value. squeeze_ratio (float): Squeeze ratio in the SE Layer. Default: 0.25. expansion_ratio (float): Expansion ratio in the MBConv bottleneck. Default: 4. norm_layer (Callable[..., nn.Module]): Normalization function. Default: None (setting to None will produce a `BatchNorm2d(eps=1e-3, momentum=0.01)`). activation_layer (Callable[..., nn.Module]): Activation function Default: nn.GELU. head_dim (int): Dimension of the attention heads. mlp_ratio (int): Expansion ratio of the MLP layer. Default: 4. mlp_dropout (float): Dropout probability for the MLP layer. Default: 0.0. attention_dropout (float): Dropout probability for the attention layer. Default: 0.0. num_classes (int): Number of classes. Default: 1000. Ng?rrEir stem_channelsrblock_channels block_layersr|stochastic_depth_probrK.rJrIrHrrr num_classesr cDtt|d}|ttjdd}t |t|}t|D]?\}}|d|zdks|d|zdkrtd|d|d |d |d @t j t||dd || d dt||ddddd|_ t|dd d}||_t j|_|g|ddz}|}t#jd|t'|}d}t+|||D]]\}}}|jt/||| | || || | ||||||||z|jdj}||z }^t j t jdt jt j|dt j|d|dt jt j|d|d |_|dS)Nr*gMbP?g{Gz?)epsmomentumrr#z Input size z of block z$ is not divisible by partition size zx. Consider changing the partition size or the input size. Current configuration yields the following block input sizes: rr"F)rrKrJrNrTT)rrKrJrNrOr6)rFrGrIrHrKrJr|rrrrrrr)rN) r[r\rrr BatchNorm2dr0rrrr_rstemr&rrblocksnplinspacer>tolistzipr,rrAdaptiveAvgPool2dFlattenrrTanh classifier _init_weights)rfrrrrrr|rrKrJrIrHrrrr input_channelsblock_input_sizesrblock_input_sizerFrGrp_idx in_channel out_channel num_layersrks r%r\zMaxVit.__init__Ms#: D!!!   TDIIIJ 5Z^ATATUU%./@%A%A   !C!"^3q8857W#!" #&7tt#s(Ot  ttS t3itt %t"Xc29n56#t$#3 >2%t()t*+t./t01t2!3t67t8 9ttttttlF + + + + + + +r'rFrrrrrr|weightsprogresskwargsc |ot|dt|jd|jdd|jddksJt|d|jd|dd} t d ||||||| d|} |*| ||d | S) Nr  categoriesmin_sizerr#rr2)rrrrr|rrT)r, check_hashr$)rrmetapoprload_state_dictget_state_dict) rrrrrr|r+r,r-rmodels r%_maxvitr9s$fmSl9S5T5TUUU|J'*gl:.Fq.IIIIIflGL4LMMML*55J  #%!3%     E g44hSW4XXYYY Lr'c neZdZedeeddejedddddd d id d d dZ e Z dS)rz9https://download.pytorch.org/models/maxvit_t-bc5ab103.pthr2) crop_size resize_size interpolationir1zLhttps://github.com/pytorch/vision/tree/main/references/classification#maxvitz ImageNet-1KgT@g|?5.X@)zacc@1zacc@5gZd;@gK7]@zThese weights reproduce closely the results of the paper using a similar training recipe. They were trained with a BatchNorm2D momentum of 0.99 instead of the more correct 0.01.)r/ num_paramsr0recipe_metrics_ops _file_size_docs)url transformsr4N) rqrrrsrrrrBICUBICr IMAGENET1K_V1DEFAULTr$r'r%rrsG G7 3CO`Oh   /""d##   !g  M.GGGr'r pretrained)r+T)r+r,c ht|}tddgdgdddd||d|S) a Constructs a maxvit_t architecture from `MaxViT: Multi-Axis Vision Transformer `_. Args: weights (:class:`~torchvision.models.MaxVit_T_Weights`, optional): The pretrained weights to use. See :class:`~torchvision.models.MaxVit_T_Weights` below for more details, and possible values. By default, no pre-trained weights are used. progress (bool, optional): If True, displays a progress bar of the download to stderr. Default is True. **kwargs: parameters passed to the ``torchvision.models.maxvit.MaxVit`` base class. Please refer to the `source code `_ for more details about this class. .. autoclass:: torchvision.models.MaxVit_T_Weights :members: @)rKi)r"r"rr" g?)rrrr|rrr+r,r$)rverifyr9)r+r,r-s r%rrs_.%%g..G  ***!\\!      r')NF)r`s ######AAAAAAAAAAAAAAAA HHHHHHHHHH999999TTTTTTTTHHHHHHHH<<<<<<RRRRRRRR111111   uS#XSRU`chmnqsvnvhwsCxCDQVWZ\_W_Q`La##S#U\####TTTTTRYTTTnFFFFF29FFFR     ry   bi4