jddlmZmZddlZddlmZddlmcmZddlm Z ddl m Z ddl m Z ddlmZddlmZdd lmZd d lmZmZmZd d lmZd dlmZmZmZdZGddejZGddejZ GddejZ!GddejZ"GddejZ#dZ$GddejZ%GddejZ&Gdd ejZ'Gd!d"ejZ(Gd#d$ejZ)Gd%d&ejZ*d'd(iZ+Gd)d*eZ,Gd+d,eZ-dd-d.d/Z.eed0e,j/f1dd2d.d3ee,d4e*fd5Z0eed0e-j1f1dd2d.d3ee-d4e*fd6Z2dS)7)ListOptionalN)Tensor) BatchNorm2d)InstanceNorm2d)Conv2dNormActivation) OpticalFlow)_log_api_usage_once)register_modelWeights WeightsEnum)handle_legacy_interface) grid_samplemake_coords_grid upsample_flow)RAFT raft_large raft_smallRaft_Large_WeightsRaft_Small_Weightsc6eZdZdZddddeffdZdZxZS) ResidualBlockzr?r@r)r<rBrCs@r5rErEJsZDDHI8       r6rEcDeZdZdZeddejdfd ZdZdZ xZ S)FeatureEncoderzThe feature encoder, used both as the actual feature encoder, and as the context encoder. It must downsample its input by 8. @rN`)r rr r )blocklayersstridesr ctt|dkrtdt|t d|d|d|dd|_|||d|d||d |_|||d|d ||d  |_|||d |d||d |_ tj |d|d d |_ | D]}t|tj r(tj|jd dDt|tjtjfrX|j%tj|jd|j%tj|jdtt-t/d|}|d|_d |z|_dS)Nz0The expected number of layers is 5, instead got r rTrr)r first_strider rG)r!fan_outr0)mode nonlinearityc|dkS)Nr )ss r5z)FeatureEncoder.__init__..s AFr6)r(r)len ValueErrorr convnormrelu_make_2_blockslayer1layer2layer3r,Conv2dconvmodules isinstanceinitkaiming_normal_weightrr constant_r"listfilter output_dimdownsample_factor)r1rRrSrTr mnum_downsamplesr4s r5r)zFeatureEncoder.__init__zs  v;;!  ]PSTZP[P[]]^^ ^1 vayZQwqzX\   ))%F1IR\krstku)vv ))%F1IR\krstku)vv ))%F1IR\krstku)vv IfQiBBB  1 1A!RY'' 1''yv'VVVVA0ABCC 18'G%%ah2226%G%%afa000d6*:*:G#D#DEEFF *!"O!3r6ch|||||}||||d}tj||S)N)r rr)r, Sequential)r1rRr2r3r rXblock1block2s r5rdzFeatureEncoder._make_2_blockssF{LZP\]]]|\jQRSSS}VV,,,r6c||}||}||}||}||}|Sr8)rcrerfrgrir1r:s r5r<zFeatureEncoder.forwardsV   a  KKNN KKNN KKNN IIaLLr6) r=r>r?r@rr,rr)rdr<rBrCs@r5rLrLts{ %-C\fhft4444444@---       r6rLc2eZdZdZddddfd ZdZxZS) MotionEncoderzThe motion encoder, part of the update block. Takes the current predicted flow and the correlation features as input and returns an encoded version of these. rQrPrNrP) corr_layers flow_layersr3ctt|dkrtdt|t|dvrtdt|t ||ddd|_t|dkr%t |d|ddd|_ntj|_t d|ddd |_ t |d|ddd|_ t |d |d z|dz dd|_ ||_ dS) Nr z5The expected number of flow_layers is 2, instead got rr z8The number of corr_layers should be 1 or 2, instead got rrr r!r rWr`) r(r)rarbr convcorr1 convcorr2r,r- convflow1 convflow2rir3)r1in_channels_corrrrr3r4s r5r)zMotionEncoder.__init__sg  {  q gUXYdUeUegghh h {  6 ) )jX[\gXhXhjjkk k-.> A[_mnooo {  q 1+a.+a.]aopqqqDNN[]]DN-aQD^_```-k!nk!nY]klmmm) Ok"o -|a/?D^_    )r6c8||}||}|}||}||}t j||gd}||}t j||gdSNrdim)rrrrtorchcatri)r1flow corr_featurescorr flow_orig corr_flows r5r<zMotionEncoder.forwards~~m,,~~d## ~~d##~~d##ItTl222 IIi(( y)Y/Q7777r6rJrCs@r5r}r}sg 9CPYhk)))))))0 8 8 8 8 8 8 8r6r}c(eZdZdZfdZdZxZS)ConvGRUzConvolutional Gru unit.cttj||z||||_tj||z||||_tj||z||||_dS)Nr!padding)r(r)r,rhconvzconvrconvqr1 input_size hidden_sizer!rr4s r5r)zConvGRU.__init__s Y{Z7R]gnooo Y{Z7R]gnooo Y{Z7R]gnooo r6cntj||gd}tj||}tj||}tj|tj||z|gd}d|z |z||zz}|Sr)rrsigmoidrrtanhr)r1hr:hxzrqs r5r<zConvGRU.forwards Y1v1 % % % M$**R.. ) ) M$**R.. ) ) Jtzz%)QUAJA">">">?? @ @ UaK!a% r6rJrCs@r5rrsR!!ppppp r6rc|Sr8r])r_s r5_pass_through_hrs Hr6c0eZdZdZdddfd ZdZxZS)RecurrentBlockzRecurrent block, part of the update block. Takes the current hidden state and the concatenation of (motion encoder output, context) as input. Returns an updated hidden state. )rrV)rVr)rr )r rrc$tt|t|kr/tdt|dt|t|dvrtdt|t |||d|d|_t|dkr%t |||d|d|_n t|_||_dS) NzSkernel_size should have the same length as padding, instead got len(kernel_size) = z and len(padding) = rz.kernel_size should either 1 or 2, instead got rrrr!rr r) r(r)rarbrconvgru1convgru2rrrs r5r)zRecurrentBlock.__init__sC  {  s7|| + +[fijufvfv[[MPQXMYMY[[  {  6 ) )`cR]N^N^``aa a!{ TU`ghi`j     {  q #%;KXYNdklmdnDMM,DM&r6c^|||}|||}|Sr8)rr)r1rr:s r5r<zRecurrentBlock.forward s- MM!Q   MM!Q  r6rJrCs@r5rrse @PYi''''''',r6rc(eZdZdZfdZdZxZS)FlowHeadzFlow head, part of the update block. Takes the hidden state of the recurrent unit as input, and outputs the predicted "delta flow". cttj||dd|_tj|ddd|_tjd|_dS)Nr rrr Tr&)r(r)r,rhconv1conv2r/r0)r1r2rr4s r5r)zFlowHead.__init__se Y{KAFFF Y{Aq!<<< GD))) r6cx||||Sr8)rr0rr{s r5r<zFlowHead.forwards*zz$))DJJqMM22333r6rJrCs@r5rrsQ ***** 4444444r6rc(eZdZdZfdZdZxZS) UpdateBlockzThe update block which contains the motion encoder, the recurrent block, and the flow head. It must expose a ``hidden_state_size`` attribute which is the hidden state size of its recurrent block. ct||_||_||_|j|_dSr8)r(r)motion_encoderrecurrent_block flow_headrhidden_state_size)r1rrrr4s r5r)zUpdateBlock.__init__&sA ,."!0!<r6c|||}tj||gd}|||}||}||fSr)rrrrr)r1 hidden_statecontextrrmotion_featuresr: delta_flows r5r<zUpdateBlock.forward.sa--dMBB Iw0a 8 8 8++L!<< ^^L11 Z''r6rJrCs@r5rr sQ =====(((((((r6rc.eZdZdZddfd ZdZxZS) MaskPredictorzMask predictor to be used when upsampling the predicted flow. It takes the hidden state of the recurrent unit as input and outputs the mask. This is not used in the raft-small model. ?) multiplierctt||dd|_t j|ddd|_||_dS)Nr ri@rrr)r(r)rconvrelur,rhrir)r1r2rrr4s r5r)zMaskPredictor.__init__>s\ ,[+RVdefff Ik9aCCC %r6cj||}||}|j|zSr8)rrirr{s r5r<zMaskPredictor.forwardKs/ MM!   IIaLL""r6rJrCs@r5rr7sb @D % % % % % % %#######r6rcFeZdZdZddddedeffdZdZdZd ZxZ S) CorrBlockaThe correlation block. Creates a correlation pyramid with ``num_levels`` levels from the outputs of the feature encoder, and then indexes from this pyramid to create correlation features. The "indexing" of a given centroid pixel x' is done by concatenating its surrounding neighbors that are within a ``radius``, according to the infinity norm (see paper section 3.2). Note: typo in the paper, it should be infinity norm, not 1-norm. rG num_levelsradiusrrct||_||_t jdg|_|d|zdzdzz|_dS)Nrr r)r(r)rrrtensor corr_pyramidr3)r1rrr4s r5r)zCorrBlock.__init__[s\ $ +0<??*; '!f*q.Q)>>r6c Z |j|jkr td|jd|jddd|jdz zz t fd|jddDr,td d |jddd d d zd |||}|j\}}}}}}|||z|z|||}|g|_t|jdz D]3}tj |dd}|j |4dS)aLBuild the correlation pyramid from two feature maps. The correlation volume is first computed as the dot product of each pair (pixel_in_fmap1, pixel_in_fmap2) The last 2 dimensions of the correlation volume are then pooled num_levels times at different resolutions to build the correlation pyramid. z;Input feature maps should have the same shape, instead got z (fmap1.shape) != z (fmap2.shape)r rc3$K|] }|kV dSr8r]).0 fmap_size min_fmap_sizes r5 z*CorrBlock.build_pyramid..ys(KKYy=(KKKKKKr6NzuFeature maps are too small to be down-sampled by the correlation pyramid. H and W of feature maps should be at least z; got: zv. Remember that input images to the model are downsampled by 8, so that means their dimensions should be at least 8 * z = .)r!r) shaperbrany_compute_corr_volumereshaperrangeF avg_pool2dappend) r1fmap1fmap2 corr_volume batch_sizerw num_channelsrrs @r5 build_pyramidzCorrBlock.build_pyramidhs ;%+ % %IekIImrmxIII Q4?Q#678 KKKK%+bcc:JKKK K K \>K\\TYT_`b`c`cTd\\6C\\HI=GX\\\ //u== /:/@, Aq,1!))*q.1*1 E  "7 7 7{I^{{fsfy{{ r6cZ|j\}}}}|||||z}|||||z}tj|dd|}||||d||}|tjtj|z S)Nrr )rrrmatmul transposesqrtr)r1rrrrrrrs r5rzCorrBlock._compute_corr_volumes).& L!Q :|QU;; :|QU;;|EOOAq11599yyQ1a33ejl!;!;<<<r?r@intr)rrrrBrCs@r5rrQs-.Q ? ? ?c ?s ? ? ? ? ? ? 2 2 2D<=======r6rc2eZdZddfd ZddefdZxZS)rN)mask_predictorctt|||_||_||_||_||_t|jdstddS)a RAFT model from `RAFT: Recurrent All Pairs Field Transforms for Optical Flow `_. args: feature_encoder (nn.Module): The feature encoder. It must downsample the input by 8. Its input is the concatenation of ``image1`` and ``image2``. context_encoder (nn.Module): The context encoder. It must downsample the input by 8. Its input is ``image1``. As in the original implementation, its output will be split into 2 parts: - one part will be used as the actual "context", passed to the recurrent unit of the ``update_block`` - one part will be used to initialize the hidden state of the recurrent unit of the ``update_block`` These 2 parts are split according to the ``hidden_state_size`` of the ``update_block``, so the output of the ``context_encoder`` must be strictly greater than ``hidden_state_size``. corr_block (nn.Module): The correlation block, which creates a correlation pyramid from the output of the ``feature_encoder``, and then indexes from this pyramid to create correlation features. It must expose 2 methods: - a ``build_pyramid`` method that takes ``feature_map_1`` and ``feature_map_2`` as input (these are the output of the ``feature_encoder``). - a ``index_pyramid`` method that takes the coordinates of the centroid pixels as input, and returns the correlation features. See paper section 3.2. It must expose an ``out_channels`` attribute. update_block (nn.Module): The update block, which contains the motion encoder, the recurrent unit, and the flow head. It takes as input the hidden state of its recurrent unit, the context, the correlation features, and the current predicted flow. It outputs an updated hidden state, and the ``delta_flow`` prediction (see paper appendix A). It must expose a ``hidden_state_size`` attribute. mask_predictor (nn.Module, optional): Predicts the mask that will be used to upsample the predicted flow. The output channel must be 8 * 8 * 9 - see paper section 3.3, and Appendix B. If ``None`` (default), the flow is upsampled using interpolation. rzIThe update_block parameter should expose a 'hidden_state_size' attribute.N) r(r)r feature_encodercontext_encoder corr_block update_blockr hasattrrb)r1rrrrr r4s r5r)z RAFT.__init__sH D!!!..$(,t(*=>> jhii i j jr6 num_flow_updatesc |j\}}}}||f|jddkr%td|d|d|jdd|dzdks|dzdkrtd|d|d |tj||gd }tj|d d \} } | jdd|dz|dzfkrtd |j| | ||} | jdd|dz|dzfkrtd|j j } | jd| z } | dkr!td| jdd| dtj | | | gd \}}tj |}tj|}t||dz|dz| j}t||dz|dz| j}g}t%|D]}|}|j|}||z }| ||||\}}||z}|jdn||}t-||z |}|||S)Nrz6input images should have the same shape, instead got (z, z) != rrz:input image H and W should be divisible by 8, instead got z (h) and z (w)rr )chunksrz2The feature encoder should downsample H and W by 8z2The context encoder should downsample H and W by 8rzThe context encoder outputs zA channels, but it should have at strictly more than hidden_state=z channels)r)rup_mask)rrbrrrchunkrrrrrsplitrrr0rrrrdetachrr rr)r1image1image2rrrrrfmapsrr context_outrout_channels_contextrrcoords0coords1flow_predictionsrrrrupsampled_flows r5r<z RAFT.forwardsG$l Aq! q6V\"##& & &vVWvv[\vvcicoprpspsctvvww wA Q!mZ[mmfgmmmnn n$$UY/?Q%G%G%GHH{5::: u ;rss QQ/ / /QRR R %%eU333**622  RSS !a1fa1f%5 5 5QRR R !-?*036GG 1 $ $c{/@/CccGXccc !& K:KMa9bhi j j j gz,// &//":qAvqAv>>AA%,OO":qAvqAv>>AA%,OO'(( 4 4Ann&&G O9979SSMW$D'+'8'8wP]_c'd'd $L* *G"19ddt?R?RS_?`?`G*71BWUUUN  # #N 3 3 3 3r6)r)r=r>r?r)r r<rBrCs@r5rrsmei/j/j/j/j/j/j/jb1 1 1 1 1 1 1 1 1 1 r6rmin_size)rPrPc eZdZdZedeieddddiddidd d d d d ddZedeieddddiddiddd d d d ddZedeieddddiddidd d ddZ edeieddddiddidd d d dZ ed!eieddd"d#d$iid d d%dZ ed&eieddd"d#d'iid d d(dZ e Z d)S)*rThe metrics reported here are as follows. ``epe`` is the "end-point-error" and indicates how far (in pixels) the predicted flow is from its true value. This is averaged over all pixels of all images. ``per_image_epe`` is similar, but the average is different: the epe is first computed on each image independently, and then averaged over all images. This corresponds to "Fl-epe" (sometimes written "F1-epe") in the original paper, and it's only used on Kitti. ``fl-all`` is also a Kitti-specific metric, defined by the author of the dataset and used for the Kitti leaderboard. It corresponds to the average of pixels whose epe is either <3px, or <5% of flow's 2-norm. zBhttps://download.pytorch.org/models/raft_large_C_T_V1-22a6c225.pthi@9P$https://github.com/princeton-vl/RAFTepeg?߾?g{P@gu@gޓZs1@ per_image_epefl_allzSintel-Train-CleanpasszSintel-Train-Finalpassz Kitti-TraingNbX9`j@g/$!4@These weights were ported from the original paper. They are trained on :class:`~torchvision.datasets.FlyingChairs` + :class:`~torchvision.datasets.FlyingThings3D`. num_paramsrecipe_metrics_ops _file_size_docsurl transformsmetazBhttps://download.pytorch.org/models/raft_large_C_T_V2-1bb1363a.pthChttps://github.com/pytorch/vision/tree/main/references/optical_flowgH}?g&S@g_L @gea0@These weights were trained from scratch on :class:`~torchvision.datasets.FlyingChairs` + :class:`~torchvision.datasets.FlyingThings3D`.zGhttps://download.pytorch.org/models/raft_large_C_T_SKHT_V1-0b8c9e55.pthg ףp= ?gq= ףp @)zSintel-Test-CleanpasszSintel-Test-Finalpassa0 These weights were ported from the original paper. They are trained on :class:`~torchvision.datasets.FlyingChairs` + :class:`~torchvision.datasets.FlyingThings3D` and fine-tuned on Sintel. The Sintel fine-tuning step is a combination of :class:`~torchvision.datasets.Sintel`, :class:`~torchvision.datasets.KittiFlow`, :class:`~torchvision.datasets.HD1K`, and :class:`~torchvision.datasets.FlyingThings3D` (clean pass). zGhttps://download.pytorch.org/models/raft_large_C_T_SKHT_V2-ff5fadd5.pthgv?gK7@a/ These weights were trained from scratch. They are pre-trained on :class:`~torchvision.datasets.FlyingChairs` + :class:`~torchvision.datasets.FlyingThings3D` and then fine-tuned on Sintel. The Sintel fine-tuning step is a combination of :class:`~torchvision.datasets.Sintel`, :class:`~torchvision.datasets.KittiFlow`, :class:`~torchvision.datasets.HD1K`, and :class:`~torchvision.datasets.FlyingThings3D` (clean pass). zIhttps://download.pytorch.org/models/raft_large_C_T_SKHT_K_V1-4a6a5039.pthz Kitti-Testr+gffffff@a These weights were ported from the original paper. They are pre-trained on :class:`~torchvision.datasets.FlyingChairs` + :class:`~torchvision.datasets.FlyingThings3D`, fine-tuned on Sintel, and then fine-tuned on :class:`~torchvision.datasets.KittiFlow`. The Sintel fine-tuning step was described above. zIhttps://download.pytorch.org/models/raft_large_C_T_SKHT_K_V2-b5c70766.pthg(\@a These weights were trained from scratch. They are pre-trained on :class:`~torchvision.datasets.FlyingChairs` + :class:`~torchvision.datasets.FlyingThings3D`, fine-tuned on Sintel, and then fine-tuned on :class:`~torchvision.datasets.KittiFlow`. The Sintel fine-tuning step was described above. N)r=r>r?r@rr _COMMON_METAC_T_V1C_T_V2 C_T_SKHT_V1 C_T_SKHT_V2 C_T_SKHT_K_V1 C_T_SKHT_K_V2DEFAULTr]r6r5rrs  W P  !<+0&/+0&/177KK  >   F*W P  ![+0&/+0&/177KK  >   F(' U  !<*/*/     K6' U  ![*/*/     K4G W  !<x.    M0G W  ![x.    M.GGGr6rc eZdZdZedeieddddiddidd d d d d ddZedeieddddiddiddd d d d ddZeZ dS)rr&zBhttps://download.pytorch.org/models/raft_small_C_T_V1-ad48884c.pthir'r(gQ@gZd; @go@g3G9@r)r,gp= G@g rh@r-r.r5zBhttps://download.pytorch.org/models/raft_small_C_T_V2-01064c6d.pthr9gHPs?gC @g1%d@g"lxz<9@r:N) r=r>r?r@rr r;r<r=rBr]r6r5rrs  W P   <+0&/+0&/177KK >   F(W P   [+0&/+0&/177KK >   F(GGGr6rF)weightsprogressc |ddpt|||}|ddpt|||}|ddpt|| }|dd}|at|j| | | }|d| z }t |j|z| || }t | | }t||| }|d d}||rt| d d}td|||||d|}|*| | |d|S)Nr)rRrSr rrrr)rrrr3r`r)r2r)rrrr rQr)r2rr)rrrrr T)rE check_hashr]) poprLrr}r3rrrrrload_state_dictget_state_dict)rDrEfeature_encoder_layersfeature_encoder_blockfeature_encoder_norm_layercontext_encoder_layerscontext_encoder_blockcontext_encoder_norm_layercorr_block_num_levelscorr_block_radiusmotion_encoder_corr_layersmotion_encoder_flow_layersmotion_encoder_out_channels!recurrent_block_hidden_state_sizerecurrent_block_kernel_sizerecurrent_block_paddingflow_head_hidden_sizeuse_mask_predictorkwargsrrrrrrrrr models r5_raftr]s:jj!2D99^#,BOi>>>Ojj!2D99^#,BOi>>>OL$//x9H]fw3x3x3xJ::nd33L&'4224     6b9<]](%25II93+    )JXmnnn ".Ramvwww ZZ 0$77N"4&9     ''!%       E g44hSW4XXYYY Lr6 pretrained)rDTrDreturnc t|}tdid|d|dddtdtdddtd t d d d d d ddddddddddddddd|S)aRAFT model from `RAFT: Recurrent All Pairs Field Transforms for Optical Flow `_. Please see the example below for a tutorial on how to use this model. Args: weights(:class:`~torchvision.models.optical_flow.Raft_Large_Weights`, optional): The pretrained weights to use. See :class:`~torchvision.models.optical_flow.Raft_Large_Weights` below for more details, and possible values. By default, no pre-trained weights are used. progress (bool): If True, displays a progress bar of the download to stderr. Default is True. **kwargs: parameters passed to the ``torchvision.models.optical_flow.RAFT`` base class. Please refer to the `source code `_ for more details about this class. .. autoclass:: torchvision.models.optical_flow.Raft_Large_Weights :members: rDrErKrMrLrMrNrOrPrQrGrRrSr~rTrrUrPrVrWrrXrrYrQrZTr])rverifyr]rrrrDrEr[s r5rrGs0!''00G       65   ,m   $2>   65 ,m $/;  a ! $.:  $-9! "%(C# &+.#' (%5$4) *!1 0+ ."c/ 2 4 5  r6c t|}td id|d|dddtdtddd td d d d dddddddddddddddddd|S)!aRAFT "small" model from `RAFT: Recurrent All Pairs Field Transforms for Optical Flow `__. Please see the example below for a tutorial on how to use this model. Args: weights(:class:`~torchvision.models.optical_flow.Raft_Small_Weights`, optional): The pretrained weights to use. See :class:`~torchvision.models.optical_flow.Raft_Small_Weights` below for more details, and possible values. By default, no pre-trained weights are used. progress (bool): If True, displays a progress bar of the download to stderr. Default is True. **kwargs: parameters passed to the ``torchvision.models.optical_flow.RAFT`` base class. Please refer to the `source code `_ for more details about this class. .. autoclass:: torchvision.models.optical_flow.Raft_Small_Weights :members: rDrErK) rdrNrOrPrLrMrN)rdrdrNrOrOrPNrQrGrRr rS)rOrT)rNrdrURrVrOrW)r rX)rrYrPrZFr])rrar]rErrbs r5rrs .!''00G       54   .o   $2>   54 .o $(4  a ! $)5  $,8! "%'B# &+-"' (%)D) *!%+ ."c/ 2!5 5  r6)3typingrrrtorch.nnr,torch.nn.functional functionalrrtorch.nn.modules.batchnormrtorch.nn.modules.instancenormrtorchvision.opsrtransforms._presetsr utilsr _apir rr_utilsrrrr__all__ModulerrErLr}rrrrrrrrr;rrr]r?rr=rr]r6r5rts!!!!!!!! 222222888888000000......((((((7777777777,,,,,,@@@@@@@@@@ , , , , , BI, , , ^' ' ' ' ' bi' ' ' T44444RY444n(8(8(8(8(8BI(8(8(8Vbi$       RY   F 4 4 4 4 4ry 4 4 4 (((((")(((.#####BI###4^=^=^=^=^= ^=^=^=Bc c c c c 29c c c N  [[[[[[[[|66666666v  PPPPPf,0B0N!OPPP:>3338$673\`333QP3l,0B0I!JKKK:>2228$672\`222LK222r6