˲jOddlmZmZddlZddlmZddlmZmZddl m Z m Z m Z ddl mZddlmZdd lmZgd ZGd d eZGd deZGddeeZGddeZGddeeZGddeZGddeeZGddeZGddeeZGddeZdS))AnyOptionalN)Tensor) functionalinit) ParameterUninitializedBufferUninitializedParameter) SyncBatchNorm)LazyModuleMixin)Module) BatchNorm1dLazyBatchNorm1d BatchNorm2dLazyBatchNorm2d BatchNorm3dLazyBatchNorm3dr c eZdZUdZdZgdZeed<eed<e eed<e ed<e ed< ddedede ede de d d f fd Z ddZ ddZ dZdZfdZxZS) _NormBasez,Common base of _InstanceNorm and _BatchNorm.)track_running_statsmomentumeps num_featuresaffinerrrrrh㈵>皙?TNreturnc ||d}t||_||_||_||_||_|jrIttj |fi||_ ttj |fi||_ n,| dd| dd|jr| dtj|fi|| dtj|fi|||| dtj d dtjid |D|nB| dd| dd| dd|dS) Ndevicedtypeweightbias running_mean running_varnum_batches_trackedrr#c&i|]\}}|dk ||Sr#.0kvs KC:\Projects\Historical\venv\Lib\site-packages\torch/nn/modules/batchnorm.py z&_NormBase.__init__..Ks#OOO1!w,,q!,,,r)super__init__rrrrrrtorchemptyr$r%register_parameterregister_bufferzerosonestensorlongitemsreset_parameters selfrrrrrr"r#factory_kwargs __class__s r0r5z_NormBase.__init__&s%+U;; (   #6 ; 2#EK $O$O$O$OPPDK!%+l"M"Mn"M"MNNDII  # #Hd 3 3 3  # #FD 1 1 1  # >   L K KN K K     uz,II.II     /  .  % *PO(<(<(>(>OOO     6 6   6 6 6   5 5 5  !6 = = = r2c|jrN|j|jd|jdSdSNr )rr&zero_r'fill_r(rAs r0reset_running_statsz_NormBase.reset_running_statsUs`  # -   # # % % %   " "1 % % %  $ * * , , , , ,  - -r2c||jr4tj|jtj|jdSdSN)rIrrones_r$zeros_r%rHs r0r?z_NormBase.reset_parameters]sR   """ ; # Jt{ # # # K " " " " " # #r2ctrK)NotImplementedErrorrAinputs r0_check_input_dimz_NormBase._check_input_dimcs!!r2c&djdi|jS)Nzj{num_features}, eps={eps}, momentum={momentum}, affine={affine}, track_running_stats={track_running_stats}r+)format__dict__rHs r0 extra_reprz_NormBase.extra_reprfs1 ? 88> P PAE P P r2c X|dd}||dkrc|jr\|dz} | |vrS|j)|jjt jdkr|jnt jdtj|| <t|||||||dS)Nversionrr(metarr*) getrr(r"r6r<r=r4_load_from_state_dict) rA state_dictprefixlocal_metadatastrict missing_keysunexpected_keys error_msgsrXnum_batches_tracked_keyrCs r0r[z_NormBase._load_from_state_dictls!$$Y55 Ow{{0H{'-/D&D #&j88/;075<;O;OOO,,auz::: 23 %%            r2rrTTNNrN)__name__ __module__ __qualname____doc___version __constants__int__annotations__floatrboolr5rIr?rRrVr[ __classcell__rCs@r0rrsF66HXXXM JJJuo LLL$'$(- - - - 5/ -  - " -  - - - - - - ^----#### """            r2rc beZdZ ddededeeded ed df fd Zd ed efd Z xZ S) _BatchNormrrTNrrrrrrcN||d}tj|||||fi|dSNr!)r4r5r@s r0r5z_BatchNorm.__init__sP%+U;; #x1D  HV     r2rQc |||jd}n|j}|jrN|jrG|j@|jd|jdt |jz }n|j} |jrd}n|jduo|jdu} tj ||jr|jr|jnd|jr|jr|jnd|j |j |||j S)Nr ?T)rRrtrainingrr(add_rnr&r'F batch_normr$r%r)rArQexponential_average_factor bn_trainings r0forwardz_BatchNorm.forwards1 e$$$ = ), & &)- & = ?T5 ?'3(--a000=(14uT=U7V7V1V..15.  = UKK,4T4;Kt;SK | = $($< D  $(M WT5M WD  SW K I  & H   r2rd) rfrgrhrlrnrror5rrrprqs@r0rsrss$'$(       5/    "          . V. . . . . . . . . r2rscZeZdZUeed<eed< d d fd Zd fd Zd d ZxZS) _LazyNormBaser$r%rrTNrc||d}tjd||ddfi|||_||_|jr"t di||_t di||_|jrctdi||_tdi||_ tj ddtj id| D|_dSdS)Nr!rFr#c&i|]\}}|dk ||Sr*r+r,s r0r1z*_LazyNormBase.__init__..s#KKKDAqa7ll1alllr2r+r3)r4r5rrr r$r%r r&r'r6r<r=r>r() rArrrrr"r#rBrCs r0r5z_LazyNormBase.__init__s%%+U;;        #6 ; A0BB>BBDK.@@@@DI  #  3 E En E ED 2DD^DDD ',|((j(LKN$8$8$:$:KKK((D $ $ $  r2c|s-|jdkr$tdSdSdS)Nr)has_uninitialized_paramsrr4r?)rArCs r0r?z_LazyNormBase.reset_parameterssP,,.. '43D3I3I GG $ $ & & & & & ' '3I3Ir2c |r|jd|_|jrxt |jt sJt |jt sJ|j|jf|j|jf|j r@|j |jf|j |jf| dSdSrE) rshaperr isinstancer$r r% materializerr&r'r?rPs r0initialize_parametersz#_LazyNormBase.initialize_parameterss  ( ( * * $ % AD { <!$+/EFFFFF!$)-CDDDDD ''):(<=== %%t'8&:;;;' !--&( ,,&(  ! ! # # # # # $ $r2rdre) rfrgrhr rmr5r?rrprqs@r0rrs """"              D''''''$$$$$$$$r2rceZdZdZdZdS)ra Applies Batch Normalization over a 2D or 3D input. Method described in the paper `Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift `__ . .. math:: y = \frac{x - \mathrm{E}[x]}{\sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta The mean and standard-deviation are calculated per-dimension over the mini-batches and :math:`\gamma` and :math:`\beta` are learnable parameter vectors of size `C` (where `C` is the number of features or channels of the input). By default, the elements of :math:`\gamma` are set to 1 and the elements of :math:`\beta` are set to 0. At train time in the forward pass, the variance is calculated via the biased estimator, equivalent to ``torch.var(input, unbiased=False)``. However, the value stored in the moving average of the variance is calculated via the unbiased estimator, equivalent to ``torch.var(input, unbiased=True)``. Also by default, during training this layer keeps running estimates of its computed mean and variance, which are then used for normalization during evaluation. The running estimates are kept with a default :attr:`momentum` of 0.1. If :attr:`track_running_stats` is set to ``False``, this layer then does not keep running estimates, and batch statistics are instead used during evaluation time as well. .. note:: This :attr:`momentum` argument is different from one used in optimizer classes and the conventional notion of momentum. Mathematically, the update rule for running statistics here is :math:`\hat{x}_\text{new} = (1 - \text{momentum}) \times \hat{x} + \text{momentum} \times x_t`, where :math:`\hat{x}` is the estimated statistic and :math:`x_t` is the new observed value. Because the Batch Normalization is done over the `C` dimension, computing statistics on `(N, L)` slices, it's common terminology to call this Temporal Batch Normalization. Args: num_features: number of features or channels :math:`C` of the input eps: a value added to the denominator for numerical stability. Default: 1e-5 momentum: the value used for the running_mean and running_var computation. Can be set to ``None`` for cumulative moving average (i.e. simple average). Default: 0.1 affine: a boolean value that when set to ``True``, this module has learnable affine parameters. Default: ``True`` track_running_stats: a boolean value that when set to ``True``, this module tracks the running mean and variance, and when set to ``False``, this module does not track such statistics, and initializes statistics buffers :attr:`running_mean` and :attr:`running_var` as ``None``. When these buffers are ``None``, this module always uses batch statistics. in both training and eval modes. Default: ``True`` Shape: - Input: :math:`(N, C)` or :math:`(N, C, L)`, where :math:`N` is the batch size, :math:`C` is the number of features or channels, and :math:`L` is the sequence length - Output: :math:`(N, C)` or :math:`(N, C, L)` (same shape as input) Examples:: >>> # With Learnable Parameters >>> m = nn.BatchNorm1d(100) >>> # Without Learnable Parameters >>> m = nn.BatchNorm1d(100, affine=False) >>> input = torch.randn(20, 100) >>> output = m(input) c|dkr=|dkr'td|ddSdSNrzexpected 2D or 3D input (got D input)dim ValueErrorrPs r0rRzBatchNorm1d._check_input_dimSW 99;;!   q 0 0RUYY[[RRRSS S   0 0r2NrfrgrhrirRr+r2r0rr s5DDLTTTTTr2rceZdZdZeZdZdS)raRA :class:`torch.nn.BatchNorm1d` module with lazy initialization. Lazy initialization based on the ``num_features`` argument of the :class:`BatchNorm1d` that is inferred from the ``input.size(1)``. The attributes that will be lazily initialized are `weight`, `bias`, `running_mean` and `running_var`. Check the :class:`torch.nn.modules.lazy.LazyModuleMixin` for further documentation on lazy modules and their limitations. Args: eps: a value added to the denominator for numerical stability. Default: 1e-5 momentum: the value used for the running_mean and running_var computation. Can be set to ``None`` for cumulative moving average (i.e. simple average). Default: 0.1 affine: a boolean value that when set to ``True``, this module has learnable affine parameters. Default: ``True`` track_running_stats: a boolean value that when set to ``True``, this module tracks the running mean and variance, and when set to ``False``, this module does not track such statistics, and initializes statistics buffers :attr:`running_mean` and :attr:`running_var` as ``None``. When these buffers are ``None``, this module always uses batch statistics. in both training and eval modes. Default: ``True`` c|dkr=|dkr'td|ddSdSrrrPs r0rRz LazyBatchNorm1d._check_input_dimurr2N)rfrgrhrir cls_to_becomerRr+r2r0rrXs94 MTTTTTr2rceZdZdZdZdS)ra Applies Batch Normalization over a 4D input. 4D is a mini-batch of 2D inputs with additional channel dimension. Method described in the paper `Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift `__ . .. math:: y = \frac{x - \mathrm{E}[x]}{ \sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta The mean and standard-deviation are calculated per-dimension over the mini-batches and :math:`\gamma` and :math:`\beta` are learnable parameter vectors of size `C` (where `C` is the input size). By default, the elements of :math:`\gamma` are set to 1 and the elements of :math:`\beta` are set to 0. At train time in the forward pass, the standard-deviation is calculated via the biased estimator, equivalent to ``torch.var(input, unbiased=False)``. However, the value stored in the moving average of the standard-deviation is calculated via the unbiased estimator, equivalent to ``torch.var(input, unbiased=True)``. Also by default, during training this layer keeps running estimates of its computed mean and variance, which are then used for normalization during evaluation. The running estimates are kept with a default :attr:`momentum` of 0.1. If :attr:`track_running_stats` is set to ``False``, this layer then does not keep running estimates, and batch statistics are instead used during evaluation time as well. .. note:: This :attr:`momentum` argument is different from one used in optimizer classes and the conventional notion of momentum. Mathematically, the update rule for running statistics here is :math:`\hat{x}_\text{new} = (1 - \text{momentum}) \times \hat{x} + \text{momentum} \times x_t`, where :math:`\hat{x}` is the estimated statistic and :math:`x_t` is the new observed value. Because the Batch Normalization is done over the `C` dimension, computing statistics on `(N, H, W)` slices, it's common terminology to call this Spatial Batch Normalization. Args: num_features: :math:`C` from an expected input of size :math:`(N, C, H, W)` eps: a value added to the denominator for numerical stability. Default: 1e-5 momentum: the value used for the running_mean and running_var computation. Can be set to ``None`` for cumulative moving average (i.e. simple average). Default: 0.1 affine: a boolean value that when set to ``True``, this module has learnable affine parameters. Default: ``True`` track_running_stats: a boolean value that when set to ``True``, this module tracks the running mean and variance, and when set to ``False``, this module does not track such statistics, and initializes statistics buffers :attr:`running_mean` and :attr:`running_var` as ``None``. When these buffers are ``None``, this module always uses batch statistics. in both training and eval modes. Default: ``True`` Shape: - Input: :math:`(N, C, H, W)` - Output: :math:`(N, C, H, W)` (same shape as input) Examples:: >>> # With Learnable Parameters >>> m = nn.BatchNorm2d(100) >>> # Without Learnable Parameters >>> m = nn.BatchNorm2d(100, affine=False) >>> input = torch.randn(20, 100, 35, 45) >>> output = m(input) c|dkr%td|ddSNzexpected 4D input (got rrrPs r0rRzBatchNorm2d._check_input_dim? 99;;!  Luyy{{LLLMM M  r2Nrr+r2r0rrz5EENNNNNNr2rceZdZdZeZdZdS)raUA :class:`torch.nn.BatchNorm2d` module with lazy initialization. Lazy initialization is done for the ``num_features`` argument of the :class:`BatchNorm2d` that is inferred from the ``input.size(1)``. The attributes that will be lazily initialized are `weight`, `bias`, `running_mean` and `running_var`. Check the :class:`torch.nn.modules.lazy.LazyModuleMixin` for further documentation on lazy modules and their limitations. Args: eps: a value added to the denominator for numerical stability. Default: 1e-5 momentum: the value used for the running_mean and running_var computation. Can be set to ``None`` for cumulative moving average (i.e. simple average). Default: 0.1 affine: a boolean value that when set to ``True``, this module has learnable affine parameters. Default: ``True`` track_running_stats: a boolean value that when set to ``True``, this module tracks the running mean and variance, and when set to ``False``, this module does not track such statistics, and initializes statistics buffers :attr:`running_mean` and :attr:`running_var` as ``None``. When these buffers are ``None``, this module always uses batch statistics. in both training and eval modes. Default: ``True`` c|dkr%td|ddSrrrPs r0rRz LazyBatchNorm2d._check_input_dimrr2N)rfrgrhrirrrRr+r2r0rr94 MNNNNNr2rceZdZdZdZdS)ra Applies Batch Normalization over a 5D input. 5D is a mini-batch of 3D inputs with additional channel dimension as described in the paper `Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift `__ . .. math:: y = \frac{x - \mathrm{E}[x]}{ \sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta The mean and standard-deviation are calculated per-dimension over the mini-batches and :math:`\gamma` and :math:`\beta` are learnable parameter vectors of size `C` (where `C` is the input size). By default, the elements of :math:`\gamma` are set to 1 and the elements of :math:`\beta` are set to 0. At train time in the forward pass, the standard-deviation is calculated via the biased estimator, equivalent to ``torch.var(input, unbiased=False)``. However, the value stored in the moving average of the standard-deviation is calculated via the unbiased estimator, equivalent to ``torch.var(input, unbiased=True)``. Also by default, during training this layer keeps running estimates of its computed mean and variance, which are then used for normalization during evaluation. The running estimates are kept with a default :attr:`momentum` of 0.1. If :attr:`track_running_stats` is set to ``False``, this layer then does not keep running estimates, and batch statistics are instead used during evaluation time as well. .. note:: This :attr:`momentum` argument is different from one used in optimizer classes and the conventional notion of momentum. Mathematically, the update rule for running statistics here is :math:`\hat{x}_\text{new} = (1 - \text{momentum}) \times \hat{x} + \text{momentum} \times x_t`, where :math:`\hat{x}` is the estimated statistic and :math:`x_t` is the new observed value. Because the Batch Normalization is done over the `C` dimension, computing statistics on `(N, D, H, W)` slices, it's common terminology to call this Volumetric Batch Normalization or Spatio-temporal Batch Normalization. Args: num_features: :math:`C` from an expected input of size :math:`(N, C, D, H, W)` eps: a value added to the denominator for numerical stability. Default: 1e-5 momentum: the value used for the running_mean and running_var computation. Can be set to ``None`` for cumulative moving average (i.e. simple average). Default: 0.1 affine: a boolean value that when set to ``True``, this module has learnable affine parameters. Default: ``True`` track_running_stats: a boolean value that when set to ``True``, this module tracks the running mean and variance, and when set to ``False``, this module does not track such statistics, and initializes statistics buffers :attr:`running_mean` and :attr:`running_var` as ``None``. When these buffers are ``None``, this module always uses batch statistics. in both training and eval modes. Default: ``True`` Shape: - Input: :math:`(N, C, D, H, W)` - Output: :math:`(N, C, D, H, W)` (same shape as input) Examples:: >>> # With Learnable Parameters >>> m = nn.BatchNorm3d(100) >>> # Without Learnable Parameters >>> m = nn.BatchNorm3d(100, affine=False) >>> input = torch.randn(20, 100, 35, 45, 10) >>> output = m(input) c|dkr%td|ddSNzexpected 5D input (got rrrPs r0rRzBatchNorm3d._check_input_dim1rr2Nrr+r2r0rrrr2rceZdZdZeZdZdS)raUA :class:`torch.nn.BatchNorm3d` module with lazy initialization. Lazy initialization is done for the ``num_features`` argument of the :class:`BatchNorm3d` that is inferred from the ``input.size(1)``. The attributes that will be lazily initialized are `weight`, `bias`, `running_mean` and `running_var`. Check the :class:`torch.nn.modules.lazy.LazyModuleMixin` for further documentation on lazy modules and their limitations. Args: eps: a value added to the denominator for numerical stability. Default: 1e-5 momentum: the value used for the running_mean and running_var computation. Can be set to ``None`` for cumulative moving average (i.e. simple average). Default: 0.1 affine: a boolean value that when set to ``True``, this module has learnable affine parameters. Default: ``True`` track_running_stats: a boolean value that when set to ``True``, this module tracks the running mean and variance, and when set to ``False``, this module does not track such statistics, and initializes statistics buffers :attr:`running_mean` and :attr:`running_var` as ``None``. When these buffers are ``None``, this module always uses batch statistics. in both training and eval modes. Default: ``True`` c|dkr%td|ddSrrrPs r0rRz LazyBatchNorm3d._check_input_dimSrr2N)rfrgrhrirrrRr+r2r0rr6rr2rceZdZdZ ddededeed ed ed eed dffd Z dZ dZ de d e fdZ eddZxZS)r aApplies Batch Normalization over a N-Dimensional input. The N-D input is a mini-batch of [N-2]D inputs with additional channel dimension) as described in the paper `Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift `__ . .. math:: y = \frac{x - \mathrm{E}[x]}{ \sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta The mean and standard-deviation are calculated per-dimension over all mini-batches of the same process groups. :math:`\gamma` and :math:`\beta` are learnable parameter vectors of size `C` (where `C` is the input size). By default, the elements of :math:`\gamma` are sampled from :math:`\mathcal{U}(0, 1)` and the elements of :math:`\beta` are set to 0. The standard-deviation is calculated via the biased estimator, equivalent to `torch.var(input, unbiased=False)`. Also by default, during training this layer keeps running estimates of its computed mean and variance, which are then used for normalization during evaluation. The running estimates are kept with a default :attr:`momentum` of 0.1. If :attr:`track_running_stats` is set to ``False``, this layer then does not keep running estimates, and batch statistics are instead used during evaluation time as well. .. note:: This :attr:`momentum` argument is different from one used in optimizer classes and the conventional notion of momentum. Mathematically, the update rule for running statistics here is :math:`\hat{x}_\text{new} = (1 - \text{momentum}) \times \hat{x} + \text{momentum} \times x_t`, where :math:`\hat{x}` is the estimated statistic and :math:`x_t` is the new observed value. Because the Batch Normalization is done for each channel in the ``C`` dimension, computing statistics on ``(N, +)`` slices, it's common terminology to call this Volumetric Batch Normalization or Spatio-temporal Batch Normalization. Currently :class:`SyncBatchNorm` only supports :class:`~torch.nn.DistributedDataParallel` (DDP) with single GPU per process. Use :meth:`torch.nn.SyncBatchNorm.convert_sync_batchnorm()` to convert :attr:`BatchNorm*D` layer to :class:`SyncBatchNorm` before wrapping Network with DDP. Args: num_features: :math:`C` from an expected input of size :math:`(N, C, +)` eps: a value added to the denominator for numerical stability. Default: ``1e-5`` momentum: the value used for the running_mean and running_var computation. Can be set to ``None`` for cumulative moving average (i.e. simple average). Default: 0.1 affine: a boolean value that when set to ``True``, this module has learnable affine parameters. Default: ``True`` track_running_stats: a boolean value that when set to ``True``, this module tracks the running mean and variance, and when set to ``False``, this module does not track such statistics, and initializes statistics buffers :attr:`running_mean` and :attr:`running_var` as ``None``. When these buffers are ``None``, this module always uses batch statistics. in both training and eval modes. Default: ``True`` process_group: synchronization of stats happen within each process group individually. Default behavior is synchronization across the whole world Shape: - Input: :math:`(N, C, +)` - Output: :math:`(N, C, +)` (same shape as input) .. note:: Synchronization of batchnorm statistics occurs only while training, i.e. synchronization is disabled when ``model.eval()`` is set or if ``self.training`` is otherwise ``False``. Examples:: >>> # xdoctest: +SKIP >>> # With Learnable Parameters >>> m = nn.SyncBatchNorm(100) >>> # creating process group (optional) >>> # ranks is a list of int identifying rank ids. >>> ranks = list(range(8)) >>> r1, r2 = ranks[:4], ranks[4:] >>> # Note: every rank calls into new_group for every >>> # process group created, even if that rank is not >>> # part of the group. >>> process_groups = [torch.distributed.new_group(pids) for pids in [r1, r2]] >>> process_group = process_groups[0 if dist.get_rank() <= 3 else 1] >>> # Without Learnable Parameters >>> m = nn.BatchNorm3d(100, affine=False, process_group=process_group) >>> input = torch.randn(20, 100, 35, 45, 10) >>> output = m(input) >>> # network is nn.BatchNorm layer >>> sync_bn_network = nn.SyncBatchNorm.convert_sync_batchnorm(network, process_group) >>> # only single gpu per process is currently supported >>> ddp_sync_bn_network = torch.nn.parallel.DistributedDataParallel( >>> sync_bn_network, >>> device_ids=[args.local_rank], >>> output_device=args.local_rank) rrTNrrrrr process_grouprc \||d} tj|||||fi| ||_dSru)r4r5r) rArrrrrrr"r#rBrCs r0r5zSyncBatchNorm.__init__sV%+U;; #x1D  HV   +r2c|dkr%td|ddS)Nrz expected at least 2D input (got rrrPs r0rRzSyncBatchNorm._check_input_dims< 99;;??U UUUVV V ?r2cV|ddkrtddS)Nr rz9SyncBatchNorm number of input channels should be non-zero)sizerrPs r0_check_non_zero_input_channelsz,SyncBatchNorm._check_non_zero_input_channelss4 ::a==A  K   r2rQc R|||||jd}n|j}|jrU|jrN|jJ|jd|jd|jz }n|j} |jrd}n|jduo|j du} |jr|jr|jnd}|jr|jr|j nd}|oB|jo;tj otj }|r|jjdtjfvr.t%dtjtj jj}|jr|j}tj |}|dk}|s*t/j||||j|j|||jS|sJt9j||j|j|||j||| S)Nrwr rxTcudaz4SyncBatchNorm expected input tensor to be on GPU or )rRrrryrr(rzitemr&r'r6 distributed is_availableis_initializedr"type_C_get_privateuse1_backend_namergroupWORLDrget_world_sizer{r|r$r%rsync_batch_normapply) rArQr}r~r&r' need_syncr world_sizes r0rzSyncBatchNorm.forwards e$$$ ++E222 = ), & &)- & = ;T5 ;+777  $ ) )! , , ,}$-043K3P3P3R3R-R**-1]*  = UKK,4T4;Kt;SK &*] Xd6N XD  TX %)M WT5M WD  SW   3  3!..00 3!0022   '| 6688)!Bx==??BB "-39M! 3 $ 2 *99-HHJ"QI <  *     ;"(  *   r2c|}t|tjjjjrtj|j|j|j |j |j |}|j rCtj 5|j |_ |j|_dddn #1swxYwY|j|_|j|_|j|_|j|_t'|dr |j|_|D]/\}}|||||0~|S)aaConverts all :attr:`BatchNorm*D` layers in the model to :class:`torch.nn.SyncBatchNorm` layers. Args: module (nn.Module): module containing one or more :attr:`BatchNorm*D` layers process_group (optional): process group to scope synchronization, default is the whole world Returns: The original :attr:`module` with the converted :class:`torch.nn.SyncBatchNorm` layers. If the original :attr:`module` is a :attr:`BatchNorm*D` layer, a new :class:`torch.nn.SyncBatchNorm` layer object will be returned instead. Example:: >>> # Network with nn.BatchNorm layer >>> # xdoctest: +REQUIRES(env:TORCH_DOCTEST_CUDA) >>> module = torch.nn.Sequential( >>> torch.nn.Linear(20, 100), >>> torch.nn.BatchNorm1d(100), >>> ).cuda() >>> # creating process group (optional) >>> # ranks is a list of int identifying rank ids. >>> ranks = list(range(8)) >>> r1, r2 = ranks[:4], ranks[4:] >>> # Note: every rank calls into new_group for every >>> # process group created, even if that rank is not >>> # part of the group. >>> # xdoctest: +SKIP("distributed") >>> process_groups = [torch.distributed.new_group(pids) for pids in [r1, r2]] >>> process_group = process_groups[0 if dist.get_rank() <= 3 else 1] >>> sync_bn_module = torch.nn.SyncBatchNorm.convert_sync_batchnorm(module, process_group) Nqconfig)rr6nnmodules batchnormrsr rrrrrno_gradr$r%r&r'r(ryhasattrrnamed_children add_moduleconvert_sync_batchnorm)clsmoduler module_outputnamechilds r0rz$SyncBatchNorm.convert_sync_batchnorm6sH feh.8C D D 7!H22#  * M} 5]__55+1=M()/M&555555555555555*0)rs ********UUUUUUUUUU888888!!!!!!   s s s s s s s s l> > > > > > > > B9$9$9$9$9$OY9$9$9$xITITITITIT*ITITITXTTTTTmZTTTDJNJNJNJNJN*JNJNJNZNNNNNmZNNNDJNJNJNJNJN*JNJNJNZNNNNNmZNNND[[[[[J[[[[[r2