@j=ddlZddlmZmZmZddlZddlmcmZ ddlmZm Z gdZ Gddej Z Gddej ZGd d ej ZGd d ej ZGd dej ZdS)N)ListOptionalTuple)nnTensor)ResBlock MelResNet Stretch2dUpsampleNetworkWaveRNNc>eZdZdZd deddffd ZdedefdZxZS) rafResNet block based on *Efficient Neural Audio Synthesis* :cite:`kalchbrenner2018efficient`. Args: n_freq: the number of bins in a spectrogram. (Default: ``128``) Examples >>> resblock = ResBlock() >>> input = torch.rand(10, 128, 512) # a random spectrogram >>> output = resblock(input) # shape: (10, 128, 512) n_freqreturnNc Httjtj||ddtj|tjdtj||ddtj||_dS)NF in_channels out_channels kernel_sizebiasTinplace)super__init__r SequentialConv1d BatchNorm1dReLUresblock_model)selfr __class__s JC:\Projects\Historical\venv\Lib\site-packages\torchaudio/models/wavernn.pyrzResBlock.__init__s  m I&v1SX Y Y Y N6 " " GD ! ! ! I&v1SX Y Y Y N6 " "   specgramc2|||zS)zPass the input through the ResBlock layer. Args: specgram (Tensor): the input sequence to the ResBlock layer (n_batch, n_freq, n_time). Return: Tensor shape: (n_batch, n_freq, n_time) )r r!r%s r#forwardzResBlock.forward(s""8,,x77r$)r __name__ __module__ __qualname____doc__intrrr( __classcell__r"s@r#rrs|     s  T        8 86 8 8 8 8 8 8 8 8r$rc PeZdZdZ ddedededed ed d f fd Zd ed efdZxZS)r aMelResNet layer uses a stack of ResBlocks on spectrogram. Args: n_res_block: the number of ResBlock in stack. (Default: ``10``) n_freq: the number of bins in a spectrogram. (Default: ``128``) n_hidden: the number of hidden dimensions of resblock. (Default: ``128``) n_output: the number of output dimensions of melresnet. (Default: ``128``) kernel_size: the number of kernel size in the first Conv1d layer. (Default: ``5``) Examples >>> melresnet = MelResNet() >>> input = torch.rand(10, 128, 512) # a random spectrogram >>> output = melresnet(input) # shape: (10, 128, 508) r n_res_blockrn_hiddenn_outputrrNcVtfdt|D}tjtj||dtjtjdg|tj|dR|_dS)Nc.g|]}tS)r).0_r5s r# z&MelResNet.__init__..Is!DDDAXh''DDDr$FrTrr)rrr) rrrangerrrrrmelresnet_model)r!r4rr5r6r ResBlocksr"s ` r#rzMelResNet.__init__Ds DDDD{1C1CDDD !} I&x[_d e e e N8 $ $ GD ! ! !   I(q Q Q Q    r$r%c,||S)zPass the input through the MelResNet layer. Args: specgram (Tensor): the input sequence to the MelResNet layer (n_batch, n_freq, n_time). Return: Tensor shape: (n_batch, n_output, n_time - kernel_size + 1) )r>r's r#r(zMelResNet.forwardSs##H---r$r2rrrr3r)r0s@r#r r 4s   vw     -0  BE  WZ  or          . .6 . . . . . . . .r$r c@eZdZdZdededdffd ZdedefdZxZS) r aUpscale the frequency and time dimensions of a spectrogram. Args: time_scale: the scale factor in time dimension freq_scale: the scale factor in frequency dimension Examples >>> stretch2d = Stretch2d(time_scale=10, freq_scale=5) >>> input = torch.rand(10, 100, 512) # a random spectrogram >>> output = stretch2d(input) # shape: (10, 500, 5120) time_scale freq_scalerNcdt||_||_dSN)rrrDrC)r!rCrDr"s r#rzStretch2d.__init__ms+ $$r$r%cj||jd|jdS)zPass the input through the Stretch2d layer. Args: specgram (Tensor): the input sequence to the Stretch2d layer (..., n_freq, n_time). Return: Tensor shape: (..., n_freq * freq_scale, n_time * time_scale) )repeat_interleaverDrCr's r#r(zStretch2d.forwardss1))$/2>>PPQUQ`bdeeer$r)r0s@r#r r _s  %3%C%D%%%%%% f f6 f f f f f f f fr$r cxeZdZdZ ddeedededed ed ed d ffd Zded eeeffdZ xZ S)r aUpscale the dimensions of a spectrogram. Args: upsample_scales: the list of upsample scales. n_res_block: the number of ResBlock in stack. (Default: ``10``) n_freq: the number of bins in a spectrogram. (Default: ``128``) n_hidden: the number of hidden dimensions of resblock. (Default: ``128``) n_output: the number of output dimensions of melresnet. (Default: ``128``) kernel_size: the number of kernel size in the first Conv1d layer. (Default: ``5``) Examples >>> upsamplenetwork = UpsampleNetwork(upsample_scales=[4, 4, 16]) >>> input = torch.rand(10, 128, 10) # a random spectrogram >>> output = upsamplenetwork(input) # shape: (10, 128, 1536), (10, 128, 1536) r2rr3upsample_scalesr4rr5r6rrNc<td}|D]}||z}||_|dz dz|z|_t ||||||_t |d|_g} |D]} t | d} tj ddd| dzdzfd| fd} tjj | j d| dzdzz | | | | tj| |_dS)NrrF)rrrpaddingr?)rr total_scaleindentr resnetr resnet_stretchrConv2dtorchinit constant_weightappendrupsample_layers)r!rLr4rr5r6rrQupsample_scale up_layersscalestretchconvr"s r#rzUpsampleNetwork.__init__sE  - * *N > )KK +"Q1,{:  VXxUU ' Q77 $ # #Eq))G9AAuqy1};MXY[`WahmD HM # #DK A 1F G G G   W % % %   T " " " "!}i8r$r%c`||d}||}|d}|d}||}|ddddd|j|j f}||fS)aPass the input through the UpsampleNetwork layer. Args: specgram (Tensor): the input sequence to the UpsampleNetwork layer (n_batch, n_freq, n_time) Return: Tensor shape: (n_batch, n_freq, (n_time - kernel_size + 1) * total_scale), (n_batch, n_output, (n_time - kernel_size + 1) * total_scale) where total_scale is the product of all elements in upsample_scales. rN)rS unsqueezerTsqueezer[rR)r!r% resnet_outputupsampling_outputs r#r(zUpsampleNetwork.forwards H--77:: ++M:: %--a00 %%a(( 00::-55a88AAAt{dk\?Y9YZ -//r$rA) r*r+r,r-rr.rrrr(r/r0s@r#r r s&99c99 9  9  99 999999>005+@00000000r$r ceZdZdZ ddeededed ed ed ed ed edededdffd ZdededefdZe j j ddede ede ee effdZxZS)r aWWaveRNN model from *Efficient Neural Audio Synthesis* :cite:`wavernn` based on the implementation from `fatchord/WaveRNN `_. The original implementation was introduced in *Efficient Neural Audio Synthesis* :cite:`kalchbrenner2018efficient`. The input channels of waveform and spectrogram have to be 1. The product of `upsample_scales` must equal `hop_length`. See Also: * `Training example `__ * :class:`torchaudio.pipelines.Tacotron2TTSBundle`: TTS pipeline with pretrained model. Args: upsample_scales: the list of upsample scales. n_classes: the number of output classes. hop_length: the number of samples between the starts of consecutive frames. n_res_block: the number of ResBlock in stack. (Default: ``10``) n_rnn: the dimension of RNN layer. (Default: ``512``) n_fc: the dimension of fully connected layer. (Default: ``512``) kernel_size: the number of kernel size in the first Conv1d layer. (Default: ``5``) n_freq: the number of bins in a spectrogram. (Default: ``128``) n_hidden: the number of hidden dimensions of resblock. (Default: ``128``) n_output: the number of output dimensions of melresnet. (Default: ``128``) Example >>> wavernn = WaveRNN(upsample_scales=[5,5,8], n_classes=512, hop_length=200) >>> waveform, sample_rate = torchaudio.load(file) >>> # waveform shape: (n_batch, n_channel, (n_time - kernel_size + 1) * hop_length) >>> specgram = MelSpectrogram(sample_rate)(waveform) # shape: (n_batch, n_channel, n_freq, n_time) >>> output = wavernn(waveform, specgram) >>> # output shape: (n_batch, n_channel, (n_time - kernel_size + 1) * hop_length, n_classes) r2r3rrL n_classes hop_lengthr4n_rnnn_fcrrr5r6rNc t||_|dzr|dz n|dz|_||_| dz|_||_||_ttj |j|_ d} |D]} | | z} | |jkrtd| d|t|||| | ||_tj||jzdz||_tj||d|_tj||jz|d|_tjd|_tjd|_tj||jz||_tj||jz||_tj||j|_dS) NrNrz/Expected: total_scale == hop_length, but found z != T) batch_firstr)rrr_padrjn_auxrirhr.mathlog2n_bits ValueErrorr upsamplerLinearfcGRUrnn1rnn2rrelu1relu2fc1fc2fc3)r!rLrhrir4rjrkrrr5r6rQr\r"s r#rzWaveRNN.__init__s &(3aH[1__[QN  ] $"ty8899  - * *N > )KK $/ ) )l{ll`jllmm m'fhX`bmnn )FTZ/!3U;;F5%T::: F54:-u$GGG WT*** WT*** 9UTZ/669TDJ.559T4>22r$waveformr%c|ddkrtd|ddkrtd|d|d}}|d}tjd|j|j|j}tjd|j|j|j}|\}}| dd}| dd}fdtdD}|d d d d |d|df}|d d d d |d|df} |d d d d |d|d f} |d d d d |d |d f} tj | d ||gd } | } | } | |\} }| | z} | } tj | | gd } | |\} }| | z} tj | | gd } | } | } tj | | gd } | } | } | } | dS)aPass the input through the WaveRNN model. Args: waveform: the input waveform to the WaveRNN layer (n_batch, 1, (n_time - kernel_size + 1) * hop_length) specgram: the input spectrogram to the WaveRNN layer (n_batch, 1, n_freq, n_time) Return: Tensor: shape (n_batch, 1, (n_time - kernel_size + 1) * hop_length, n_classes) rz*Require the input channel of waveform is 1z*Require the input channel of specgram is 1r)dtypedevicerNc$g|] }j|z Sr9rp)r:ir!s r#r<z#WaveRNN.forward...s444a4:>444r$r3NrmrIdim)sizertrcrVzerosrjrrru transposer=catrbrwryrzr}r{r~r|r)r!rr% batch_sizeh1h2auxaux_idxa1a2a3a4xresr;s` r#r(zWaveRNN.forwards ==  q IJJ J ==  q IJJ J%--a00(2B2B12E2E(]]1%% [J (.QYQ` a a a [J (.QYQ` a a a h// #%%a++mmAq!!4444588444 AAAwqzGAJ.. / AAAwqzGAJ.. / AAAwqzGAJ.. / AAAwqzGAJ.. / Ix))"--x<" E E E GGAJJyyB1 G Iq"g2 & & &yyB1 G Iq"g2 & & & HHQKK JJqMM Iq"g2 & & & HHQKK JJqMM HHQKK{{1~~r$lengthsc|j}|j}tjj|jjf}|\}||jjz}g}| \}}}tj d|j f||} tj d|j f||} tj |df||} fdtdD} t|D]Ԋ|ddddf} fd| D\}}}}tj | | |gd} | } | d| \}} | | dz} tj | |gd}|d| \}} | | dz} tj | |gd} t%j| } tj | |gd} t%j| } | }t%j|d}tj|d} d | zd jzd z z d z } || tj|dd d|fS) aInference method of WaveRNN. This function currently only supports multinomial sampling, which assumes the network is trained on cross entropy loss. Args: specgram (Tensor): Batch of spectrograms. Shape: `(n_batch, n_freq, n_time)`. lengths (Tensor or None, optional): Indicates the valid length of each audio in the batch. Shape: `(batch, )`. When the ``specgram`` contains spectrograms with different durations, by providing ``lengths`` argument, the model will compute the corresponding valid output lengths. If ``None``, it is assumed that all the audio in ``waveforms`` have valid length. Default: ``None``. Returns: (Tensor, Optional[Tensor]): Tensor The inferred waveform of size `(n_batch, 1, n_time)`. 1 stands for a single channel. Tensor or None If ``lengths`` argument was provided, a Tensor of shape `(batch, )` is returned. It indicates the valid length in time axis of the output Tensor. Nr)rrcXg|]&}ddj|zj|dzzddf'S)Nrr)r:rrr!s r#r<z!WaveRNN.infer..xsCXXX!SDJNTZ1q5-AA111DEXXXr$rmc.g|]}|ddddfSrFr9)r:ars r#r<z!WaveRNN.infer..~s+%D%D%DQa111aj%D%D%Dr$rrrNrP)rrrVr functionalpadrorurQrrrjr=rrwryrbrzFrelur}r~rsoftmax multinomialfloatrsrZstackpermute)r!r%rrroutputb_sizer;seq_lenrrr aux_splitm_ta1_ta2_ta3_ta4_tinplogits posteriorrrs` @@r#inferz WaveRNN.inferKs<8&**8di5KLL h// #   99G!%]]__7 [!VTZ0u M M M [!VTZ0u M M M K F% @ @ @XXXXXuUVxxXXX w  A111aaa7#C%D%D%D%D)%D%D%D "D$d 1c4.a000A AIIakk!nnb11EArBqE A)QI1---CIIcmmA..33EArBqE A 1d)+++Atxx{{##A 1d)+++Atxx{{##AXXa[[F &a000I!)Q//5577AADK#-.4A MM!    {6""**1a33W<rs} ((((((((((     8 8 8 8 8ry 8 8 8F(.(.(.(.(. (.(.(.Vfffff fffBD0D0D0D0D0biD0D0D0NR=R=R=R=R=biR=R=R=R=R=r$