˲j&p"ddlZddlmZddlmZmZmZmZmZm Z ddl Z ddl m Z m Z ddlmZmZmZmZmZmZmZmZmZmZgdZeddZGd d e jZ dd e jd edefdZGddeZGddeZ dS)N) namedtuple)AnyCallableDictListOptionalTuple))sparse_semi_structured_from_dense_cutlass'sparse_semi_structured_to_dense_cutlass) fallback_dispatchersemi_sparse_addmmsemi_sparse_detachsemi_sparse_indicessemi_sparse_linearsemi_sparse_mmsemi_sparse_scaled_mm semi_sparse_tsemi_sparse_valuessemi_sparse_view)SparseSemiStructuredTensor!SparseSemiStructuredTensorCUTLASS$SparseSemiStructuredTensorCUSPARSELTto_sparse_semi_structured_SEMI_STRUCTURED_SPARSE_CONFIGz=sparse_min_rows sparse_min_cols dense_min_rows dense_min_colsceZdZUdZdZeed<eej e fed<dZ e ed<dZ e ed<dZe ed<eed <eeefed <eejed <eejed <eejed <eejed<eejed<e ed<eed<gdZe d(dejd eejd eejd eejdeejdeejde dede fdZdefdZdeeeeeje ee fffdZedeeje ee fdejfdZejjZede fdZ!ed)d*dZ"edejddfdZ#ed ejdejfd!Z$d"Z%edejddfd#Z&dd$d%ejd&eejdejfd'Z'dS)+ra This class implementes semi-structured sparsity as a Tensor subclass. Semi-structured sparsity describes a sparsity pattern where n in every 2n elements are sparse, depending on the datatype. It is also referred to as 2:4 sparsity or fine-grained structured sparsity. There are two backends available for semi_structred sparsity, either cuSPARSELt or CUTLASS. This class is meant to serve as a base class for both implementations. SparseSemiStructuredCUTLASS and SparseSemiStructuredCUSPARSELT both inherit from this class and define three backend-specific items. Note that as such, this class cannot be insantiated directly. -`_DTYPE_SHAPE_CONSTRAINTS` - A dictionary holding backend specific dense/sparse min shape constraints - `def from_dense()` - backend specific compression routines - `def _mm()` - backend specifc mm op (either torch._cslt_sparse_mm or torch._sparse_semi_structured_(mm|addmm)) r_DEFAULT_ALG_ID_DTYPE_SHAPE_CONSTRAINTSF_FORCE_CUTLASS_FUSE_TRANSPOSE_PROTOTYPE_WARNING_SHOWNBACKENDSPARSE_DISPATCHpackedmetapacked_tmeta_tcompressed_swizzled_bitmaskfuse_transpose_cusparseltalg_id_cusparselt)r#r$r%r&r'shape requires_gradc |jsTtjdtd|_|t j|||} n||} ntd| j | j | j | d} t j j ||fi| } || _|| _|| _|| _|| _|| _|| _| S)a0 Create a new instance of the tensor subclass from the compressed sparse representation. We have the option to create the subclass with the compressed representations of both X and X', for training. For inference, we only need a single representation (either X or X'), while the corresponding other set will be None. Depending on the backend selected, certain fields will be set to None. (CUSPARSELT vs CUTLASS) Args: shape: The shape of the original dense tensor packed: The compressed representation of the original dense tensor meta: The metadata of the original dense tensor, if it is stored separately packed_t: The compressed representation of the transposed original dense tensor meta_t: The metadata of the transposed original dense tensor, if it is stored separately compressed_swizzled_bitmask: The masks used by the CUTLASS backend to determine which threads should participate in the computation. Used for pointwise ops. fuse_transpose_cusparselt: When running with cuSPARSELt, we have the option to fuse a transposition with a matmul, which is useful in the case of 2:4 sparse training. alg_id_cusparselt: The algorithm id to use when using cuSPARSELT, will have effect on performance Returns: torch.Tensor: A torch.Tensor wrapper subclass. Raises: ValueError: If all of the tensor arguments are None. zThe PyTorch API of SparseSemiStructuredTensor is in prototype stage and will change in the near future. Please open a Github issue for features requests and see our documentation on the torch.sparse module for further information about the project.TNz3At least one of packed or packed_t must be provided)devicedtypelayoutr+)r warningswarn UserWarning_load_dispatch_tabletorch_dynamoallow_in_graph ValueErrorr-r.r/Tensor_make_wrapper_subclassr#r$r%r&r'r(r)) clsr*r#r$r%r&r'r(r)r+previous_tensorkwargstensors MC:\Projects\Historical\venv\Lib\site-packages\torch/sparse/semi_structured.py__new__z"SparseSemiStructuredTensor.__new__KsN+ . MH    ,0C (  $ $ & & & M ( ( - - -  $OO  !&OORSS S&,$*%,*    4S%JJ6JJ  " -H*+D(#4  returncTt|dsJ|jjd|jdS)Nr*z(shape=))hasattr __class____name__r*)selfs r>__repr__z#SparseSemiStructuredTensor.__repr__s6tW%%%%%.)??$*????r@cttfdj}jjjjf}||fS)Nc(t|duSN)getattr)xrGs r>z?SparseSemiStructuredTensor.__tensor_flatten__..sWT1--T9r@)listfilter __slots__r*r(r)r+)rG inner_tensors tensor_metas` r>__tensor_flatten__z-SparseSemiStructuredTensor.__tensor_flatten__sZ 99994> J J   J  *  "    k))r@rSc |\}}}}|||dd|dd|dd|dd|dd||| S)Nr#r$r%r&r' r*r#r$r%r&r'r(r)r+)get) r:rRrS outer_size outer_strider*r(r)r+s r>__tensor_unflatten__z/SparseSemiStructuredTensor.__tensor_unflatten__sNYJ(*;]s $$Xt44""6400"&&z488 $$Xt44(5(9(9-t))'@/'    r@c|j|jvrt|jd|jd|j|j||||S)NzI only supports a specific set of operations, can't perform requested op (rC)_overloadpacketr"NotImplementedErrorrF)r:functypesargsr<s r>__torch_dispatch__z-SparseSemiStructuredTensor.__torch_dispatch__sp  s': : :%<@@/3}@@@ 9s"4#78udFSSSr@Nc6t|ddtjjjt tjjjttjjjttjjj ttjjj ttjjj ttjjjt tjjjt$tjjjt$tjjjt*tjjjt.tjjjttjjjt4i |_||j|dSdSdS)zT Loads the op overload sparse dispatch table for the current class. r"N)rLr4opsatenvaluesrindicesr is_same_sizer detach_detachrtrviewrmmrmatmuladdmmr linearr_to_copy _scaled_mmrr"update)r:custom_dispatch_tables r>r3z/SparseSemiStructuredTensor._load_dispatch_tables 3)4 0 0 8 %'9 &(; +-@ &(; %'9  - #%5 !> %~ $&7 %'9 ')< )+@#C %0#**+@AAAAA# 9 8 10r@original_tensorc .|jstd|jd|dkr%td|d|std|j|jvrtd|jd|d |j\}}|j|jj}|j|jj }||ks||zs ||ks||zrtd |jd |d |d dS)z_ Assert that the given tensor is valid for semi-structured sparse compression. zError original_tensor.device= z= is not supported! Only CUDA tensors are currently supported.zError original_tensor.dim = z; is not supported! Only 2d tensors are currently supported.zXError original_tensor is not contiguous!Only contiguous tensors are currently supported.zError original_tensor.dtype z is not a supported dtype for !zError original_tensor.shape zS is not supported! Both dimensions must be larger or equal than and a multiple of (z, rCN) is_cuda RuntimeErrorr-dim is_contiguousr.rr*sparse_min_rowssparse_min_cols)r:rtmnmin_rowsmin_colss r> _validate_device_dim_dtype_shapez;SparseSemiStructuredTensor._validate_device_dim_dtype_shapes & =1G===     A % %;/B/B/D/D;;;  ,,.. C   (D D Dj/Djjdgjjj  $1/0EFV/0EFV x<<1x<<1x<<1x<<k/DkkS[kk_gkkk ,8_pad_dense_inputz+SparseSemiStructuredTensor._pad_dense_inputs  A%%%% 1/ 0ABQ/ 0ABQ%&LLALLA2==a$%LLALLA2==a  x 8&**;Ha8RSS S r@c|jd}tj|tj||j|jS)N)r.r-)r*r4rleyer.r-)rGcols r>to_densez#SparseSemiStructuredTensor.to_dense+s4jnxei4:dkRRRSSSr@ctrKr]r:rts r> from_densez%SparseSemiStructuredTensor.from_dense/s!!r@biasBrc trKr)rGrrr<s r>_mmzSparseSemiStructuredTensor._mm3s "!r@)FrFrK)rAN)(rF __module__ __qualname____doc__rint__annotations__rr4r.rrboolrr strrrr8rQ staticmethodSizer?rHr rrT classmethodrZ_C_disabled_torch_function_impl__torch_function__rrar3rrrrrr@r>rr&s"OS"5;0N#NOOOO ND   !OT!!!%*d*** LLL(H,---- U\ """" 5<    u|$$$$ U\ """"!)%,!7777####WWWI+0!"#PPzP&Pu|$ P 5<( P & P&.el%;P$(PPPPP\Pd@#@@@@ * tCy% D#t ;<< = * * * * 5:tS$67     [ ,?TcTTT[TBBBB[B.(u|(PT((([(T5<EL[*TTT"":V"""["(, """ <"u|$ "  """"""r@rFrt transposedrAc|rtjdtdtjrt jjnt jj}| |S)a This function converts a dense tensor into a sparse semi-structured tensor. It will return a SparseSemiStructuredTensor, a subclass of torch.Tensor. This function will check to ensure the dense tensor has the right dtype, size, dims, and device. We currently only support semi-structured sparse tensors for 2d CUDA tensors. Additionally, your tensor must be a positive multiple of the mininum sparse block size, given in `_DTYPE_TO_SHAPE_CONSTRAINTS` for each dtype (float32, float16, bfloat16, int8). Args: original_tensor (Tensor): the dense tensor to convert transposed (bool, optional): deprecated arg to be removed in another release. Do not use. Returns: SparseSemiStructuredTensor: A sparse semi-structured tensor created from the given original_tensor Raises: None Example: >>> # xdoctest: +REQUIRES(env:TORCH_DOCTEST_CUDA) >>> A = torch.Tensor([0, 0, 1, 1]).tile((128, 32)).half().cuda() tensor([[0., 0., 1., ..., 0., 1., 1.], [0., 0., 1., ..., 0., 1., 1.], [0., 0., 1., ..., 0., 1., 1.], ..., [0., 0., 1., ..., 0., 1., 1.], [0., 0., 1., ..., 0., 1., 1.], [0., 0., 1., ..., 0., 1., 1.]], device='cuda:0', dtype=torch.float16) >>> A_sparse = to_sparse_semi_structured(A) SparseSemiStructuredTensor(shape=torch.Size([128, 128])) >>> A_sparse.values() tensor([[1., 1., 1., ..., 1., 1., 1.], [1., 1., 1., ..., 1., 1., 1.], [1., 1., 1., ..., 1., 1., 1.], ..., [1., 1., 1., ..., 1., 1., 1.], [1., 1., 1., ..., 1., 1., 1.], [1., 1., 1., ..., 1., 1., 1.]], device='cuda:0', dtype=torch.float16), >>> A_sparse.indices() tensor([[-4370, -4370, -4370, ..., -4370, -4370, -4370], [-4370, -4370, -4370, ..., -4370, -4370, -4370], [-4370, -4370, -4370, ..., -4370, -4370, -4370], ..., [-4370, -4370, -4370, ..., -4370, -4370, -4370], [-4370, -4370, -4370, ..., -4370, -4370, -4370], [-4370, -4370, -4370, ..., -4370, -4370, -4370]], device='cuda:0', dtype=torch.int16)) zSetting transpose from `to_sparse_semi_structured` is deprecated and will be removed in a future release. `SparseSemiStructuredTensor` only support contiguous input tensors.rv) stacklevel) r0r1 FutureWarningrrr4sparserrr)rtrSPARSE_SUBCLASSs r>rr=snb   R       & 4 ? 66 \ >  % %o 6 66r@c ^eZdZdZdZejeddddejeddddej eddddej eddddiZ e d ej d dfd Zfd Ze dd ej d dfdZdddej deej d ej fdZxZS)ra This class implements semi-structured sparsity for the CUTLASS backend. In this implementation, the specified elements and metadata are stored seprately, in packed and meta respectively. When _FORCE_CUTLASS is set, or when cuSPARSELt is not available, this subclass calls into _sparse_semi_structured_(mm|addmm) and sparse_semi_structured_from_dense for conversion to the compressed format. cutlass @rtrAc ||t|\}}||j||ddd|jS)Nr#r$r%r&r'r+)rr r*r+)r:rtsparse_tensor_cutlassmeta_tensor_cutlasss r>rz,SparseSemiStructuredTensorCUTLASS.from_densesd ,,_=== 6o F F ! s  !($(,)7    r@c|j|jJ|jjdkrt|j|jnt S)Nrv)r$r#ndimr superr)rGrEs r>rz*SparseSemiStructuredTensorCUTLASS.to_densesay$)@)@@ y~"" 4      !!##  r@rc jtj||d\}}}}}||j|||||dS)a This function takes in a unpruned dense tensor and runs a (branchless) static sort across a 4x4 tile. It greedily picks the largest values in the tile, upholding the 2:4 sparsity constraint across both rows and columns. The algorithm used to prune the matrix is implemented in `_sparse_semi_structured_tile`. Then it creates the packed and meta tensors for the compressed sparse representation of the pruned dense tensor. It also calculates the packed_t and meta_t tensors for the compressed sparse representation of the transposed pruned dense tensor. Since we cannot transpose the compressed representations, we store both for the fw/bw pass respectively. Finally, this function also computes a compressed swizzled bitmask that encodes the sparsity pattern This can be used in the backward pass to mask the gradients. [9 1 7 4] [9 0 7 0] [1 2 3 0] [0 2 0 0] [8 3 5 4] -> prune 4x4 tile -> [8 0 0 4] -> pack to CUTLASS semi-structured -> packed [1 2 6 2] [0 0 6 2] -> metadata -> pack to transposed CUTLASS -> packed_t semi-structured representation -> metadata_t -> compute swizzled bitmask -> compressed_swizzled_bitmask The equivalent PyTorch code to create the same five outputs from the dense tensor can be found below: ``` from torch.sparse import SparseSemiStructuredTensorCUTLASS from torch.sparse._semi_structured_conversions import _sparse_semi_structured_tile, _compute_compressed_swizzled_bitmask pruned = _sparse_semi_structured_tile(dense) packed_cutlass, meta_cutlass = sparse_semi_structured_from_dense_cutlass(pruned) packed_t_cutlass, meta_t_cutlass = sparse_semi_structured_from_dense_cutlass(pruned.t().contiguous()) bitmask = _compute_compressed_swizzled_bitmask(pruned) SparseSemiStructuredTensorCUTLASS(dense.shape, packed_cutlass, meta_cutlass, packed_t_cutlass, meta_t_cutlass, bitmask) ``` T algorithm use_cutlassFrr4_sparse_semi_structured_tiler*r:rtrr#r$r%r&r's r>prune_dense_static_sortz9SparseSemiStructuredTensorCUTLASS.prune_dense_static_sortshb  . yd        ' s  !(C    r@Nrrrc t|trtd|jj}|jdks |jdkrt d|d|j|jt d|d|!tj |j|j|}n!tj ||j|j|}|d|j dS)NZ`SparseSemiStructuredTensor @ SparseSemiStructuredTensor` is not supported by the hardwarerv`)` matmul: Broadcasting is not implemented$` matmul: operation is not supportedr) isinstancerr7rErFrr]r#r$r4_sparse_semi_structured_mm_sparse_semi_structured_addmmr*)rGrrr<cls_nameress r>rz%SparseSemiStructuredTensorCUTLASS._mms a3 4 4 l >* 9>>QVq[[%GHGGG  ; $)"3%BHBBB |6t{DIqQQ9$+ty!A' 'r@r)rFrrrr!r4int8rfloat16bfloat16float32rrr8rrrrr __classcell__)rEs@r>rrsj  G 222sBCC 55b"aCC 66r2q!DD 55b"aCC   #l ,   [ $      68< < #l< %< < < [< ~BF(((((0(>( ((((((((r@rc NeZdZdZdZejeddddejeddddej eddddej eddddiZ e dej ddfdZe ddej dd fd Zd d dej deej dej fdZd S)ra The cuSPARSELt backend expects the specified elements and the metadata to be stored in a single tensor: packed = [ specified elements of original tensor | metadata ] For an original tensor of size (m, k) we expect the first m * k // 2 elements to be the kept elements The rest of the tensor is metadata. Since there is only one tensor, we only use the packed and packed_t attributes respectively. cuSPARSELt also supports transposition fusion, which is necessary for performant 2:4 sparse training, as well as specifying alg_id, a config that affects the performance of the matmul depending on matmul sizes. cusparseltrrrrtrAc ||||jtj|ddddtjtj|j S)NrV)rr*r4_cslt_compressrrrr+rs r>rz/SparseSemiStructuredTensorCUSPARSELT.from_dense sc ,,_===s!''88(,&@&P8H)7    r@rrc jtj||d\}}}}}||j|||||dS)a This function does the same thing as described in SparseSemiStructuredCUTLASS, but uses the cuSPASRELt metadata layout and sparse matmul. The only functional difference is that cuSPARSELt stores `metadata` and `packed` together into a single tensor. [9 1 7 4] [9 0 7 0] [1 2 3 0] [0 2 0 0] [8 3 5 4] -> prune 4x4 tile -> [8 0 0 4] -> pack to cuSPARSELT semi-structured -> packed [1 2 6 2] [0 0 6 2] -> pack to transposed cuSPARSELt -> packed_t semi-structured representation -> compute swizzled bitmask -> compressed_swizzled_bitmask The equivalent PyTorch code to create the same three outputs from the dense tensor can be found below: ``` from torch.sparse import SparseSemiStructuredTensorCUSPARSELT from torch.sparse._semi_structured_conversions import _sparse_semi_structured_tile, _compute_compressed_swizzled_bitmask pruned = _sparse_semi_structured_tile(dense) packed_cusparselt = torch._cslt_compress(pruned) packed_t_cusparselt = torch._cslt_compress(pruned.t().contiguous()) bitmask = _compute_compressed_swizzled_bitmask(pruned) SparseSemiStructuredTensorCUSPARSELT(dense.shape, packed_cutlass, None, packed_t_cutlass, None, bitmask) ``` Frrrrs r>rzrz(SparseSemiStructuredTensorCUSPARSELT._mmgs a3 4 4 l  9>>QVq[[%VDN+VVV  7dj %YDN+YYuTZGXGXYY_defel_m_mYY $ YY9:YYY    dj 8 8%\DN+\\uTZGXGX\\_defel_m_m\\(, \\AB\\\  :, , ,%`DN+``uTZGXGX``_defel_m_m``(, ```  ; %QDN+QQQ ' !%!?- C#<E355777# Er@r)rFrrrr!r4rrrrrrrr8rrrrrr@r>rr sR  G ;;BBKK 222r2rBB 55b"aCC 66r2q!DD   #l /   [ 683 3 #l3 %3 3 3 [3 lBF*F*F*F*F(0(>*F *F*F*F*F*F*Fr@r)F)!r0 collectionsrtypingrrrrrr r4)torch.sparse._semi_structured_conversionsr r !torch.sparse._semi_structured_opsr r rrrrrrrr__all__rr8rrrrrrr@r>rs""""""================                            ",$C"" T"T"T"T"T"T"T"T"rA7A7\A7A7 A7A7A7A7HH(H(H(H(H((BH(H(H(VEFEFEFEFEF+EEFEFEFEFEFr@