̲j] ddlZddlZddlmZmZmZmZddlmZddl Z ddl m Z m Z ddl mZmZmZgdZdedeefd Zded eedeefd Zd edefd Ze jfd edededefdZd edefdZ dd edeedefdZeded ededefdZd edefdZd ede jdefdZdS)N)AnyListOptionalSet) deprecated)_ShardingIterDataPipeSHARDING_PRIORITIES)DataPipe DataPipeGraph traverse_dps)apply_random_seedapply_shardingapply_shuffle_seedapply_shuffle_settingsget_all_graph_pipesgraphreturnc:t|tSN)_get_all_graph_pipes_helperset)rs PC:\Projects\Historical\venv\Lib\site-packages\torch/utils/data/graph_settings.pyrrs &ucee 4 44id_cachecg}|D]Z\}\}}||vr |||||t ||[|Sr)itemsaddappendextendr)rrresultsdp_iddatapipe sub_graphs rrrs!G(- II$$) H    Ux   29hGGHHHH Nrr"c~t|tp(t|dotj|jS)Nr) isinstancerhasattrinspectismethodrr"s r_is_sharding_datapiper*)s> h 5 6 6 *++ 6  X4 5 5rnum_of_instances instance_idcTt|}dfd ||S)z Apply dynamic sharding over the ``sharding_filter`` DataPipe that has a method ``apply_sharding``. RuntimeError will be raised when multiple ``sharding_filter`` are presented in the same branch. Ncp|D]\}}d}t|ry|td|d|tj|j}t |jdkr|n| |}||}||dS)NzoSharding twice on a single pipeline is likely unintended and will cause data loss. Sharding already applied to z while trying to apply to )sharding_group)valuesr* RuntimeErrorr' signaturerlen parameters) r prev_applieddpr#appliedsig_helperr,r+r0s rr:zapply_sharding.._helper=s"\\^^ ( (MB G$R(( +&d7Cdd_add '(9::s~&&**%%&6 DDDD%%(+n&& GIw ' ' ' '' ( (rr)r )r"r+r,r0rr:s ``` @rrr0sV  " "E(((((((((, GENNN Orct|doAt|do1tj|jotj|jS)N set_shuffleset_seed)r&r'r(r<r=r)s r_is_shuffle_datapiper>XsT-(( 0 Hj ) ) 0  X1 2 2 0  X. / / rshufflec||St|}t|}d|D}|s-|r+tjd|}|g}|D]}|||S)aE Traverse the graph of ``DataPipes`` to find and set shuffle attribute. Apply the method to each `DataPipe` that has APIs of ``set_shuffle`` and ``set_seed``. Args: datapipe: DataPipe that needs to set shuffle attribute shuffle: Shuffle option (default: ``None`` and no-op to the graph) Nc0g|]}t||S)r>).0pipes r z*apply_shuffle_settings..ss&JJJ$/CD/I/IJJJJrz`shuffle=True` was set, but the datapipe does not contain a `Shuffler`. Adding one at the end. Be aware that the default buffer size might not be sufficient for your task.)r rwarningswarnr?r<)r"r?r all_pipes shufflersshufflers rrras  " "E#E**IJJ)JJJI     [   ##%%  &&W%%%% Orz`apply_shuffle_seed` is deprecated since 1.12 and will be removed in the future releases. Please use `apply_random_seed` instead.)categoryrngc"t||Sr)r )r"rLs rrrs Xs + ++rcTt|dotj|jS)Nr=)r&r'r(r=r)s r_is_random_datapiperOs% 8Z ( ( PW-=h>O-P-PPrct|}t|}t}g}|D]Z}t||vrt |r7|||t|[|D]j}ttj dtj  | }| |k|S)a6 Traverse the graph of ``DataPipes`` to find random ``DataPipe`` with an API of ``set_seed``. Then set the random seed based on the provided RNG to those ``DataPipe``. Args: datapipe: DataPipe that needs to set randomness rng: Random number generator to generate random seeds rB)dtype) generator)r rridrOrrinttorchemptyint64random_itemr=)r"rLrrHcacherandom_datapipesrD random_seeds rr r s  " "E#E**I EEE   d88u    t $ $  # #D ) ) ) IIbhh    ## K%+ . . . 6 6 6 E E J J L L   k"""" Orr) r'rFtypingrrrrtyping_extensionsrrU(torch.utils.data.datapipes.iter.shardingrr torch.utils.data.graphr r r __all__rrTrboolr*DEFAULTrr>r FutureWarningrrO Generatorr rBrrrfs}++++++++++++(((((( IHHHHHHHHH   5}5h5555   $'H  (^    H'. %%%%%  %%%%P837   !)$      F .  ,,,,,,  ,Q(QtQQQQu8r