[jddlmZddlmZddlmZddlmZddlZddl m Z ddl m Z dd l mZd d lmZd d lmZd d lmZmZmZd dl mZmZmZmZmZGddeZGddZGddZGddZdS)) nullcontext)partial)repeat) TypedDictN)PretrainedConfig)ContinuousBatchingConfig)get_available_devices)PagedAttentionCache)%ContinuousBatchingLogitsProcessorList) TMP_TOKEN_IDFutureRequestStatelogger)CudaGraphBufferaligned_divideattn_mask_is_neededbuild_attention_mask pad_to_pow2c~eZdZUdZejed<ejeeejfzdzed<ejed<ejed<ejeeejfzed<e ed<e eee fzed <e ejed <e ejed <ejed <e ed <ejdzed<ejed<e ed<dS)PagedAttentionArgsaThe keyword arguments for a forward pass using paged attention, passed directly as model forward kwargs. Attributes: input_ids: Input token IDs tensor of shape `(1, total_query_tokens)`. attention_mask: Attention mask tensor or dictionary mapping layer types to masks. Can be `None` if the attention implementation doesn't require explicit masks. position_ids: Position IDs tensor of shape `(1, total_query_tokens)`. cu_seq_lens_q: Cumulative sequence lengths for queries, used for variable-length batching. cu_seq_lens_k: Cumulative sequence lengths for keys/values. Can be a tensor or dictionary mapping layer types (e.g., "full_attention", "sliding_attention") to tensors for hybrid models. max_seqlen_q: Maximum query sequence length in the batch. max_seqlen_k: Maximum key/value sequence length. Can be an int or dictionary for hybrid models. write_index: List of tensors indicating where to write new KV states in the cache, one per attention group. read_index: List of tensors indicating which cache positions to read from, one per attention group. logits_indices: Tensor indicating which positions in the output should be used for next-token prediction. cache: The [`PagedAttentionCache`] instance managing the KV cache. block_table: Block table for paged KV cache. If provided, uses `flash_attn_with_kvcache` for fused attention + cache update. More information in src/transformers/integrations/flash_paged.py logits_processor_args: List of tensors containing the arguments for the logits processors, one per request. use_cache: Whether to use caching (always `False` in continuous batching as the cache is managed externally). input_idsNattention_mask position_ids cu_seq_lens_q cu_seq_lens_k max_seqlen_q max_seqlen_k write_index read_indexlogits_indicescache block_tablelogits_processor_args use_cache) __name__ __module__ __qualname____doc__torchTensor__annotations__dictstrintlistr boolgC:\Projects\DSCons\.venv\Lib\site-packages\transformers\generation\continuous_batching\input_outputs.pyrrs,|L4U\(9#::TAAAA,<<$sEL'8"99999S#X&&&&el####U\""""L   $$$$ <'''OOOOOr3rceZdZUdZdZeed<dedede de j de j d e d d fd Zd e d d fd Z d.ddde jjded d fdZe jd.ded d fdZd/dZd ee jeee jfffdZde jde jde jd d fdZd/dZd eeeeeee d zffdZ!deede ded ed!ed"ed d fd#Z"d"ed efd$Z#d.d"ed e$fd%Z%d ee je je jffd&Z&d eed'ffd(Z'd0d*ed e jj(d zfd+Z)d,e jj(d d fd-Z*d S)1ContinuousBatchingIOszA class to hold inputs and outputs for a continuous batching forward pass, using static tensors as storage. The class is meant to be self-contained, so once a set of inputs have been created, the class can be used to update the batch alone.  static_inputsr"configcontinuous_batching_configdevice model_dtypelogit_processorreturnNc||_||_||_||_|j|_|jd|_t|dddn|j|_|j |_ d|_ d|_ d|_ dt|jD|_dt|jD|_d|_g|_i|_t)|_|j|_|j|_|||d |jd kr%t:j|j nd|_ dS) a3Initialize the continuous batching I/O manager. Args: - cache: The [`PagedAttentionCache`] instance managing the KV cache. Meant to be unique. - config: The model's pretrained configuration. - continuous_batching_config: The continuous batching configuration. - device: The device to allocate tensors on. If the device is CPU, then the memory is pinned. - model_dtype: The data type for model computations. - logit_processor: The [`ContinuousBatchingLogitsProcessorList`] object used to process the logits. rsliding_windowNr cg|]}dSrr2.0_s r4 z2ContinuousBatchingIOs.__init__..lsCCCaCCCr3cg|]}dSrBr2rCs r4rFz2ContinuousBatchingIOs.__init__..ms D D Dq D D Dr3F)r=T full_resetcudar;)!r"r;r9r<max_requests_per_batchcuda_graph_booleansuse_cuda_graph_varlengetattrr@return_logprobs num_q_tokens max_kv_readnum_request_in_batchrange num_groupstrue_read_sizestrue_write_sizesuse_block_tablerequests_in_batchreq_id_to_new_token_positionrgraphsread_trash_index_read_trash_indexwrite_trash_index_write_trash_index_setup_static_tensors_reset_static_tensorstyper*rJStreamcompute_streamselfr"r9r:r;r<r=s r4__init__zContinuousBatchingIOs.__init__Nsc$   &&@&W#%?%STU%V"#*63CT#J#J#RaaX^Xm9I$%!CC51A+B+BCCC D DE%2B,C,C D D D$;=<>)'6'8'8 !&!7"'"9 ""?"CCC ""d"333GM{V\G\G\ej//t{/CCCbfr3c|jj}|jj}|j}|jj|jjz}|jjdkottdk}|j |j z}t|dzdd}tj||ftj|j||_tj|j dftj|j|_||j|jdd|f|_|jdd|f|_|jdd|dzf|_|jd d|f|_|jd d|dzf} |jd d|dzf} |jd d|f|_i|_|jjr | |jd <|jjr | |jd<|jrdnd} tj| |dzftj|j||_|jd|_t> |j!d|_"d|_#t> |j!d|_$tK|j&rUi|_'|j!D]3} tjdd|||zf|j(|j||j'| <4nd|_'|jj)dkr|nd} tj| ||jj)ftj|j||_*tj||ftj+|j||_,tj|||zftj+|j||_-dS)aAllocates static tensors for generation inputs and outputs. This is called only once at init time, to avoid repeated allocations and enable CUDA graphs. All tensors are allocated with maximum possible sizes. The allocated tensors are: - `_bulk_input_tensor`: Storage for all the small inputs: `input_ids`, `position_ids`, `cumulative_seqlens_q`, `logits_indices`, `cumulative_seqlens_k`, `carry_over_ids`. - `attention_mask`: Optional attention masks (only for eager/SDPA implementations) - `write_index` and `read_index` storage: Cache indexing tensors for each attention group - `output_ids`: Storage for generated token IDs and maybe log probabilities if return_logprobs is True cpur )dtyper; pin_memoryrkr;rNr full_attentionsliding_attention)sizerkr;rl).r"rUmax_batch_tokensrL num_blocks block_sizer;rblenr r8tensors_requiredrr*emptyint32_bulk_input_tensorlogits_processors_defaults fill_defaultsrrcumulative_seqlens_qr!carry_over_idscumulative_seqlens_knum_full_attention_groupsnum_sliding_attention_groupsrP output_idszero_total_seqlen_qr-fromkeyskeystotal_seqlen_krrrr9rr<max_blocks_per_requestr#int64write_index_storageread_index_storage)rfr=rUrurL num_pagesrl bulk_lines bulk_columns#full_attention_cumulative_seqlens_k&sliding_attention_cumulative_seqlens_knum_output_rows layer_typens r4r`z+ContinuousBatchingIOs._setup_static_tensorszs Z* :6!%!<J)DJ,AA [%.S37L7N7N3O3ORS3S '/*JJ %&6&:ArBB "'+  &ek$+Zd# # # +0+  -q 1T[+ + + ' %%d&EFFF04E5E4E1EF 3A7H8H7H4HI$($;A?[AWZ[A[?[<[$\!"5a9P:P9P6PQ.2.EaIeKadeKeIeFe.f+151HLhNdghNhLhIh1i."5a9J:J9J6JK>@! : / ^:]D %&6 7 : 2 d=cD %&9 : $3:!!+ .2 35;t{gq    .2mmD>   27+Q 0)>N2NO*;) 333#J// #'D *;a??JJQ ; & (I J+;!    $); ) *%+dk^h$ $ $  #(+ %55 6ekRVR]jt# # # r3Fotherstream non_blockingc|j|_|j|_|j|_|jdd|_|jdd|_|j|_|j|_t|j|_|j |_ t|j |_ |tj |n t}|5|j|j||jr"|j|j|nM|j|j||jdkr!|j|j||jP|jI|jD]/}|j||j||0ddddS#1swxYwYdS)Nrr)rQrRrSrVrWrXrr-rrrr*rJrrr|copy_r#rrrr)rfrrr maybe_streamrs r4_transfer_inputsz&ContinuousBatchingIOs._transfer_inputss1". ,%)%>" $ 4QQQ 7!%!6qqq!9 $ 4#2#D$788!.!$"3444:4Fuz((000KMM  w w  $ * *4+BQ] * ^ ^ ^# g!''(8|'TTTT)//0HWc/ddd#a'',2243JYe2fff".53G3S"&"5":":"<"<wwJ(4::4;Nz;Ziu:vvvv w w w w w w w w w w w w w w w w w wsC/GGGrIc|r|jdn|j}|r|jdn|j}|r |jdznt |j|j}|jd|jd|dzf |r|j |j|jd<d|_ |j d| |j ddd|f |jD]n}d|j|<d|j|<|jQ|j|ddddd|d||zft'j|jjo|r|jddd|fd|jddd|f|j|jddd||zf|jdS|jr(|jddd|fddS|jddd|f|j|jddd||zf|jdS)a Reset static tensors for the next batch. For efficiency, this only resets the portions of tensors that were actually used in the previous batch, using the attributes num_q_tokens and max_kv_read. If a (full_reset) is requested, the entire tensor storage is reset. r Nr)rrtrQrrRrLminr|r8rr}rr!rrrrrfill_r*finfor<r#r_r]rX)rfrIq_lenkv_lenb_sizers r4raz+ContinuousBatchingIOs._reset_static_tensorss6@V(--b111TEV5?U(--b111TEU4>w,q00CHY[_[vDwDw  4$"4 4k k ABHHJJJ  \<@<[D #D$6$8$8 9 GVG$**,,, 7F7 #))+++3  J,-D j )./D  +".#J/111fuf>N>N0NOUUK 0115  W  QQQZ ( . .r 2 2 2  $QQQY / 5 5d6M N N N  #AAA'7'7$7 8 > >t?U V V V V V  ! W  QQQZ ( . .r 2 2 2 2 2  $QQQY / 5 5d6M N N N  #AAA'7'7$7 8 > >t?U V V V V Vr3c|dg|_i|_|j|jdSdS)z4Reset all relevant states for a new generation loop.TrHN)rarYrZrd synchronizerfs r4resetzContinuousBatchingIOs.resetsU ""d"333!#,.)   *   + + - - - - - + *r3c|j|jfS)z:Get the cumulative sequence lengths for the current batch.)rrrs r4get_cumulative_seqlensz,ContinuousBatchingIOs.get_cumulative_seqlens(s($*CCCr3rrprev_output_idscdSNr2)rfrrrs r4carry_over_tokensz'ContinuousBatchingIOs.carry_over_tokens,s  r3cJ|j|jdSdSr)rdrrs r4retrieve_device_outputsz-ContinuousBatchingIOs.retrieve_device_outputs1s.   *   + + - - - - - + *r3c|jdd|jf}|jrG|jdd|jft j}nd}|j||fS)Nrr rk)rrStolistrPviewr*float32rY)rf new_tokenslogprobss r4prepare_batch_updatez*ContinuousBatchingIOs.prepare_batch_update5s_Q(C$*C(C%CDKKMM   q*ED,E*E'EFKKRWR_K``ggiiHHH%z8;;r3rYlogits_processorsuse_decode_fast_pathrQrR use_paddingc 0 |std|o|jdk|_||_|jrdn||_t ||_|dt|j j D|_ dt|j j D|_ g|_i|_g}g}dg} g} d|jD} dt|j j D} |jdkrdn"dt|j j D} t%|D]\}}|j}|j}|j}|j ||}|xj|z c_||j|t|||z| | d |zt5|j||_|D]R\}}| || |d |zt5|j|||j|<S|jr2|j |j|||jdd|fn#|j |j||| | |j!r@| | d d z tDg|_| d |j|j<|j||#||j$|j%d | d |_&|j'E| D]0\}}tQ|j'|| ||d kr|j)nd 1|rI|*|}||jz }| tW|j&|nd}tYtZj.tZj/|j0}|||j1dt |<|||j2dt |<|| |j3dt | <|| |j4dt | <| D]b\}}|d }||j5|<|tW|||||j|dt |<c|jstYtZj.tZj6|j0}t%| D]@\}}|||j7|dt |f<t ||j |<A| Rt%| D]D\}}|||j8|dt |f<t ||j |<AdSdSdS)aPrepare tensors and metadata for the next model forward pass, using the given requests as data. This method: 1. Resets the static tensors from the previous batch 2. Iterates through requests to accumulate input_ids, position_ids, and sequence lengths 3. Extends read/write indices for cache management 4. Builds attention masks if needed (for eager/SDPA implementations) 5. Converts accumulated lists to tensors and copies them to static storage This method also modifies the `position_offset` attribute of each request to track progress and adds a temporary token at the end of the requests for which there will a new token. zNo requests in batchrcg|]}dSrBr2rCs r4rFz?ContinuousBatchingIOs.prepare_batch_tensors..asHHHaHHHr3cg|]}dSrBr2rCs r4rFz?ContinuousBatchingIOs.prepare_batch_tensors..bs I I Iq I I Ir3ci|]}|dgSrBr2)rDrs r4 z?ContinuousBatchingIOs.prepare_batch_tensors..lscccJ QCcccr3cg|]}gSr2r2rCs r4rFz?ContinuousBatchingIOs.prepare_batch_tensors..ms@@@ar@@@r3Ncg|]}gSr2r2rCs r4rFz?ContinuousBatchingIOs.prepare_batch_tensors..ns8b8b8b8b8b8br3rr )rY arg_storagers)rrrr@rrm)9 ValueErrorr#numelrXrQrRrxrSrarTr"rUrVrWrYrZrr enumeratestateposition_offset query_length get_seqlens_kextendtokens_to_processappendmaxritemsrfill_block_table request_idextend_read_and_write_indices has_new_tokenrprepare_tensor_argsr|r8rrrr@_get_num_sequencesrrr*tensorr{r;rrrr!rrrr)rfrYrrrQrRrrrrr!rrr i future_stater past_lengthr seqlens_krlayer_type_seqlen_klayer_type_seqlens_knum_sequences_in_next_batchfake_sequences to_tensorrto_index_tensorgroup_write_indicesgroup_read_indicess r4prepare_batch_tensorsz+ContinuousBatchingIOs.prepare_batch_tensors?s*! 5344 4 4T8H8N8N8P8PST8T( $ 4E11+$'(9$:$:! ""$$$ IH51F+G+GHHH I IE$*2G,H,H I I I!#,.)  !scc$B[B`B`BbBbccc@@5)>#?#?@@@ !-22TT8b8bU4:K`EaEa8b8b8b  )):;;# 8# 8OA| &E/K'4L 00lKKI  ! !\ 1 ! !   U4 5 5 5   k;3M N N O O O ' '(&BCCC+7.'FTUWFX1%2BC  " ) ), 7 7 7 7 --//0B0D0DE .   326   *4H4N4N4P4P  0 0$#'#6z#B)=)=:DH[:[:[4#6#6ab   *.*A*Ak*A*Z*Z '84;TTN ' 't/BN(S(S T T T TNEL DKPPP ,59Y+?+?'Y'(1:<1H1H-C ---.AJK_A`A`!"=C(<$=$="=>5>Y~5N5N1c.11121E0J0J0L0L q q ,J,1"5N..sE%%%" D,=j,I4:K_ae f f%%%r3rrr rr.r)rQrRrrr unsqueezerrrr!r|r8rr"rXr#rNrrrTrUrWrrrVrrrrrxpopitem) rfrq_sizekv_size num_sequenceskwargsrwrite_index_sizeread_index_sizerrk_lens ` r4get_model_kwargsz&ContinuousBatchingIOs.get_model_kwargss""T%66//K/HH $nWfW-77::*7F73==a@@34Gma6G4GH*.~ ~>"&"9$:L:N:NP^Q^P^:^"_#'#6#>44B*?C?S](N]N):;;Y]   $   &D ' D,F %%%%&*&7&<&<&>&>%%%!'+&:!Q@Q~tz,-- V VA)4Rvv$:OPQ:R  = ! ( ()A!EVFVEVBV)W X X X1$$"#-8"U''d>RST>U < ' '(?CSOCS@S(T U U U U&*%>%D%D%F%F m m !J 2;O ": .".#.SD4G 4S7;7J:7VWZ\c]c\cekfkekWk7l'(4 t(--// 0 0A 5 5&,_&=&E&E&G&G&JF? #%+N%;%C%C%E%Ea%HF> "".+12B+C+K+K+M+Ma+P'( r3c*|j|j|jfS)a.Returns the tensors used inside the generation step that are not inputs to the model forward pass. In synchronous batching, there is no carry over, so the only tensor that will be used is output_ids, but we still return 3 tensors to have the same interface as when using async batching.)rrrs r4 get_cb_kwargsz#ContinuousBatchingIOs.get_cb_kwargss"DOT_DDr3.cp|jr|jfS|j|jg|jRSr)rXrQrRrvaluesrs r4_get_graph_keyz$ContinuousBatchingIOs._get_graph_keysA   (%' '!4#3Qd6G6N6N6P6PQQQr3prefixc|}|j|}|tj|d||S)NzCreating graph for key = )rr[ get_graphrinfo)rfrkeygraphs r4rzContinuousBatchingIOs.get_graph%sP!!## %%c** = K6>>c>> ? ? ? r3rc|}|j||tjd|dS)NzSetting graph for key = )rr[ set_graphrr)rfrrs r4rzContinuousBatchingIOs.set_graph-sJ!!## c5))) 11122222r3Fr>N)r)+r&r'r(r)r8r/r,r rrr*r;rkr rgr`rJrcr1rno_gradrartupler+r-r.rrrr0rfloatrrrrrrr CUDAGraphrrr2r3r4r6r6Fs M3*g"*g!*g%= *g  *g [ *g?*g *g*g*g*gXW 5ZW _cW W W W v_dww,w6;j6GwW[w wwww@U]__(W(W(W(W(W(W_(WT....DelDelARNc$t|||tjd|||_t|||||||_tjrtjnd|_tjrtjnd|_ tjrtjnd|_ dS)Nrir"r9r:r;r<r=) r6r*r;host_io device_iorJ is_availableEventh2d_over compute_overd2h_overres r4rgzHostDeviceIOPair.__init__4s-'A<&&#+     /'A#+    /4j.E.E.G.GQ ((***T 27*2I2I2K2KUEJ,,...QU.3j.E.E.G.GQ ((***T r3c|j|j|j|j|jfD]}||dSr)r rr rrrr)rfevents r4rzHostDeviceIOPair.resetSsj  mT%6 F $ $E !!### $ $r3rcJ|j|j|ddS)NT)rr)r rr )rfrs r4transfer_inputs_h2dz$HostDeviceIOPair.transfer_inputs_h2dZs( %%dnVRV%WWWWWr3c|tj|n t}|5|jj|jjdddddS#1swxYwYdS)NTr)r*rJrrr rrr )rfrrs r4transfer_outputs_d2hz%HostDeviceIOPair.transfer_outputs_d2h]s4:4Fuz((000KMM  X X L # ) )$.*CRV ) W W W X X X X X X X X X X X X X X X X X Xs,A++A/2A/r)r&r'r(r rrr*r;rkr rgrrJrcrrr2r3r4rr3sR"R!R%= R  R [ R?R RRRR>$$$$X%**;XXXXXX5:+│1->│ ¦ │2->│ ¦ │3->│ ¦ ¦ │ └───┼───┴───────────┬─────────────┴─┬─┼───────────┴───┼───────────────┐ ¦ GPU │ │ COMPUTE 0 │ COMPUTE 1 │█│ COMPUTE 2 │ COMPUTE 3 │ ¦ │ └───────────────┼───┬───────────┼─┴─┬─────────────┼───┬───────────┼───┤ D2H │ │0<-│ │1<-│ │2<-│ │3<-│ │ └───┘ └───┘ └───┘ └───┘ with: - CPU: actions happening on the CPU (host-side) - GPU: actions happening on the GPU (device-side) - H2D: host to device transfer - D2H: device to host transfer and: - PR N: preparation of batch N - ->N: host to device transfer of batch N - COMPUTE N: compute step for batch N - <-N: device to host transfer of batch N - UP N: update of batch N You can see that the GPU is almost always busy, except where the █ is. Proper ordering of steps is ensured through the use of CUDA events and streams. r"r9r:r;r<r=r>Nctjs.tdtjd|_fdt dD|_tj|_tj|_ tj|_ d|jdj _ d|jdj _ d|jdj _ d|jdj _ j |_ dS)Nz>Async batching requires CUDA, but torch.cuda.is_available() = rc :g|]}tS)r )r)rDrEr"r9r:r;r=r<s r4rFz7ContinuousBatchingAsyncIOs.__init__..sL     +E' /        r3rnrKr )r*rJr  RuntimeError current_pairrTio_pairsrc h2d_stream d2h_streamrdr r rures ``````r4rgz#ContinuousBatchingAsyncIOs.__init__sCz&&(( dbEJD[D[D]D]bbcc c          1XX      *++6+::*++6+::#j//v/>>26 a /48 a"126 a /48 a"1 % 6r3cT|j|jjSr)rrr rrs r4rz1ContinuousBatchingAsyncIOs.get_cumulative_seqlenss!}T./7NNPPPr3rYrrrQrRrc|j|j}|j|||||||jj|dSr)rrr rrrinfer_carry_over_ids)rfrYrrrQrRrio_pairs r4rz0ContinuousBatchingAsyncIOs.prepare_batch_tensorssk- 12-- 02F Vacn    &,,T-F-F-H-HIIIIIr3cz|j|jjj}|jd|jz jj}dt |jD}t |D]!\}}||}||||<"tj |tj S)aInfers the ids of the tokens to carry over from batch N to batch N+1. In asynchronous batching mode, we can schedule a request for batch N+1 without knowing the token predicted for that request in batch N. For that reason, we might need to carry over tokens just predicted in batch N before launching the forward pass of batch N+1. This method computes the ids of the tokens to carry over.r cg|]}dS)rr2rCs r4rFzCContinuousBatchingAsyncIOs.infer_carry_over_ids..sCCC"CCCr3Nr) rrr rZrTrurrgetr*rr{)rf!next_req_id_to_new_token_position!prev_req_id_to_new_token_positionrrreq_idnew_token_positions r4r#z/ContinuousBatchingAsyncIOs.infer_carry_over_idss -1M$:K,L,T,q),0M!d>O:O,P,X,u)CCeD,A&B&BCCC ##D#I#I#K#KLL 7 7IAv!B!F!Fv!N!N !-5612|N%+>>>>r3Fc |j|j}||j|j|j|j|j|j |S)Nr) rrrr record_eventrrd wait_eventr r)rfrr$s r4rz+ContinuousBatchingAsyncIOs.get_model_kwargsst- 12##DO444 $$W%5666 &&w'7888 11k1JJJr3c|j|j}|jd|jz }|jj|jj|jjfS)aReturns the tensors used inside the generation step that are not inputs to the model forward pass. Those tensors could be retrieved using this object, but it would trigger a recompile if using torch.compile. They are: - output_ids: the output ids of the current batch - prev_output_ids: the output ids of the previous batch, required to carry over outputs tokens of the previous batch to the input tokens of the next batch. - carry_over_ids: a mask representing how to carry over tokens. r )rrr rr)rfr previous_pairs r4rz(ContinuousBatchingAsyncIOs.get_cb_kwargssL}T%67  a$*;&;<  " 1  # .  " -  r3rrrc|d|f}|dk}|d|d}|d|d}||z|dd|z zz|d<dS)aAs explained in the infer_carry_over_ids method, we might need to carry over tokens just predicted in batch N before launching the forward pass of batch N+1. This method performs the carry over, and is recorded in CUDA graphs if they are enabled.rrNr )r/rt)rfrrrcarried_over_idscarried_over_masks r4rz,ContinuousBatchingAsyncIOs.carry_over_tokenss+1n+<=+r16688+,?innQ.?.?,?@-.A q0A0A.AB'*;;ilaRcNc>dd ! r3c:|j|jjjSr)rrr rrs r4rz%ContinuousBatchingAsyncIOs.output_ids s}T./9DDr3cnd|jd}|j|jj|S)Nz(IO ))r)rrr r)rfrs r4rz$ContinuousBatchingAsyncIOs.get_graphs9,),,,}T./9CC6CRRRr3rcZ|j|jj|dSr)rrr r)rfrs r4rz$ContinuousBatchingAsyncIOs.set_graphs) d'(2<>> ""7#7888$$T_555 $$W%5666 r3c$d|jz |_dS)zeSwitch to the other IO pair so the next batch reads from / writes into a fresh set of static buffers.r N)rrs r4r:z(ContinuousBatchingAsyncIOs.swap_io_pairs%s 11r3c|j|j}|j|jSr)rrrrr rr;s r4rz/ContinuousBatchingAsyncIOs.prepare_batch_update*s:- 12$$&&&33555r3cd|_|jD]}||j|j|jdS)zWReset all state for a new generation session. Used in persistent mode between sessions.rN)rrrrrr rdr;s r4rz ContinuousBatchingAsyncIOs.reset/sm}  G MMOOOO ##%%% ##%%% '')))))r3rr)'r&r'r(r)r rrr*r;rkr rgrr+r-r.rr0rr1r/rr#rrrrpropertyrrJrrrrXrr:rrrr2r3r4rrcs ++Z#7"#7!#7%= #7  #7 [ #7?#7 #7#7#7#7LQelDelARrIs#""""" ======PPPPPP******&&&&&&GGGGGG>>>>>>>>>>jjjjjjjjjjjjjj$$$$$$$$Nj3j3j3j3j3j3j3j3Z-X-X-X-X-X-X-X-X`S*S*S*S*S*S*S*S*S*S*r3