j'W d dl Z d dlZd dlZd dlZd dlZd dlZd dlZd dlmZ d dl m
Z
m Z m Z m
Z
mZmZmZ d dlmZ d dlZd dlZd dlmZ ddlmZmZmZm Z m Z ddl m Z G d d
e Z G d d e Z! G d
de Z" G d de Z# G d de Z$de%de&fdZ'ej( ej) ej* ej+ ej, ej- dZ.d de/de0dej1 fdZ2de/dej1 fdZ3de/dej1 fd Z4dS ) N)Path)AnyCallableDictListOptionalTupleUnion)URLError)Image )_flip_byte_ordercheck_integrity download_and_extract_archiveextract_archiveverify_str_arg)
VisionDatasetc
e Zd ZdZddgZg dZdZdZg dZe d Z
e d Z e d
Z e d Z
d"deeef dedee dee dedd
f fd
Zd Zd Zd Zdedeeef fdZdefdZ e defd Z e defd Z e de eef fd Z defd Z!d#d Z"defd!Z# xZ$S )$MNISTam `MNIST `_ Dataset.
Args:
root (str or ``pathlib.Path``): Root directory of dataset where ``MNIST/raw/train-images-idx3-ubyte``
and ``MNIST/raw/t10k-images-idx3-ubyte`` exist.
train (bool, optional): If True, creates dataset from ``train-images-idx3-ubyte``,
otherwise from ``t10k-images-idx3-ubyte``.
download (bool, optional): If True, downloads the dataset from the internet and
puts it in root directory. If dataset is already downloaded, it is not
downloaded again.
transform (callable, optional): A function/transform that takes in a PIL image
and returns a transformed version. E.g, ``transforms.RandomCrop``
target_transform (callable, optional): A function/transform that takes in the
target and transforms it.
z!http://yann.lecun.com/exdb/mnist/z.https://ossci-datasets.s3.amazonaws.com/mnist/))train-images-idx3-ubyte.gz f68b3c2dcbeaaa9fbdd348bbdeb94873)train-labels-idx1-ubyte.gz d53e105ee54ea40749a09fcbcd1e9432)t10k-images-idx3-ubyte.gz 9fb629c4189551a2d022fa330f9573f3)t10k-labels-idx1-ubyte.gz ec29112dd5afa0611ce80d1b7f02629cz training.ptztest.pt
z0 - zeroz1 - onez2 - twoz 3 - threez4 - fourz5 - fivez6 - sixz 7 - sevenz 8 - eightz9 - ninec 8 t j d | j S )Nz%train_labels has been renamed targetswarningswarntargetsselfs KC:\Projects\Historical\venv\Lib\site-packages\torchvision/datasets/mnist.py train_labelszMNIST.train_labels@ s
=>>>| c 8 t j d | j S )Nz$test_labels has been renamed targetsr r$ s r& test_labelszMNIST.test_labelsE s
<===|r( c 8 t j d | j S )Nz train_data has been renamed datar! r" datar$ s r&
train_datazMNIST.train_dataJ s
8999yr( c 8 t j d | j S )Nz test_data has been renamed datar, r$ s r& test_datazMNIST.test_dataO s
7888yr( TNFroottrain transformtarget_transformdownloadreturnc t ||| || _ | r#| \ | _ | _ d S |r| | st d |
\ | _ | _ d S )N)r3 r4 z;Dataset not found. You can use download=True to download it) super__init__r2 _check_legacy_exist_load_legacy_datar- r# r5
_check_exists RuntimeError
_load_data)r% r1 r2 r3 r4 r5 __class__s r& r9 zMNIST.__init__T s EUVVV
# # % % &*&<&<&>&> #DIt| F MMOOO!!## ^ \]] ]"&//"3"3 4<<z,MNIST._check_legacy_exist..p sO
KOOBGLL)> E E F F
r( )rC rD existsrF all
training_file test_file)r% processed_folder_existss ` r& r: zMNIST._check_legacy_existk sk "$'..1F"G"G & 5
TXTfhlhvSw
r( c | j r| j n| j }t j t
j | j | d S )NT) weights_only) r2 rL rM torchloadrC rD rE rF )r% data_files r& r; zMNIST._load_legacy_datat sC +/*HD&&$. z"',,t' zMNIST._load_dataz s| #':96MMM
rw||DOZ H HII#':96MMM
!"',,t
"K"KLLW} r( indexc | j | t | j | }}t j | d }| j | | }| j | | }||fS )z
Args:
index (int): Index
Returns:
tuple: (image, target) where target is index of the target class.
Lmode)r- intr# r fromarraynumpyr3 r4 r% r] imgtargets r& __getitem__zMNIST.__getitem__ s i&DL,?(@(@V ociikk444 > %..%%C ,**622FF{r( c * t | j S rB )lenr- r$ s r& __len__z
MNIST.__len__ s 49~~ r( c b t j | j | j j d S )NrawrC rD rE r1 r? __name__r$ s r& rY zMNIST.raw_folder s! w||DIt~'>FFFr( c b t j | j | j j d S )N processedrn r$ s r& rF zMNIST.processed_folder s! w||DIt~'> LLLr( c > d t | j D S )Nc i | ]\ }}|| S rt )rG i_classs r&
z&MNIST.class_to_idx.. s CCCiaCCCr( ) enumerateclassesr$ s r& class_to_idxzMNIST.class_to_idx s CC9T\+B+BCCCCr( c D t fd j D S )Nc 3 K | ]y\ }}t t j j t j t j | d V zdS )r N)r rC rD rE rY splitextbasename)rG url_r% s r& rI z&MNIST._check_exists.. s|
Q
BGLL "':J:J27K[K[\_K`K`:a:abc:d e e f f
r( )rK resourcesr$ s `r& r< zMNIST._check_exists s=
.
r( c | rdS t j | j d | j D ]\ }}g }| j D ]O}| | } t
|| j || n,# t $ r }| | Y d}~Fd}~ww xY w nId| d}t | j | D ] \ }}|d| dt | d z
} t | dS )
z4Download the MNIST data if it doesn't exist already.NTexist_ok)
download_rootfilenamemd5zError downloading z:
zTried z, got:
) r< rC makedirsrY r mirrorsr r appendzipstrr= ) r% r r errorsmirrorr eserrs r& r5 zMNIST.download sG F
DOd3333 "^ & &MHcF ,
&
& +++ 0DO^floppppp MM!$$$ HHHH 6666#&t|V#<#< ? ?KFC>&>>#c((>>>>AA"1oo% & &s A))
B3B
Bc & | j du rdnd}d| S )NTTrainTestSplit: )r2 )r% splits r&
extra_reprzMNIST.extra_repr s% :--6 r( )TNNFr6 N)%ro
__module__ __qualname____doc__r r rL rM ry propertyr' r* r. r0 r
r r boolr r r9 r: r; r> rb r r rh rk rY rF r rz r< r5 r
__classcell__r? s @r& r r s " ,8G
I "MI G X X X X (,/3
4 4CI 4 4 H% 4
#8, 4
4
4 4 4 4 4 4.
] ] ] sCx , GC G G G XG M# M M M XM Dd38n D D D XD
t
& & & &2!C ! ! ! ! ! ! ! !r( r c ( e Zd ZdZdgZg dZg dZdS ) FashionMNISTa `Fashion-MNIST `_ Dataset.
Args:
root (str or ``pathlib.Path``): Root directory of dataset where ``FashionMNIST/raw/train-images-idx3-ubyte``
and ``FashionMNIST/raw/t10k-images-idx3-ubyte`` exist.
train (bool, optional): If True, creates dataset from ``train-images-idx3-ubyte``,
otherwise from ``t10k-images-idx3-ubyte``.
download (bool, optional): If True, downloads the dataset from the internet and
puts it in root directory. If dataset is already downloaded, it is not
downloaded again.
transform (callable, optional): A function/transform that takes in a PIL image
and returns a transformed version. E.g, ``transforms.RandomCrop``
target_transform (callable, optional): A function/transform that takes in the
target and transforms it.
z;http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/))r 8d4fb7e6c68d591d4c3dfef9ec88bf0d)r 25c81989df183df01b3e8a0aad5dffbe)r bef4ecab320f06d8554ea6380940ec79)r bb300cfdad3c16e7a12a480ee83cd310)
z T-shirt/topTrouserPulloverDressCoatSandalShirtSneakerBagz
Ankle bootNro r r r r r ry rt r( r& r r sB MMG I yxxGGGr( r c ( e Zd ZdZdgZg dZg dZdS )KMNISTa{ `Kuzushiji-MNIST `_ Dataset.
Args:
root (str or ``pathlib.Path``): Root directory of dataset where ``KMNIST/raw/train-images-idx3-ubyte``
and ``KMNIST/raw/t10k-images-idx3-ubyte`` exist.
train (bool, optional): If True, creates dataset from ``train-images-idx3-ubyte``,
otherwise from ``t10k-images-idx3-ubyte``.
download (bool, optional): If True, downloads the dataset from the internet and
puts it in root directory. If dataset is already downloaded, it is not
downloaded again.
transform (callable, optional): A function/transform that takes in a PIL image
and returns a transformed version. E.g, ``transforms.RandomCrop``
target_transform (callable, optional): A function/transform that takes in the
target and transforms it.
z-http://codh.rois.ac.jp/kmnist/dataset/kmnist/))r bdb82020997e1d708af4cf47b453dcf7)r e144d726b3acfaa3e44228e80efcd344)r 5c965bf0a639b31b8f53240b1b52f4d7)r 7320c461ea6c1c855c0b718fb2a4b134)
okisutsunahamayarewoNr rt r( r& r r sA ??G I KJJGGGr( r c e Zd ZdZdZdZdZh dZ ee j
e j z Z e
ee e
ee ez
e
ee ez
dg ee j z ee j
ee j
dZdeeef ded ed
d f fd Zed
efd
Zed
efd Zed
efd Zed
efd Zed
efd Z d Z d
e fdZ ddZ xZ!S )EMNISTaH `EMNIST `_ Dataset.
Args:
root (str or ``pathlib.Path``): Root directory of dataset where ``EMNIST/raw/train-images-idx3-ubyte``
and ``EMNIST/raw/t10k-images-idx3-ubyte`` exist.
split (string): The dataset has 6 different splits: ``byclass``, ``bymerge``,
``balanced``, ``letters``, ``digits`` and ``mnist``. This argument specifies
which one to use.
train (bool, optional): If True, creates dataset from ``training.pt``,
otherwise from ``test.pt``.
download (bool, optional): If True, downloads the dataset from the internet and
puts it in root directory. If dataset is already downloaded, it is not
downloaded again.
transform (callable, optional): A function/transform that takes in a PIL image
and returns a transformed version. E.g, ``transforms.RandomCrop``
target_transform (callable, optional): A function/transform that takes in the
target and transforms it.
z4https://biometrics.nist.gov/cs_links/EMNIST/gzip.zip 58c8d27c78d21e728a6bc7b3cc06412e)byclassbymergebalancedlettersdigitsmnist> cru jklmr pr uvwxyzzN/Ar1 r kwargsr6 Nc t |d| j | _ | | | _ | | | _ t j |fi | | j | j | _
d S )Nr ) r splitsr _training_filerL
_test_filerM r8 r9 classes_split_dictry )r% r1 r r r? s r& r9 zEMNIST.__init__' su #E7DK@@
! 0 0 7 7 //(((((.tz: r( c d| dS )N training_.ptrt r s r& r zEMNIST._training_file. s %5%%%%r( c d| dS )Ntest_r rt r s r& r zEMNIST._test_file2 s !u!!!!r( c . d| j d| j rdnd S )Nzemnist--r2 test)r r2 r$ s r& _file_prefixzEMNIST._file_prefix6 s% III&GggIIIr( c \ t j | j | j d S )NrV rC rD rE rY r r$ s r& images_filezEMNIST.images_file: & w||DO0A-U-U-UVVVr( c \ t j | j | j d S )NrW r r$ s r& labels_filezEMNIST.labels_file> r r( c R t | j t | j fS rB )rX r rZ r r$ s r& r> zEMNIST._load_dataB s# t /00/$BR2S2SSSr( c L t d | j | j fD S )Nc 3 4 K | ]}t | V d S rB r rG rH s r& rI z'EMNIST._check_exists..F * ZZT?4((ZZZZZZr( rK r r r$ s r& r< zEMNIST._check_existsE * ZZd6FHX5YZZZZZZr( c | rdS t j | j d t | j | j | j t j | j d }t j | D ]J}|
d r3t t j || | j Kt j
| dS )z5Download the EMNIST data if it doesn't exist already.NTr )r r gzipz.gz)r< rC r rY r r r rD rE listdirendswithr shutilrmtree)r% gzip_folder gzip_files r& r5 zEMNIST.downloadH s F
DOd3333$TXT_RVRZ[[[[gll4?F;; K00 W WI!!%((
W [) D DdoVVV
k"""""r( r )"ro r r r r r r _merged_classessetstringr
ascii_letters _all_classessortedlistascii_lowercaser r
r r r r9 staticmethodr r r r r r r> r r< r5 r r s @r& r r sK & AC
,C
MFaaaO3v}v';;<>??F44 >??@@7TT&"8 9 99$v}%%fm$$
;U39 - ;c ;S ;T ; ; ; ; ; ; & & & & \& "S " " " \" Jc J J J XJ WS W W W XW WS W W W XWT T T[t [ [ [ [
#
#
#
#
#
#
#
#r( r c
B e Zd ZU dZddddddZddgdd gd
d gd Zeeee eef f e
d
< g dZ d de ee
f dee dedededdf fd
Zedefd Zedefd ZdefdZd Zd dZd ede eef fd Zdefd Z xZS )!QMNISTa` `QMNIST `_ Dataset.
Args:
root (str or ``pathlib.Path``): Root directory of dataset whose ``raw``
subdir contains binary files of the datasets.
what (string,optional): Can be 'train', 'test', 'test10k',
'test50k', or 'nist' for respectively the mnist compatible
training set, the 60k qmnist testing set, the 10k qmnist
examples that match the mnist testing set, the 50k
remaining qmnist testing examples, or all the nist
digits. The default is to select 'train' or 'test'
according to the compatibility argument 'train'.
compat (bool,optional): A boolean that says whether the target
for each example is class number (for compatibility with
the MNIST dataloader) or a torch vector containing the
full qmnist information. Default=True.
download (bool, optional): If True, downloads the dataset from
the internet and puts it in root directory. If dataset is
already downloaded, it is not downloaded again.
transform (callable, optional): A function/transform that
takes in a PIL image and returns a transformed
version. E.g, ``transforms.RandomCrop``
target_transform (callable, optional): A function/transform
that takes in the target and transforms it.
train (bool,optional,compatibility): When argument 'what' is
not specified, this boolean decides whether to load the
training set or the testing set. Default: True.
r2 r nist)r2 r test10ktest50kr )zbhttps://raw.githubusercontent.com/facebookresearch/qmnist/master/qmnist-train-images-idx3-ubyte.gz ed72d4157d28c017586c42bc6afe6370)z`https://raw.githubusercontent.com/facebookresearch/qmnist/master/qmnist-train-labels-idx2-int.gz 0058f8dd561b90ffdd0f734c6a30e5e4)zahttps://raw.githubusercontent.com/facebookresearch/qmnist/master/qmnist-test-images-idx3-ubyte.gz 1394631089c404de565df7b7aeaf9412)z_https://raw.githubusercontent.com/facebookresearch/qmnist/master/qmnist-test-labels-idx2-int.gz 5b5b05890a5e13444e108efe57b788aa)z[https://raw.githubusercontent.com/facebookresearch/qmnist/master/xnist-images-idx3-ubyte.xz 7f124b3b8ab81486c9d8c2749c17f834)zYhttps://raw.githubusercontent.com/facebookresearch/qmnist/master/xnist-labels-idx2-int.xz 5ed0e788978e45d4a8bd4b7caec3d79d)r2 r r r r NTr1 whatcompatr r6 c ||rdnd}t |dt | j | _ || _ |dz | _ | j | _ | j | _ t j
||fi | d S )Nr2 r r r ) r tuplesubsetskeysr r rS rL rM r8 r9 )r% r1 r r r2 r r? s r& r9 zQMNIST.__init__ s < #/77D"4t|7H7H7J7J1K1KLL !^ u///////r( c | j | j | j \ \ }}}t j | j t j t j | d S Nr r r r rC rD rE rY r} r~ )r% r r s r& r zQMNIST.images_file s] nT\$)%<= a!w||DORW-=-=bg>N>Ns>S>S-T-TUV-WXXXr( c | j | j | j \ }\ }}t j | j t j t j | d S r r )r% r r s r& r zQMNIST.labels_file s] nT\$)%<= 8Cw||DORW-=-=bg>N>Ns>S>S-T-TUV-WXXXr( c L t d | j | j fD S )Nc 3 4 K | ]}t | V d S rB r r s r& rI z'QMNIST._check_exists.. r r( r r$ s r& r< zQMNIST._check_exists r r( c t | j }|j t j k rt d|j | dk rt d t | j }| dk r$t d| | j
dk rD|ddd d d d f }|ddd d f }nN| j
d k rC|dd d d d d f }|dd d d f }||fS )
Nz/data should be of dtype torch.uint8 instead of z zQMNIST._load_data sn ,T-=>> : $ $ Zdj Z Z[[ [ ?? ! ! [\\ \/0@AAFFHH 1 $ $ bGL^L^L`L` b bcc c 9 ! !%AAA
&,,..D agqqqj)//11GG
Y)
#
#111 %++--D effaaai(..00GW} r( c | rdS t j | j d | j | j | j }|D ] \ }}t || j | dS )zDownload the QMNIST data if it doesn't exist already.
Note that we only download what has been asked for (argument 'what').
NTr )r )r< rC r rY r r r r )r% r r r s r& r5 zQMNIST.download s F
DOd3333t|DI 67 H HHC (do3 G G G G G H Hr( r] c 6 | j | | j | }}t j | d }| j | | }| j rt |d }| j | | }||fS )Nr_ r` r ) r- r# r rc rd r3 r rb r4 re s r& rh zQMNIST.__getitem__ s i& U(;Vociikk444 > %..%%C ; $^^F ,**622FF{r( c d| j S )Nr )r r$ s r& r zQMNIST.extra_repr s $$$$r( )NTTr ) ro r r r r r r r r r __annotations__ry r
r r r r r9 r r r r< r> r5 rb rh r r r s @r& r r X s : Fv_effG
+ 3 3ItCeCHo../ @ G fj
0
0#t)$
0,4SM
0JN
0^b
0ux
0
0
0
0
0
0
0 YS Y Y Y XY YS Y Y Y XY[t [ [ [ [ ( H H H H
sCx
%C % % % % % % % %r( r br6 c H t t j | d d S )Nhex )rb codecsencode)r* s r& get_intr0 s v}Q&& + ++r( )
TrD strictc :
t | d 5 }|
ddd n # 1 swxY w Y t j dk r"t
dd }|dz }|dz }ngt
dd }t
dd t
dd dz z t
d d dz dz z }d|cxk rd k sn J d
|cxk rd k sn J t
| }
fd t
| D }t j d
k rZt
t | D ]=}t ||
dd d
d ||< >t j t
|d|dz z } t j dk r'| dk rt | } | j d t# j | k s|rJ | j | S )zRead a SN3 file in "Pascal Vincent" format (Lush file 'libidx/idx-io.lsh').
Argument may be a filename, compressed filename, or file object.
rbNlittler r
r r r1 r6 c V g | ]%}t d |dz z d |dz z &S )r; r
r )r0 )rG ru r- s r&
z1read_sn3_pascalvincent_tensor.. s; EEEaa1q5kAQK/0 1 1EEEr( big) byteorderF)r@ signed)r offset)openreadsysr@ r0 SN3_PASCALVINCENT_TYPEMAPrangerj rb
from_bytesto_bytesrQ
frombuffer bytearray element_sizer shapenpprodview) rD r7 fmagicndty
torch_typer ru parsedr- s @r& r r sW
dD Qvvxx } QqS ""
S[
c\
T!A#Y
T!A#Y
'$qs)"4"4s":
:WT!A#Y=O=ORU=UX[=[
[ <<<>!A$--X-"F"FRW`e>ffAaDD
iooZbSTf
W
W
WF } V%8%8%:%:Q%>%>!&)) <?bgajj ( ( ( ( 6 6;?s 377c t | d }|j t j k rt d|j | dk r$t
d| | S )NFr7 ,x should be of dtype torch.uint8 instead of r
z%x should have 1 dimension instead of )r r rQ r r! r" r# r$ rD r s r& rZ rZ ! s{ %d5999Aw%+ PqwPPQQQ||~~Q QQRRR 6688Or( c t | d }|j t j k rt d|j | dk r$t
d| |S )NFrX rY r z%x should have 3 dimension instead of )r r rQ r r! r" r# rZ s r& rX rX * ss %d5999Aw%+ PqwPPQQQ||~~Q QQRRR Hr( )T)5r. rC os.pathr r rE r! pathlibr typingr r r r r r r
urllib.errorr rd rN rQ PILr utilsr r r r r visionr r r r r r bytesrb r0 r int8int16int32float32float64rF r r Tensorr rZ rX rt r( r& rj s
D D D D D D D D D D D D D D D D D D ! ! ! ! ! ! s s s s s s s s s s s s s s ! ! ! ! ! !u! u! u! u! u!M u! u! u!py y y y y5 y y y8K K K K KU K K K8Q# Q# Q# Q# Q#U Q# Q# Q#hS% S% S% S% S%U S% S% S%l,u , , , , ,
{ z
" " "T "U\ " " " "J# %,
#
%,
r(