j@*dZddlZddlmZmZddlZddlmZddlm Z m Z m Z ddl m Z mZddlmZdd lmZmZmZmZmZmZmZmZmZmZmZdd lmZdd lm Z e!eeZ"ee Z#ej$d d e#De%Z&ee%dzej'dd(e"dee"fZ)edZ*e*+dgddddddeddddddf dee%dzej,dfdedededee&dzej'dfd ed!e)d"ee-ej'd#d$d%fd&ee-ej'd'd(fd)ee-ej'd*d+d,fd-ed.ed/dfd0Z.e*+d1gd2 dRd4ee%ej,d5fd ed.ed/dfd6Z/e*+d7d8d9g dSd4ee%ej,d:fd-ed!e)d.ed/df d;Z0e*+dee%dzej'd?d@fdAee%dzej'dBfd.ed/df dCZ1e*+dDdEdFg dTdDee%ej,dGfd.ed/dfdHZ2e*+dIgdJ dUd4ee%ej,d:fdKee-ej'dLdMfdNee-ej'dOdPfd.ed/df dQZ3dS)VaContains commands to interact with datasets on the Hugging Face Hub. Usage: # list datasets on the Hub hf datasets ls # list datasets with a search query hf datasets ls --search "code" # get info about a dataset hf datasets info HuggingFaceFW/fineweb N) Annotatedget_args)execute_raw_sql_query)CLIErrorRepositoryNotFoundErrorRevisionNotFoundError) DatasetSort_TExpandDatasetProperty_T) DatasetCard) REPO_LIST_DEFAULT_LIMIT AuthorOpt FilterOptLimitOpt RevisionOpt SearchOptTokenOptapi_object_to_dict get_hf_apimake_expand_properties_parser typer_factory)list_repo_files_cmd)outDatasetSortEnumci|]}||Sr).0ss _/var/www/html/Carbon-Document/venv/lib/python3.11/site-packages/huggingface_hub/cli/datasets.py r 8s/L/L/L1/L/L/L)typezComma-separated properties to return. When used, only the listed properties (and id) are returned. Example: '--expand=downloads,likes,tags'. Valid: z, .)helpcallbackz"Interact with datasets on the Hub.)r$z list | ls)zhf datasets lsz*hf datasets ls --sort downloads --limit 10zhf datasets ls --search "code"z*hf datasets ls --filter benchmark:officialz$hf datasets ls HuggingFaceFW/finewebz'hf datasets ls HuggingFaceFW/fineweb -Rz.hf datasets ls HuggingFaceFW/fineweb --tree -h)examplesFrepo_idzVDataset ID (e.g. `username/repo-name`) to list files from. If omitted, lists datasets.searchauthorfiltersortz Sort results.limitexpandhuman_readablez--human-readablez-hz=Show sizes in human readable format (only for listing files).as_treez--treez3List files in tree format (only for listing files). recursivez --recursivez-Rz0List files recursively (only for listing files).revisiontokenreturnc ||tjd|tjd|tjd|tjd|tkrtjd|tjdt|d||| | | S|rtjd | rtjd |rtjd | tjd t | } |r|jnd} d| |||| ||D}tj|dS)zList datasets on the Hub, or files in a dataset repo. When called with no argument, lists datasets on the Hub. When called with a dataset ID, lists files in that dataset repo. Nz'Cannot use --search when listing files.z'Cannot use --author when listing files.z'Cannot use --filter when listing files.z%Cannot use --sort when listing files.z&Cannot use --limit when listing files.z'Cannot use --expand when listing files.dataset)r' repo_typer.r/r0r1r2z(Cannot use --tree when listing datasets.z-Cannot use --recursive when listing datasets.z2Cannot use --human-readable when listing datasets.z,Cannot use --revision when listing datasets.r2c,g|]}t|Srr)r dataset_infos r zdatasets_ls..s0     <((   r!)r*r)r(r+r,r-) typer BadParameterr rrvalue list_datasetsrtable)r'r(r)r*r+r,r-r.r/r0r1r2apisort_keyresultss r datasets_lsrDGs\  $%NOO O  $%NOO O  $%NOO O  $%LMM M + + +$%MNN N  $%NOO O")    M !KLLLR !PQQQW !UVVV !OPPP 5 ! ! !C!+tzztH  -- .     GIgr! leaderboard)z4hf datasets leaderboard SWE-bench/SWE-bench_VerifiedzLhf datasets leaderboard SWE-bench/SWE-bench_Verified --limit 5 --format jsonzIhf datasets ls --filter benchmark:official # list available leaderboards dataset_idz?The benchmark dataset ID (e.g. `SWE-bench/SWE-bench_Verified`).c*t|}||}d|d|D}tj|gddddd tjd |r%tjd |d jd dSdS)zList model scores from a dataset leaderboard. This command helps find the best models for a task or compare models by benchmark scores. Use 'hf datasets ls --filter benchmark:official' to list available leaderboards.r7)r'c,g|]}t|Srr9rentrys rr;z(datasets_leaderboard..s!JJJU!%((JJJr!N)rankmodel_idr>sourcerMright)rLr>)headersid_key alignmentszPUse 'hf datasets ls --filter benchmark:official' to list available leaderboards.zUse 'hf models info rz' to get details about a model.)rget_dataset_leaderboardrr@hintrM)rGr,r2rArErCs rdatasets_leaderboardrUs 5 ! ! !C--j-AAKJJk&5&6IJJJGI777#g66  H _```b ` A(?```aaaaabbr!infoz&hf datasets info HuggingFaceFW/finewebz9hf datasets info my-dataset --expand downloads,likes,tagsz+The dataset ID (e.g. `username/repo-name`).ct|} ||||}nJ#t$r}td|d|d}~wt$r}td|d|d|d}~wwxYwt j|dS) z$Get info about a dataset on the Hub.r7)r'r1r-z Dataset 'z ' not found.Nz Revision 'z' not found on 'z'.)rr:rrrrdict)rGr1r-r2rArVes r datasets_inforZs 5 ! ! !CU XfUU "DDD;:;;;<.s)TTT%emu{e?S?S?S?S?Sr!cDg|]}|j|j|j|jdS)r\r]urlsize)r_r]rdrerJs rr;z$datasets_parquet..s;af5<%+eiQVQ[\\r!rcrd)rPrQN)rlist_dataset_parquet_filesrr@)rGr\r]r2rAentriesfilteredrCs ` rdatasets_parquetris 5 ! ! !C,,Z,OOGTTTT7TTTHjrGIgAAA%PPPPPPr!sqlzhf datasets sql "SELECT COUNT(*) AS rows FROM read_parquet('https://huggingface.co/api/datasets/cfahlgren1/hub-stats/parquet/models/train/0.parquet')"zhf datasets sql "SELECT * FROM read_parquet('https://huggingface.co/api/datasets/cfahlgren1/hub-stats/parquet/models/train/0.parquet') LIMIT 5" --format jsonzRaw SQL query to execute.c t||}n/#t$r"}tt||d}~wwxYwt j|dS)zAExecute a raw SQL query with DuckDB against dataset parquet URLs.) sql_queryr2N)r ImportErrorrstrrr@)rjr2resultrYs r datasets_sqlrpsf&&EBBB &&&s1vvA%&Ifs A;Acard)z&hf datasets card HuggingFaceFW/finewebz1hf datasets card HuggingFaceFW/fineweb --metadataz?hf datasets card HuggingFaceFW/fineweb --metadata --format jsonz-hf datasets card HuggingFaceFW/fineweb --textmetadataz --metadataz'Output only the metadata from the card.textz--textz(Output only the text body (no metadata).cR|r|rtdtj||}|r-tj|jdS|rtj|jdStj|jtj d|ddS)z7Get the dataset card (README) for a dataset on the Hub.z---metadata and --text are mutually exclusive.r7zUse `hf datasets card z/ --metadata` to extract only the card metadata.N) rr loadrrXdatato_dictrscontentrT)rGrrrsr2rqs r datasets_cardrys HDHFGGG  Je 4 4 4Dg ""$$%%%%% g   e*eeefffffr!)rFN)NNNra)FFN)4__doc__enumtypingrrr<huggingface_hub._dataset_viewerrhuggingface_hub.errorsrrrhuggingface_hub.hf_apir r huggingface_hub.repocardr _cli_utilsr rrrrrrrrrr _file_listingr_outputrsorted_EXPAND_PROPERTIES _SORT_OPTIONSEnumrnrOptionjoin ExpandOpt datasets_clicommandArgumentboolrDrUrZrirpryrr!rrs   &&&&&&&& AAAAAA[[[[[[[[[[IIIIIIII000000                          /.....VHH%<==>>'' $)-/L/Lm/L/L/LSVWWW $JELEeienenoAeBeBEEE../ABB  }"FGGG     -    7OO  d tuuu wO  O  O O $ /*** , O O O  '4sttt vO$  X$YZZZ \%O,  ]D/abbb d-O45O6 7O8 9OOO  Odbb#~u~3tuuuuvb b b bbbb( 0C! #~u~3`aaaab     " ptZ^ Q Q#~u~3`aaaab Q cDj,%,z@j"k"k"kk l Q S4Z3U!V!V!VV W Q  Q  Q Q Q Q   c j   3,GHHHH I         mrej gg#~u~3`aaaabglel<>ghhhhig D,%,x6`aaaa bg  g  ggggggr!