Load dataset error

Parameters

• 
  with_community_datasets (bool, optional, defaults to True) —
  Include the community provided datasets.
  
• 
  with_details (bool, optional, defaults to False) —
  Return the full details on the datasets instead of only the short name.
  

>>> from datasets import list_datasets
>>> list_datasets()
['acronym_identification',
 'ade_corpus_v2',
 'adversarial_qa',
 'aeslc',
 'afrikaans_ner_corpus',
 'ag_news',
 ...
]

Parameters

• 
  path (str) —
  Path or name of the dataset.
  Depending on path, the dataset builder that is used comes from a generic dataset script (JSON, CSV, Parquet, text etc.) or from the dataset script (a python file) inside the dataset directory.

  For local datasets:

  • if path is a local directory (containing data files only)
    -> load a generic dataset builder (csv, json, text etc.) based on the content of the directory
    e.g. './path/to/directory/with/my/csv/data'.
  • if path is a local dataset script or a directory containing a local dataset script (if the script has the same name as the directory)
    -> load the dataset builder from the dataset script
    e.g. './dataset/squad' or './dataset/squad/squad.py'.

  For datasets on the Hugging Face Hub (list all available datasets and ids with datasets.list_datasets())

  • if path is a dataset repository on the HF hub (containing data files only)
    -> load a generic dataset builder (csv, text etc.) based on the content of the repository
    e.g. 'username/dataset_name', a dataset repository on the HF hub containing your data files.
  • if path is a dataset repository on the HF hub with a dataset script (if the script has the same name as the directory)
    -> load the dataset builder from the dataset script in the dataset repository
    e.g. glue, squad, 'username/dataset_name', a dataset repository on the HF hub containing a dataset script 'dataset_name.py'.

• 
  name (str, optional) —
  Defining the name of the dataset configuration.
  
• 
  data_dir (str, optional) —
  Defining the data_dir of the dataset configuration. If specified for the generic builders (csv, text etc.) or the Hub datasets and data_files is None,
  the behavior is equal to passing os.path.join(data_dir, **) as data_files to reference all the files in a directory.
  
• 
  data_files (str or Sequence or Mapping, optional) —
  Path(s) to source data file(s).
  
• 
  split (Split or str) —
  Which split of the data to load.
  If None, will return a dict with all splits (typically datasets.Split.TRAIN and datasets.Split.TEST).
  If given, will return a single Dataset.
  Splits can be combined and specified like in tensorflow-datasets.
  
• 
  cache_dir (str, optional) —
  Directory to read/write data. Defaults to "~/.cache/huggingface/datasets".
  
• 
  features (Features, optional) —
  Set the features type to use for this dataset.
  
• 
  download_config (DownloadConfig, optional) —
  Specific download configuration parameters.
  
• 
  download_mode (DownloadMode, defaults to REUSE_DATASET_IF_EXISTS) —
  Download/generate mode.
  
• 
  ignore_verifications (bool, defaults to False) —
  Ignore the verifications of the downloaded/processed dataset information (checksums/size/splits/…).
  
• 
  keep_in_memory (bool, defaults to None) —
  Whether to copy the dataset in-memory. If None, the dataset
  will not be copied in-memory unless explicitly enabled by setting datasets.config.IN_MEMORY_MAX_SIZE to
  nonzero. See more details in the improve performance section.
  
• 
  save_infos (bool, defaults to False) —
  Save the dataset information (checksums/size/splits/…).
  
• 
  revision (Version or str, optional) —
  Version of the dataset script to load.
  As datasets have their own git repository on the Datasets Hub, the default version “main” corresponds to their “main” branch.
  You can specify a different version than the default “main” by using a commit SHA or a git tag of the dataset repository.
  
• 
  use_auth_token (str or bool, optional) —
  Optional string or boolean to use as Bearer token for remote files on the Datasets Hub.
  If True, or not specified, will get token from "~/.huggingface".
  
• 
  task (str) —
  The task to prepare the dataset for during training and evaluation. Casts the dataset’s Features to standardized column names and types as detailed in datasets.tasks.
  
• 
  streaming (bool, defaults to False) —
  If set to True, don’t download the data files. Instead, it streams the data progressively while
  iterating on the dataset. An IterableDataset or IterableDatasetDict is returned instead in this case.

  Note that streaming works for datasets that use data formats that support being iterated over like txt, csv, jsonl for example.
  Json files may be downloaded completely. Also streaming from remote zip or gzip files is supported but other compressed formats
  like rar and xz are not yet supported. The tgz format doesn’t allow streaming.

• 
  num_proc (int, optional, defaults to None) —
  Number of processes when downloading and generating the dataset locally.
  Multiprocessing is disabled by default.

  Added in 2.7.0

• 
  **config_kwargs (additional keyword arguments) —
  Keyword arguments to be passed to the BuilderConfig
  and used in the DatasetBuilder.
  

• if split is not None: the dataset requested,
• if split is None, a DatasetDict with each split.

or IterableDataset or IterableDatasetDict: if streaming=True

• if split is not None, the dataset is requested
• if split is None, a ~datasets.streaming.IterableDatasetDict with each split.

>>> from datasets import load_dataset
>>> ds = load_dataset('rotten_tomatoes', split='train')


>>> data_files = {'train': 'train.csv', 'test': 'test.csv'}
>>> ds = load_dataset('namespace/your_dataset_name', data_files=data_files)

>>> from datasets import load_dataset
>>> ds = load_dataset('csv', data_files='path/to/local/my_dataset.csv')


>>> from datasets import load_dataset
>>> ds = load_dataset('json', data_files='path/to/local/my_dataset.json')


>>> from datasets import load_dataset
>>> ds = load_dataset('path/to/local/loading_script/loading_script.py', split='train')

>>> from datasets import load_dataset
>>> ds = load_dataset('rotten_tomatoes', split='train', streaming=True)

>>> from datasets import load_dataset
>>> ds = load_dataset('imagefolder', data_dir='/path/to/images', split='train')

Parameters

• 
  dataset_path (str) —
  Path (e.g. "dataset/train") or remote URI (e.g.
  "s3://my-bucket/dataset/train") of the Dataset or DatasetDict directory where the dataset will be
  loaded from.
  
• 
  fs (~filesystems.S3FileSystem or fsspec.spec.AbstractFileSystem, optional) —
  Instance of the remote filesystem used to download the files from.

  Deprecated in 2.9.0

  fs was deprecated in version 2.9.0 and will be removed in 3.0.0.
  Please use storage_options instead, e.g. storage_options=fs.storage_options.

• 
  keep_in_memory (bool, defaults to None) —
  Whether to copy the dataset in-memory. If None, the dataset
  will not be copied in-memory unless explicitly enabled by setting datasets.config.IN_MEMORY_MAX_SIZE to
  nonzero. See more details in the improve performance section.
  
• 
  storage_options (dict, optional) —
  Key/value pairs to be passed on to the file-system backend, if any.

  Added in 2.9.0

• If dataset_path is a path of a dataset directory: the dataset requested.
• If dataset_path is a path of a dataset dict directory, a DatasetDict with each split.

>>> from datasets import load_from_disk
>>> ds = load_from_disk('path/to/dataset/directory')

Parameters

• 
  path (str) —
  Path or name of the dataset.
  Depending on path, the dataset builder that is used comes from a generic dataset script (JSON, CSV, Parquet, text etc.) or from the dataset script (a python file) inside the dataset directory.

  For local datasets:

  • if path is a local directory (containing data files only)
    -> load a generic dataset builder (csv, json, text etc.) based on the content of the directory
    e.g. './path/to/directory/with/my/csv/data'.
  • if path is a local dataset script or a directory containing a local dataset script (if the script has the same name as the directory)
    -> load the dataset builder from the dataset script
    e.g. './dataset/squad' or './dataset/squad/squad.py'.

  For datasets on the Hugging Face Hub (list all available datasets and ids with datasets.list_datasets())

  • if path is a dataset repository on the HF hub (containing data files only)
    -> load a generic dataset builder (csv, text etc.) based on the content of the repository
    e.g. 'username/dataset_name', a dataset repository on the HF hub containing your data files.
  • if path is a dataset repository on the HF hub with a dataset script (if the script has the same name as the directory)
    -> load the dataset builder from the dataset script in the dataset repository
    e.g. glue, squad, 'username/dataset_name', a dataset repository on the HF hub containing a dataset script 'dataset_name.py'.

• 
  name (str, optional) —
  Defining the name of the dataset configuration.
  
• 
  data_dir (str, optional) —
  Defining the data_dir of the dataset configuration. If specified for the generic builders (csv, text etc.) or the Hub datasets and data_files is None,
  the behavior is equal to passing os.path.join(data_dir, **) as data_files to reference all the files in a directory.
  
• 
  data_files (str or Sequence or Mapping, optional) —
  Path(s) to source data file(s).
  
• 
  cache_dir (str, optional) —
  Directory to read/write data. Defaults to "~/.cache/huggingface/datasets".
  
• 
  features (Features, optional) —
  Set the features type to use for this dataset.
  
• 
  download_config (DownloadConfig, optional) —
  Specific download configuration parameters.
  
• 
  download_mode (DownloadMode, defaults to REUSE_DATASET_IF_EXISTS) —
  Download/generate mode.
  
• 
  revision (Version or str, optional) —
  Version of the dataset script to load.
  As datasets have their own git repository on the Datasets Hub, the default version “main” corresponds to their “main” branch.
  You can specify a different version than the default “main” by using a commit SHA or a git tag of the dataset repository.
  
• 
  use_auth_token (str or bool, optional) —
  Optional string or boolean to use as Bearer token for remote files on the Datasets Hub.
  If True, or not specified, will get token from "~/.huggingface".
  
• 
  **config_kwargs (additional keyword arguments) —
  Keyword arguments to be passed to the BuilderConfig
  and used in the DatasetBuilder.
  

>>> from datasets import load_dataset_builder
>>> ds_builder = load_dataset_builder('rotten_tomatoes')
>>> ds_builder.info.features
{'label': ClassLabel(num_classes=2, names=['neg', 'pos'], id=None),
 'text': Value(dtype='string', id=None)}

Parameters

• 
  path (str) — path to the dataset processing script with the dataset builder. Can be either:
  • a local path to processing script or the directory containing the script (if the script has the same name as the directory),
    e.g. './dataset/squad' or './dataset/squad/squad.py'
  • a dataset identifier on the Hugging Face Hub (list all available datasets and ids with datasets.list_datasets())
    e.g. 'squad', 'glue' or 'openai/webtext'

• 
  revision (Union[str, datasets.Version], optional) —
  If specified, the dataset module will be loaded from the datasets repository at this version.
  By default:
  • it is set to the local version of the lib.
  • it will also try to load it from the main branch if it’s not available at the local version of the lib.
    Specifying a version that is different from your local version of the lib might cause compatibility issues.

• 
  download_config (DownloadConfig, optional) —
  Specific download configuration parameters.
  
• 
  download_mode (DownloadMode, defaults to REUSE_DATASET_IF_EXISTS) —
  Download/generate mode.
  
• 
  dynamic_modules_path (str, defaults to ~/.cache/huggingface/modules/datasets_modules) —
  Optional path to the directory in which the dynamic modules are saved. It must have been initialized with init_dynamic_modules.
  By default the datasets and metrics are stored inside the datasets_modules module.
  
• 
  data_files (Union[Dict, List, str], optional) —
  Defining the data_files of the dataset configuration.
  
• 
  **download_kwargs (additional keyword arguments) —
  Optional attributes for DownloadConfig which will override the attributes in download_config if supplied,
  for example use_auth_token.
  

>>> from datasets import get_dataset_config_names
>>> get_dataset_config_names("glue")
['cola',
 'sst2',
 'mrpc',
 'qqp',
 'stsb',
 'mnli',
 'mnli_mismatched',
 'mnli_matched',
 'qnli',
 'rte',
 'wnli',
 'ax']

Parameters

• 
  path (str) — path to the dataset processing script with the dataset builder. Can be either:
  • a local path to processing script or the directory containing the script (if the script has the same name as the directory),
    e.g. './dataset/squad' or './dataset/squad/squad.py'
  • a dataset identifier on the Hugging Face Hub (list all available datasets and ids with datasets.list_datasets())
    e.g. 'squad', 'glue' or`'openai/webtext'

• 
  revision (Union[str, datasets.Version], optional) —
  If specified, the dataset module will be loaded from the datasets repository at this version.
  By default:
  • it is set to the local version of the lib.
  • it will also try to load it from the main branch if it’s not available at the local version of the lib.
    Specifying a version that is different from your local version of the lib might cause compatibility issues.

• 
  download_config (DownloadConfig, optional) —
  Specific download configuration parameters.
  
• 
  download_mode (DownloadMode, defaults to REUSE_DATASET_IF_EXISTS) —
  Download/generate mode.
  
• 
  data_files (Union[Dict, List, str], optional) —
  Defining the data_files of the dataset configuration.
  
• 
  use_auth_token (str or bool, optional) —
  Optional string or boolean to use as Bearer token for remote files on the Datasets Hub.
  If True, or not specified, will get token from "~/.huggingface".
  
• 
  **config_kwargs (additional keyword arguments) —
  Optional attributes for builder class which will override the attributes if supplied.
  

>>> from datasets import get_dataset_infos
>>> get_dataset_infos('rotten_tomatoes')
{'default': DatasetInfo(description="Movie Review Dataset.
 is a dataset of containing 5,331 positive and 5,331 negative processed
ences from Rotten Tomatoes movie reviews...), ...}

Parameters

• 
  path (str) — path to the dataset processing script with the dataset builder. Can be either:
  • a local path to processing script or the directory containing the script (if the script has the same name as the directory),
    e.g. './dataset/squad' or './dataset/squad/squad.py'
  • a dataset identifier on the Hugging Face Hub (list all available datasets and ids with datasets.list_datasets())
    e.g. 'squad', 'glue' or 'openai/webtext'

• 
  config_name (str, optional) —
  Defining the name of the dataset configuration.
  
• 
  data_files (str or Sequence or Mapping, optional) —
  Path(s) to source data file(s).
  
• 
  download_config (DownloadConfig, optional) —
  Specific download configuration parameters.
  
• 
  download_mode (DownloadMode, defaults to REUSE_DATASET_IF_EXISTS) —
  Download/generate mode.
  
• 
  revision (Version or str, optional) —
  Version of the dataset script to load.
  As datasets have their own git repository on the Datasets Hub, the default version “main” corresponds to their “main” branch.
  You can specify a different version than the default “main” by using a commit SHA or a git tag of the dataset repository.
  
• 
  use_auth_token (str or bool, optional) —
  Optional string or boolean to use as Bearer token for remote files on the Datasets Hub.
  If True, or not specified, will get token from "~/.huggingface".
  
• 
  **config_kwargs (additional keyword arguments) —
  Optional attributes for builder class which will override the attributes if supplied.
  

>>> from datasets import get_dataset_split_names
>>> get_dataset_split_names('rotten_tomatoes')
['train', 'validation', 'test']

Parameters

• 
  path (str) — Path to the dataset processing script with the dataset builder. Can be either:
  • a local path to processing script or the directory containing the script (if the script has the same name
    as the directory),
    e.g. './dataset/squad' or './dataset/squad/squad.py'.
  • a dataset identifier on the Hugging Face Hub (list all available datasets and ids with list_datasets())
    e.g. 'squad', 'glue' or 'openai/webtext'.

• 
  local_path (str) —
  Path to the local folder to copy the dataset script to.
  
• 
  download_config (DownloadConfig, optional) —
  Specific download configuration parameters.
  
• 
  **download_kwargs (additional keyword arguments) —
  Optional arguments for DownloadConfig which will override
  the attributes of download_config if supplied.
  

Parameters

• 
  with_community_metrics (bool, optional, default True) — Include the community provided metrics.
  
• 
  with_details (bool, optional, default False) — Return the full details on the metrics instead of only the short name.
  

>>> from datasets import list_metrics
>>> list_metrics()
['accuracy',
 'bertscore',
 'bleu',
 'bleurt',
 'cer',
 'chrf',
 ...
]

Parameters

• 
  path (str) —
  path to the metric processing script with the metric builder. Can be either:
  • a local path to processing script or the directory containing the script (if the script has the same name as the directory),
    e.g. './metrics/rouge' or './metrics/rogue/rouge.py'
  • a metric identifier on the HuggingFace datasets repo (list all available metrics with datasets.list_metrics())
    e.g. 'rouge' or 'bleu'

• 
  config_name (str, optional) — selecting a configuration for the metric (e.g. the GLUE metric has a configuration for each subset)
  
• 
  process_id (int, optional) — for distributed evaluation: id of the process
  
• 
  num_process (int, optional) — for distributed evaluation: total number of processes
  
• 
  cache_dir (Optional str) — path to store the temporary predictions and references (default to ~/.cache/huggingface/metrics/)
  
• 
  experiment_id (str) — A specific experiment id. This is used if several distributed evaluations share the same file system.
  This is useful to compute metrics in distributed setups (in particular non-additive metrics like F1).
  
• 
  keep_in_memory (bool) — Whether to store the temporary results in memory (defaults to False)
  
• 
  download_config (Optional datasets.DownloadConfig — specific download configuration parameters.
  
• 
  download_mode (DownloadMode, default REUSE_DATASET_IF_EXISTS) — Download/generate mode.
  
• 
  revision (Optional Union[str, datasets.Version]) — if specified, the module will be loaded from the datasets repository
  at this version. By default, it is set to the local version of the lib. Specifying a version that is different from
  your local version of the lib might cause compatibility issues.
  

>>> from datasets import load_metric
>>> accuracy = load_metric('accuracy')
>>> accuracy.compute(references=[1, 0], predictions=[1, 1])
{'accuracy': 0.5}

Parameters

• 
  path (str) — path to the dataset processing script with the dataset builder. Can be either:
  • a local path to processing script or the directory containing the script (if the script has the same name as the directory),
    e.g. './dataset/squad' or './dataset/squad/squad.py'
  • a dataset identifier on the Hugging Face Hub (list all available datasets and ids with datasets.list_datasets())
    e.g. 'squad', 'glue' or 'openai/webtext'

• 
  local_path (str) — path to the local folder to copy the datset script to.
  
• 
  download_config (Optional datasets.DownloadConfig) — specific download configuration parameters.
  
• 
  **download_kwargs (additional keyword arguments) — optional attributes for DownloadConfig() which will override the attributes in download_config if supplied.