> ## Documentation Index
> Fetch the complete documentation index at: https://pioneer-kelton-add-decoder-inference-prices.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Pioneer datasets: create, version, inspect, and delete

> Pioneer stores and versions your training datasets automatically. Learn how to create them via generation or auto-labeling, then list, inspect, and delete them.

Datasets in Pioneer are collections of labeled examples used to train and evaluate models. Each dataset has a name you define, and Pioneer versions it automatically as you add or regenerate data. You reference a dataset by name when starting a training job or running an evaluation — so the name you choose is the stable identifier you'll use throughout your workflow.

## How datasets are created

You create datasets in two ways:

**Synthetic data generation** — Use `POST /generate` to have Pioneer produce labeled examples from a description of your domain and the labels you care about. This is the fastest way to bootstrap a dataset without any existing labeled data.

```bash theme={null}
curl -X POST https://api.pioneer.ai/generate \
  -H "X-API-Key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "task_type": "ner",
    "dataset_name": "my-ner-dataset",
    "labels": ["person", "company", "product"],
    "num_examples": 100,
    "domain_description": "Tech industry news articles"
  }'
```

`POST /generate` also supports task types beyond NER — pass a different `task_type` and its required field:

| `task_type`      | Required field                     | Produces                                              |
| ---------------- | ---------------------------------- | ----------------------------------------------------- |
| `ner`            | `labels`                           | Named-entity recognition dataset                      |
| `classification` | `labels`                           | Text classification dataset                           |
| `custom`         | `prompt`                           | Free-form prompt-based dataset                        |
| `decoder`        | `domain_description`               | Instruction-tuning (chat format) dataset              |
| `records`        | `fields`                           | Structured records                                    |
| `fields`         | `input_fields` and `output_fields` | Structured records with separate input/output schemas |

The endpoint returns `202` immediately with a `job_id`; generation itself runs asynchronously. Once generated or labeled, examples are stored in your dataset automatically. Poll `GET /generate/jobs/:job_id` until `status` is `ready` (or `failed`, in which case check the `error` field) before starting training.

## Uploading your own dataset

Uploading your own data: Use `POST/felix/datasets/upload/url` if you already have labeled data. This is a three-step process:

#### Step 1. Get a presigned upload URL

```bash theme={null}
 curl -X POST https://api.pioneer.ai/felix/datasets/upload/url \
    -H "X-API-Key: YOUR_API_KEY" \
    -H "Content-Type: application/json" \                                                                               
    -d '{
      "dataset_name": "my-ner-dataset",                                                                                 
      "dataset_type": "ner",
      "type": "training",
      "filename": "data.jsonl"
    }'      
```

Only `dataset_name` is required — `dataset_type` defaults to `"ner"` if omitted, and accepts `"ner"`, `"classification"`, `"custom"`, or `"decoder"` (the `type` field is `"training"` by default; `"benchmark"` is rejected here since benchmark datasets are system-managed). The response includes 'presigned\_url', 'dataset\_id', and 'version\_number'.

#### Step 2. Upload the file directly to S3

```bash theme={null}
curl -X PUT "<presigned_url from step 1 response>" \                                                                  
   --upload-file ./data.jsonl
```

This is a direct HTTP PUT to S3. Do not include your API key here.

#### Step 3. Trigger processing

```bash theme={null}
curl -X POST https://api.pioneer.ai/felix/datasets/upload/process \
    -H "X-API-Key: YOUR_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{                                                                                                               
      "dataset_id": "<dataset_id from step 1>"
    }' 
```

This call returns immediately (`202`) with status `uploading`; the dataset then moves through the remaining statuses in the background: initialized → uploading → converting → validating → ready

Poll `GET /felix/datasets/{name}/{version}` until `status` is `ready` (or `failed`, in which case check `processing_error`) before starting a training job. You can also pass `latest` in place of a version number to always fetch the newest version.

## Listing your datasets

Retrieve all datasets in your account:

```bash theme={null}
curl https://api.pioneer.ai/felix/datasets \
  -H "X-API-Key: YOUR_API_KEY"
```

The response lists each dataset by name along with metadata such as creation time and version count. Datasets with status `failed` are excluded by default — pass `include_failed=true` to see them too.

## Inspecting a dataset

To see the versions and details of a specific dataset, pass its name:

```bash theme={null}
curl https://api.pioneer.ai/felix/datasets/my-ner-dataset \
  -H "X-API-Key: YOUR_API_KEY"
```

This returns version history and example counts, which is useful for confirming the dataset is ready before training.

## Deleting a dataset

```bash theme={null}
curl -X DELETE https://api.pioneer.ai/felix/datasets/my-ner-dataset \
  -H "X-API-Key: YOUR_API_KEY"
```

Deleting a dataset by name soft-deletes it and all its versions — S3 data is preserved in case you need to restore it. If any evaluation or training job is currently `pending`/`running` against the dataset, the delete is rejected with `409 Conflict` until that job finishes. Once deletion succeeds, already-completed training jobs and evaluations remain queryable, but you can no longer start new jobs referencing it.

<Note>
  Dataset storage is free. You are not charged for storing datasets in Pioneer, regardless of size or number of versions.
</Note>

## Dataset endpoints summary

| Method   | Endpoint                                     | Description                                                                                            |
| -------- | -------------------------------------------- | ------------------------------------------------------------------------------------------------------ |
| `GET`    | `/felix/datasets`                            | List all datasets                                                                                      |
| `GET`    | `/felix/datasets/:name`                      | Get all versions for a dataset                                                                         |
| `GET`    | `/felix/datasets/:name/:version`             | Get status and metadata for a specific version (`:version` also accepts `latest`)                      |
| `GET`    | `/felix/datasets/:name/:version/preview`     | Preview a sample of rows without downloading the full file                                             |
| `GET`    | `/felix/datasets/:name/:version/download`    | Download a version as `jsonl`, `csv`, or `parquet`                                                     |
| `DELETE` | `/felix/datasets/:name`                      | Soft-delete a dataset and all its versions (rejected with `409` while a job/eval is actively using it) |
| `DELETE` | `/felix/datasets/:name/:version`             | Soft-delete a specific version                                                                         |
| `POST`   | `/felix/datasets/upload/url`                 | Get presigned S3 URL for direct upload                                                                 |
| `POST`   | `/felix/datasets/upload/process`             | Trigger processing after S3 upload                                                                     |
| `POST`   | `/felix/datasets/merge`                      | Merge multiple datasets of the same type into one new dataset                                          |
| `POST`   | `/felix/datasets/:name/:version/push-to-hub` | Push a dataset version to HuggingFace Hub                                                              |
| `POST`   | `/felix/datasets/pull-from-hub`              | Import a dataset from HuggingFace Hub                                                                  |
| `POST`   | `/felix/datasets/preview-from-hub`           | Preview a HuggingFace Hub dataset before importing it                                                  |
| `POST`   | `/generate`                                  | Start a synthetic data generation job (`202` + `job_id`, async)                                        |
| `GET`    | `/generate/jobs/:job_id`                     | Poll generation job status                                                                             |
| `POST`   | `/generate/ner/label-existing`               | Auto-label raw text for NER (synchronous — returns results directly, no `job_id`)                      |
| `POST`   | `/generate/classification/label-existing`    | Auto-classify raw text (synchronous — returns results directly, no `job_id`)                           |

<Note>
  **Data Privacy:** If you would like to opt out of having your data used in Fastino's model training, please email [support@fastino.ai](mailto:support@fastino.ai) and we will ensure your data is excluded from our training pipelines.
</Note>