Pioneer datasets: create, version, inspect, and delete

Pioneer stores and versions your training datasets automatically. Learn how to create them via generation or auto-labeling, then list, inspect, and delete them.

Datasets in Pioneer are collections of labeled examples used to train and evaluate models. Each dataset has a name you define, and Pioneer versions it automatically as you add or regenerate data. You reference a dataset by name when starting a training job or running an evaluation — so the name you choose is the stable identifier you’ll use throughout your workflow.

How datasets are created

You create datasets in two ways:

Synthetic data generation — Use POST /generate to have Pioneer produce labeled examples from a description of your domain and the labels you care about. This is the fastest way to bootstrap a dataset without any existing labeled data.

curl -X POST https://api.pioneer.ai/generate \
  -H "X-API-Key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "task_type": "ner",
    "dataset_name": "my-ner-dataset",
    "labels": ["person", "company", "product"],
    "num_examples": 100,
    "domain_description": "Tech industry news articles"
  }'

POST /generate also supports task types beyond NER — pass a different task_type and its required field:

task_type	Required field	Produces
ner	labels	Named-entity recognition dataset
classification	labels	Text classification dataset
custom	prompt	Free-form prompt-based dataset
decoder	domain_description	Instruction-tuning (chat format) dataset
records	fields	Structured records
fields	input_fields and output_fields	Structured records with separate input/output schemas

The endpoint returns 202 immediately with a job_id; generation itself runs asynchronously. Once generated or labeled, examples are stored in your dataset automatically. Poll GET /generate/jobs/:job_id until status is ready (or failed, in which case check the error field) before starting training.

Uploading your own dataset

Uploading your own data: Use POST/felix/datasets/upload/url if you already have labeled data. This is a three-step process:

Step 1. Get a presigned upload URL

 curl -X POST https://api.pioneer.ai/felix/datasets/upload/url \
    -H "X-API-Key: YOUR_API_KEY" \
    -H "Content-Type: application/json" \                                                                               
    -d '{
      "dataset_name": "my-ner-dataset",                                                                                 
      "dataset_type": "ner",
      "type": "training",
      "filename": "data.jsonl"
    }'      

Only dataset_name is required — dataset_type defaults to "ner" if omitted, and accepts "ner", "classification", "custom", or "decoder" (the type field is "training" by default; "benchmark" is rejected here since benchmark datasets are system-managed). The response includes ‘presigned_url’, ‘dataset_id’, and ‘version_number’.

Step 2. Upload the file directly to S3

curl -X PUT "<presigned_url from step 1 response>" \                                                                  
   --upload-file ./data.jsonl

This is a direct HTTP PUT to S3. Do not include your API key here.

Step 3. Trigger processing

curl -X POST https://api.pioneer.ai/felix/datasets/upload/process \
    -H "X-API-Key: YOUR_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{                                                                                                               
      "dataset_id": "<dataset_id from step 1>"
    }' 

This call returns immediately (202) with status uploading; the dataset then moves through the remaining statuses in the background: initialized → uploading → converting → validating → ready

Poll GET /felix/datasets/{name}/{version} until status is ready (or failed, in which case check processing_error) before starting a training job. You can also pass latest in place of a version number to always fetch the newest version.

Listing your datasets

Retrieve all datasets in your account:

curl https://api.pioneer.ai/felix/datasets \
  -H "X-API-Key: YOUR_API_KEY"

The response lists each dataset by name along with metadata such as creation time and version count. Datasets with status failed are excluded by default — pass include_failed=true to see them too.

Inspecting a dataset

To see the versions and details of a specific dataset, pass its name:

curl https://api.pioneer.ai/felix/datasets/my-ner-dataset \
  -H "X-API-Key: YOUR_API_KEY"

This returns version history and example counts, which is useful for confirming the dataset is ready before training.

Deleting a dataset

curl -X DELETE https://api.pioneer.ai/felix/datasets/my-ner-dataset \
  -H "X-API-Key: YOUR_API_KEY"

Deleting a dataset by name soft-deletes it and all its versions — S3 data is preserved in case you need to restore it. If any evaluation or training job is currently pending/running against the dataset, the delete is rejected with 409 Conflict until that job finishes. Once deletion succeeds, already-completed training jobs and evaluations remain queryable, but you can no longer start new jobs referencing it.

Dataset storage is free. You are not charged for storing datasets in Pioneer, regardless of size or number of versions.

Dataset endpoints summary

Method	Endpoint	Description
GET	/felix/datasets	List all datasets
GET	/felix/datasets/:name	Get all versions for a dataset
GET	/felix/datasets/:name/:version	Get status and metadata for a specific version (:version also accepts latest)
GET	/felix/datasets/:name/:version/preview	Preview a sample of rows without downloading the full file
GET	/felix/datasets/:name/:version/download	Download a version as jsonl, csv, or parquet
DELETE	/felix/datasets/:name	Soft-delete a dataset and all its versions (rejected with 409 while a job/eval is actively using it)
DELETE	/felix/datasets/:name/:version	Soft-delete a specific version
POST	/felix/datasets/upload/url	Get presigned S3 URL for direct upload
POST	/felix/datasets/upload/process	Trigger processing after S3 upload
POST	/felix/datasets/merge	Merge multiple datasets of the same type into one new dataset
POST	/felix/datasets/:name/:version/push-to-hub	Push a dataset version to HuggingFace Hub
POST	/felix/datasets/pull-from-hub	Import a dataset from HuggingFace Hub
POST	/felix/datasets/preview-from-hub	Preview a HuggingFace Hub dataset before importing it
POST	/generate	Start a synthetic data generation job (202 + job_id, async)
GET	/generate/jobs/:job_id	Poll generation job status
POST	/generate/ner/label-existing	Auto-label raw text for NER (synchronous — returns results directly, no job_id)
POST	/generate/classification/label-existing	Auto-classify raw text (synchronous — returns results directly, no job_id)

Data Privacy: If you would like to opt out of having your data used in Fastino’s model training, please email [e-mail] and we will ensure your data is excluded from our training pipelines.