Gretel AI Releases a New Multilingual Synthetic Financial Dataset on HuggingFace 🤗 for AI Developers Tackling Personally Identifiable Information PII Detection

Gretel's Navigator tool assists developers in creating customized synthetic datasets for PII detection by allowing them to tailor the datasets to their specific needs4. This tool helps developers rapidly create large-scale, diverse, privacy-preserving datasets that accurately reflect the characteristics and challenges of their domain. As a result, PII detection models are well-prepared for real-world scenarios and unique document types.

Key features of the synthetic financial document dataset generated by Navigator include extensive records, coverage of financial document formats, synthetic PII, full-length documents, multilingual support, and quality assurance2. The dataset can be used for training NER models, testing PII scanning systems, evaluating de-identification systems, and developing data privacy solutions for the financial industry.

The quality of the dataset is ensured through the LLM-as-a-Judge technique using the Mistral-7B language model. Each generated record is evaluated based on conformance, quality, toxicity, bias, and groundedness, ensuring the dataset's integrity and reliability for training robust PII detection models.

Gretel's multilingual Financial Document Dataset has several key features:

1. Extensive Records: The dataset contains 55,940 records, which are partitioned into 50,776 training samples and 5,164 test samples2.
2. Coverage of Financial Document Formats: It includes 100 distinct financial document formats, with 20 specific subtypes for each format3. This covers everything from customer support logs to insurance policies and AML compliance audits.
3. Synthetic PII: The dataset contains 29 distinct types of personally identifiable information (PII), which are aligned with Python Faker library generators for easy detection and replacement3.
4. Full-Length Documents: The average length of documents in the dataset is 1,357 characters.
5. Multilingual Support: The dataset supports multiple languages, including English, Spanish, Swedish, German, Italian, Dutch, and French3.
6. Quality Assurance: The LLM-as-a-Judge technique with the Mistral-7B language model is used to ensure data quality and evaluate conformance, quality, toxicity, bias, and groundedness.

These features make the dataset a valuable resource for AI developers and organizations working with sensitive data, enabling them to build more effective and domain-specific PII detection systems4.