Self Checks
Summary
The default delimiter for parser_config.delimiter is set in at least 11 places across the codebase, with 4 distinct values. The shipped defaults diverge by callback site, and the divergence is large enough to produce a real chunk-count difference for the same input text. There is no single source of truth.
The codebase has never had a guiding question of "what should the default delimiter be?" — every new parser path has copy-pasted whichever default was nearby, and the inconsistency has accumulated silently.
Environment
- RAGFlow: v0.27.0 (
ec9c08d80, tag v0.27.0)
- Affected files:
rag/app/naive.py, deepdoc/parser/txt_parser.py, rag/app/email.py, rag/app/book.py, rag/nlp/__init__.py, web/src/components/chunk-method-dialog/use-default-parser-values.ts, web/src/pages/dataset/dataset-setting/index.tsx, web/src/components/delimiter-form-field.tsx
The 11 defaults
Grouped by category:
Group 1 — UI defaults (the value the user sees and submits)
| Location |
Default |
parsed_dels |
web/src/components/chunk-method-dialog/use-default-parser-values.ts:30 |
'\n' |
['\n'] (1) |
web/src/pages/dataset/dataset-setting/index.tsx:71 |
'\n' |
['\n'] (1) |
web/src/components/delimiter-form-field.tsx:78 (fallback when field.value is undefined) |
'\n' |
['\n'] (1) |
The three agree: '\n' (single newline). This is the user's "safe minimum" — no surprises.
Group 2 — Python function signature defaults (effectively dead code)
| Location |
Default |
parsed_dels |
rag/nlp/__init__.py:1366 naive_merge |
'\n。；！？' |
['\n', '。', '；', '！', '？'] (5, Chinese-only) |
rag/nlp/__init__.py:1429 naive_merge_with_images |
'\n。；！？' |
same |
rag/nlp/__init__.py:1785 naive_merge_docx |
'\n。；！？' |
same |
deepdoc/parser/txt_parser.py:30 RAGFlowTxtParser.__call__ |
'\n!?;。；！？' |
['\n', '!', '?', ';', '。', '；', '！', '？'] (8, full set) |
deepdoc/parser/txt_parser.py:35 RAGFlowTxtParser.parser_txt |
'\n!?;。；！？' |
same |
These trigger only when the function is called without parser_config, which only happens in unit tests. In production (rag/svr/task_executor.py:360), chunker.chunk() is always called with parser_config=parser_config_for_chunk, so these defaults are unreachable for real users.
But they encode the maintainers' mental model of "the default delimiter" — and the two camps disagree: 3 functions say '\n。；！？', 2 say '\n!?;。；！？'.
Group 3 — parser_config.get defaults (the ones that ship)
| Location |
Default |
parsed_dels |
rag/app/naive.py:1137 (txt) |
'\n!?;。；！？' |
8 delims |
rag/app/naive.py:1149 (markdown) |
'\n!?;。；！？' |
8 delims |
rag/app/naive.py:1030 (docx) |
'\n!?。；！？' |
7 delims (no ;) |
rag/app/naive.py:1304 (image) |
'\n!?。；！？' |
7 delims (no ;) |
rag/app/email.py:102 |
'\n!?。；！？' |
7 delims (no ;) |
rag/app/book.py:184 |
'\n。；！？' |
5 delims (Chinese-only) |
These are the ones that actually reach users. The parser_config.get fallback is consulted when the user has set parser_config but the delimiter key is missing (e.g. they didn't fill the field). Across 5 file types, the divergence is:
- txt, markdown — full set including
;
- docx, image, email — same set minus
;
- book — Chinese-only, no English punctuation at all
There is no consistent reason for this split. Both .txt and .docx typically contain English text; both contain semicolons. The ; is a hard sentence-clause separator in English prose. Removing it from .docx parsing causes chunking inconsistency: the same paragraph of text with a semicolon produces one chunk in .txt and a different (possibly oversize) chunk in .docx.
Why the split is almost certainly drift, not design
I traced the defaults through git history. The diverging values predate the recent refactors (#17383 / #17203 / #17808) — they've been in the codebase for a long time. There's no commit message or comment that explains why docx and email chose '\n!?。；！？' while txt and markdown chose '\n!?;。；！？'.
The closest thing to a designed choice is the book.py:184 default '\n。；！？' (Chinese-only). The book parser is commonly used for Chinese/foreign-language books, and a Chinese-only default makes sense for that audience. But the rest of the divergence is unexplained.
Reproduction
This is a 10-line audit script anyone can run from the repo root:
from rag.nlp.delim import parse_delimiter_field
defaults = [
("chunk-method-dialog/use-default-parser-values.ts:30", "\n"),
("dataset-setting/index.tsx:71", "\n"),
("delimiter-form-field.tsx:78", "\n"),
("naive_merge signature", "\n。；！？"),
("naive_merge_with_images signature", "\n。；！？"),
("naive_merge_docx signature", "\n。；！？"),
("TxtParser.__call__ signature", "\n!?;。；！？"),
("TxtParser.parser_txt signature", "\n!?;。；！？"),
("naive.py:1137 (txt)", "\n!?;。；！？"),
("naive.py:1149 (markdown)", "\n!?;。；！？"),
("naive.py:1030 (docx)", "\n!?。；！？"),
("naive.py:1304 (image)", "\n!?。；！？"),
("email.py:102", "\n!?。；！？"),
("book.py:184 (book)", "\n。；！？"),
]
distinct_values = sorted(set(v for _, v in defaults))
print(f"Distinct defaults in production: {len(distinct_values)}")
for v in distinct_values:
print(f" {v!r} -> parsed: {parse_delimiter_field(v)}")Expected output (on any current main):
Distinct defaults in production: 4
'\n' -> parsed: ['\n']
'\n!?;。；！？' -> parsed: ['\n', '!', '?', ';', '。', '；', '！', '？']
'\n!?。；！？' -> parsed: ['\n', '!', '?', '。', '；', '！', '？']
'\n。；！？' -> parsed: ['\n', '。', '；', '！', '？']
The user-visible effect
Measured on a 2101-token sample with . after every sentence (no \n):
| Default |
parsed_dels count |
Chunks for the sample |
Per-file-type scope |
'\n' (UI) |
1 |
1 (oversized, 2102 tokens) |
hypothetical fall-through |
'\n!?;。；！？' (txt, markdown) |
8 |
3 (490–1002 tokens) |
.txt, .md |
'\n!?。；！？' (docx, image, email) |
7 |
3 (490–1002 tokens) |
.docx, image, .eml |
'\n。；！？' (book, naive_merge signatures) |
5 |
1 (oversized, 2102 tokens) |
.book, hypothetical fall-through |
For text with normal newlines the four values converge because newlines dominate the splits. For text without newlines but with punctuation — OCR'd PDFs, copy-pasted prose, normalized text — the values diverge:
'\n!?;。；！？' (txt, markdown) splits correctly on every sentence boundary.'\n!?。；！？' (docx, image, email) splits correctly on every sentence boundary too, except at ; — so a paragraph like "The rain; the sun; the wind" becomes one oversized chunk instead of three.'\n。；！？' (book) misses every English sentence boundary because it has no English punctuation. The same paragraph with English punctuation produces one giant chunk.
This is a latent bug. PR #17203 didn't touch it because the reporter's data had newlines. Users with OCR-normalized or paragraph-stripped text will hit it. The robustness fix is to consolidate to one default.
Suggested fix
A single source of truth for the shipped default. Three concrete options:
Option A — One constant, all parser_config.get calls use it
Add a constant:
# rag/nlp/delim.py
DEFAULT_DELIMITER = "\n!?;。；！？"
Replace every parser_config.get("delimiter", "...") call with parser_config.get("delimiter", DEFAULT_DELIMITER). Apply the same to the UI default in chunk-method-dialog/use-default-parser-values.ts (or leave the UI default at '\n' since that's the safe minimum).
Option B — One parser_config.get default + per-file-type English-only delisting
For parsers that don't want extra English punctuation, use the function signature default. But keep the parser_config.get at the full set for all parsers. This loses the ability to have parser-specific defaults but gains consistency.
Option C — Just fix the silent divergence
The minimum viable fix: change naive.py:1030, naive.py:1304, email.py:102 to use '\n!?;。；！？' (the same as txt/markdown). Add ; back. This alone removes the "no ; for docx/email" inconsistency in 3 lines.
The function signature defaults in rag/nlp/__init__.py can stay as-is (dead code) but ideally also collapse to one value.
Acceptance criteria
Any of these would prevent the issue:
- One shipped default for the main
parser_config.delimiter field, defined as a constant and reused across all parser_config.get calls.
- An
assert in the test suite that fails if parser_config.get("delimiter", ...) defaults diverge across parsers.
- A note in the developer docs (or in the
delim.py module docstring) explaining the intent behind the shipped default and why one value fits all parsers.
Workaround (today)
There's no per-dataset escape hatch. If a user hits the English-punctuation gap in .docx, they have to switch the parser to naive (general) or pre-process the file to substitute ; for ; (or vice versa). Not documented.
Related
Self Checks
custom delimiterbug but not the defaults specificallySummary
The default delimiter for
parser_config.delimiteris set in at least 11 places across the codebase, with 4 distinct values. The shipped defaults diverge by callback site, and the divergence is large enough to produce a real chunk-count difference for the same input text. There is no single source of truth.The codebase has never had a guiding question of "what should the default delimiter be?" — every new parser path has copy-pasted whichever default was nearby, and the inconsistency has accumulated silently.
Environment
ec9c08d80, tagv0.27.0)rag/app/naive.py,deepdoc/parser/txt_parser.py,rag/app/email.py,rag/app/book.py,rag/nlp/__init__.py,web/src/components/chunk-method-dialog/use-default-parser-values.ts,web/src/pages/dataset/dataset-setting/index.tsx,web/src/components/delimiter-form-field.tsxThe 11 defaults
Grouped by category:
Group 1 — UI defaults (the value the user sees and submits)
parsed_delsweb/src/components/chunk-method-dialog/use-default-parser-values.ts:30'\n'['\n'](1)web/src/pages/dataset/dataset-setting/index.tsx:71'\n'['\n'](1)web/src/components/delimiter-form-field.tsx:78(fallback whenfield.valueisundefined)'\n'['\n'](1)The three agree:
'\n'(single newline). This is the user's "safe minimum" — no surprises.Group 2 — Python function signature defaults (effectively dead code)
parsed_delsrag/nlp/__init__.py:1366naive_merge'\n。；！？'['\n', '。', '；', '！', '？'](5, Chinese-only)rag/nlp/__init__.py:1429naive_merge_with_images'\n。；！？'rag/nlp/__init__.py:1785naive_merge_docx'\n。；！？'deepdoc/parser/txt_parser.py:30RAGFlowTxtParser.__call__'\n!?;。；！？'['\n', '!', '?', ';', '。', '；', '！', '？'](8, full set)deepdoc/parser/txt_parser.py:35RAGFlowTxtParser.parser_txt'\n!?;。；！？'These trigger only when the function is called without
parser_config, which only happens in unit tests. In production (rag/svr/task_executor.py:360),chunker.chunk()is always called withparser_config=parser_config_for_chunk, so these defaults are unreachable for real users.But they encode the maintainers' mental model of "the default delimiter" — and the two camps disagree: 3 functions say
'\n。；！？', 2 say'\n!?;。；！？'.Group 3 —
parser_config.getdefaults (the ones that ship)parsed_delsrag/app/naive.py:1137(txt)'\n!?;。；！？'rag/app/naive.py:1149(markdown)'\n!?;。；！？'rag/app/naive.py:1030(docx)'\n!?。；！？';)rag/app/naive.py:1304(image)'\n!?。；！？';)rag/app/email.py:102'\n!?。；！？';)rag/app/book.py:184'\n。；！？'These are the ones that actually reach users. The
parser_config.getfallback is consulted when the user has setparser_configbut thedelimiterkey is missing (e.g. they didn't fill the field). Across 5 file types, the divergence is:;;There is no consistent reason for this split. Both
.txtand.docxtypically contain English text; both contain semicolons. The;is a hard sentence-clause separator in English prose. Removing it from.docxparsing causes chunking inconsistency: the same paragraph of text with a semicolon produces one chunk in.txtand a different (possibly oversize) chunk in.docx.Why the split is almost certainly drift, not design
I traced the defaults through git history. The diverging values predate the recent refactors (#17383 / #17203 / #17808) — they've been in the codebase for a long time. There's no commit message or comment that explains why docx and email chose
'\n!?。；！？'while txt and markdown chose'\n!?;。；！？'.The closest thing to a designed choice is the
book.py:184default'\n。；！？'(Chinese-only). The book parser is commonly used for Chinese/foreign-language books, and a Chinese-only default makes sense for that audience. But the rest of the divergence is unexplained.Reproduction
This is a 10-line audit script anyone can run from the repo root:
Expected output (on any current
main):The user-visible effect
Measured on a 2101-token sample with
.after every sentence (no\n):parsed_delscount'\n'(UI)'\n!?;。；！？'(txt, markdown)'\n!?。；！？'(docx, image, email)'\n。；！？'(book, naive_merge signatures)For text with normal newlines the four values converge because newlines dominate the splits. For text without newlines but with punctuation — OCR'd PDFs, copy-pasted prose, normalized text — the values diverge:
'\n!?;。；！？'(txt, markdown) splits correctly on every sentence boundary.'\n!?。；！？'(docx, image, email) splits correctly on every sentence boundary too, except at;— so a paragraph like "The rain; the sun; the wind" becomes one oversized chunk instead of three.'\n。；！？'(book) misses every English sentence boundary because it has no English punctuation. The same paragraph with English punctuation produces one giant chunk.This is a latent bug. PR #17203 didn't touch it because the reporter's data had newlines. Users with OCR-normalized or paragraph-stripped text will hit it. The robustness fix is to consolidate to one default.
Suggested fix
A single source of truth for the shipped default. Three concrete options:
Option A — One constant, all
parser_config.getcalls use itAdd a constant:
Replace every
parser_config.get("delimiter", "...")call withparser_config.get("delimiter", DEFAULT_DELIMITER). Apply the same to the UI default inchunk-method-dialog/use-default-parser-values.ts(or leave the UI default at'\n'since that's the safe minimum).Option B — One
parser_config.getdefault + per-file-type English-only delistingFor parsers that don't want extra English punctuation, use the function signature default. But keep the
parser_config.getat the full set for all parsers. This loses the ability to have parser-specific defaults but gains consistency.Option C — Just fix the silent divergence
The minimum viable fix: change
naive.py:1030,naive.py:1304,email.py:102to use'\n!?;。；！？'(the same as txt/markdown). Add;back. This alone removes the "no;for docx/email" inconsistency in 3 lines.The function signature defaults in
rag/nlp/__init__.pycan stay as-is (dead code) but ideally also collapse to one value.Acceptance criteria
Any of these would prevent the issue:
parser_config.delimiterfield, defined as a constant and reused across allparser_config.getcalls.assertin the test suite that fails ifparser_config.get("delimiter", ...)defaults diverge across parsers.delim.pymodule docstring) explaining the intent behind the shipped default and why one value fits all parsers.Workaround (today)
There's no per-dataset escape hatch. If a user hits the English-punctuation gap in
.docx, they have to switch the parser tonaive(general) or pre-process the file to substitute;for;(or vice versa). Not documented.Related
custom delimiterbug. The defaults divergence is the prerequisite for the "no;" sub-case of that bug.chunk_token_numignored bug. The defaults divergence is one of several open angles.naive_merge/naive_merge_with_images/RAGFlowTxtParser. Doesn't touch defaults either.naive_merge_docx. Doesn't touch defaults.book.py/paper.pychunking. Doesn't touch defaults.