Skip to main content

Module chunk

Module chunk 

Source
Expand description

Budget-driven text segmentation. The size budget is injected as a fits predicate, so the caller owns measurement; markdown mode keeps code fences self-contained across segment boundaries.

Structsยง

Chunker ๐Ÿ”’

Enumsยง

Cut ๐Ÿ”’
Where the next segment ends.

Functionsยง

assemble ๐Ÿ”’
Joins the reopened head, body, and an optional closing fence into a segment.
chunk
Lazily splits text using a monotonic fits predicate.
discover ๐Ÿ”’
Extends lines to cover at least upto lines of rest, or until rest is exhausted. Each entry pairs the byte offset past a line with the fence info open after it (markdown mode only).
fence_line_info ๐Ÿ”’
The info string of a triple-backtick fence line, or None when the line is not a fence delimiter.
reopen ๐Ÿ”’
toggle_fence ๐Ÿ”’
Toggles fence when line is a code-fence delimiter: an outside delimiter opens a fence carrying its info string, an inside delimiter closes it.

Type Aliasesยง

Line ๐Ÿ”’
Byte offset past a line, paired with the fence info open after it.