Record · corpus.blog/posts/01a08783-2996-72ba-a876-ac53e58d286d

Japanese and CJK Text Preprocessing for RAG - Tokenization, Normalization, and Chunk Boundaries | hidekazu-konishi.com

hidekazu-konishi.com · published 2 August 2026

A record is what was published and who pointed at it. The text of the post is not held here: read it at the source, or at its Wayback capture.

Read the post at hidekazu-konishi.com Wayback capture

0blogs citing

First cited: —
Most recent: —
Rank this month: not ranked
External links: 25
Words captured: 10,434

Cited by

No blog we hold has cited this post yet.

Links from this post

Link

Host

https://github.com/openai/tiktoken

github.com

https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/capabilities/get-token-count

docs.cloud.google.com

https://www.unicode.org/reports/tr29/

unicode.org

https://unicode.org/reports/tr15/

unicode.org

http://www.unicode.org/reports/tr14/

unicode.org

https://www.unicode.org/reports/tr11/

unicode.org

https://www.unicode.org/reports/tr50/

unicode.org

https://www.unicode.org/versions/Unicode17.0.0/

unicode.org

https://docs.opensearch.org/latest/analyzers/language-analyzers/cjk/

docs.opensearch.org

https://docs.opensearch.org/latest/analyzers/language-analyzers/icu/

docs.opensearch.org

https://docs.opensearch.org/latest/analyzers/language-analyzers/index/

docs.opensearch.org

https://docs.opensearch.org/latest/analyzers/tokenizers/ngram/

docs.opensearch.org

https://docs.opensearch.org/latest/install-and-configure/additional-plugins/index/

docs.opensearch.org

https://lucene.apache.org/core/10_5_0/analysis/kuromoji/org/apache/lucene/analysis/ja/JapaneseTokenizer.html

lucene.apache.org

https://lucene.apache.org/core/10_5_0/analysis/kuromoji/org/apache/lucene/analysis/ja/JapaneseTokenizer.Mode.html

lucene.apache.org

https://lucene.apache.org/core/10_5_0/analysis/nori/org/apache/lucene/analysis/ko/KoreanTokenizer.DecompoundMode.html

lucene.apache.org

https://lucene.apache.org/core/10_5_0/analysis/common/org/apache/lucene/analysis/cjk/CJKBigramFilter.html

lucene.apache.org

https://lucene.apache.org/core/10_5_0/analysis/common/org/apache/lucene/analysis/cjk/CJKWidthFilter.html

lucene.apache.org

https://github.com/WorksApplications/Sudachi

github.com

https://github.com/WorksApplications/SudachiDict

github.com

https://docs.aws.amazon.com/opensearch-service/latest/developerguide/supported-plugins.html

docs.aws.amazon.com

https://docs.aws.amazon.com/opensearch-service/latest/developerguide/custom-packages.html

docs.aws.amazon.com

https://docs.aws.amazon.com/opensearch-service/latest/developerguide/serverless-genref.html

docs.aws.amazon.com

https://docs.aws.amazon.com/bedrock/latest/userguide/titan-embedding-models.html

docs.aws.amazon.com

https://platform.claude.com/docs/en/build-with-claude/token-counting

platform.claude.com

Elsewhere on hidekazu-konishi.com (14)

Amazon Bedrock Knowledge Bases Retrieval Quality Engineering - Chunking Strategies, Hybrid Search, Metadata Filtering, Reranking, and Query Decomposition | hidekazu-konishi.com Read at the source ↗

hidekazu-konishi.com

Amazon Bedrock Knowledge Bases Ingestion and Isolation Patterns - Direct Ingestion, Sync Strategies, S3 Metadata Design, and Tenant Boundaries | hidekazu-konishi.com Read at the source ↗

hidekazu-konishi.com

Vector Database Selection on AWS - OpenSearch, Aurora pgvector, MemoryDB, and S3 Vectors with Index Algorithm Internals | hidekazu-konishi.com Read at the source ↗

hidekazu-konishi.com

LLM Token Counter and Context Budget Planner - Estimate Tokens and Visualize Context Window Usage | hidekazu-konishi.com Read at the source ↗

hidekazu-konishi.com

Character Encoding Converter Tool - Japanese Encoding Conversion and Mojibake Repair | hidekazu-konishi.com Read at the source ↗

hidekazu-konishi.com

Multimodal Document Intelligence Pipeline on AWS - From Documents, Images, Audio, and Video to RAG with Amazon Bedrock Data Automation | hidekazu-konishi.com Read at the source ↗

hidekazu-konishi.com

Production RAG Architecture on Amazon Bedrock - Knowledge Bases, Vector Stores, Retrieval, Guardrails, and Evaluation | hidekazu-konishi.com Read at the source ↗

hidekazu-konishi.com

Amazon S3 Vectors Design Decision Guide - When to Use It, When Not to, and Tiered Designs with Amazon OpenSearch Service | hidekazu-konishi.com Read at the source ↗

hidekazu-konishi.com

Agentic RAG Architecture on Amazon Bedrock - Query Planning, Retrieval Grading, and Self-Correction with AgentCore, Knowledge Bases, and Reranking | hidekazu-konishi.com Read at the source ↗

hidekazu-konishi.com

GraphRAG Architecture on AWS - Multi-Hop Retrieval with Amazon Bedrock Knowledge Bases and Neptune Analytics | hidekazu-konishi.com Read at the source ↗

hidekazu-konishi.com

RAG and Prompting Techniques History and Timeline - Chain-of-Thought, ReAct, RAG, and GraphRAG | hidekazu-konishi.com Read at the source ↗

hidekazu-konishi.com

AWS History and Timeline regarding Amazon OpenSearch Service - Overview, Functions, Features, Summary of Updates, and Introduction | hidekazu-konishi.com Read at the source ↗

hidekazu-konishi.com

Amazon Bedrock Glossary - A Reference for AI Engineers and Architects | hidekazu-konishi.com Read at the source ↗

hidekazu-konishi.com

LLM Output Verification Patterns - Grounding Checks, Self-Verification, Cross-Model Review, and Citation Enforcement | hidekazu-konishi.com Read at the source ↗

hidekazu-konishi.com