Record · corpus.blog/posts/01a08c57-7924-7181-9d13-082f02bf2b6e

Working with Chinese, Japanese, and Korean text in Generative AI pipelines

tonybaloney.github.io · published 12 March 2024

A record is what was published and who pointed at it. The text of the post is not held here: read it at the source, or at its Wayback capture.

Read the post at tonybaloney.github.io Wayback capture

1blogs citing

First cited: 1 Jun 2024
Most recent: 1 Jun 2024
Rank this month: not ranked
Links from this post: 20
Words captured: 3,041

Cited by

Blog

In the post

Date

Pamela Fox

Truncating conversation history for OpenAI chat completions Read at the source ↗

1 Jun 2024

Links from this post

Link

Host

https://www.statista.com/statistics/262946/most-common-languages-on-the-internet/

statista.com

https://learn.microsoft.com/azure/ai-studio/how-to/model-catalog

learn.microsoft.com

https://huggingface.co/docs/transformers/main/en/tokenizer_summary

huggingface.co

https://learn.microsoft.com/azure/ai-services/openai/concepts/models

learn.microsoft.com

https://www.unicode.org/versions/Unicode15.0.0/ch06.pdf

unicode.org

https://www.kanshudo.com/kanji/%E7%8C%AB

kanshudo.com

https://tatoeba.org/en/downloads

tatoeba.org

https://unicode.org/charts/PDF/UAC00.pdf

unicode.org

https://www.unicode.org/L2/L2006/06310-hangul-decompose9.pdf

unicode.org

https://learn.microsoft.com/azure/search/vector-search-ranking

learn.microsoft.com

https://learn.microsoft.com/azure/search/semantic-search-overview

learn.microsoft.com

https://techcommunity.microsoft.com/t5/ai-azure-ai-services-blog/azure-ai-search-outperforming-vector-search-with-hybrid/ba-p/3929167

techcommunity.microsoft.com

https://python.langchain.com/docs/modules/data_connection/document_transformers/character_text_splitter

python.langchain.com

https://python.langchain.com/docs/modules/data_connection/document_transformers/recursive_text_splitter

python.langchain.com

https://www.w3.org/TR/jlreq/

w3.org

https://standards.globalspec.com/std/1206038/JIS%20X%204051

standards.globalspec.com

https://python.langchain.com/docs/modules/data_connection/document_transformers/split_by_token

python.langchain.com

https://github.com/azure-Samples/azure-search-openai-demo

github.com

https://support.microsoft.com/office/using-right-to-left-languages-in-office-17d8a34d-36d6-49ad-b765-257cb7cd22e2

support.microsoft.com

https://github.com/python-openxml/python-docx

github.com