본문으로 건너뛰기

About Now Projects Blog AI Curator

About Now Projects Blog AI Curator

Tag

tokenizer

1 posts tagged tokenizer.

All cpp unilink async-io architecture LLM api-design AI Curator backpressure boost-asio channel concurrency dependency-injection transformer zero-copy abstraction AI alignment astro attention base-model benchmarking buffer builder builder-pattern chat-model cmake communication concepts crtp deadlock debugging decoder decoding design-patterns diagnostics embedding encoder facade factory-pattern fine-tuning framer instruction-tuning latency logit loss markdown memory meta networking next-token-prediction observability open-source ownership packaging pimpl pretraining probability protocol queue realtime release RLHF runtime-stats sampling self-attention softmax stream sveltia tcp temperature testing token tokenizer top-k top-p transport vector vocabulary wrapper

Posts

2026.07.03Study
Tokenizer와 Embedding
Tokenizer의 토큰 분할, Token ID 매핑, Embedding Layer 및 Position Embedding의 구조와 역할