ARTIFICIAL INTELLIGENCE AND NLP APPLICATIONS Natural Language Processing with Python from the Transformer Architecture to RAG Systems Review Article Keywords: NLP • Transformer • BERT • LLM • RAG • spaCy • HuggingFace • LangChain • Python ABSTRACT Natural Language Processing (NLP) is one of the fastest-growing subfields of artificial intelligence, studying the capacity of machines to understand, interpret and generate human language. The paper 'Attention Is All You Need', published in 2017, and the Transformer architecture created a paradigm shift in this field; BERT (2018), GPT-3 (2020) and the Large Language Models (LLMs) that followed carried NLP from research laboratories into corporate production environments. This review article addresses comprehensively the historical evolution of NLP, core technical concepts (tokenisation, embedding, the attention mechanism), the Transformer, BERT and GPT architectures, the principal NLP tasks (sentiment analysis, named entity recognition, machine translation, text summarisation, question answering), Retrieval-Augmented Generation (RAG) systems, the spaCy and HuggingFace Transformers libraries and the LangChain framework. Each section is supported by Python code examples that make the topic concrete. The article also covers the ethical dimensions of NLP, problems of bias and fairness, and assessments of the future of artificial intelligence.
Keywords: natural language processing, NLP, transformer, BERT, GPT, large language model, LLM, sentiment analysis, named entity recognition, machine translation, RAG, retrieval-augmented generation, spaCy, HuggingFace, LangChain, Python, attention mechanism ABSTRACT Natural Language Processing (NLP) is one of the fastest-growing subfields of artificial intelligence, studying the capacity of machines to understand, interpret, and generate human language. The 'Attention Is All You Need' paper published in 2017 and the Transformer architecture created a paradigm shift in this field; BERT (2018), GPT-3 (2020), and the subsequent Large Language Models (LLMs) moved NLP from research laboratories to corporate production environments. This review article comprehensively covers the historical evolution of NLP, core technical concepts (tokenization, embedding, attention mechanism), Transformer, BERT, and GPT architectures, major NLP tasks (sentiment analysis, named entity recognition, machine translation, text summarization, question answering), Retrieval-Augmented Generation (RAG) systems, spaCy and HuggingFace Transformers libraries, and the LangChain framework. Each section is supported by Python code examples that concretize the topic.
Keywords: natural language processing, NLP, transformer, BERT, GPT, large language model, LLM, sentiment analysis, named entity recognition, machine translation, RAG, retrieval-augmented generation, spaCy, HuggingFace, LangChain, Python, attention mechanism 1. Introduction Human language, with its complexity and ambiguity, constitutes one of the most difficult problems in computer science. The same word can carry different meanings in different contexts; the same meaning can be expressed with different sentence structures; the emotional tone of a text reflects the writer's intent beyond the words themselves. The effort to cope with that complexity gave rise to the discipline of Natural Language Processing (NLP).
NLP's historical journey begins with the rule-based systems of the 1950s and extends through statistical models and neural networks to the Transformer revolution confirmed in 2017 by the work 'Attention Is All You Need' published by Vaswani et al. That revolution transformed fundamentally not only research but the scale and quality of practical applications. ChatGPT reaching a million users within five days in 2022 makes concrete the speed and scope of that transformation.
According to the assessment of the Thallam NST Journal (2025), the last five years have been shaped by the dominance of large-scale pre-trained language models — BERT, GPT-3, PaLM, LLaMA. These models have entered production in dozens of sectors: text translation, code writing, legal document analysis, medical report summarisation and customer service automation. In the 2024-2025 period, Retrieval-Augmented Generation (RAG) architectures are becoming standardised at enterprise scale as the most effective way of overcoming LLMs' limitation on knowledge currency. The aim of this article is to set out comprehensively the meeting point of NLP and artificial intelligence — with technical depth, practical Python examples and current research frameworks. The article serves as both a theoretical and an applied resource, offering information for researchers in the field, software developers and corporate decision-makers.
2. The Historical Evolution of NLP Figure 1. A Timeline of the Historical Evolution of NLP / AI Technologies (1950–2025) 2.1 The Rule-Based Period (1950–1990) The first period of NLP rests on systems in which language was defined by grammatical rules and those rules were explicitly coded. In the 1950s Alan Turing's paper 'Computing Machinery and Intelligence' raised the question of whether machines can think; that question became in time the philosophical ground of NLP research. ELIZA (1966, MIT) and SHRDLU (1970) showed both how superficial rule-based approaches remained and how striking the results they could produce were.
In the 1980s expert systems and context-free grammars dominated NLP research. Prolog-based natural language parsers attracted academic interest in this period; but the systems could not scale to processing real-world linguistic diversity and could not easily be adapted to a new language or domain. According to the PeerJ CS assessment of Tucudean et al. (2024), the constraint of that period lay precisely here: 'RNNs and LSTMs were effective for short sequences; but vanishing gradients and non-parallel processing limited the capture of long dependencies.' 2.2 The Statistical Period (1990–2012) In the early 1990s statistical approaches began to replace rule-based systems. N-gram language models trained on large text corpora, hidden Markov models (HMM) and support vector machines (SVM) were the dominant tools of this period. Google Translate's move to statistical machine translation in 2006 proved the industrial applicability of that paradigm.
Word2Vec (Mikolov et al., 2013) represents the closing masterpiece of this period. Representing words with 300-dimensional vectors — turning semantic similarity into geometric distance — an approach made concrete by analogies such as 'King − Man + Woman = Queen', placed the concept of the embedding at the centre of NLP. 2.3 The Deep Learning and Transformer Period (2013–Present) The years 2013-2017 are the transitional period in which deep learning architectures dominated NLP. LSTM and GRU networks brought significant advances in processing sequential data; but the paper 'Attention Is All You Need' published by Vaswani et al. in 2017 changed that picture fundamentally. The Transformer architecture removed recurrent structures entirely and placed the attention mechanism at the centre of computation. This made parallel training possible and opened the way to developing models with billions of parameters. BERT by Devlin et al. (2019), the GPT series by Radford et al. and the T5, RoBERTa, XLNet and ALBERT models that followed launched a race continually rewriting NLP benchmarks. Accuracy above eighty per cent on the GLUE benchmark and an F1 score above ninety on the SQuAD QA data set symbolise the technical outputs of this period (arxiv, 2025). With ChatGPT's public release in 2022, these technologies completed their transition from research into daily life.
3. Core Technical Concepts 3.1 Tokenisation Tokenisation is the process of splitting a text sequence into smaller units (tokens) the model can process. There are three basic approaches: word level, character level and subword level. The great majority of modern LLMs use BPE (Byte Pair Encoding) or WordPiece subword tokenisation, which keeps the vocabulary under control while still handling rare words. HuggingFace's BERT tokenizer is the reference implementation of this approach.
Example 1: Tokenisation with the HuggingFace BERT Tokenizer from transformers import BertTokenizer # Load the BERT tokenizer tokenizer = BertTokenizer.from_pretrained('bert-base-uncased') text = "Artificial intelligence has revolutionised the field of NLP." # Tokenisation tokens = tokenizer.tokenize(text) print("Tokens:", tokens) # Output: ['artificial', 'intelligence', 'has', 'revolution', '##ised', 'the', 'field', 'of', 'nlp', '.'] # Convert to IDs token_ids = tokenizer.encode(text, return_tensors='pt') print("Token IDs:", token_ids) # Encode with padding for batch processing encoded = tokenizer( [text, "BERT is a powerful language model."], padding=True, truncation=True, max_length=128, return_tensors='pt' print("input_ids shape:", encoded['input_ids'].shape) 3.2 Embedding Representations Word embeddings are numerical representations expressing the semantic and syntactic properties of words as dense vectors. Word2Vec and GloVe produce static embeddings (the word 'bank' receives the same vector whatever context it is used in), whereas BERT and its derivatives produce context-dependent dynamic embeddings. That difference is of decisive importance in solving the problem of polysemy.
Example 2: Computing Semantic Similarity with Sentence-Transformers from sentence_transformers import SentenceTransformer from sklearn.metrics.pairwise import cosine_similarity # Load the model model = SentenceTransformer('paraphrase-multilingual-mpnet-base-v2') sentences = [ "Artificial intelligence is transforming the business world.", "Machine learning is revolutionising industry.", "The weather has been changeable in Türkiye." # Generate embeddings embeddings = model.encode(sentences) # Cosine similarity matrix similarity = cosine_similarity(embeddings) print("Similarity of sentences 1 vs 2:", round(similarity[0][1], 3)) print("Similarity of sentences 1 vs 3:", round(similarity[0][2], 3)) # Expected output: high similarity between 1 and 2 (close meaning) # low similarity between 1 and 3 (different topic) 3.3 The Attention Mechanism The attention mechanism is the mathematical framework enabling a model to focus in a weighted way on different parts of the input sequence while producing output. Scaled Dot-Product Attention, presented in the Transformer paper by Vaswani et al. (2017), is defined by the formula: Attention(Q, K, V) = softmax(QK^T / √d_k)V In this formula the Q (Query), K (Key) and V (Value) matrices are transformations learned from the input embedding vectors. Scaling by √d_k prevents gradient explosion. Multi-Head Attention is an extension in which this mechanism is applied in parallel across different representational subspaces. That structure allows the model to focus on syntactic and semantic relationships at the same time.
4. The Transformer Architecture Figure 2. The Transformer Architecture: Encoder-Decoder Structure (Vaswani et al., 2017) 4.1 The Encoder-Decoder Structure The Transformer architecture consists of two basic components: the Encoder and the Decoder. The Encoder converts the input text into a semantic representation; the Decoder uses that representation to produce the output text. Each Encoder layer consists of Multi-Head Self-Attention and Feed-Forward Network sub-layers, with a residual connection and layer normalisation around each sub-layer. Positional Encoding is a critical component enabling the Transformer to capture sequential structure. Since there is no recurrent structure, the model does not naturally know the order of words; sine- and cosine-based positional encodings inject that information into the embedding vectors. According to the assessment of Tucudean et al. (2024), this architecture 'overcomes the constraints RNNs could not surmount in capturing long dependencies and in parallel training.' 4.2 BERT: The Bidirectional Encoder BERT (Bidirectional Encoder Representations from Transformers) is a pre-trained language model developed by Google in 2018 using only the encoder part of the Transformer. Two basic features distinguish BERT from earlier models. The first is bidirectional context: while traditional language models process text left to right, BERT learns each word's context simultaneously from preceding and following words through masked language modelling (MLM).
According to the comprehensive BERT review by Springer Nature (2025), this model is applied in a wide variety of NLP tasks such as sentence boundary detection, tokenisation, grammatical error detection, dependency parsing, named entity recognition, part-of-speech tagging, question-answering systems, machine translation, sentiment analysis and cross-language transfer. BERT's architecture is offered in two sizes: BERT-Base (12 layers, 110M parameters) and BERT-Large (24 layers, 340M parameters).
Example 3: Fine-Tuning with BERT — Text Classification from transformers import (BertTokenizer, BertForSequenceClassification, Trainer, TrainingArguments) from datasets import Dataset import torch # Load model and tokenizer model_name = 'bert-base-uncased' tokenizer = BertTokenizer.from_pretrained(model_name) model = BertForSequenceClassification.from_pretrained(model_name, num_labels=2) # Prepare sample data data = { 'text': [ 'This product is great, I am very satisfied!', 'It was a terrible experience, I did not like it at all.', 'Quality product, I recommend it.', 'My money went to waste, it did not work.' ], 'label': [1, 0, 1, 0] # 1=positive, 0=negative } def tokenize_fn(example):
return tokenizer(example['text'], truncation=True, padding='max_length', max_length=128) dataset = Dataset.from_dict(data).map(tokenize_fn, batched=True) dataset = dataset.train_test_split(test_size=0.25) # Training arguments training_args = TrainingArguments( output_dir='./bert_classification', num_train_epochs=3, per_device_train_batch_size=8, evaluation_strategy='epoch', save_strategy='epoch', load_best_model_at_end=True, trainer = Trainer( model=model, args=training_args, train_dataset=dataset['train'], eval_dataset=dataset['test'], # trainer.train() # Enable for actual training 4.3 The GPT Series: Autoregressive Decoder Models The GPT (Generative Pre-trained Transformer) series is an autoregressive family of models using only the decoder part of the Transformer. Each token is generated looking only at preceding tokens (left-to-right attention). GPT-1 (2018) introduced this approach while GPT-3 (2020), with 175 billion parameters, proved its 'few-shot learning' capacity; GPT-4 (2023) gained multimodal capabilities.
According to the assessment of Medium/Sinha (2024), 'in 2024 models such as GPT-4, PaLM and LLaMA can produce human-like text with billions of parameters, answer questions and display increasingly high fluency in translating between languages.' Open-source models such as LLaMA 2 and 3 (Meta), Mistral and Gemma have opened that capacity to academic and corporate researchers.
5. Core NLP Tasks and Python Applications Figure 3. The NLP Task Ecosystem: Six Main Categories and Areas of Use 5.1 Sentiment Analysis Sentiment analysis is the task of determining the emotional tone of a piece of text — positive, negative or neutral. Customer feedback analysis, social media monitoring and product review assessment are the principal areas of application. According to Springer Nature's (2025) BERT review, BERT-based sentiment analysis models achieve an F1 score of ninety-four per cent on the Stanford SST-2 data set.
Example 4: Sentiment Analysis with a HuggingFace Pipeline # Sentiment analysis pipeline — automatic model selection sentiment_analyser = pipeline('sentiment-analysis', model='cardiffnlp/twitter-xlm-roberta-basesentiment') reviews = [ 'This library is really excellent, it was very useful!', 'It was disappointing, it did not meet my expectations.', 'Not bad, but there are aspects that need improving.', for review in reviews:
result = sentiment_analyser(review)[0] print(f"Text : {review[:50]}...") print(f"Sentiment : {result['label']} (confidence: {result['score']:.2%})") print('-' * 60) # Alternative approach with spaCy + TextBlob from textblob import TextBlob nlp = spacy.load('en_core_web_sm') text_en = 'The product quality is excellent and delivery was fast!' blob = TextBlob(text_en) print(f"Polarity: {blob.sentiment.polarity:.2f}") # -1 → +1 print(f"Subjectivity: {blob.sentiment.subjectivity:.2f}") # 0 → 1 5.2 Named Entity Recognition (NER) NER is a classification task detecting entities in predefined categories such as person name, organisation name, place name, date and currency. It plays a critical role in legal document analysis, biomedical information extraction, news feed processing and business process automation. In the LegalLens-2024 shared task, a spaCy + DeBERTa combination achieved an F1 score of 86.3 per cent in the legal NER subtask (Meghdadi and Inkpen, 2024). Example 5: Entity Recognition and Visualisation with spaCy from spacy import displacy # Load the model nlp = spacy.load('en_core_web_trf') # English transformer model text = """Apple CEO Tim Cook visited Istanbul on 15 November 2024 and held strategic partnership talks with Koç Holding. The value of the agreement was stated to be 500 million dollars.""" doc = nlp(text) # Print the entities print("Entities detected:") for entity in doc.ents:
print(f" {entity.text:<30} [{entity.label_}]") # Example output:
# Apple [ORG] # Tim Cook [PER] # 15 November 2024 [DATE] # Istanbul [LOC] # Koç Holding [ORG] # 500 million dollars [MONEY] # HTML visualisation (for use in Jupyter/browser) # displacy.render(doc, style='ent', jupyter=True) 5.3 Machine Translation Machine translation is the automatic conversion of text from one language to another. With the emergence of the Transformer architecture, neural machine translation (NMT) has come to dominate the market; Helsinki-NLP's MarianMT, Meta's NLLB-200 (No Language Left Behind) and Google's T5 model form the pioneering open-source frameworks in this field. NLLB-200 is a comprehensive model able to translate across 200 language pairs, designed particularly for low-resource languages. Example 6: Machine Translation with Helsinki-NLP MarianMT from transformers import MarianMTModel, MarianTokenizer # English → Turkish translation model model_name = 'Helsinki-NLP/opus-mt-en-tr' tokenizer = MarianTokenizer.from_pretrained(model_name) model = MarianMTModel.from_pretrained(model_name) def translate(text_list, src_lang='en', tgt_lang='tr'):
"""Translates a list of texts into the target language.""" tokenized = tokenizer(text_list, return_tensors='pt', padding=True, truncation=True, max_length=512) translated = model.generate(**tokenized, num_beams=4, max_length=512, early_stopping=True) return [tokenizer.decode(t, skip_special_tokens=True) for t in translated] english_texts = [ 'Natural language processing is transforming the way we interact with computers.', 'Large language models have billions of parameters.', 'Deep learning techniques have revolutionized NLP research.' translation_results = translate(english_texts) for source, target in zip(english_texts, translation_results): print(f'EN: {source}') print(f'TR: {target}') print() 5.4 Text Summarisation Text summarisation is the task of presenting a long document in a shorter form while preserving its essence. Extractive summarisation selects important sentences from the original document, while abstractive summarisation generates new sentences. The BART and T5 models are among the architectures displaying the highest performance in abstractive summarisation.
Example 7: Abstractive Text Summarisation with BART # Summarisation pipeline summariser = pipeline('summarization', model='facebook/bart-large-cnn') long_text = """ Natural language processing (NLP) is a subfield of linguistics, computer science, and artificial intelligence concerned with the interactions between computers and human language, in particular how to program computers to process and analyze large amounts of natural language data. The goal is a computer capable of understanding the contents of documents, including the contextual nuances of the language within them. The technology can then accurately extract information and insights contained in the documents, as well as categorize and organize the documents themselves. The field has seen rapid advances in recent years, driven by the development of transformer-based models like BERT and GPT, which have achieved state-of-the-art results on many NLP benchmarks. summary = summariser(long_text, max_length=100, min_length=30, do_sample=False) print("Summary:", summary[0]['summary_text']) # mT5 alternative for multilingual summarisation # summariser_ml = pipeline('summarization', model='csebuetnlp/mT5_multilingual_XLSum') 5.5 Question Answering Systems Question answering (QA) systems produce an answer from a relevant text or knowledge base when given a question. Extractive QA selects the answer as a span within the text, while generative QA constructs new sentences. On the SQuAD data set, BERT-based models have approached or in some subtasks surpassed human performance. Example 8: Extractive Question Answering with BERT # QA pipeline qa_model = pipeline('question-answering', model='deepset/xlm-roberta-base-squad2') context = """ The Transformer architecture was introduced in 2017 by the Google Brain team in a paper titled 'Attention Is All You Need'. By using the attention mechanism, this architecture removed the need for recurrent structures and made parallel training possible. BERT is a pre-trained language model developed by Google in 2018 that uses only the encoder part of the Transformer.
questions = [ "In which year was the Transformer architecture introduced?", "Who developed BERT?", "What does the attention mechanism provide?", for question in questions:
answer = qa_model(question=question, context=context) print(f"Question : {question}") print(f"Answer : {answer['answer']}") print(f"Confidence : {answer['score']:.2%}") print() 6. NLP Libraries: spaCy, NLTK and a Comparison 6.1 spaCy: Production-Focused NLP spaCy is an open-source Python library developed by Explosion and offered under the slogan 'industrial-strength NLP'. Standing out for its speed, production maturity and transformer support, spaCy offers tokenisation, morphological analysis, dependency parsing, NER, text classification and semantic similarity computation components under a single consistent API. From spaCy v3 onwards, RoBERTa- and BERT-based transformer pipelines are supported, and NER accuracy above ninety per cent is achieved on the OntoNotes 5.0 benchmark (spaCy, 2025).
Example 9: Comprehensive Text Analysis with spaCy nlp = spacy.load('en_core_web_sm') text = """OpenAI released GPT-4 in March 2023.
Microsoft invested $10 billion in the company.
Sam Altman serves as the CEO of OpenAI.""" doc = nlp(text) # 1. Tokenisation and part-of-speech tagging print("=== Part-of-Speech Tags ===") for token in doc[:8]:
print(f"{token.text:<15} {token.pos_:<8} {token.dep_:<15} {token.lemma_}") # 2. Entity recognition print("\n=== Entities ===") for ent in doc.ents:
print(f"{ent.text:<25} {ent.label_:<12} {spacy.explain(ent.label_)}") # 3. Dependency parsing print("\n=== Dependency Relations ===") for token in doc:
if token.dep_ in ('nsubj', 'dobj', 'ROOT'):
print(f"{token.text:<15} --{token.dep_}--> {token.head.text}") # 4. Sentence boundaries print("\n=== Sentences ===") for i, sent in enumerate(doc.sents):
print(f"Sentence {i+1}: {sent.text.strip()[:80]}") 6.2 NLTK: A Platform for Teaching and Research NLTK (Natural Language Toolkit) is a long-established library used for NLP research and teaching in Python. Compared with spaCy, NLTK stands out for broader language support, a comprehensive collection of corpora and lexicons, and a modular structure that is comprehensible from an educational standpoint. According to DSStream's (undated) comparison, 'NLTK is popular among researchers and educators because of its comprehensive documentation and large community; it is a good choice if you want to build a model from scratch.' Example 10: Basic NLP Operations with NLTK import nltk from nltk.tokenize import word_tokenize, sent_tokenize from nltk.corpus import stopwords from nltk.stem import PorterStemmer, WordNetLemmatizer from nltk import FreqDist, pos_tag # Download the required resources (on first run) # nltk.download(['punkt','stopwords','wordnet','averaged_perceptron_tagger']) text = """Natural language processing enables machines to understand human language. Modern NLP systems use transformer architectures. These models are trained on massive text corpora.""" # Sentence and word tokenisation sentences = sent_tokenize(text) words = word_tokenize(text.lower()) print(f"Sentence count: {len(sentences)}, Word count: {len(words)}") # Stop word removal stop_words = set(stopwords.words('english')) clean = [w for w in words if w.isalpha() and w not in stop_words] print(f"Token count after stop word removal: {len(clean)}") # Stemming and Lemmatisation stemmer = PorterStemmer() lemmatizer = WordNetLemmatizer() print("\nWord | Stem | Lemma") for word in ['processing', 'enabling', 'systems', 'trained']: print(f"{word:<15} {stemmer.stem(word):<15} {lemmatizer.lemmatize(word)}") # Frequency analysis freq = FreqDist(clean) print("\nFive most frequent words:") for word, count in freq.most_common(5):
print(f" {word:<15} : {count} times") Table 1. spaCy and NLTK: A Feature Comparison Feature spaCy NLTK Primary Goal Production environment Research and teaching Speed Very fast (optimised with Cython) Moderate (Python-based) API Design Object-oriented, fluent Modular, functional Transformer Support Yes (v3+, native) No (add-on required) Language Support 60+ languages (ready models) 80+ languages (corpus-based) Visualisation displacy (built in) Limited (with matplotlib) Learning Curve Moderate Low Note. Compiled on the basis of DSStream (undated); spaCy.io (2025); Medium/Srivastava (2023). 7. Retrieval-Augmented Generation (RAG) Figure 4. The RAG Pipeline: A Six-Stage Process from Documents to LLM Response 7.1 What Is RAG and Why Does It Matter?
Retrieval-Augmented Generation (RAG) is an architectural approach developed to solve LLMs' knowledge cutoff limitation and the problem of hallucination. Introduced in the 2020 NeurIPS paper by Lewis et al., this approach combines retrieval with a generative model, enabling the LLM to access private or current data.
According to Ferrer's (2025) assessment in KDnuggets: 'While LLMs such as GPT and LLaMA are excellent at answering questions, they are limited by their training data. They cannot access private data or learn beyond their training cutoff date. RAG fills that gap.' According to LangCopilot's (2025) market analysis, RAG frameworks became in 2025 an inseparable component of the standard enterprise AI stack.
7.2 RAG Pipeline Components A complete RAG pipeline consists of six main components. The first component is the document loading and chunking stage. LangCopilot's (2025) research shows that optimal chunk size is achieved at 256-512 tokens with an overlap ratio of ten to twenty per cent. The second component is embedding generation: each chunk is converted into a dense vector for semantic similarity search.
The third component is the vector database: FAISS (Facebook AI), Chroma, Pinecone or Weaviate are the principal platforms used for this role. The fourth component is query-time retrieval: the user's question is converted into an embedding and the most similar chunks are found by cosine similarity. The fifth component is context construction and prompt engineering. The sixth component is the LLM processing the context-enriched prompt and producing an answer. Example 11: A Basic RAG Pipeline with LangChain from langchain_community.document_loaders import TextLoader from langchain.text_splitter import RecursiveCharacterTextSplitter from langchain_community.embeddings import HuggingFaceEmbeddings from langchain_community.vectorstores import FAISS from langchain.chains import RetrievalQA from langchain_community.llms import HuggingFacePipeline from transformers import pipeline as hf_pipeline # 1. Load the document loader = TextLoader('documents/company_policy.txt', encoding='utf-8') documents = loader.load() # 2. Chunking splitter = RecursiveCharacterTextSplitter( chunk_size=512, chunk_overlap=64, separators=['\n\n', '\n', '.', ' '] chunks = splitter.split_documents(documents) print(f"Total chunk count: {len(chunks)}") # 3. Embedding model and vector database embedding_model = HuggingFaceEmbeddings( model_name='sentence-transformers/paraphrase-multilingual-mpnet-base-v2' vector_db = FAISS.from_documents(chunks, embedding_model) vector_db.save_local('faiss_index') # 4. LLM and QA chain llm_pipeline = hf_pipeline( 'text-generation', model='microsoft/DialoGPT-medium', max_new_tokens=256 llm = HuggingFacePipeline(pipeline=llm_pipeline) qa_chain = RetrievalQA.from_chain_type( llm=llm, chain_type='stuff', retriever=vector_db.as_retriever(search_kwargs={'k': 3}), return_source_documents=True # 5. Query question = 'According to the company leave policy, how many days of annual leave are there?' result = qa_chain({'query': question}) print(f"Answer: {result['result']}") print(f"Number of source chunks: {len(result['source_documents'])}") 7.3 Advanced RAG Techniques Beyond naive RAG, several advanced techniques stand out for improving retrieval quality. According to Monigatti's (2025) review in Towards Data Science: 'Advanced RAG has developed as a new paradigm with targeted improvements aimed at remedying some of the limitations of the naive RAG paradigm.' • Hybrid Search: a combination of dense vector retrieval with keyword-based BM25 search. This approach captures both semantic and lexical similarity.
• Re-Ranking: reordering the retrieved chunks with a cross-encoder model. Speed is prioritised in the first stage and accuracy in the second. • Sentence Window Retrieval: expanding the context with neighbouring sentences when a chunk is retrieved, producing coherent information.
• Parent-Child Chunking: combining precise retrieval with small chunks and rich context presentation with large parent chunks.
Example 12: FAISS Vector Search and Similarity Scores import faiss from sentence_transformers import SentenceTransformer # Load the model model = SentenceTransformer('paraphrase-multilingual-mpnet-base-v2') # Sample corpus corpus = [ 'Transformer models revolutionised NLP.', 'BERT is a bidirectional language model.', 'GPT is used for text generation.', 'RAG systems give LLMs current information.', 'spaCy is optimised for production environments.', 'Python is a popular language for machine learning.' # Generate embeddings corpus_embeddings = model.encode(corpus, normalize_embeddings=True) # Build the FAISS index dim = corpus_embeddings.shape[1] index = faiss.IndexFlatIP(dim) # Inner Product (equivalent to cosine if normalised) index.add(corpus_embeddings.astype(np.float32)) # Search query = 'How do large language models work?' query_embedding = model.encode([query], normalize_embeddings=True).astype(np.float32) k = 3 # Three nearest results scores, indices = index.search(query_embedding, k) print(f"Query: {query}\n") for i, (score, idx) in enumerate(zip(scores[0], indices[0])): print(f"{i+1}. (score: {score:.3f}) {corpus[idx]}") 8. Large Language Models (LLMs) and Areas of Application 8.1 Prompt Engineering Prompt engineering is the discipline of systematically designing the input text (prompt) in order to obtain the desired output from an LLM. Effective prompt design has three basic components: a role definition (the system prompt), a task description and context information, and few-shot examples. The difference in output quality between a poorly designed and a well-designed prompt can be equivalent to, or greater than, the improvement from fine-tuning a small model.
Example 13: Structured Data Extraction with the Anthropic Claude API import anthropic import json client = anthropic.Anthropic() # Read from the ANTHROPIC_API_KEY environment variable system_prompt = """You are a data extraction assistant.
Extract structured information from the text passage you are given.
Always respond in JSON format.""" news_text = """ Tesla CEO Elon Musk announced on 18 November 2024 that the company would establish an electric vehicle production facility in Istanbul. The total value of the investment is stated to be 2 billion dollars and it is expected to create 5,000 jobs. The facility is planned to enter production in 2027. user_message = f"""Extract the following from the news text below as JSON: - person: person name - company: company name - place: location - date: date - investment_amount: investment amount - employment: number of employees - opening_year: facility opening year Text: {news_text} message = client.messages.create( model='claude-sonnet-4-6', max_tokens=512, system=system_prompt, messages=[{'role': 'user', 'content': user_message}] result_text = message.content[0].text try:
data = json.loads(result_text) print(json.dumps(data, ensure_ascii=False, indent=2)) except json.JSONDecodeError:
print(result_text) 8.2 Chain-of-Thought Prompting Chain-of-Thought (CoT) prompting is a technical prompt strategy that increases accuracy in complex tasks by encouraging the LLM to explain its reasoning process step by step. The research of Wei et al. (2022) shows that CoT prompting markedly improves LLM performance, particularly in arithmetic reasoning and multi-step problem-solving tasks.
Example 14: Agent-Based Reasoning with LangChain from langchain.agents import create_react_agent, AgentExecutor from langchain.tools import Tool from langchain_anthropic import ChatAnthropic from langchain import hub # LLM definition llm = ChatAnthropic(model='claude-sonnet-4-6', temperature=0) # Tool definitions def calculate(expression: str) -> str:
"""Evaluates a mathematical expression.""" try:
return str(eval(expression)) except Exception as e:
return f'Error: {str(e)}' def search_text(query: str) -> str:
"""Searches the document database.""" # In a real application a vector DB search is performed return f"Results found for '{query}': [document_1, document_2]" tools = [ Tool(name='Calculator', func=calculate, description='Used for mathematical calculation'), Tool(name='Document_Search', func=search_text, description='Searches the document database'), # Create the ReAct agent prompt = hub.pull('hwchase17/react') agent = create_react_agent(llm, tools, prompt) agent_executor = AgentExecutor( agent=agent, tools=tools, verbose=True, max_iterations=5 # A complex query result = agent_executor.invoke({ 'input': 'Search for the 2024 NLP trends report and', ' ': 'then calculate 45 million * 0.08' }) print(result['output']) 9. Model Performance Evaluation 9.1 NLP Benchmarks Standard benchmark data sets and evaluation metrics are used to compare NLP models systematically. GLUE (General Language Understanding Evaluation) and SuperGLUE represent multi-task benchmark families measuring general language understanding capacity. SQuAD 1.1 and 2.0 are the industry standard for question-answering tasks. CoNLL-2003 is the reference data set widely used for NER evaluation. Example 15: NLP Model Evaluation with sklearn from sklearn.metrics import (classification_report, f1_score, confusion_matrix, accuracy_score) # Sample prediction results (sentiment analysis) true_labels = [1, 0, 1, 1, 0, 0, 1, 0, 1, 0] predicted_labels = [1, 0, 1, 0, 0, 1, 1, 0, 1, 0] label_names = ['Negative', 'Positive'] # Detailed classification report print("=== Classification Report ===") print(classification_report( true_labels, predicted_labels, target_names=label_names )) # Token-level F1 for NER (with the seqeval library) # pip install seqeval from seqeval.metrics import classification_report as ner_report true_ner = [['B-PER', 'I-PER', 'O', 'B-ORG', 'O']] predicted_ner = [['B-PER', 'I-PER', 'O', 'B-LOC', 'O']] print("\n=== NER Evaluation ===") print(ner_report(true_ner, predicted_ner, output_dict=False)) Table 2. A Benchmark Performance Comparison of the Principal NLP Models Model Parameters GLUE Score SQuAD 2.0 F1 Distinguishing Feature BERT-Base 110M 80.4 76.3 Bidirectional, MLM pre-training RoBERTa 125M 88.5 86.8 Optimised version of BERT DeBERTa-v3 184M 91.9 88.4 Disentangled attention and virtual tokens GPT-4 ~1T ~90+ ~90+ Multimodal, general purpose (estimated) LLaMA 3.1-70B 70B 89.3 88.0 Open source, efficient Note. Compiled on the basis of arxiv (2025); Tucudean et al. PeerJ CS (2024); Springer Nature (2025). Some values are estimates.
10. Ethical Dimensions: Bias, Fairness and Responsible AI 10.1 Bias in NLP Models NLP models tend to learn the societal biases and stereotypes present in their training data. Gender bias, racial bias, age discrimination and a lack of cultural representation are the most documented problems in this field. Early research on Word2Vec embeddings showed that the word 'doctor' sits closer to male names and the word 'nurse' closer to female names, revealing how such bias can be reproduced in job advertisement ranking systems.
According to the assessment of the Thallam NST Journal (2025): 'The ethical dimension of NLP covers many concerns. Bias and fairness describe a model's tendency to produce erroneous behaviour systematically throughout its training process. Privacy cannot be overlooked either: language models can leak personal information from training data.' These risks make the integration of responsible AI development frameworks into NLP practice unavoidable. 10.2 Responsible NLP: Mitigation Strategies The research community has developed several basic strategies for coping with bias. • Data auditing and cleaning: systematic examination of training data for representational inequality and, where necessary, rebalancing.
• Debiasing techniques: detecting and neutralising bias vectors in the embedding space (Gender Debiasing, for example).
• Diverse evaluation data sets: data sets such as WinoBias, StereoSet and BBQ have been developed to measure model bias.
• Human oversight: accompanying automation with a human review process in high-risk applications.
• Model cards and documentation: transparent documentation of model capabilities, limitations and known biases (Model Cards for Model Reporting, Mitchell et al.).
11. NLP for Turkish: Particular Challenges and Models 11.1 The Morphological Complexity of Turkish Turkish, with its agglutinative structure, presents distinctive challenges for NLP: a single word root can produce extremely long word forms with multiple inflectional and derivational suffixes. Although extreme examples such as 'Türkleştiremeyebileceklerimizdenmişsinizcesine' appear frequently in the academic literature, forms combining five to seven morphemes are quite common in everyday language. This means that word-level tokenisation is inadequate for Turkish and that morphology-aware approaches are essential. 11.2 BERTurk and Turkish NLP Models Language models trained specifically for Turkish have developed significantly in recent years. BERTurk (dbmdz/bert-base-turkish-cased and uncased) is a BERT model accessible on HuggingFace and pre-trained on large Turkish text corpora. XLM-RoBERTa, a multilingual model covering 100 languages including Turkish, is also widely used. Example 16: Masked Language Modelling with Turkish BERT from transformers import pipeline, BertTokenizer, BertForMaskedLM # Load the Turkish BERT model model_name = 'dbmdz/bert-base-turkish-cased' masking = pipeline('fill-mask', model=model_name) # Masked prediction sentences = [ 'Yapay zeka teknolojisi [MASK] dünyasını değiştiriyor.', 'Transformer modeller NLP [MASK] tarihsel bir dönüşüm yarattı.', '[MASK] Dil Modelleri büyük metin derlemleri üzerinde eğitilir.', for sentence in sentences:
predictions = masking(sentence, top_k=3) print(f"\nInput: {sentence}") for p in predictions:
print(f" {p['token_str']:<20} (score: {p['score']:.3f})") # The zeyrek library for morphological analysis # pip install zeyrek import zeyrek analyser = zeyrek.MorphAnalyzer() words = ['gidiyorum', 'öğrenciler', 'Türkiye', 'yapabileceğim'] for word in words:
analysis = analyser.lemmatize(word) print(f"{word} → {analysis[0][1]}") 12. Future Trends and Research Directions 12.1 Multimodal LLMs The boundaries of NLP are expanding beyond text. Multimodal models can process text, image, audio and video data within a unified architecture. GPT-4V, Claude 3 Opus and Gemini Ultra are the pioneering systems of this generation, combining image understanding and text generation capabilities in a single model. According to the Thallam NST Journal's (2025) forecast, 'the future will include real-time multimodal dialogue systems integrating speech, facial recognition and textual input.' 12.2 Small Language Models (SLM) and Efficiency The high computational cost of models at GPT-4 scale is accelerating research into Small Language Models (SLM). Models such as Phi-3 (Microsoft), Gemma 2B (Google) and LLaMA 3.2-3B offer billions of parameters at a scale able to run on smartphones and edge devices. According to the finding of Tucudean et al. (2024), the compression approaches developed for BERT 'succeed in reducing the memory footprint without degrading performance, making an important contribution to NLP's use in edge architectures.' 12.3 Quantum NLP The integration of quantum computers into NLP stands out as an interesting research direction over the long term. As the Thallam NST Journal (2025) explains: 'Quantum computers are poised to transform NLP by increasing computational efficiency for large-scale models. While traditional NLP models rest on classical matrix multiplications, quantum-enhanced transformers will use quantum superposition and entanglement, offering the potential for exponential speed-up.' 13. Conclusion NLP continues to be one of the fastest-evolving disciplines of artificial intelligence, on a journey from the rule-based systems of the 1950s to today's Large Language Models with hundreds of billions of parameters. The 2017 Transformer turning point on that journey transformed NLP fundamentally in both architectural and practical terms: parallel training, context-sensitive embeddings and transfer learning are the three basic gains of that revolution. While BERT and its derivatives offer performance approaching human level in understanding-oriented tasks such as sentiment analysis, entity recognition and question answering, the GPT family brought into practice systems that both understand and generate text. RAG architectures address at enterprise level the most fundamental limitations of those generative models — knowledge currency and hallucination. Frameworks such as LangChain and LlamaIndex accelerate the assembly of these components in production environments.
The Python ecosystem has been the most important agent in the democratisation of access to these technologies. HuggingFace Transformers' model hub, spaCy's production-focused pipelines and LangChain's agent frameworks open NLP's doors to a broad range of users from researchers to corporate developers.
Even so, ethical problems — bias, privacy, disinformation and accountability — remain critical agenda items for the societal acceptance of these technologies. Multimodal models, efficient small language models and possible developments in quantum NLP are among the topics that will shape the research agenda of the coming decade. Developing these technologies responsibly, transparently and inclusively is the shared responsibility of researchers, practitioners and policymakers alike. References Anthropic. (2025). Claude API documentation. https://docs.anthropic.com Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. Proceedings of NAACL-HLT 2019, 4171–4186. https://arxiv.org/abs/1810.04805 DSStream. (undated). The grand tour of NLP: spaCy vs. NLTK. https://www.dsstream.com/post/the-grand-tour-of-nlp-spacy-vs-nltk Explosion AI. (2025). spaCy: Industrial-strength NLP in Python. https://spacy.io Ferrer, J. (2025, April 21). Building a RAG application using LlamaIndex. KDnuggets. https://www.kdnuggets.com/building-a-rag-application-using-llamaindex Gupta, A. (2024). BERT applications in natural language processing: A review. Artificial Intelligence Review, Springer Nature. https://doi.org/10.1007/s10462-025-11162-5 HuggingFace. (2025). Transformers: State-of-the-art machine learning. https://huggingface.co/docs/transformers Jain, A. (2025, November). Build a RAG system using Python, LangChain, and OpenAI. Medium. https://medium.com/@abhishekjainaj Kaushik, S. (2024, November). Mastering sentiment analysis with pretrained NLP models. Medium. https://medium.com/@kaushiksimran827 LangChain. (2025). Build a RAG agent with LangChain.
https://docs.langchain.com/oss/python/langchain/rag LangCopilot. (2025, September 18). Best RAG frameworks 2025: LangChain vs LlamaIndex vs Haystack vs RAGFlow. https://langcopilot.com/posts/2025-09-18-top-ragframeworks-2024-complete-guide Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., ... & Kiela, D. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. Advances in Neural Information Processing Systems, 33, 9459–9474.
Meghdadi, N., & Inkpen, D. (2024). uOttawa at LegalLens-2024: Transformer-based classification experiments. arXiv preprint arXiv:2410.21139. https://arxiv.org/pdf/2410.21139 Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781. Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., ... & Gebru, T. (2019). Model cards for model reporting. Proceedings of FAT* 2019, 220–229. Monigatti, L. (2025, January). Advanced retrieval-augmented generation: From theory to LlamaIndex implementation. Towards Data Science.
https://towardsdatascience.com/advanced-retrieval-augmented-generation Sinha, Y. (2024, September). AI for natural language processing (NLP) in 2024: Latest trends and advancements. Medium. https://medium.com/@yashsinha12354 Srivastava, P. (2023). spaCy vs. NLTK: A comprehensive comparison. Medium. https://medium.com/@prabhuss73 Thallam, N. S. T. (2025). Natural language processing (NLP) beyond 2024: The future of human-AI communication and language models. NST Journal of Scientific and Engineering Research, 12(1), 182–190.
Tucudean, G., Bucos, M., Dragulescu, B., & Caleanu, C. D. (2024). Natural language processing with transformers: A review. PeerJ Computer Science. https://doi.org/10.7717/peerj-cs.2222 Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., ... & Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems, 30.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Chi, E., Le, Q., & Zhou, D. (2022). Chain-ofthought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903.
arxiv. (2025). Advancements in natural language processing: Exploring transformer-based architectures for text understanding. arXiv:2503.20227.
https://arxiv.org/abs/2503.20227
