<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[Rasel's Developer Journal]]></title><description><![CDATA[Rasel's Developer Journal]]></description><link>https://rasel-dev.hashnode.dev</link><generator>RSS for Node</generator><lastBuildDate>Mon, 16 Nov 2026 23:41:53 GMT</lastBuildDate><atom:link href="https://rasel-dev.hashnode.dev/rss.xml" rel="self" type="application/rss+xml"/><language><![CDATA[en]]></language><ttl>60</ttl><item><title><![CDATA[Building KnowledgeHub: My Journey into AI-Powered Document Search and RAG]]></title><description><![CDATA[Building KnowledgeHub, a full-stack AI document assistant using Next.js, NestJS, PostgreSQL, embeddings, and RAG.]]></description><link>https://rasel-dev.hashnode.dev/building-knowledgehub-my-journey-into-ai-powered-document-search-and-rag</link><guid isPermaLink="true">https://rasel-dev.hashnode.dev/building-knowledgehub-my-journey-into-ai-powered-document-search-and-rag</guid><category><![CDATA[Next.js]]></category><category><![CDATA[nestjs]]></category><category><![CDATA[TypeScript]]></category><category><![CDATA[Artificial Intelligence]]></category><category><![CDATA[RAG ]]></category><dc:creator><![CDATA[Md. Rasel]]></dc:creator><pubDate>Sun, 15 Nov 2026 03:38:46 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/covers/6af83d26ebd6f2495630000a/1ff8ef22-3bb5-4516-aeba-6dc9f5f9d19a.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<h1>Introduction</h1>
<p>As a frontend developer, I have always been interested in building applications that solve real-world problems.</p>
<p>After working with React, Next.js, Angular, and TypeScript, I decided to expand my knowledge into backend development and AI application development.</p>
<p>Rather than learning technologies individually, I wanted to build a practical full-stack project that would help me understand how modern AI-powered applications work.</p>
<p>That idea led me to build <strong>KnowledgeHub – AI Document Assistant</strong>.</p>
<p>In this article, I will explain the project overview, architecture, technology stack, implemented features, document-processing workflow, and how I approached embeddings and Retrieval-Augmented Generation (RAG).</p>
<h1>1. What Is KnowledgeHub?</h1>
<p>KnowledgeHub is a full-stack document management and AI-assisted search application.</p>
<p>The main goal is to help users upload PDF and DOCX documents, extract their text, divide the content into smaller chunks, generate embeddings, and search through their document collections.</p>
<p>The application also includes a question-answering pipeline designed around the RAG architecture.</p>
<h3>The problem I wanted to solve</h3>
<p>Imagine having multiple documents containing important information:</p>
<ul>
<li>Study materials</li>
<li>Research papers</li>
<li>Technical documentation</li>
<li>Business reports</li>
<li>Project documentation</li>
</ul>
<p>Finding specific information in a large collection of documents can be time-consuming.</p>
<p>KnowledgeHub aims to simplify this process by providing document management, content search, and a question-answering interface.</p>
<h1>2. Technology Stack</h1>
<p>I selected the following technologies to build the application.</p>
<h3>Frontend</h3>
<ul>
<li>Next.js</li>
<li>React</li>
<li>TypeScript</li>
<li>Tailwind CSS</li>
<li>Lucide React</li>
</ul>
<h3>Backend</h3>
<ul>
<li>NestJS</li>
<li>TypeScript</li>
<li>Prisma ORM</li>
<li>PostgreSQL</li>
<li>JWT authentication</li>
</ul>
<h3>AI and document processing</h3>
<ul>
<li>OpenAI API integration</li>
<li>Text embeddings</li>
<li>PDF text extraction</li>
<li>DOCX text extraction</li>
<li>Text chunking</li>
<li>Retrieval-Augmented Generation (RAG)</li>
</ul>
<h3>Testing and development</h3>
<ul>
<li>Jest</li>
<li>Postman</li>
<li>Swagger</li>
<li>Git and GitHub</li>
</ul>
<h1>3. System Architecture</h1>
<p>I separated the application into frontend, backend, database, and AI service layers.</p>
<p>The frontend handles user interactions, while the NestJS backend manages business logic, authentication, document processing, and AI-related operations.</p>
<p>The database stores user information, document metadata, chunks, and embeddings.</p>
<h3>High-level architecture</h3>
<pre><code class="language-text">             USER
               |
               v
       NEXT.JS FRONTEND
               |
          HTTP / REST
               |
               v
        NESTJS BACKEND
               |
       +-------+--------+
       |       |        |
       v       v        v
   AUTH    DOCUMENTS   AI SERVICES
       |       |        |
       +-------+--------+
               |
               v
          PRISMA ORM
               |
               v
          POSTGRESQL
</code></pre>
<p>This separation makes the application easier to maintain and extend.</p>
<h1>4. Implemented Features</h1>
<h2>4.1 User Authentication</h2>
<p>I implemented user registration and login using NestJS and JWT-based authentication.</p>
<p>Passwords are hashed before being stored in the database.</p>
<p>Authentication tokens are handled through HTTP-only cookies.</p>
<p>Protected backend endpoints use authentication guards to identify the current user and restrict access to their data.</p>
<p><strong>Benefits:</strong></p>
<ul>
<li>Individual user accounts.</li>
<li>Protected application routes and APIs.</li>
<li>User-specific document management.</li>
</ul>
<h2>4.2 Dynamic Dashboard</h2>
<p>The dashboard provides an overview of the user's document collection.</p>
<p>It retrieves document information from the backend and displays:</p>
<ul>
<li>Total documents.</li>
<li>Ready documents.</li>
<li>Processing or uploaded documents.</li>
<li>Failed documents.</li>
<li>Recent document information.</li>
</ul>
<p>Users can also search their document list and upload new files directly from the dashboard.</p>
<p><strong>Benefits:</strong> Users can monitor their documents and processing status from one place.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6af83d26ebd6f2495630000a/ec35e1bd-e875-45d7-b8c7-3da53c95b2fd.png" alt="" style="display:block;margin:0 auto" />

<h2>4.3 Document Upload</h2>
<p>KnowledgeHub supports PDF and DOCX uploads.</p>
<p>I implemented file uploads using Multer in the NestJS backend.</p>
<p>The current upload limit is 10 MB.</p>
<p>The upload process works as follows:</p>
<ol>
<li>The user selects a document.</li>
<li>The frontend validates its file extension and size.</li>
<li>The document is sent to the backend using multipart FormData.</li>
<li>Multer receives and stores the file.</li>
<li>The backend creates the document record and starts the processing workflow.</li>
<li>The document-processing pipeline starts.</li>
</ol>
<p><strong>Benefits:</strong> Users can upload their own documents without manually copying their content into the application.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6af83d26ebd6f2495630000a/59bb8adc-0c26-49bf-9901-8091ec493c2d.png" alt="" style="display:block;margin:0 auto" />

<h2>4.4 PDF and DOCX Text Extraction</h2>
<p>After uploading a document, the backend extracts its textual content.</p>
<p>The extracted content becomes the input for the next processing stage.</p>
<p>For example, an uploaded document might contain:</p>
<pre><code class="language-text">Introduction to Database Management

A database is an organized collection of data.
Database indexes help improve data retrieval.
</code></pre>
<p>The application extracts this text so that it can be processed and searched.</p>
<p><strong>Benefits:</strong> The application can work with the content inside documents instead of only storing their filenames.</p>
<h2>4.5 Text Chunking</h2>
<p>Large documents can contain thousands of words.</p>
<p>Processing an entire document as one large text block is not always practical for retrieval-based AI applications.</p>
<p>To solve this, I implemented a text chunking service.</p>
<p>The current configuration uses approximately:</p>
<ul>
<li>Chunk size: 1,200 characters.</li>
<li>Chunk overlap: 200 characters.</li>
</ul>
<p>For example:</p>
<pre><code class="language-plaintext">Original document
       |
       v
+----------------------+
| Full extracted text  |
+----------------------+
       |
       v
+----------------------+
| Chunk 1              |
| First section        |
+----------------------+
       |
       v
+----------------------+
| Chunk 2              |
| Overlapping section  |
+----------------------+
       |
       v
+----------------------+
| Chunk 3              |
| Next section         |
+----------------------+
</code></pre>
<p>The overlap helps preserve context between consecutive chunks.</p>
<p><strong>Benefits:</strong></p>
<ul>
<li>Smaller text sections.</li>
<li>More focused retrieval.</li>
<li>Better preparation for embedding generation.</li>
</ul>
<img src="https://cdn.hashnode.com/uploads/covers/6af83d26ebd6f2495630000a/ea4a745d-5cd5-44e4-8b4b-283d0b48fb05.png" alt="" style="display:block;margin:0 auto" />

<h2>4.6 Embedding Generation</h2>
<p>Embeddings convert text into numerical vectors.</p>
<p>For example:</p>
<pre><code class="language-text">"How does database indexing work?"
                |
                v
        Embedding Model
                |
                v
       [0.12, -0.45, 0.78, ...]
</code></pre>
<p>These vectors can be compared to estimate the similarity between text sections.</p>
<p>I implemented an embedding service with two provider options:</p>
<ul>
<li>OpenAI embedding provider.</li>
<li>Mock embedding provider.</li>
</ul>
<p>The current development configuration uses a mock provider with 1536-dimensional test vectors.</p>
<p>An important limitation is that these mock vectors are not genuine semantic embeddings. They help test the processing and storage pipeline, but they do not understand the meaning of text.</p>
<p>Real semantic search requires a compatible real embedding model.</p>
<p><strong>Benefits:</strong> The embedding service establishes the foundation for vector-based document retrieval.</p>
<h2>4.7 Embedding Storage</h2>
<p>After generating embeddings, I store them alongside document chunk information in PostgreSQL.</p>
<p>The application uses Prisma ORM to manage database operations.</p>
<p>Each document chunk contains information such as:</p>
<ul>
<li>Chunk content.</li>
<li>Chunk index.</li>
<li>Character count.</li>
<li>Embedding vector.</li>
<li>Embedding model name.</li>
<li>Embedding generation timestamp.</li>
</ul>
<p>The embedding data is currently stored using PostgreSQL JSONB.</p>
<p><strong>Benefits:</strong> Embeddings remain associated with their original document chunks and can be retrieved during search.</p>
<h2>4.8 Document Management</h2>
<p>Users can open individual document detail pages.</p>
<p>The details page displays:</p>
<ul>
<li>Document information.</li>
<li>Processing status.</li>
<li>Extracted text.</li>
<li>Chunk information.</li>
<li>Embedding status.</li>
</ul>
<p>Users can also generate embeddings for eligible documents and delete documents.</p>
<p><strong>Benefits:</strong> Users can inspect document processing and manage their uploaded files.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6af83d26ebd6f2495630000a/82fb5172-fcfc-467f-9238-db31783df7b3.png" alt="" style="display:block;margin:0 auto" />

<h2>4.9 Document Search</h2>
<p>I implemented a search API and a dedicated frontend search page.</p>
<p>The search service:</p>
<ol>
<li>Receives a user's search query.</li>
<li>Generates a query embedding.</li>
<li>Retrieves the user's embedded document chunks.</li>
<li>Calculates cosine similarity.</li>
<li>Sorts results by similarity.</li>
<li>Returns the requested matching chunks.</li>
</ol>
<p>Example request:</p>
<pre><code class="language-json">{
  "query": "IELTS writing task 2 essay",
  "limit": 5
}
</code></pre>
<p>Cosine similarity is used to compare the direction of two vectors.</p>
<p>The current search pipeline is implemented and tested. However, because I am using mock embeddings, the results are not genuine semantic search results yet.</p>
<p><strong>Benefits:</strong> The architecture allows users to search across document content rather than relying only on document titles.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6af83d26ebd6f2495630000a/b7a3b101-6750-4383-a792-c3c0266f6516.png" alt="" style="display:block;margin:0 auto" />

<h2>4.10 Question Answering with RAG</h2>
<p>One of the most interesting parts of KnowledgeHub is the question-answering pipeline.</p>
<p>RAG stands for Retrieval-Augmented Generation.</p>
<p>It combines document retrieval with language-model generation.</p>
<p>The general workflow is:</p>
<pre><code class="language-text">      USER QUESTION
             |
             v
      SEARCH SERVICE
             |
             v
   RETRIEVE DOCUMENT CHUNKS
             |
             v
     PREPARE CONTEXT
             |
             v
       AI LANGUAGE MODEL
             |
             v
      ANSWER + CITATIONS
</code></pre>
<p>I implemented a QA service that retrieves document chunks, prepares citations, and supports different QA providers.</p>
<p>The project currently includes:</p>
<ul>
<li>Mock QA provider.</li>
<li>OpenAI chat-completion integration.</li>
<li>Context preparation.</li>
<li>Source citation handling.</li>
<li>Error handling.</li>
</ul>
<p>The active configuration currently uses the mock QA provider.</p>
<p>The OpenAI integration is implemented, but real AI answer generation still needs validation with available API credits.</p>
<p><strong>Benefits:</strong> The RAG architecture provides a way to generate answers based on retrieved document information and show the sources used.</p>
<h2>4.11 AI Chat Interface</h2>
<p>I also created a dedicated chat page in Next.js.</p>
<p>Users can submit questions through the chat interface.</p>
<p>The frontend communicates with the QA endpoint and displays the returned answer and citations.</p>
<p>The interface includes:</p>
<ul>
<li>Message display.</li>
<li>Question input.</li>
<li>Loading states.</li>
<li>Error handling.</li>
<li>Example questions.</li>
<li>Source links.</li>
</ul>
<p>This makes the question-answering workflow accessible through a familiar conversational interface.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6af83d26ebd6f2495630000a/65d0ea28-ff2b-4f12-ad11-36af7a030660.png" alt="" style="display:block;margin:0 auto" />

<h1>5. Database Design</h1>
<p>KnowledgeHub uses PostgreSQL with Prisma ORM.</p>
<p>The main database models include:</p>
<ul>
<li>User</li>
<li>RefreshToken</li>
<li>Document</li>
<li>DocumentChunk</li>
</ul>
<p>The relationships are:</p>
<pre><code class="language-text">USER
 |
 +---- DOCUMENT
          |
          +---- DOCUMENT CHUNK
          |
          +---- DOCUMENT CHUNK
          |
          +---- DOCUMENT CHUNK

USER
 |
 +---- REFRESH TOKEN
</code></pre>
<p>A user can own multiple documents, and each document can contain multiple chunks.</p>
<p>This structure allows the application to retrieve document chunks belonging to the authenticated user.</p>
<h1>6. API Architecture</h1>
<p>The NestJS backend exposes REST API endpoints.</p>
<p>Some of the main endpoints include:</p>
<table>
<thead>
<tr>
<th>Method</th>
<th>Endpoint</th>
<th>Purpose</th>
</tr>
</thead>
<tbody><tr>
<td>POST</td>
<td>/api/users/register</td>
<td>User registration</td>
</tr>
<tr>
<td>POST</td>
<td>/api/auth/login</td>
<td>Login</td>
</tr>
<tr>
<td>GET</td>
<td>/api/auth/me</td>
<td>Current user</td>
</tr>
<tr>
<td>POST</td>
<td>/api/documents/upload</td>
<td>Upload document</td>
</tr>
<tr>
<td>GET</td>
<td>/api/documents</td>
<td>Retrieve documents</td>
</tr>
<tr>
<td>GET</td>
<td>/api/documents/:id</td>
<td>Document details</td>
</tr>
<tr>
<td>GET</td>
<td>/api/documents/:id/chunks</td>
<td>Retrieve chunks</td>
</tr>
<tr>
<td>POST</td>
<td>/api/documents/:id/embed</td>
<td>Generate embeddings</td>
</tr>
<tr>
<td>DELETE</td>
<td>/api/documents/:id</td>
<td>Delete document</td>
</tr>
<tr>
<td>POST</td>
<td>/api/search</td>
<td>Search documents</td>
</tr>
<tr>
<td>POST</td>
<td>/api/qa/ask</td>
<td>Ask a question</td>
</tr>
</tbody></table>
<h1>7. Testing</h1>
<p>I used Jest to test backend services.</p>
<p>The previously confirmed backend test results were:</p>
<ul>
<li>12 test suites passed.</li>
<li>49 tests passed.</li>
</ul>
<p>I also used Postman to test API endpoints and verify document upload, search, and question-answering workflows.</p>
<p>Testing helped me understand the importance of error handling, validation, and reliable backend services.</p>
<h1>8. Challenges and Lessons Learned</h1>
<p>Building KnowledgeHub has been a valuable learning experience.</p>
<p>Some of the challenges I encountered were:</p>
<p><strong>Learning NestJS:</strong> Understanding modules, controllers, services, dependency injection, and authentication helped me expand beyond frontend development.</p>
<p><strong>Document processing:</strong> Handling uploaded files, extracting text, and tracking processing status required careful backend design.</p>
<p><strong>Database relationships:</strong> Designing relationships between users, documents, and chunks improved my understanding of relational databases.</p>
<p><strong>Embeddings:</strong> Learning how text can be represented as vectors introduced me to a new area of AI application development.</p>
<p><strong>RAG architecture:</strong> Understanding the difference between retrieval and generation helped me understand how document-based AI assistants work.</p>
<h1>9. Current Limitations and Future Improvements</h1>
<p>KnowledgeHub is an ongoing learning project.</p>
<p>Some capabilities are implemented as development pipelines rather than fully validated AI functionality.</p>
<p>My next improvements include:</p>
<ul>
<li>Testing with genuine semantic embeddings.</li>
<li>Enabling real AI-generated answers.</li>
<li>Exploring PostgreSQL pgvector.</li>
<li>Improving retrieval quality.</li>
<li>Adding conversation history.</li>
<li>Implementing streaming responses.</li>
<li>Improving production deployment and file storage.</li>
</ul>
<h1>10. Conclusion</h1>
<p>Building KnowledgeHub has helped me explore full-stack development beyond the frontend.</p>
<p>I have gained practical experience with NestJS, PostgreSQL, Prisma, document processing, embeddings, search, and RAG architecture.</p>
<p>My goal is to continue improving the application and gradually turn the current learning implementation into a more complete AI-powered document assistant.</p>
<p>You can explore the source code here:</p>
<p><strong>GitHub Repository:</strong> <a href="https://github.com/raselcsedev/knowledgehub">https://github.com/raselcsedev/knowledgehub</a></p>
<p>Thank you for reading about my project and learning journey!</p>
]]></content:encoded></item></channel></rss>