Chatbots are a thing again, thanks to GenAI and the ease with which you can create such bots with OpenAI APIs or other bespoke models. They pop-up on most websites — funneling users to a product, providing assistance or just more details about the product.
Let me show you how I built one in one on a Saturday evening, maybe spending about 2 hours. Thoughts about how I would improve this at the end.
The codebase is available on my Github — https://github.com/mrsauravsahu/poc-chatbot-genai
The good and the bad ones
While some companies have already gotten their implementation pretty much right, there are many others who still suffer from providing the default ChatGPT like responses, given the users can just type in anything.
So the idea is to build a bot as quickly as possible, but incorporating a few basic guardrails to ensure no irrelevant questions are answered.
My LLM-App Architecture so far
This is the simplified architecture I have so far. The Backend can evolve from a simple server like mine to an entire platform with more convoluted compute system which has data sanitization, caching and auditing in place. For example, my payobills app has Kubernetes based compute instead of this simple server.
Comment on this post ‘arch’ if you want to see how to generate architecture diagrams like this with ChatGPT.
The Backend — v1.0.0-static-messages
It’s obviously a bad idea to directly communicate with OpenAPI in your application, and having a backend server as an intermediary can be used to parse through messages and it can help make it easier to the end state architecture I described earlier.
I want the application to get instant feedback about messages so WebSockets seems like the right way to go. I know SignalR can help me implement this really easily, but feel free to explore other technologies your team is experienced with to achieve this functionality.
First, I created the boilerplate static backend. The prompt I used to create the backend is this —
Give me a list of commands and the code to generate the backend app. — The app should be in .NET 8 using signalR — The backend should be able to connect to the SvelteKit app you generated above — For now, for any message received from the SvelteKit app, the response should be sent back as ‘Okay, got it. Let me help you with that’.
The UI/frontend v1.0.0-signalr
I am a big big fan of Svelte and just as I thought, creating a Chatbot in Svelte’s application framework SvelteKit was a breeze. Obviously, you should choose the right library/framework based on your team.
The prompt I used to generate the site was —
Generate a svelte kit project which has a chat window which connects to a signalR ASP.NET Core backend — Make the UI simple but clean — dark mode — The app should be responsive and use mobile first code — As a user, I should have a text input at the bottom of the screen to be able to send and receive messages
Now the UI started working and I could see my basic looking chat UI. The UI generated had zero CSS for some reason, we’ll get to that.
Like any other application, I also got CORS errors 😂, so I had to proxy to the vite config in the SvelteKit app, again ChatGPT to the rescue —
give me changes to make to viteconfig to proxy the /chathub to my backend running at 5073
The LLM connection
I wanted to get something out there, and OpenAI’s APIs make it really easy. I’ll show you how to use other APIs in future posts.
I didn’t have a free trial for OpenAI’s APIs so I added $5.00 to my account over at https://platform.openai.com/
Next was the time to use the OpenAI API. I had done some reading at https://learnprompting.org/docs/intro. The docs are available explained the text generation API pretty well, and just to confirm I created a basic prompt and curl to test it out. And it worked pretty well — tried both positive and negative tests on this.
You can think of adding more context and preserving context as well, but this just shows the basics. Feel free to comment on how you would achieve robust assistant with this single API.
I have added the basic guards using the system prompt to atleast have a net to catch any obviously misleading messages sent to the bot via the App. The backend server exists to make it extensible, and can help do a lot of things to make this production ready —
- Giving you the ability to add rigid rules via code
- Adding extra guards before sending the message to OpenAI and after receiving the response from them.
- You can make the responses fairly deterministic by generating seed numbers or setting the temperature right — more in OpenAI’s docs
- Insert user and session specific context
Connecting the Backend to LLM
Because I have SignalR, all I need to do is create a class to parse and pass on the user’s message to OpenAI. But we can also make it generic to make sure we can use other LLMs later (just saying) — don’t forget SOLID principles.
Because I already had the custom crafted curl to invoke OpenAI’s Text Generation API, I used this prompt to generate this OpenAIClient class for my backend —
Make a reusable class for my ASP.NET Backend — take in an object as a parameter, with one property Text string and get the response to be sent using the below API call —
Make a reusable class for my ASP.NET Backend - take in an object as a
parameter, with one property Text string and get the response to be sent using the
below API call - replace the user message into the <USER-QUESTION> in the
body
```
curl --location 'https://api.openai.com/v1/chat/completions' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer TOKEN' \
--data '{
"model": "gpt-3.5-turbo",
"messages": [
{
"role": "system",
"content": "You are an assistant for a finance website which has features like - listing the bills that a user has to pay; get the list of payments made for these bills; mark a bill as paid; create new bills to be paid monthly on a specific date, or a prepaid bill to be renewed every X days. With all questions, respond back with a relevance score of whether the question that is asked is relevant to this assistant, respond with '\''I'\''m not sure if I can help with that'\'' in the right language if the relevance score is less than 75%; You HAVE TO ALWAYS RETURN CONTENT IN JSON, and it should have a content (string) property and the relevance_score (double value) property"
},
{
"role": "user",
"content": "<USER-QUESTION>"
}
]
}'
```
At this point, all our coding for the logic was done. The app started giving me responses from OpenAI’s API through SignalR and I was able to basically use the assistant. All I needed some formatting and CSS.
I hadn’t done CSS in a while so I felt like writing some, it was fun, but obviously you can explain and get it generated from ChatGPT too. You obviously know the final UI, but here it is again.
The responses aren’t perfect. And I would modify the prompt to make sure all responses also generate tags like “GET /bills”, “GET /balances?bill=amex”, essentially helping the backend to all more details from the data store, but this was a fun little project.
I haven’t shared most of my work on LLMs so far and I have most of my projects stored secretly on my private git, but I can get it ready if I see enough enthusiasm from you. Parts of it will show up in my upcoming ‘Beyond FullStack Engineering’ series on my YouTube
As always this is just the beginning, and you can do a lot more with OpenAI APIs and GenAI and LLMs in general. Happy to hear your thoughts, hit me up on LinkedIn.
— Sahu
