en_US
en_US zh_TW
en_US
en_US zh_TW
Home Assistant step-by-step tutorial

A truly smart home with brains! Home Assistant customized AI voice assistant ChatGPT, Gemini makes your smart home come alive – Home Assistant step-by-step tutorial EP 9

, Modified:
Neil
Neil

Have you seen Iron Man? The conversation between Tony and Jarvis must be the dream of many people (at least it is mine🤣). Today I will show you how to use the most popular AI (ChatGPT, Google Gemini) to customize the HA voice assistant to make the original A mentally retarded assistant becomes a truly "smart" assistant!

However, because I just got the ESP32-S3-BOX-3, I need to spend more time to install and test it. Today, I can only show you how to use Home Assistant, and I can’t talk yet ~ I will show you how to use ESP32-S3 in the next episode -BOX-3 Let your Home Assistant speak~

Why do we need smarter voice assistants?

鋼鐵人 with Jarvis

You must have used "last generation" voice assistants such as Siri and Google Assistant, right? Commonly known as "mentally retarded assistant" 🤣 Because voice assistants of the previous generation such as Siri and Google Assistant have a common problem, that is, they can only understand fixed instructions. Rather than understanding, it is actually more like "can recognize", and recognition The method is hard-coded into the program by engineers, so if you don’t use precise words, these last-generation voice assistants will not understand what you are saying at all.

The preset voice assistant of the Home Assistant system is also in the same situation. This has also resulted in the proportion of voice control used in my smart home being less than 1%, because the success rate is too low. If you want to use it, it is better to just take it. It’s faster to control with a mobile phone🤬

In recent months, Google has allowed the built-in voice assistant in Android to be replaced with Gemini (but not Nest smart home speakers), and Apple has also reported that it is preparing to cooperate with ChatGPT. In the past, this "previous generation of mentally retarded assistants" should There will be improvements soon, and Home Assistant has also launched LLM (Large Language Model, large language model) that can be used by customized voice assistants in recent months. In order to make the smart home as localized as possible, let us start trying it. Give it a try!

Step 1. Create Home Assistant voice assistant

Step 1-1. "Settings" on the left ➜ "Voice Assistant"

左側「設定」 ➜ 「語音助理」

Step 1-2. Click "Add Assistant"

(If there is already an assistant here, you can skip it directly)

點擊「新增助理」

Step 1-3. Enter the "name" yourself and use the default options for everything else.

自行輸入「名稱」

Step 1-4 Test whether the assistant is normal

Go back to "Overview", click the speech bubble button in the upper right corner, and try talking to the assistant. If there is a response, it means success!

At this time, it is still a "mentally retarded assistant", don't worry, next we will connect ChatGPT and Google Gemini to make it smart! 👍

測試助理是否正常

Step 2. Obtain the OpenAI API key

Step 2-1. Register/Login OpenAI Platform

This is different from the commonly used ChatGPT, which comes first OpenAI Platform Complete registration and log in.

Step 2-2. Create OpenAI secret key

  1. Click "API keys" in the left column
  2. Click "Create new secret key" in the upper right corner
  3. 「Create secret key」Create secret key
Create secret key

Step 2-3. Save secret key

"Copy" copy it, find a place to save it, you will use it later.

保存 secret key

Step 3. Add “OpenAI Conversation” integration

Step 3-1. "Settings" on the left side of Home Assistant ➜ "Devices and Services"

Home Assistant 左側「設定」 ➜ 「裝置與服務」

Step 3-2. Install “OpenAI Conversation”

  1. Click "Add Integration" in the lower right corner
  2. Find "OpenAI Conversation" and click
  3. Fill in the API key in Step 2-3. Obtained secret key
  4. Click "Send"
安裝「OpenAI Conversation」

Step 4. Obtain Google AI API key

Step 4-1. Register/Login Google AI Studio

to Google AI Studio Complete registration and log in.

Step 4-2. Create Google AI API key

  1. Click "Get API key" in the left column
  2. Click "Create API key"
  3. Select "Google Cloud project" (if it doesn't exist, follow the guided steps to add one)
  4. Click "Create AI key in existing project"
  5. Copy "Copy" in the fill-in window and save the API key for later use.
建立 Google AI API key

Step 5. Add “Google Generative AI” integration

Step 5-1. "Settings" on the left side of Home Assistant ➜ "Devices and Services"

Home Assistant 左側「設定」 ➜ 「裝置與服務」

Step 5-2. Install "Google Generative AI"

  1. Click "Add Integration" in the lower right corner
  2. Find "Google Generative AI" and click
  3. Fill in the API key in Step 4-2. Obtained API key
  4. Click "Send"
安裝「Google Generative AI」

Step 6. Set the conversation assistant used by Home Assistant voice assistant

It’s a bit verbose, but to put it simply, it’s to replace the original Home Assistant’s retarded voice assistant with ChatGPT or Google Gemini, whichever you like🤣

Step 6-1. Select Home Assistant voice assistant

  1. Home Assistant "Settings" on the left ➜ "Voice Assistant"
  2. Choose your voice assistant
選擇 Home Assistant 語音助理

Step 6-2. Set up conversation assistant

  1. In the menu, you can choose between Home Assistant, ChatGPT, and Google Generative AI, choose the one you like (here is a demonstration using Google)

💡 There is a ⚙️ on the right side of the menu for advanced settings. It is recommended to try to adjust it after you have figured out the functions~

  1. Click "Update"
設定對話助理

You're done!

congratulations! Your Home Assistant has become a real smart assistant~ Try talking to your Home Assistant~

You can also try to make your voice assistant more personalized through the advanced settings mentioned in Step 6-2.

大功告成！

Next episode preview

The next episode will show you how to use ESP32-S3-BOX-3 to create a “talking” voice assistant in Home Assistant!

Stay tuned~

19 COMMENTS

  1. May I ask which device should I use to make voice calls? Can I also make calls using the assistant on the smart watch? Thank you

    • Hi George, first make sure the API key is filled in correctly, and remember to top up before you can use it! API calls and subscriptions to ChatGPT are separate

      • If the API key is not filled in correctly, it should not be configured into the integration. It seems that it may need to be stored in value.

  3. Please tell me, when I set up Google Generative AI and tried to set the device On/Off or other functions appeared, sorry I had a problem talking to google generative ai:500 An internal error has occurred. Please retry if this part is still there Do I need to make other settings?
    When I try to get the device status and currently turned on lights the reply information is stable.
    Thanks!

    • Hi Yean, I checked and the 500 error refers to "Your input context is too long." (The input background information is too long). Reduce the number of authorized devices and try again.

  4. Please tell me that after I completed the settings and wanted to control the device On/Off or other functions through Google Generative AI, I was prompted with the message "sorry i had a problem talking to google generative ai: 500 An internal error has occurred. Please retry this part" Are there any other settings that need to be done? Thank you!

    • Hi Richard, Google has a free (Free of charge) plan to choose from. Of course, there are still usage restrictions, but it is enough if you try it. If you like it, consider paying.

      • Hi Neil~ I just checked to see if the API services of GOOGLE AI and OPEN AI are free as long as I use them within the traffic limit. If you exceed the traffic limit, you will have to pay. Is that true? Sorry for the many questions~Thank you

      • Hi Richard, OpenAI should have a billing plan set up so that the API can be opened, and it must be recharged before it can be used.
        Or you can also read the official one Pricing page

      • It turns out that this is the case. My OpenAI has not set a billing plan, so no wonder an Error Code appears. So if I use Google AI, is it as I mentioned above that it is free within the traffic, and it will ask you to pay if it exceeds the traffic, or do I need to go What about choosing a payment mechanism? It's a bit messy~

      • Hi Richard,
        I choose the free plan with Google. My understanding is that it is free within the quota and cannot be used when the quota is exceeded.

  6. Hi Neil:
    I currently follow your instructions and successfully use Google Generative AI as an assistant. However, when using ChatGPT as an assistant, the dialogue shows OPEN AI Error Code: 429. Check the OPEN AI error code comparison table, which means that my frequent use has exceeded the usage limit. Yes Are there any charges?

    • Hi Richard,
      Look at the official oneillustrateIt's probably because the calls are too frequent. You can try again every other day. OpenAI's API usage also requires payment. Please check your OpenAI bill to avoid expiration.

LEAVE A REPLY

Please enter your comment!
Please enter your name here

you might like
recent discussions
  • Jerry 2025/04/14Thank you very much Neil for your selfless sharing. The websites you provided are of great help to me. I will save it first and digest it slowly^^ I also look forward to Neil's industry news and teaching articles in the futureHome Assistant Floor plan A new experience in smart home control with visual floor plan! – Home Assistant step-by-step tutorial EP 6
  • Neil 2025/04/13Hi Jerry, I usually get information about HA from these places: Website: – Home Assistant official Release notes – Automata – Home Automation Community – Weijia Home Intelligence FB Group: – Taiwan Home Assistant enthusiasts exchange – Home Assistant YouTube channel: – Smart Home Solver – Everything Smart Home – Home Assistant Programming language: Strictly speaking, YAML is not really a programming language (let me make this clear in case some people mind 🤣), but if we include these data formats or style languages that “look like” programming languages, and sort them by depth of use in HA (the deeper the bottom, the deeper the bottom): – YAML (data format): a language for configuration files used in many places in the HA environment. – CSS (style language): used to adjust the interface style (font size, color, image, position, etc.). – JSON (data format): If you use Node-RED, you will probably come across it. – Jinja2 (Python template syntax): You may encounter it when you explore automation or Dashboard in depth. ※ But in fact, even if HA does not know the above, it can still make a smart home complete. You don’t have to put so much pressure on yourself, but if you are interested, it is also good to know more. It is actually difficult to generalize about the support of devices. Now, even if the home appliances themselves do not support it, there are still many different ways to connect traditional home appliances to HA. In addition, cases like Haier do require in-depth research. What I usually do is to crawl a lot of information on the Internet, and usually find attempts made by others. It just so happens that when I was looking up other information recently, I saw someone sharing how Haier water heaters were connected to HA. But it is indeed like what you said, if you get involved for a while, you will understand these details more and more. 🤣Home Assistant Floor plan A new experience in smart home control with visual floor plan! – Home Assistant step-by-step tutorial EP 6
  • Jerry 2025/04/10Hello Neil: After observing for 1 or 2 days, everything seems to be normal. Maybe it’s because the system was not stable during the initial setup period!? The more I play, the deeper I feel. I also find that I still have many shortcomings. I wonder if Neil can share and teach everyone how to fish? For example, recommend good websites, forums, FB groups, youtube videos, or give some guidance. In addition to the YAML language, would it be helpful to know some CSS syntax? Do I need to know other languages? I can even start a new teaching direction. When I see non-big-name products that claim to support APP (such as products from mainland China), I can teach everyone how to identify whether they can be connected to HA and how to ask customer service? Where can I see supported articles or platforms? This kind of experience sharing, I think, will be very helpful to everyone. I am currently looking to buy a heat pump water heater from Haier. Haier is a big manufacturer in the world and has its own APP. I have also found Haier's kit from HACS, but it does not seem to support heat pump water heaters. Does that mean it cannot be connected to HA? Or is there any channel to check it? There are also many flat panel lights and fan function lamps on the other side of the strait, which also claim to support APP connection, but they are all small manufacturers and it is difficult to tell whether they can be connected to HA. These are the problems I am facing now. If the home appliances I buy are different from what Neil shared, I don’t know where to start. Maybe I'm still too inexperienced. If I play for a few more months, maybe the above problems won't be a problem anymore, haha^^Home Assistant Floor plan A new experience in smart home control with visual floor plan! – Home Assistant step-by-step tutorial EP 6
  • Neil 2025/04/09Hi Jerry, I tested it myself and it was fine, but please note that because it is controlled through the cloud, there is an extra layer of brand mediation, so it is not suitable for fast switching. I tested it myself and it took about 1 second to complete the synchronization after clicking. Remember to put the entity in entities, otherwise the status display may have problemsHome Assistant Floor plan A new experience in smart home control with visual floor plan! – Home Assistant step-by-step tutorial EP 6
  • Jerry 2025/04/07Thanks Neil for clarifying this. I tested it later and found that when using the "button panel" to control the switch of the Electrolux air conditioner (the same model as Neil), the device will respond to turning on or off in about 1 second, but the "status icon" for turning on is much slower, and it takes about 30 seconds. The "status icon" for turning off is normal, and the "status icons" of other negative ion switches and the "status icons" of the monitoring light switch are also normal, which made me think there was a problem with the synchronization. I'll test it again, maybe it's caused by the wrong setting.Home Assistant Floor plan A new experience in smart home control with visual floor plan! – Home Assistant step-by-step tutorial EP 6
  • Neil 2025/04/07Hi Jerry, 1. This method is through cloud integration, which is actually using HA to call Xiaomi/Electrolux to control the device. 2. It is not normal. If this happens, you may need to check whether the entity you display on the floor plan can truly represent the status of the device. If not, you may need to try other entities of the same device.Home Assistant Floor plan A new experience in smart home control with visual floor plan! – Home Assistant step-by-step tutorial EP 6
  • Jerry 2025/04/07The problem is solved. Thanks again to Neil for his help^^ I have two more questions about concepts. Neil, do I have any misunderstandings? 1. I connected Xiaomi and Electrolux devices to HA through HACS. Is it still connected through WiFi? Before connecting, you need to obtain API key and Access Token from the official. This means that the communication between HA and the device still needs to go through the official. It’s just that the official APP can be replaced and all integrated into the HA APP for control? 2. After testing, when controlling the device through physical operation or the original official APP, the HA app cannot know the status of the device synchronously, but it can know the opposite. Is this normal? For example, the Xiaomi electric fan at home was turned on by someone physically before going out, but when the person is outside, he cannot know through the HA app that the device was forgotten to be turned off. He has to connect to the Xiaomi app to know!?Home Assistant Floor plan A new experience in smart home control with visual floor plan! – Home Assistant step-by-step tutorial EP 6
  • Jerry 2025/04/07Xiaomi Home can be used in Singapore by selecting the countryHome Assistant must-install HACS plug-in installation steps detailed explanation – Home Assistant step-by-step tutorial EP 17
  • Ghost 2025/04/07Xiaomi Home currently does not support Taiwan. For now, you can only use Xiaomi Miot Auto. As for Xiaomi Cloud Map Extractor, I currently have a 2FA problem. There seems to be no simple solution. I haven’t tried it successfully yet.Home Assistant must-install HACS plug-in installation steps detailed explanation – Home Assistant step-by-step tutorial EP 17
  • Neil 2025/04/07Jerry, it seems that the click action cannot penetrate the part where the picture changes because of the overlapping pictures. Try adding pointer-events: none to the style.Home Assistant Floor plan A new experience in smart home control with visual floor plan! – Home Assistant step-by-step tutorial EP 6
you might like

© Copyright - SpaceN (https://neiltw.com/en)