ESP32-S3-BOX-3 deploys Home Assistant localized voice assistant, 3 personalized teaching methods – Home Assistant step-by-step teaching EP 10

Everyone should be familiar with controlling smart homes through voice. Even on mobile phones, we often use Hey Siri or OK Google... to call voice assistants. Continuation of the previous article:

We replaced the Home Assistant voice assistant with Google Gemini or ChatGPT, so that the response of the Home Assistant voice assistant would not be as stiff as Siri and would be smarter. It could even be assigned a personality according to your own preferences, but at that time it could only interact with typing. .

Today I will show you how to use ESP32-S3-BOX-3 to build your own Home Assistant. This is better than synchronizing the device to HomeKit or Google Home through HomeKit Bridge or Google Home integration, and then using Siri or Google Assistant voice control. It is more direct and you can create a more suitable voice assistant according to your own preferences👍

Let’s take a look at the final effect first:

Great! It’s so easy to experience the life of an ancient emperor🤣

Without further ado, let’s get started!


ESP32-S3-BOX-3 unboxing

ESP32-S3-BOX-3 開箱圖

Contents:

  1. ESP32-S3-BOX-3 Me: This is the host computer. It can be used directly regardless of other accessories being plugged in.
  2. DOCK: Base, with interfaces on it to connect other devices in series.
  3. SENSOR: It has the same function as DOCK, but it has additional temperature and humidity sensors, infrared transmitter/receiver, radar sensor, rechargeable battery slot, and MicroSD slot. Although I can’t use them today, I use this one because it’s more handsome🤣
  4. BRACKET: Adapter accessory, you can use this to install the host on other devices.
  5. BREAD: Adapter accessory, used to connect to breadboard.
  6. transmission line
  7. RGD module, Dupont line

For what we are going to do today, you can actually buy "ESP32-S3-BOX-3B", which includes the host and DOCK, and the others will not be used. Think about it when you want to buy it.


Step 1. Install and launch the ESPHome add-on

Step 1-1. "Settings" on the left side of Home Assistant ➜ "Add-ons"

Step 1-1. Home Assistant 左側「設定」 ➜ 「附加元件」

Step 1-2. Install "ESPHome"

  1. Click "Add-on Store" in the lower right corner
右下角點選「附加元件商店」
  1. Find "ESPHome" and click
找到「ESPHome」並點擊
  1. Install "ESPHome"
安裝「ESPHome」
  1. Turn on Watchdog, automatically update, and display in the sidebar
  2. Click "Start"
開啟 Watchdog、自動更新、於側邊欄顯示、點擊「啟動」
  1. "ESPHome" startup completed!
「ESPHome」啟動完成!

Step 2. Install the openWakeWord add-on

Step 2-1. "Settings" on the left side of Home Assistant ➜ "Add-ons"

Step 2-1. Home Assistant 左側「設定」 ➜ 「附加元件」

Step 2-2. Install "openWakeWord"

  1. Click "Add-on Store" in the lower right corner
右下角點選「附加元件商店」
  1. Find "openWakeWord" and click
找到「openWakeWord」並點擊
  1. Install "openWakeWord"
安裝「openWakeWord」
  1. Turn on Watchdog and automatically update
  2. Click "Start"
開啟 Watchdog、自動更新、點擊「啟動」
  1. "openWakeWord" startup completed!
「openWakeWord」啟動完成!

Step 3. Add "openWakeWord" integration

Step 3-1. "Settings" on the left side of Home Assistant ➜ "Devices and Services"

Step 3-1. Home Assistant 左側「設定」 ➜ 「裝置與服務」

Step 3-2. Set up "openWakeWord" integration

  1. At this time, "openWakeWord" will appear in the "Scanned" section, click "Settings"
  2. The system will ask whether to use Wyoming Protocol, click "Send"
  3. Click "Finish" in the success message window
這時候會在「已掃描」區塊出現「openWakeWord」,點擊「設定」系統會詢問是否使用 Wyoming Protocal,點擊「傳送」在成功訊息視窗點擊「完成」

Step 4. Set the voice assistant wake word

Step 4-1. Home Assistant "Settings" on the left ➜ "Voice Assistant"

Step 4-1. Home Assistant 左側「設定」 ➜ 「語音助理」

Step 4-2. Select voice assistant

Step 4-2. 選擇語音助理

Step 4-3. Set wake word

  1. Wake word engine selects "openwakeword"
  2. Choose a wake-up word that you like. Here is an example of choosing "ok nabu"
喚醒詞引擎選擇「openwakeword」喚醒詞選一個你自己喜歡的,這裡示範選擇「ok nabu」

Now that you have set the wake word, proceed to the installation and settings of ESP32-S3-BOX-3!


Step 5. Install and set up ESP32-S3-BOX-3

⚠️Attention! If you have installed S3 BOX in the past, be sure to go to "Settings > Devices and Services > ESPHome" to find the S3 BOX and delete it⚠️

Step 5-1. Flash ESP32-S3-BOX-3 firmware

  1. Go to the ESPHome project website:https://esphome.io/projects/index.html
  2. Select "Voice assistant"
  3. Select "ESP32-S3-BOX-3"
  4. Click "Connect"
前往 ESPHome 專案網站:https://esphome.io/projects/index.html選擇「Voice assistant」選擇「ESP32-S3-BOX-3」點擊「Connect」
  1. Connect ESP32-S3-BOX-3 to your computer through the transmission cable
  2. Select the ESP32-S3-BOX-3 device
  3. Click "Connect"
將 ESP32-S3-BOX-3 透過傳輸線連接到你的電腦選擇 ESP32-S3-BOX-3 裝置點擊「連線」
  1. Select "Install ESP32 S3 Box 3"
  2. Click "Install"
  3. Click "Next"
選擇「Install ESP32 S3 Box 3」點擊「Install」點擊「Next」

Step 5-2. Set up WiFi connection

Enter your home WiFi connection information and click "Connect"

輸入家中的 WiFi 連線資訊並點擊「Connect」

Step 5-3. Add the device to Home Assistant

  1. Click "Add to Home Assistant"
點擊「Add to Home Assistant」
  1. Confirm Home Assistant physical address
    The default is "http://homeassistant.local:8123". If you change it when installing Home Assistant, remember to change it to a URL that can connect to Home Assistant!
  2. Click "Open Link"
確認 Home Assistant 實體位址預設是 "http://homeassistant.local:8123",如果你在安裝 Home Assistant 時有自己變更,記得這裡要換成可以連接到 Home Assistant 的 URL!點擊「Open Link」
  1. At this time you will automatically be taken to the Home Assistant integration list, select "OK" in the pop-up window
  2. Select "ESP32 S3 Box 3" in the Add discovered device menu
  3. Select "Send" in the pop-up window
這時候你會自動被帶到 Home Assistant 整合列表中,在彈跳視窗中選擇「好」在新增所發現的裝置選單中選擇「ESP32 S3 Box 3」彈跳視窗中選擇「傳送」

Step 6. Check whether the ESP32-S3-BOX-3 integration is successful

  1. Home Assistant left column "Settings" > "Devices and Services"
  2. Find ESPHome click device count
Home Assistant 左側欄「設定」>「裝置與服務」找到 ESPHome 點擊裝置數

At this time, you can see the ESP32-S3-BOX-3 related device information, and you can mute and switch the screen, then you have successfully integrated it!

這時候有看到 ESP32-S3-BOX-3 相關裝置資訊,並可以進行靜音、螢幕開關那你就整合成功啦!

Try talking to the voice assistant!

Also featured – Personalize your voice assistant

Just switching to a voice assistant with a brain must still be too boring, right? ! Try changing your profile, customizing your display picture, and customizing your wake word!

Change persona

  1. Click on the right side of the conversation assistant menu⚙️
  2. Add the role you want it to play in the instructions
  3. Click "Send"
點擊對話助理選單右側 ⚙️在指示中新增你想要它扮演的角色點擊「傳送」

In this way, you can change it into a variety of characters you like! Share with me what kind of persona do you like to change into? 🤣

Customize voice assistant display picture

Frankly speaking, the default display image is really boring, right? Then try changing the picture! You can even change it to a photo of yourself (if you are narcissistic enough🤣), as if there is another self helping you handle big and small things, cool~

Before starting, take a look at the picture below to understand the 6 states of the Home Assistant voice assistant:

Home Assistant 語音助理的 6 種狀態
Image credit: Home Assistant

① Enter ESPHome web UI

Usually it will be in the sidebar. If not, go to "Settings > Add-ons" to find it.

進入 ESPHome web UI

② Put ESP32-S3-BOX-3 into management (ADOPT)

This will allow you to adjust various settings. Once completed you will see the status in the upper right corner change from DISCOVERED to ONLINE

ESP32-S3-BOX-3 納入管理 (ADOPT)
Image credit: Home Assistant
ESP32-S3-BOX-3 納入管理 (ADOPT)

③ Select picture material or create picture material

In Home Assistant Github repository There are materials shared for everyone to use. If you are like me and have no artistic talent, you can just find one you like from there.

Of course, you can also draw it yourself, or let AI draw it for you, and refer to the format and specifications to produce relevant picture material files:

  1. Size: 320×240. If the provided material is not in 4:3 ratio, the remaining area will be filled with background color.
  2. Image format: PNG, JPEG, SVG.
  3. In the image, you can use two background colors to roughly distinguish whether the voice assistant is active or inactive:
    • loading, idle state: dark background.
    • listening, thinking, reply status: light background.
    • error status: dark or light background is fine.
  4. If the background of your image is left transparent, you can set the background color in the configuration in the following steps.
  5. Prepare 6 pictures and put them into a folder
  6. Place the folder containing 6 pictures into the folder of config/ESPHome

④ Edit ESP32-S3-BOX-3 configuration file

Find ESP32-S3-BOX-3 in ESPHome web UI and click "EDIT"

編輯 ESP32-S3-BOX-3 設定檔

At this time you will see the yaml editor, like this:

編輯 ESP32-S3-BOX-3 設定檔

If you choose Home Assistant Github repository Material library, place the following code into substitutions In the block:
※ Remember to change the path to the material path you want

YAML
substitutions:
  loading_illustration_file: https://github.com/jlpouffier/home-assistant-s3-box-community-illustrations/raw/main/jarvis_female/illustrations/loading.png
  idle_illustration_file: https://github.com/jlpouffier/home-assistant-s3-box-community-illustrations/raw/main/jarvis_female/illustrations/idle.png
  listening_illustration_file: https://github.com/jlpouffier/home-assistant-s3-box-community-illustrations/raw/main/jarvis_female/illustrations/listening.png
  thinking_illustration_file: https://github.com/jlpouffier/home-assistant-s3-box-community-illustrations/raw/main/jarvis_female/illustrations/thinking.png
  replying_illustration_file: https://github.com/jlpouffier/home-assistant-s3-box-community-illustrations/raw/main/jarvis_female/illustrations/replying.png
  error_illustration_file: https://github.com/jlpouffier/home-assistant-s3-box-community-illustrations/raw/main/jarvis_female/illustrations/error.png

If you are making your own material, place the following code into substitutions block and set the background color of each state (if the image has a transparent background):
※ Change the example voice_assistant_gfx Replace it with the name of your material folder
※ Remember to change the background color of each status (line 8~13 in the example below)

YAML
substitutions:
  loading_illustration_file: voice_assistant_gfx/loading.png
  idle_illustration_file: voice_assistant_gfx/idle.png
  listening_illustration_file: voice_assistant_gfx/listening.png
  thinking_illustration_file: voice_assistant_gfx/thinking.png
  replying_illustration_file: voice_assistant_gfx/replying.png
  error_illustration_file: voice_assistant_gfx/error.png
  loading_illustration_background_color: '000000'
  idle_illustration_background_color: '000000'
  listening_illustration_background_color: 'FFFFFF'
  thinking_illustration_background_color: 'FFFFFF'
  replying_illustration_background_color: 'FFFFFF'
  error_illustration_background_color: '000000'

⑤ Save and install

This step will take different installation times depending on the hardware environment you are using, so please be patient! Once completed, you can see the new display image on the screen! ~

※ I used Raspberry Pi 4 + 8G RAM (Raspberry Pi 4 Model B/8G) and ran it for about 30 minutes

儲存並安裝

I unexpectedly found that this is a perfect match for my desktop🤣

ESP32-S3-BOX-3 桌面搭配

There is still a lot to play with in ESP32-S3-BOX-3, and there is also self-made firmware by foreign masters to give the BOX’s touch screen more functions. I will find time to try it in the future.

I hope today’s article can be helpful to you who have joined Home Assistant in building a localized voice assistant. You are also welcome to leave a message below to interact with me! 👋

8 COMMENTS
  1. Hi Neil,
    After I followed the settings and talked to the ESP32, it didn't wake it up. The ESP32 was always on the standby screen. Do I need to register and pay for HA Cloud to use this voice function?

  2. Hi Neil, please tell me how to solve the problem that printf outputs Chinese characters with black squares. Even after changing several fonts, it still remains the same?

    • Try it yourself :D
      I originally thought that if glyphs were not added, then the entire character set would be included.
      The result must be added
      Sorry to bother you

      – file: “https://github.com/notofonts/noto-cjk/raw/refs/heads/main/google-fonts/NotoSerifTC%5Bwght%5D.ttf”
      id: font_response
      size: 15
      glyphs: '${allowed_characters}'

  3. After using the sensor and completing the settings and firmware burning according to the content of your article, will there be entities for temperature, humidity, and infrared transmission and reception in the HA?

LEAVE A REPLY

Please enter your comment!
Please enter your name here

you might like
recent discussions
  • Neil 2025/05/29Hi Husn, I recommend using Zigbee or Matter for connection! Zigbee accesses HA through Z2M, Matter directly accesses, and WiFi accesses HA through Tuya, which is more troublesome. It is also recommended to find a rectangular one, because generally a small round smart socket inserted into a double socket will compress the space of the other socket.The Ultimate Guide to the Smart Home in 2025! – SMART HOME EP 6
  • Husn 2025/05/29I would like to ask you one more question. I want to use a smart socket to control traditional mechanical home appliances (electric fans). If it is Tuya's small round smart socket, can it be connected to HA? I see that it uses WIFI, or is it better to find one with MATTER? Thank you very much for your guidance.The Ultimate Guide to the Smart Home in 2025! – SMART HOME EP 6
  • Neil 2025/05/24Hi Andy, your understanding is correct. When the switch is turned off, the lamp will be disconnected directly. There are two ways to do this: 1. Replace the switch with a smart switch that supports the "decoupling" function, that is, the button and the relay can be separated, so that the lamp is always powered, and then the switch and lamp are linked through the smart central system. 2. [The switch must have a neutral wire] In the switch box, connect the live wire to the control wire (the control wire does not enter the switch) to change the lamp and switch from series to parallel, so that the lamp is always powered, and then set the switch in the smart central system to link the lamp★ If you are not sure about the wiring, it is recommended to find a professional plumber to handle it!A complete analysis of smart home lighting control solutions: smart switches, smart bulbs, smart relays, there is always one that suits you!
  • Andy 2025/05/24Sorry, I want to ask Neil a very basic question. Currently, we all use HomeKit. The switch has been replaced with a green-silver smart switch. Will there be any problems if the bulb is replaced with a smart bulb? After all, when the switch is turned off, the light should be disconnected directly. Will it show no response in HomeKit? Or can it be avoided by setting it as a group with other accessories? Thank you. I want to keep the position of the physical switch, but also want to control the light color and brightness. I hope you can give me some guidance.A complete analysis of smart home lighting control solutions: smart switches, smart bulbs, smart relays, there is always one that suits you!
  • raja 2025/05/24This article is written in great detail and is very helpful for those who want to get started with smart home! Looking forward to your follow-up tutorials, thank you for sharing!Home Assistant installation steps detailed, quickly build a smart home – step-by-step tutorial EP 1
  • Neil 2025/05/23Hi Husn, looks like there is a chance: https://www.facebook.com/share/p/1UyoRxwx7q/The Ultimate Guide to the Smart Home in 2025! – SMART HOME EP 6
  • Husn 2025/05/23Excuse me, but I have to ask again, is there any chance to achieve dual voice control? Can I use Xiao Ai or OK GOOGLE? In addition, because the Xiao Ai speaker I use has an infrared remote control function, if this part is changed to GOOGLE, is there any way to control it? Thank you very much for your answer!The Ultimate Guide to the Smart Home in 2025! – SMART HOME EP 6
  • Neil 2025/05/23Hi Husn, 1. Voice control: You can use Google Nest Hub for voice control. I am currently using Google Nest Audio. HA settings are in "Settings > Voice Assistant > Google Assistant" 2. Mijia and HA automation conflict: If the same automation is set in both systems, or both systems have automations to operate the same device, there may be conflicts. It is recommended to move all automations to HA management, once and for allThe Ultimate Guide to the Smart Home in 2025! – SMART HOME EP 6
  • Husn 2025/05/23Hello Neil, thank you very much for sharing. I have learned a lot. I would like to ask you a question. I have successfully set up HA with VM on NAS recently. I have also connected Xiaomi products to HA, but I still use Xiaoai for voice control. Is there any way to change this part to Siri or Google home products? For example, Google Nest Hub. I also want to ask, because I used Mijia to set up the automation before, and I still do, so if I switch to HA for setting up, will there be a conflict with Mijia?? Sorry, I have so many questions, thank you for your answers in advanceThe Ultimate Guide to the Smart Home in 2025! – SMART HOME EP 6
  • Neil 2025/05/21Hi Nichole, Home Assistant and Apple Homekit are different systems, so they don’t need to be the same account. If you want to connect a device that has been set up in Homekit to Home Assistant, you can Google to check “Homekit cannot be connected to Home Assistant”Home Assistant installation steps detailed, quickly build a smart home – step-by-step tutorial EP 1
you might like