Previously on chatCEO s24e02:local LLM for Smart Offices w/ HA
In our Monthly chatCEO Live, we’ll be starting from the challenges present in current technology, and then share our optimized solutions.Each livestream we will focus on a specific tech problem. If you have any improvement ideas, feel free to share, and we’ll work together to turn them into reality through our technology.·
In our latest livestream, we delved into the fascinating world of Local LLM (Large Language Models) technology, exploring its applications in creating smarter offices integrated with Home Assistant (HA). Let’s take a closer look at the highlights of the session and how this technology has evolved to revolutionize workplace environments.
1. Understanding the scene with Jetson!
Problem: How can we leverage AI technology to enhance the identification of spatial scenes?
Local Live LLaVa: We reproduced Dustin Franklin’s demo called Local Live LLaVa. Local Live LLaVa is an innovative solution derived from the LLaVa multimodal vision-language model, quantized with 4-bit precision and deployed on Jetson Orin for triggering user alerts. At its core, LLaVa seamlessly integrates the visual capabilities of CLIP with language embeddings from models like GPT-4, enabling AI to comprehensively understand and process multimodal inputs involving text and images, facilitating a deeper understanding of complex instructions.
By processing data locally on edge devices, LLaVa mitigates privacy concerns by ensuring sensitive information remains on-premises, enhancing privacy and security. In the livestream,we used it to capture scenes in our office and introduce the staff behind our live.
Related Links:
1.NVIDIA Jetson Generative AI Lab
2.AI NVR
2. Ongoing office mods
Problem: There are many inconveniences in traditional offices, such as waste of electricity, excessive carbon dioxide in the office, and difficulty in making appointments for meeting rooms.
Smart offices: In order to provide a better environment for colleagues in seeed studio, we have done several smart office projects together with Home Assistant. During the live streaming, five of our colleagues introduced different home assistant projects, such as light controling,environmental monitoring,smart meeting room and smart air-conditioner. What impressed me most is that it can monitor the carbon dioxide concentration in the office. When the concentration is too high, an alarm can be issued and we can make timely adjustments. This can provide a better office environment for everyone.
We will continue to optimize these projects in the future. If you want to know more about Home Assistant, check out here: Seeed’s Home Assistant Wiki
Seeed Studio offers open-source devices tailored for various industries. These devices are widely used in agriculture, smart cities, and the security sector. We have specially launched a solution web page, where you can explore more about industry cases and solutions
Another interesting thing we have tryed is 24-hour livestream. Stream Seeed Studio’s product performance and sensor data in real-time, fit for any environment, from outdoors and indoors to greenhouses.
3. local LLM (trying) to rule them all
Problem: Traditional interfaces for smart home devices have historically lacked natural language processing (NLP) capabilities, limiting user experience and functionality. Physical buttons, switches, and basic voice commands required users to adapt to technology constraints rather than vice versa. This lack of intuitive interaction made it difficult for users to fully utilize smart home devices, reducing accessibility and task efficiency.
local LLM: This project takes off with the Application Engineers at Seeed Studio embarking on a journey to leverage the advanced capabilities of the Jetson Orin AGX. Our mission is to deploy it for the operation of a LLM embedded with a Langchain agent. The objective is to evolve our office environment into a cutting-edge, GenAI-infused AIoT workspace, further enhanced by the integration of automation with the Home Assistant project. The goal is to establish an open-source framework based on RAG for building a local LLM that interacts with Home Assistant API. This endeavor seeks not only to elevate the workplace atmosphere but also to decrease energy consumption, aligning with Sustainable Development Goals (SDG).
This comprehensive project is divided into several crucial segments, each indispensable for the effective real-world deployment of this groundbreaking upgrade.
Here is the flowchart of the whole system:
And here is the picture of Local Jarvis.
Up coming LIVE session:
Next Live will be on March 29th about “TinyML and Local LLM”, We will share with you the demo we will exhibit at GTC.At the same time, we will give you an advance notice of our new product: indoor combo!
Stay tuned for more exciting updates and events from Seeed Studio. Remember to like, comment, and subscribe to stay connected with us. See you next time!
About Author
FNF is the best music game I’ve ever played!
I’m really impressed by the demonstration of Local Live LLaVa and the innovative use of the LLaVa multimodal vision-language model.