So, Google showed off Project Astra last summer, right around the same time OpenAI dropped GPT-4o with Vision. While OpenAI’s version is already out there for everyone to use since December, Google just gave us another demo of an improved Project Astra.
Yeah, it’s been a bit of a letdown. But, there’s some good news. Even though we don’t know when Project Astra will be officially available in the Gemini app, you can already play around with something similar using Google AI Studio.
Google recently added a “Stream Realtime” feature to AI Studio. It’s very similar to Project Astra and it’s a great way to see what it can do. Although AI Studio is designed for developers to test APIs, anyone can use its interface for free.
With Stream Realtime, you can share what you see using your phone or computer’s camera, or even share your computer screen, and then chat with Gemini about it.
How to Use Stream Realtime (aka Project Astra in disguise)
-
Go to aistudio.google.com on your computer or phone.
-
Sign in to your Google account.
-
Select “Stream Realtime” from the menu on the left.
-
Once you’re in Stream Realtime, you’ll see some options on the right that you can tweak. This includes options like “Output format” and “Voice.” There are currently 5 voice options available: ‘Puck’, ‘Charon’, ‘Kore’, ‘Fenrir’, and ‘Aoede’, with ‘Puck’ set as the default. Note that you can’t change the model from Gemini 2.0 Flash Experimental.
-
You can also enable some tools like “Code execution”, “Function calling”, “Automatic function response” and “Grounding”.
-
After you’ve set everything up, click on “Show Gemini” to use your camera feed, or “Share your screen” to share your computer screen. The screen sharing option isn’t available on mobile devices.
-
When I used my PC, I opted to share my screen with Gemini. I initially experienced some problems with Gemini not responding. However, a quick refresh solved this issue and it started working perfectly. You can decide to share a browser tab, an application or your entire screen with Gemini.
-
After your screen is visible, start a conversation with Gemini about its contents. To stop sharing your screen, simply click on “Stop Sharing” at the bottom of the screen.
-
To completely end the session, click the “camera” icon in the chat area to stop the stream.
-
Once the session is over, you will find video recordings, audio recordings, and transcripts of Gemini’s responses saved in the chat history.
You can also share your camera feed and talk to Gemini about it in the same way.
Things you should know:
- Gemini is great at identifying items on your screen and answering questions about them.
- Gemini can only see the portion of the app/webpage currently visible on your screen. It cannot see content until you scroll and make it visible.
- It does not have internet access in AI Studio and its knowledge is limited to its training data cutoff date, which is August 2024.