With rumors suggesting Apple’s Siri voice assistant may get an upgrade at WWDC 2024, I’ve been looking back over our thirteen years with Siri to see what, if anything, has improved. Is Apple’s voice assistant still the best around, or does it really need some dramatic improvements? In a side-by-side test against the best voice assistant software, Siri was not the best, not even close. In fact, the best will probably surprise you.
Siri started life as an app in 2010, but Apple quickly bought the voice assistant and began improving it. Siri was fully integrated into iOS the following year. Back then, we thought of Siri (and Alexa) as tools for checking the weather, setting timers, and asking simple questions, like how much is Oprah worth?
When Siri was launched, it couldn’t control many of the features on your phone, but now it has some ability to press the buttons for you and find hidden settings, if you know what they’re called.
Siri can still calculate the tip at dinner, but now it can also help you find your Apple Watch or where you parked your car. It can turn on the lights, set the thermostat, and remind you to pick up your dry cleaning as you leave the house.
Siri isn’t the only game in town, here comes Gemini and Bixby
Of course, Siri isn’t the only voice assistant in town. Samsung replaced its simpler S Voice assistant with Bixby in 2017. Bixby was introduced as an interface first and a virtual friend and helper second.
Samsung launched Bixby as a voice interface that would control all of your Samsung products, including your phone, your TV, your washing machine, and your refrigerator. It hasn’t given up on that promise, but it’s been slowly developed.
Bixby isn’t exactly an AI as we currently think of them because it doesn’t use a large language model (LLM) to understand your input or create a response. Neither does Siri, as far as we know.
Both of those assistants could get a huge upgrade soon, as Samsung has been touting its Galaxy AI features with every new phone launch, and Apple has already hinted that AI is coming to its products in a big way, perhaps at WWDC 2024.
If you don’t want to wait for an assistant that uses an LLM for its responses, you’ll want a Google Pixel with Gemini on board. Gemini is Google’s LLM, and when you load the app on your Android phone, it offers to replace Google Assistant, Google’s older voice assistant. Gemini still uses Google Assistant to do some of its job, but it also uses other Google apps and features.
The test: 25 requests for my voice assistants
I tested Siri, Bixby, and Gemini – against each other on the best phones from each manufacturer: an Apple iPhone 15 Pro Max, a Samsung Galaxy S24 Ultra, and a Google Pixel 8 Pro.
I made a list of 25 tasks I expected the AI to complete, and I wrote a simple script for each task so that I would phrase my request exactly the same way across them. For instance, when I asked, “is it gonna rain?” to Siri, I did not ask Bixby, “is it going to rain?”
For each test, I gave a ☑️ when the voice assistant gave me a good result, as I expected, and I have details when it gave me a bad result, either inaccurate information or just a wrong response.
Keep reading for a breakdown of how the phones performed, or you can jump right to the test and results.
Siri was in the middle, Bixby was the best
Siri was not the best, but it was not as unreliable as Google’s Gemini. Of the 25 tasks, Siri succeeded at 13 with no trouble, but it gave me a bad answer on seven of my requests.
For instance, when I asked Siri to “Play Archer on Netflix,” it suggested I watch totally different shows on Apple TV Plus. When I asked to “Scan a QR code,” it found information on the web about QR codes, but Gemini and Bixby knew to open the camera.
Worst of all, there was nothing that Siri could do that Gemini could not. Anything Siri could do, Bixby could do just as well, and Bixby wasn’t as fussy about it. If I asked, “turn on the Wi-Fi hotspot,” Siri just shrugged, until I realized Apple calls it a “Personal Hotspot,” and it worked. With Bixby, I could say “Wi-Fi hotspot” or “personal hotspot,” and Bixby turned it on. No matter that Samsung calls it a“mobile hotspot,” Bixby was smart enough to understand whatever I asked.
I asked the phones to “start screen recording,” and only Bixby could do it. The iPhone offered an Apple support web page to learn how to screen record for myself.
I asked the phones to “translate this to Spanish” and Bixby asked which translator I preferred, then started listening. Gemini asked me to type my response, or press the button again. Siri just didn’t understand. Ingles a español: no habla, iPhone friends.
Bixby isn’t just better, it’s much better than Siri
Some of Bixby’s capabilities are truly impressive. I asked the phones to show me photos of my son, using his name. Bixby opened the Gallery app and showed me photos of my kid. Both Siri and Gemini showed me web suggestions for movie characters with that name, even though my son is tagged in Google Photos and Apple Photos.
I asked Bixby to dismiss all of my notifications, and IT WORKED!!! They were all gone! Gemini at least opened my Notifications settings when I asked, but Siri just said, “I’m sorry, I’m afraid I can’t do that.”
In every way, Bixby was the best voice assistant. There was almost nothing it failed at doing in the 25 tasks I threw its way.
It wasn’t always perfect. I asked how many teaspoons are in ¾ of a cup, and it said “approximately 36,” but it’s exactly 36 teaspoons; there’s no need to approximate. I asked for Oprah’s net worth, and it gave me a short bio from the web, but when I clicked through, it had her money numbers inside.
The only failure was when I asked when the Orioles were playing next. As of this writing, they play tonight, but Bixby skipped the whole series they’re playing against the Blue Jays and jumped ahead to the series against the Braves next week. I would have missed the game.
What about Gemini, the assistant that actually uses AI?
Google’s Gemini was the worst of the bunch. It got the fewest answers right and the most answers wrong. I asked the phones to “remind me to get my dry cleaning when I leave here,” and both Bixby and Siri helped with location-based reminders. Google Gemini said, “I can’t assist you with that as I am only a language model.”
Gemini couldn’t help me play “Archer” on Netflix. It couldn’t find my Pixel Watch 2 automatically, even though Siri found my Apple Watch and Bixby found my Galaxy Watch. Gemini couldn’t help me find where I parked my car, even though my Pixel 8 Pro was connected to the stereo. It couldn’t start a screen recording.
When I asked Gemini to restart the phone, it said “Alright” and then did nothing.
The only time Gemini was better was understanding the nickname for the Baltimore Orioles. I started by asking “When is the next O’s game?” as anyone who grew up near Baltimore would. Google’s Gemini was the only voice assistant that could understand my question. Bixby thought I was saying, “Ohs Game,” and gave up. Siri gave me the first Oklahoma State University football game, in three months.
The test and results
| The Request | Gemini (Google Pixel 8 Pro) | Siri (Apple iPhone 15 Pro Max) | Bixby (Samsung Galaxy S24 Ultra) |
| “Call Mom” | ☑️ | ☑️ | ☑️ |
| “Send a text to XXXX saying I’m running late” | ☑️ | ☑️ | ☑️ |
| “Make an appointment for a haircut on Saturday at noon” | ☑️ | ☑️ | ☑️ |
| “Set the thermostat to 72” | ☑️ | ☑️ | ☑️ |
| Turn on/off the flashlight | ☑️ | ☑️ | ☑️ |
| “Who is the CEO of Amazon?” | ☑️ | ☑️ | ☑️ |
| “Remind me to get my dry cleaning when I leave here” | Nope: “No, I can’t assist you with that as I am only a language model” | ☑️ | ☑️ |
| “Is it gonna rain this weekend?” | ☑️ | Nope: “Yes, Saturday” (the forecast said rain Sat and Sun) | ☑️ |
| “When is the next O’s game?” | ☑️➕(understood “O’s” means Baltimore Orioles) | Nope: Oklahoma State Football season opener, August 30 | Nope: Understood Orioles, but gave wrong date |
| “Play my Spotify Discover playlist” | Nope: “Sure, I’ll use YouTube Music” | ☑️ | ☑️ |
| “Play Archer on Netflix” | Nope: “I can’t help with Netflix yet, but I’m still learning” | Nope: suggested “The Informant,” “Traitor,” and “The Dark Crystal” on Apple TV | ☑️ |
| “Set a timer for 15 minutes… How much time is left?” | ☑️ | ☑️ | ☑️ |
| “Where is my watch?” | Nope: “Check the find my app on your phone to locate your watch” | ☑️ | ☑️ |
| “What is 20% of 98 [pause] 75?” | ☑️➕ Understood I meant 98.75 | ☑️ but did not understand the pause means a decimal | ☑️ but did not understand the pause means a decimal |
| “How many teaspoons in 3/4 of a cup?” | ☑️ | ☑️ | ☑️ “approximately” |
| “What is Oprah’s net worth?” | ☑️ and cited sources | ☑️ and cited sources | Nope: Offered a bio, no net worth |
| “Turn on the wi-fi hotspot” | ☑️ | Nope: Did not understand until I said “personal hotspot” | ☑️ |
| “Tell me a joke” | ☑️ | ☑️ | ☑️ |
| “Restart my phone” | Nope: “Alright” then nothing happens | ☑️ | ☑️ |
| “Where did I park?” | Nope: “I don’t have information on where you parked” | ☑️ | ☑️ |
| “Start screen recording” | Nope: “Screen recording is not supported” | Nope: “You can learn how to screen record on Apple.com/…” | ☑️ |
| “Translate into Spanish” | ☑️ | Nope: “Sorry, I don’t understand” | ☑️ |
| “Dismiss all notifications” | Nope: opens Notifications in Settings app | Nope: “I’m sorry, I’m afraid I can’t do that” | ☑️ |
| “Scan a QR code” | ☑️ | Nope: explains QR codes instead | ☑️ |
| “Show me photos of [my son’s name]” | Nope: “There are many depictions of [my son’s name], here are a few…” | Nope: “Here are some images of [my son’s name] from the web” | ☑️ |
| FINAL RESULTS: | ☑️ = 16 | ☑️ = 16 | ☑️ = 23 |
The final analysis: Siri has a lot of work to do
The results are clear: Bixby outperformed Siri and Gemini, proving to be the most capable and versatile voice assistant. While Siri held its own in basic tasks, Bixby’s ability to control phone settings, find personal photos, and understand nuanced language set it apart.
Despite its AI pedigree, Gemini struggled with fundamental functions and lacked the contextual awareness of its competitors. As the voice assistant landscape continues to evolve, hopefully, Apple will advance Siri until it is as useful and capable as Bixby, and maybe Bixby will finally earn some respect outside of Samsung’s Galaxy.
