A tiny tot riding a hot pink bicycle with training wheels follows a ball out into a busy intersection.
Will the autonomous vehicle driving at a high rate of speed and headed in the same direction accurately identify the girl in real time and quickly slow to a stop?
The miniature-sized scenario — featuring a Barbie doll and a one-tenth scale autonomous vehicle — is one that’s playing out in UNLV’s Real Time Intelligent Systems (RTIS) Laboratory.
“We’re teaching students to design autonomous vehicles to follow the rules of the road,” said Brendan Morris, a professor of electrical and computer engineering (ECE) who heads the lab. “The car needs to know when it’s on a highway versus a residential street. The car needs to be able to stop at crosswalks, stop for other vehicles and pedestrians, and recognize the different phases of traffic lights.”
An interdisciplinary team, including students from ECE, mechanical engineering, and computer science, successfully programmed a car to complete these and other tasks on a demo track in preparation for an international competition earlier this year. While Team Street Precision didn’t make it to the final round of the Bosch Future Mobility Challenge, the extra curricular activity is an example of the real-world work happening inside RTIS. And, after receiving a special invitation from challenge organizers, the team plans to try their car out for next year’s competition.
“My goal was to explore as much as possible and learn how hardware integrates with software,” said Vansh Sehrawat, team captain and computer science major. “[The competition] is very timely and immediately applicable for real-life, hands-on experience with a car that we can program to drive autonomously while navigating complex, smart-city scenarios.”
The Real Time Intelligent Systems Lab
How we navigate the world — and how computers help us to do that — is the driving force behind the research in the RTIS Lab.
“Our focus is on machine learning, artificial intelligence, and computer vision, which is using cameras to try to understand the world, and understand activities,” Morris said. “Intelligent systems are those that are able to observe the world, learn from these observations, and understand the environment.”
Connected automated vehicles is one line of research, where UNLV students are equipping a full-size vehicle with sensors, cameras, and LiDAR technology to traverse a loop around campus. The vehicle also will communicate with sensors on nearby infrastructure and intersections using vehicle-to-everything (V2X) technology. Morris worked with the office of risk management, insurance, and claims on project parameters including restrictions to only drive at low speed on local roads around campus — no highway automated driving — and the vehicle always has an emergency driver in the driver’s seat to take over control manually. Additionally, the car employs an emergency kill switch to disengage operation.
“The idea is to have all of this infrastructure talking to each other to improve vehicle operations and to provide a fine-grained understanding of how people and cars are moving,” Morris said. “We want to create a safe, connected, and fully integrated area around campus.”
The Lincoln MKZ is currently able to demo a small circuit from the Science and Engineering Building parking lot to the Rebel Recycling Center, but the team has bigger plans for the car, including making a full loop around campus.
But to be able to get to that point, students have to test drive the car — and potentially dangerous scenarios — in a simulated world. For one group of students, that training begins with the Barbie car. But in a separate area of the RTIS lab, across the way from the students working on the one-tenth scale car, Carlos Funes sits in a full-size racing simulator chair and refines the digital twin of UNLV’s campus.
“If we have a nice reproduction of UNLV, then we can develop algorithms which we can test in the simulation first to ensure that it works the way we expect,” said Funes, who graduated with a bachelor’s in electrical engineering in May and is now pursuing a master’s. “And then once we're satisfied with that, then we can go out into the real world.”
“My work focuses on optimizing driving algorithms to mimic human behavior and improve vehicle precision, making our roads safer and smarter,” Funes added.
Steering Smart Students on the Right Trajectory
In addition to making cars and cities smarter, Morris keeps student growth central to the mission of the lab.
“The hope is that by having this program here, students can get involved and learn about and be trained to work on these autonomous vehicle systems, so that they can go work for the companies that are deploying these technologies,” said Morris, adding that one of his graduates recently landed a position at Waymo.
His mentorship extends to students beyond UNLV. Over the past six summers, more than 60 students from across the nation and UNLV have been introduced to the rigors of research as part of the NSF-funded Research Experiences for Undergraduates (REU) Smart Cities program. Students explore problems in intelligent transportation systems, automated vehicles, and V2X communication.
Morris has mentored students who have been tasked with using computer vision to track pedestrians and vehicles as they move through an intersection. The data can help inform traffic engineers of problems at intersections — like an overly congested crosswalk — and predict time-to-collision.
"Traffic operators are focused on catching congestion and accidents early," Morris explained. "When we spot problems quickly, we can get the right people and resources there faster. That cuts down on delays significantly, and it's safer too because it gives emergency crews those extra crucial minutes they need.”
Democratizing Training with Computer Vision
While Morris has applied his deep expertise in computer vision to address these and other transportation-related challenges over the past 15 years, his research trajectory has expanded to explore other mobility-related questions: Can computer vision help physical therapists, golf instructors, psychologists, surgeons, and more, monitor a person’s movements to understand how well they are able to perform a task?
“It’s called action quality assessment,” Morris said. “We’re exploring how camera-based systems can observe and understand what’s happening and give a quality score. This is kind of where my new passion is.”
Instead of using human judges to, let’s say, score a figure skating champion at the Olympics, a robot judge could be brought in. But it goes beyond simply knowing that you did a task well.
“What we really want to do is to be able to provide feedback on how to improve human performance on these tasks,” Morris said.
Most recently, his team began partnering with UNLV’s PGA Golf Management Program to provide real-time feedback to golfers, and they are looking to apply their work to other sports.
“You could have something where you set up your cell phone, you do a dive, or hit a golf ball, and then you get feedback,” Morris. “Training for some sports is really expensive and prohibitive. This kind of technology could help people get closer to the top levels where you have access to more resources and facilities. You could democratize training.”
Democratizing training and making research accessible is the philosophy present in all of Morris’ interactions with students, even high schoolers. One high school student brought his interest in piano to Morris’ research during a summer internship. The research question that surfaced: Can we trust our ears over our eyes when analyzing a piano player’s performance?
“Instead of just looking at video content, we introduced an audio component to the research,” Morris said. “He brought that personal experience into this AQA world — which was not something that we were thinking about — and it ended up actually with a paper that got published.”
As someone who struggled to get comfortable with research as an undergraduate, Morris knows firsthand the value of opening the door to experiential learning.
“Great ideas can come from all levels,” Morris said.
