analysis.md

Visualization and Qualitatively Analysis of Baseline Models

We provide some visualization videos and qualitatively analysis for baseline methods on Bench2Drive.For TCP, UniAD and VAD, we choose the best version of these models(TCP-traj,UniAD-Base,VAD-Base), and make visualization on 10 Scenarios as below.

Note: It might take some time to load all gifs. Please be patient.

Driving Skill	Scenario Name	Route ID	Success
			AD-MLP	TCP-traj	UniAD-Base	VAD-Base
Merging	MergerIntoSlowTraffic	2283	x	√	√	x
	SignalizedJunctionLeftTurn	4183	x	√	x	√
Overtaking	ParkedObstacle	25318	x	x	x	√
	HazardAtSideLane	25439	x	x	√	√
Emergency Brake	ParkingCutIn	18305	x	√	√	x
	StaticCutIn	26396	x	x	√	√
Give Way	YieldToEmergencyVehicle	3378	x	x	x	x
	InvadingTurn	2802	x	√	√	x
Traffic Sign	EnterActorFlow	3749	x	√	x	x
	VanillaNonSignalizedTurnEncounterStopsign	3905	x	√	x	x

Note: In this visualization, the definition of 'success' differs from the standard definition used in the Bench2Drive. A route may involve multiple actions, such as turning after passing through a traffic light. In this visualization, we only evaluate whether the selected segment's action is successful. For example, if the vehicle obeys the traffic light and passes through the intersection, but collides while turning, it is still considered a successful case in following traffic sign.

Merging

we visualize the behavior of three models on MergeIntoSlowTraffic and SignalizedJunctionLeftTurn scenarios to show their ability of mering . The ego vehicle should drive to off-ramp to exit the highway in MergeIntoSlowTraffic, and it should perform a left turn in SignalizedJunctionLeftTurn.

Case ID	Models	Scenario	Front Camera	BEV	Success	Qualitatively Analysis
1	TCP-traj	MergeInto SlowTraffic			√	The ego vehicle carefully changes the lane and exits highway successfully, but at a quite low speed (the video is at x2 speed).
2	UniAD-Base	MergeInto SlowTraffic			√	The ego vehicle makes a lane change at a high speed, exits highway successfully.
3	VAD-Base	MergeInto SlowTraffic			x	The ego vehicle detects the car at front right, but still drives too fast and crashes into it.
4	TCP-traj	Signalized Junction LeftTurn			√	The ego vehicle predicts appropriate trajectory and turns left at junction successfully.
5	UniAD-Base	Signalized Junction LeftTurn			x	The ego vehicle fails to detect the car coming in opposite direction and continues to drive ahead, results in crash.
6	VAD-Base	Signalized Junction LeftTurn			√	The ego vehicle detects and predicts motions of nearby vehicles, and makes left turn smoothly.

Overtaking

We visualize the behavior of three models on ParkedObstacle and HazardAtSideLane scenarios to show their ability of overtaking. The ego vehicle encounters a parked vehicle blocking part of the lane in ParkedObstacle, and encounters a slow-moving hazard blocking part of the lane in HazardAtSideLane.It should perform a lane change to avoid it.

Case ID	Models	Scenario	Front Camera	BEV	Success	Qualitatively Analysis
7	TCP-traj	ParkedObstacle			x	The ego vehicle tries to avoid the parked vehicle, but because of inaccurate and instabe planing, it still collides against it.
8	UniAD-Base	ParkedObstacle			x	The ego vehicle detects the parked car, but predicts wrong trajectory, and collides against it.
9	VAD-Base	ParkedObstacle			√	The ego vehicle detects the parked car and turns left to avoid the car and cars coming behind.
10	TCP-traj	HazardAtSideLane			x	The ego vehicle's behavior is confusing. It drives onto sidewalk and collides against cyclist.(the video is at x2 speed)
11	UniAD-Base	HazardAtSideLane			√	The ego vehicle detects cyclists and makes a left lane change smoothly.
12	VAD-Base	HazardAtSideLane			√	The ego vehicle detects cyclists, slows down and follows them for a while, and then overtakes them through the leftside of the lane without changing lane.

Emergency Brake

We visualize the behavior of three models on ParkingCutIn and StaticCutIn scenarios to show their ability of emergency brake. In these scenarios, the ego must slow down or brake to allow a vehicle cut in.

Case ID	Models	Scenario	Front Camera	BEV	Success	Qualitatively Analysis
13	TCP-traj	ParkingCutIn			√	The ego vehicle brakes and waits the parked vehicle to exit.
14	UniAD-Base	ParkingCutIn			√	The ego vehicle detects the parked vehicle, predicts its motion correctly, so the ego vehicle brakes and waits the parked vehicle to exit.
15	VAD-Base	ParkingCutIn			x	The ego vehicle detects the parked car but not stops, and leads to a series of collisions.
16	TCP-traj	StaticCutIn			x	The ego vehicle drives slowly. When the vehicle at front right attempt to change the lane, ego vehicle turns left unnecessaryly, which occurs collision.
17	UniAD-Base	StaticCutIn			√	The ego vehicle brakes when other vehicle cuts in.
18	VAD-Base	StaticCutIn			√	The ego vehicle stops at several positions that other vehicles may cut in

Give Way

We visualize the behavior of three models on YieldToEmergencyVehicle and InvadingTurn scenarios to show their ability of giving way. In YieldToEmergencyVehicle, ego must maneuver to allow the emergency vehicle behind to pass. In InvadingTurn,a vehicle coming from the opposite lane invades the ego’s lane, forcing the ego to move right to avoid a possible collision.

Case ID	Models	Scenario	Front/Back Camera	BEV	Success	Qualitatively Analysis
19	TCP-traj	YieldTo Emergency Vehicle			x	TCP model does not use back cameras, so ego can not detect the emergency vehicle behind.
20	UniAD-Base	YieldTo Emergency Vehicle			x	The ego vehicle fails to detect the emergency vehicle, and does not giveway.
21	VAD-Base	YieldTo Emergency Vehicle			x	The ego vehicle detects the emergency vehicle and tries to move right to giveway, but collides against vehicle on right lane.
22	TCP-traj	InvadingTurn			√	The ego vehicle drives slowly and moves right to avoid collision.
23	UniAD-Base	InvadingTurn			√	The ego vehicle drives in a normal spped and moves right to avoid collision.
24	VAD-Base	InvadingTurn			x	The ego vehicle moves too much, it invades the right lane and collides against other vehicle.

Traffic Sign

We visualize the behavior of three models on EnterActorFlow and VanillaNonSignalizedTurnEncounterStopsign scenarios to show their ability of following traffic sign. In EnterActorFlow, ego should follow the traffic light. In VanillaNonSignalizedTurnEncounterStopsign, ego should stop and start at stop signs.

Case ID	Models	Scenario	Front Camera	BEV	Success	Qualitatively Analysis
25	TCP-traj	EnterActorFlow			√	The ego vehicle follows the traffic light and goes through the junction.
26	UniAD-Base	EnterActorFlow			x	The ego vehicle detects the traffic light but runs a red light.
27	VAD-Base	EnterActorFlow			x	The ego vehicle does not detects the traffic light accurately，and runs a red light.
28	TCP-traj	Vanilla NonSignalized TurnEncounter Stopsign			√	The ego vehicle stops and waits at stop sign.After the opposite vehicle passes the junction, ego goes through the junction.
29	UniAD-Base	Vanilla NonSignalized TurnEncounter Stopsign			x	The ego vehicle does not stop and runs a stop.
30	VAD-Base	Vanilla NonSignalized TurnEncounter Stopsign			x	The ego vehicle stops at the sign but gets blocked, does not start again.

Conclusion

• Strategies: The three E2E-AD models implement distinct strategies. The TCP model adopts a conservative approach, favoring slower speeds to enhance obstacle avoidance and response to unexpected events, potentially at the cost of traffic flow efficiency. Conversely, the VAD model employs a more aggressive strategy, increasing the risk of excessive speed and abrupt maneuvers, which could lead to collisions. The strategy of UniAD lies between these extremes.

• Diverse Behaviors in Identical Scenarios: It is notable that in certain scenarios, such as cases 10, 11, and 12, different models trained on the same dataset exhibit varied behaviors, with more than one approach being reasonable, as seen in cases 11 and 12.

• Perception and Planning： The interplay between perception and planning is critical. A model’s failure to detect nearby vehicles often leads to incorrect routing, as demonstrated in case 5. However, even precise object detection does not always ensure correct planning decisions, as illustrated in case 10.

• Failure Cases nalysis: Several factors contribute to the failure cases observed. These include the inability to perceive other vehicles (cases 5 and 20), inaccurate motion predictions (case 8), actions that are either too mild (case 7) or too forceful (case 24), and misinterpretations of the scene (cases 29 and 30).