BUG: Add fillna at the beginning of _where not to fill NA. #60729

[sanggon6107]

• closes BUG: Series.mask incorrectly replaces positions of pd.NA in the cond argument #60729 (Replace xxxx with the GitHub issue number)
• Tests added and passed if fixing a bug or adding a new feature
• All code checks passed.
• Added type annotations to new arguments/methods/functions.
• Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

Added fillna at the beginning of _where so that we can fill pd.NA.
Since this is my first PR, please correct me if I'm mistaken. Thanks!

[WillAyd]

Is this trying to fill missing values when NaN is the missing value indicator? I don't think that is right either - the missing values should propogate for all types. We may just be missing coverage for the NaN case (which should be added to the test)

[sanggon6107]

Thanks for the feedback, @WillAyd .
I thought we could make the values propagate by filling cond with True, since _where() would finally keep the values in self alive where its cond is True.
Even if I don't fill those values here, _where would call fillna() using inplace at the below code. That's also why the result varies depending on whether inpalce=True or not.

pandas/pandas/core/generic.py

Lines 9695 to 9698 in e3b2de8

	# make sure we are boolean
	fill_value = bool(inplace)
	cond = cond.fillna(fill_value)
	cond = cond.infer_objects()

Could you explain in more detail what you mean by propagate for all type? Do you mean we need to keep NA as it is even after this line?

[sanggon6107]

Hi @WillAyd,

I've done some further investigations on this, but I still belive the current code is the simplest way to make the missing values propagate.
If we want to let NA propagate without calling fillna() here, there might be too many code changes needed. See below codes :

1. Need to change the below code so that we don't fill the missing values when caller is where() or mask(). If we don't, fillna() will fill them with inplace.

pandas/pandas/core/generic.py

Lines 9695 to 9698 in f1441b2

	# make sure we are boolean
	fill_value = bool(inplace)
	cond = cond.fillna(fill_value)
	cond = cond.infer_objects()

2. Need to change the below code as well since to_numpy() will fill the missing value using inplace when cond is a DataFrame.

pandas/pandas/core/generic.py

Lines 9703 to 9716 in f1441b2

	if not isinstance(cond, ABCDataFrame):
	# This is a single-dimensional object.
	if not is_bool_dtype(cond):
	raise TypeError(msg.format(dtype=cond.dtype))
	else:
	for _dt in cond.dtypes:
	if not is_bool_dtype(_dt):
	raise TypeError(msg.format(dtype=_dt))
	if cond._mgr.any_extension_types:
	# GH51574: avoid object ndarray conversion later on
	cond = cond._constructor(
	cond.to_numpy(dtype=bool, na_value=fill_value),
	**cond._construct_axes_dict(),
	)

3. Since extract_bool_array() fills the missing values using arg na_value=False at EABackedBlock.where(), we might need to find every single NA index from cond before we call this function(using isna() for example) and then implement additional behaviour to make those values propagate at ExtensionArray._where().

pandas/pandas/core/internals/blocks.py

Lines 1664 to 1668 in f1441b2

	def where(self, other, cond) -> list[Block]:
	arr = self.values.T
	cond = extract_bool_array(cond)

pandas/pandas/core/array_algos/putmask.py

Lines 116 to 127 in f1441b2

	def extract_bool_array(mask: ArrayLike) -> npt.NDArray[np.bool_]:
	"""
	If we have a SparseArray or BooleanArray, convert it to ndarray[bool].
	"""
	if isinstance(mask, ExtensionArray):
	# We could have BooleanArray, Sparse[bool], ...
	# Except for BooleanArray, this is equivalent to just
	# np.asarray(mask, dtype=bool)
	mask = mask.to_numpy(dtype=bool, na_value=False)
	mask = np.asarray(mask, dtype=bool)
	return mask

If _where() is trying to fill the missing values for cond anyway, I think we don't necessarily have to disfavour the current code change. Could you give me some feedback?

[sanggon6107]

FYI, it seems this has already been discussed at #53124 (comment)