Odtnhj

Sorry if this has been answered before, but I'm having trouble with the solution.

I have a 2D DataFrame with column names, where the elements contain both non-null and null values. I'd like to 'flatten' the 2D DataFrame to a 1D Series, where I preserve only the non-null data as a list in the series cell with the corresponding header.

ie: the following:

Going from (type pandas.Dataframe):

| asset | name | id |
---------------------
| a | john | 001|
| a | NaN | 002|
| NaN | dave | 003|

To (type pandas.Series):

| asset | name | id |
------------------------------------------
| [a] | [john, dave] | [001, 002, 003] |

Thank you!

EDIT: Why I would need this:

I am starting with a large DataFrame that has multiple duplicated attributes with timestamped 'rows'. At any given timestamp, the information in the rows could be added to, to deleted. I have used df.where() to return a dataframe of the unique values, and am attempting to flatten it down to one attribute collection of 'ids' per row.

In practice, the example table is from a a single GroupBy object.

Instantiate a new series using a dict comprehension (this should be faster than an apply based solution).

pd.Series(c : df[c].dropna().unique().tolist() for c in df.columns)

asset [a]
name [john, dave]
id [1, 2, 3]
dtype: object

If you want a single rowed DataFrame instead, use

pd.Series(
 c : df[c].dropna().unique().tolist() for c in df.columns
).to_frame().T

 asset name id
0 [a] [john, dave] [1, 2, 3]