All Questions
4,238
questions
1350
votes
25
answers
2.2m
views
Get a list from Pandas DataFrame column headers
I want to get a list of the column headers from a Pandas DataFrame. The DataFrame will come from user input, so I won't know how many columns there will be or what they will be called.
For example, ...
- 24.7k
580
votes
16
answers
604k
views
How to group dataframe rows into list in pandas groupby
Given a dataframe, I want to groupby the first column and get second column as lists in rows, so that a dataframe like:
a b
A 1
A 2
B 5
B 5
B 4
C 6
becomes
A [1,2]
B [5,5,4]
C [6]
How do I do this?
- 16.7k
569
votes
11
answers
1.8m
views
Get list from pandas dataframe column or row?
I have a dataframe df imported from an Excel document like this:
cluster load_date budget actual fixed_price
A 1/1/2014 1000 4000 Y
A 2/1/2014 12000 10000 Y
A 3/1/2014 ...
- 20.7k
434
votes
10
answers
1.2m
views
Creating a Pandas DataFrame from a Numpy array: How do I specify the index column and column headers?
I have a Numpy array consisting of a list of lists, representing a two-dimensional array with row labels and column names as shown below:
data = np.array([['','Col1','Col2'],['Row1',1,2],['Row2',3,4]])...
- 5,405
407
votes
6
answers
307k
views
Pandas DataFrame to List of Dictionaries
I have the following DataFrame:
customer item1 item2 item3
1 apple milk tomato
2 water orange potato
3 juice mango chips
which I want ...
- 5,295
380
votes
13
answers
463k
views
Split a Pandas column of lists into multiple columns
I have a Pandas DataFrame with one column:
import pandas as pd
df = pd.DataFrame({"teams": [["SF", "NYG"] for _ in range(7)]})
teams
0 [SF, NYG]
1 [SF, NYG]
2 ...
- 4,717
257
votes
10
answers
284k
views
Pandas column of lists, create a row for each list element
I have a dataframe where some cells contain lists of multiple values. Rather than storing multiple
values in a cell, I'd like to expand the dataframe so that each item in the list gets its own row (...
- 59.2k
176
votes
9
answers
255k
views
Python pandas insert list into a cell
I have a list 'abc' and a dataframe 'df':
abc = ['foo', 'bar']
df =
A B
0 12 NaN
1 23 NaN
I want to insert the list into cell 1B, so I want this result:
A B
0 12 NaN
1 23 ['foo', '...
- 9,309
164
votes
7
answers
527k
views
Convert List to Pandas Dataframe Column
I need to convert my list into a one-column pandas dataframe.
Current List (len=3):
['Thanks You',
'Its fine no problem',
'Are you sure']
Required Pandas DF (shape =3,):
0 Thank You
1 Its fine no ...
- 2,323
151
votes
9
answers
141k
views
Pandas DataFrame stored list as string: How to convert back to list
I have an n-by-m Pandas DataFrame df defined as follows. (I know this is not the best way to do it. It makes sense for what I'm trying to do in my actual code, but that would be TMI for this post so ...
- 6,489
132
votes
2
answers
363k
views
Get row-index values of Pandas DataFrame as list? [duplicate]
I'm probably using poor search terms when trying to find this answer. Right now, before indexing a DataFrame, I'm getting a list of values in a column this way...
list = list(df['column'])
...then ...
- 21.1k
117
votes
3
answers
120k
views
How to determine the length of lists in a pandas dataframe column
How can the length of the lists in the column be determine without iteration?
I have a dataframe like this:
CreationDate
2013-12-22 15:25:02 ...
- 16.8k
112
votes
7
answers
239k
views
Python: create a pandas data frame from a list
I am using the following code to create a data frame from a list:
test_list = ['a','b','c','d']
df_test = pd.DataFrame.from_records(test_list, columns=['my_letters'])
df_test
The above code works ...
- 24.6k
99
votes
10
answers
401k
views
Find empty or NaN entry in Pandas Dataframe
I am trying to search through a Pandas Dataframe to find where it has a missing entry or a NaN entry.
Here is a dataframe that I am working with:
cl_id a c d e ...
- 12.1k
96
votes
4
answers
222k
views
Merge multiple column values into one column in python pandas
I have a pandas data frame like this:
Column1 Column2 Column3 Column4 Column5
0 a 1 2 3 4
1 a 3 4 5
2 b 6 7 8
...
- 5,235
90
votes
1
answer
108k
views
List of Tuples to DataFrame Conversion [duplicate]
I have a list of tuples similar to the below:
[(date1, ticker1, value1),(date1, ticker1, value2),(date1, ticker1, value3)]
I want to convert this to a DataFrame with index=date1, columns=ticker1, ...
- 917
81
votes
5
answers
168k
views
python pandas flatten a dataframe to a list
I have a df like so:
import pandas
a=[['1/2/2014', 'a', '6', 'z1'],
['1/2/2014', 'a', '3', 'z1'],
['1/3/2014', 'c', '1', 'x3'],
]
df = pandas.DataFrame.from_records(a[1:],columns=a[0])
I ...
- 4,093
71
votes
3
answers
120k
views
Pandas - combine column values into a list in a new column
I have a Python Pandas dataframe df:
d = [['hello', 1, 'GOOD', 'long.kw'],
[1.2, 'chipotle', np.nan, 'bingo'],
['various', np.nan, 3000, 123.456]]
t = pd.DataFrame(data=d, columns=['A','B','...
- 2,903
68
votes
4
answers
150k
views
Convert Python list to pandas Series
What is the method to convert a Python list of strings to a pd.Series object?
(pandas Series objects can be converted to list using tolist() method--but how to do the reverse conversion?)
- 3,998
64
votes
6
answers
83k
views
How to check if a variable is either a python list, numpy array or pandas series
I have a function that takes in a variable that would work if it is any of the following three types
1. pandas Series
2. numpy array (ndarray)
3. python list
Any other type should be rejected. ...
- 4,890
63
votes
3
answers
64k
views
Pandas expand rows from list data available in column
I have a data frame like this in pandas:
column1 column2
[a,b,c] 1
[d,e,f] 2
[g,h,i] 3
Expected output:
column1 column2
a 1
b 1
c ...
- 769
61
votes
6
answers
142k
views
Selecting columns by list (and columns are subset of list)
I'm selecting several columns of a dataframe, by a list of the column names. This works fine if all elements of the list are in the dataframe.
But if some elements of the list are not in the DataFrame,...
- 645
61
votes
2
answers
164k
views
How to convert list to row dataframe with Pandas
I have a list of items like this:
A = ['1', 'd', 'p', 'bab', '']
My goal is to convert such list into a dataframe of 1 row and 5 columns. If I type pd.DataFrame(A) I get 5 rows and 1 column. What ...
- 5,800
56
votes
4
answers
59k
views
Pandas drop_duplicates method not working on dataframe containing lists
I am trying to use drop_duplicates method on my dataframe, but I am getting an
error. See the following:
error: TypeError: unhashable type: 'list'
The code I am using:
df = db.drop_duplicates()
...
- 627
55
votes
2
answers
77k
views
Merge a list of pandas dataframes
I have a list of data frames and I need to merge them together using a unique column (date). Field names are different so concat is out.
I can manually use df[0].merge(df[1],on='Date').merge(df[3],on='...
- 2,672
50
votes
7
answers
171k
views
Python pandas apply function if a column value is not NULL
I have a dataframe (in Python 2.7, pandas 0.15.0):
df=
A B C
0 NaN 11 NaN
1 two NaN ['foo', 'bar']
2 three 33 NaN
I want to apply a ...
- 9,309
47
votes
10
answers
55k
views
Pandas Series of lists to one series
I have a Pandas Series of lists of strings:
0 [slim, waist, man]
1 [slim, waistline]
2 [santa]
As you can ...
- 847
47
votes
6
answers
155k
views
Column of lists, convert list to string as a new column
I have a dataframe with a column of lists which can be created with:
import pandas as pd
lists={1:[[1,2,12,6,'ABC']],2:[[1000,4,'z','a']]}
#create test dataframe
df=pd.DataFrame.from_dict(lists,orient=...
- 2,903
45
votes
5
answers
52k
views
Remove rows with empty lists from pandas data frame
I have a data frame with some columns with empty lists and others with lists of strings:
donation_orgs donation_context
0 [] ...
- 677
40
votes
4
answers
141k
views
How to save a list as a CSV file with new lines?
I would like to save a Python list in a CSV file, for example I have a list like this:
['hello','how','are','you']
I would like to save it as:
colummn,
hello,
how,
are,
you,
I tried the following:
...
- 909
39
votes
2
answers
32k
views
"unstack" a pandas column containing lists into multiple rows [duplicate]
Say I have the following Pandas Dataframe:
df = pd.DataFrame({"a" : [1,2,3], "b" : [[1,2],[2,3,4],[5]]})
a b
0 1 [1, 2]
1 2 [2, 3, 4]
2 3 [5]
How would I "unstack" the ...
- 4,166
35
votes
2
answers
108k
views
Using map() for columns in a pandas dataframe
I have some columns in my dataframe for which I just want to keep the date part and remove the time part. I have made a list of these columns:
list_of_cols_to_change = ['col1','col2','col3','col4']
...
- 2,099
32
votes
3
answers
54k
views
set list as value in a column of a pandas dataframe
Let's say I have a dataframe df and I would like to create a new column filled with 0, I use:
df['new_col'] = 0
This far, no problem. But if the value I want to use is a list, it doesn't work:
df['...
- 3,845
27
votes
5
answers
62k
views
Convert a columns of string to list in pandas
I have a problem with the type of one of my column in a pandas dataframe. Basically the column is saved in a csv file as a string, and I wanna use it as a tuple to be able to convert it in a list of ...
- 1,263
27
votes
2
answers
68k
views
Nested List to Pandas Dataframe with headers
Basically I am trying to do the opposite of How to generate a list from a pandas DataFrame with the column name and column values?
To borrow that example, I want to go from the form:
data = [
['...
- 2,036
26
votes
8
answers
25k
views
Co-occurrence matrix from nested list of words
I have a list of names like:
names = ['A', 'B', 'C', 'D']
and a list of documents, that in each documents some of these names are mentioned.
document =[['A', 'B'], ['C', 'B', 'K'],['A', 'B', 'C', '...
- 1,070
26
votes
2
answers
57k
views
How do I create pandas DataFrame (with index or multiindex) from list of namedtuple instances?
Simple example:
from collections import namedtuple
import pandas
Price = namedtuple('Price', 'ticker date price')
a = Price('GE', '2010-01-01', 30.00)
b = Price('GE', '2010-01-02', 31.00)
l = [a, b]
...
- 4,808
25
votes
3
answers
41k
views
Read lists into columns of pandas DataFrame
I want to load lists into columns of a pandas DataFrame but cannot seem to do this simply. This is an example of what I want using transpose() but I would think that is unnecessary:
In [1]: import ...
- 17.8k
25
votes
2
answers
40k
views
How to multiply each row in pandas dataframe by a different value
I am trying to multiply each row of a pandas dataframe by a different value and wondering what the best way to do this is.
For example if I have the following dataframe:
import numpy as np
import ...
- 13.4k
24
votes
2
answers
45k
views
Splitting a list in a Pandas cell into multiple columns [duplicate]
I have a really simple Pandas dataframe where each cell contains a list. I'd like to split each element of the list into it's own column. I can do that by exporting the values and then creating a new ...
- 9,003
24
votes
3
answers
4k
views
How do I turn a dataframe into a series of lists?
I have had to do this several times and I'm always frustrated. I have a dataframe:
df = pd.DataFrame([[1, 2, 3, 4], [5, 6, 7, 8]], ['a', 'b'], ['A', 'B', 'C', 'D'])
print df
A B C D
a 1 2 ...
- 1,605
23
votes
3
answers
27k
views
Using lists in a pandas query
I am performing a query on a DataFrame:
Index Category
1 Foo
2 Bar
3 Cho
4 Foo
I would like to return the rows where the category is "Foo" or "Bar".
When I use the code:
df.query("...
- 385
19
votes
2
answers
36k
views
How to concatenate pandas column with list values into one list?
I have a dataframe with one of its column having a list at each index. I want to concatenate these lists into one list. I am using
ids = df.loc[0:index, 'User IDs'].values.tolist()
However, this ...
- 355
18
votes
6
answers
25k
views
how to assign an entire list to each row of a pandas dataframe
I have a dataframe and a list
df = pd.DataFrame({'A':[1,2,3], 'B':[4,5,6]})
mylist= [10,20,30,40,50]
I would like to have a list as element in each row of a dataframe. If I do like here,
df['C'] = ...
- 5,017
17
votes
3
answers
25k
views
Map list from dictionaries
I am new to python and have looked at numerous pages for this.
I know pandas data frames have this mapping functionlity:
dictionary = {a:1, b:2, c:6}
df['col_name'] = df.col_name.map(dictionary) #...
- 2,249
17
votes
5
answers
104k
views
Python: Store multiple dataframe in list
I have a loop that read Excel sheets in a document. I want to store them all in a list:
DF_list= list()
for sheet in sheets:
df= pd.read_excel(...)
DF_list = DF_list.append(df)
If I ...
- 183
17
votes
3
answers
24k
views
Convert a dataframe to list of tuples [duplicate]
I have a python pandas dataframe df like this:
a b
1 3
3 6
5 7
6 4
7 8
I want to transfer it to a list:
[(1,3),(3,6),(5,7),(6,4),(7,8)]
Thanks.
- 765
17
votes
6
answers
11k
views
Pandas dataframe from dictionary of list values
I have a dictionary with lists as values such as:
cols = {'animals':['dog','cat','fish'],
'colors':['red','black','blue','dog']}
I want to convert this into a dataframe in which each of the ...
- 1,744
16
votes
3
answers
25k
views
replace index values in pandas dataframe with values from list
I have a dataframe and 2 lists.
the 1st list gives a set of index values from the dataframe I want to replace
the 2nd list gives the values I want to use
I don't want to touch any of the other ...
- 1,117
16
votes
1
answer
25k
views
python create a data frame with one row by a list
in python, say I have a list [1,2,3,...,100], and I would like to use this list to create a dataframe which has one row and the row value is the list. What is the fastest and elegant way to do this?
- 1,201