All Questions
4,730
questions
1350
votes
25
answers
2.2m
views
Get a list from Pandas DataFrame column headers
I want to get a list of the column headers from a Pandas DataFrame. The DataFrame will come from user input, so I won't know how many columns there will be or what they will be called.
For example, ...
- 24.7k
675
votes
11
answers
333k
views
The difference between bracket [ ] and double bracket [[ ]] for accessing the elements of a list or dataframe
R provides two different methods for accessing the elements of a list or data.frame: [] and [[]].
What is the difference between the two, and when should I use one over the other?
- 17.5k
642
votes
27
answers
1.2m
views
Convert a list to a data frame
I have a nested list of data. Its length is 132 and each item is a list of length 20. Is there a quick way to convert this structure into a data frame that has 132 rows and 20 columns of data?
Here is ...
- 39.4k
569
votes
11
answers
1.8m
views
Get list from pandas dataframe column or row?
I have a dataframe df imported from an Excel document like this:
cluster load_date budget actual fixed_price
A 1/1/2014 1000 4000 Y
A 2/1/2014 12000 10000 Y
A 3/1/2014 ...
- 20.7k
500
votes
11
answers
358k
views
Combine a list of data frames into one data frame by row
I have code that at one place ends up with a list of data frames which I really want to convert to a single big data frame.
I got some pointers from an earlier question which was trying to do ...
- 60.3k
434
votes
10
answers
1.2m
views
Creating a Pandas DataFrame from a Numpy array: How do I specify the index column and column headers?
I have a Numpy array consisting of a list of lists, representing a two-dimensional array with row labels and column names as shown below:
data = np.array([['','Col1','Col2'],['Row1',1,2],['Row2',3,4]])...
- 5,405
407
votes
6
answers
307k
views
Pandas DataFrame to List of Dictionaries
I have the following DataFrame:
customer item1 item2 item3
1 apple milk tomato
2 water orange potato
3 juice mango chips
which I want ...
- 5,295
380
votes
13
answers
463k
views
Split a Pandas column of lists into multiple columns
I have a Pandas DataFrame with one column:
import pandas as pd
df = pd.DataFrame({"teams": [["SF", "NYG"] for _ in range(7)]})
teams
0 [SF, NYG]
1 [SF, NYG]
2 ...
- 4,717
343
votes
9
answers
283k
views
Simultaneously merge multiple data.frames in a list
I have a list of many data.frames that I want to merge. The issue here is that each data.frame differs in terms of the number of rows and columns, but they all share the key variables (which I've ...
- 4,919
270
votes
10
answers
378k
views
How do I make a list of data frames?
How do I make a list of data frames and how do I access each of those data frames from the list?
For example, how can I put these data frames in a list ?
d1 <- data.frame(y1 = c(1, 2, 3),
...
- 20.8k
176
votes
9
answers
255k
views
Python pandas insert list into a cell
I have a list 'abc' and a dataframe 'df':
abc = ['foo', 'bar']
df =
A B
0 12 NaN
1 23 NaN
I want to insert the list into cell 1B, so I want this result:
A B
0 12 NaN
1 23 ['foo', '...
- 9,309
175
votes
14
answers
255k
views
data.frame rows to a list
I have a data.frame which I would like to convert to a list by rows, meaning each row would correspond to its own list elements. In other words, I would like a list that is as long as the data.frame ...
- 70.1k
164
votes
7
answers
527k
views
Convert List to Pandas Dataframe Column
I need to convert my list into a one-column pandas dataframe.
Current List (len=3):
['Thanks You',
'Its fine no problem',
'Are you sure']
Required Pandas DF (shape =3,):
0 Thank You
1 Its fine no ...
- 2,323
151
votes
9
answers
141k
views
Pandas DataFrame stored list as string: How to convert back to list
I have an n-by-m Pandas DataFrame df defined as follows. (I know this is not the best way to do it. It makes sense for what I'm trying to do in my actual code, but that would be TMI for this post so ...
- 6,489
124
votes
8
answers
159k
views
Creating an R dataframe row-by-row
I would like to construct a dataframe row-by-row in R. I've done some searching, and all I came up with is the suggestion to create an empty list, keep a list index scalar, then each time add to the ...
- 29.6k
117
votes
3
answers
120k
views
How to determine the length of lists in a pandas dataframe column
How can the length of the lists in the column be determine without iteration?
I have a dataframe like this:
CreationDate
2013-12-22 15:25:02 ...
- 16.8k
112
votes
7
answers
239k
views
Python: create a pandas data frame from a list
I am using the following code to create a data frame from a list:
test_list = ['a','b','c','d']
df_test = pd.DataFrame.from_records(test_list, columns=['my_letters'])
df_test
The above code works ...
- 24.6k
102
votes
5
answers
56k
views
Create a data.frame where a column is a list
I know how to add a list column:
> df <- data.frame(a=1:3)
> df$b <- list(1:1, 1:2, 1:3)
> df
a b
1 1 1
2 2 1, 2
3 3 1, 2, 3
This works, but not:
> df <- data....
- 88.3k
99
votes
10
answers
401k
views
Find empty or NaN entry in Pandas Dataframe
I am trying to search through a Pandas Dataframe to find where it has a missing entry or a NaN entry.
Here is a dataframe that I am working with:
cl_id a c d e ...
- 12.1k
96
votes
4
answers
222k
views
Merge multiple column values into one column in python pandas
I have a pandas data frame like this:
Column1 Column2 Column3 Column4 Column5
0 a 1 2 3 4
1 a 3 4 5
2 b 6 7 8
...
- 5,235
83
votes
2
answers
60k
views
What is difference between dataframe and list in R?
What is difference between dataframe and list in R? Which one should be used when? Which is easier to loop over?
Exact problem: I have to first store 3 string elements like "a", "b", "c". Later for ...
- 943
82
votes
9
answers
121k
views
Merge Two Lists in R
I have two lists
first = list(a = 1, b = 2, c = 3)
second = list(a = 2, b = 3, c = 4)
I want to merge these two lists so the final product is
$a
[1] 1 2
$b
[1] 2 3
$c
[1] 3 4
Is there a simple ...
- 7,227
81
votes
5
answers
168k
views
python pandas flatten a dataframe to a list
I have a df like so:
import pandas
a=[['1/2/2014', 'a', '6', 'z1'],
['1/2/2014', 'a', '3', 'z1'],
['1/3/2014', 'c', '1', 'x3'],
]
df = pandas.DataFrame.from_records(a[1:],columns=a[0])
I ...
- 4,093
68
votes
4
answers
150k
views
Convert Python list to pandas Series
What is the method to convert a Python list of strings to a pd.Series object?
(pandas Series objects can be converted to list using tolist() method--but how to do the reverse conversion?)
- 3,998
63
votes
3
answers
64k
views
Pandas expand rows from list data available in column
I have a data frame like this in pandas:
column1 column2
[a,b,c] 1
[d,e,f] 2
[g,h,i] 3
Expected output:
column1 column2
a 1
b 1
c ...
- 769
61
votes
6
answers
142k
views
Selecting columns by list (and columns are subset of list)
I'm selecting several columns of a dataframe, by a list of the column names. This works fine if all elements of the list are in the dataframe.
But if some elements of the list are not in the DataFrame,...
- 645
61
votes
2
answers
164k
views
How to convert list to row dataframe with Pandas
I have a list of items like this:
A = ['1', 'd', 'p', 'bab', '']
My goal is to convert such list into a dataframe of 1 row and 5 columns. If I type pd.DataFrame(A) I get 5 rows and 1 column. What ...
- 5,800
55
votes
2
answers
77k
views
Merge a list of pandas dataframes
I have a list of data frames and I need to merge them together using a unique column (date). Field names are different so concat is out.
I can manually use df[0].merge(df[1],on='Date').merge(df[3],on='...
- 2,672
50
votes
4
answers
56k
views
Convert list of vectors to data frame
I'm trying to convert a list of vectors (a multidimensional array essentially) into a data frame, but every time I try I'm getting unexpected results.
My aim is to instantiate a blank list, populate ...
- 839
50
votes
4
answers
15k
views
Fast vectorized merge of list of data.frames by row
Most of the questions about merging data.frame in lists on SO don't quite relate to what I'm trying to get across here, but feel free to prove me wrong.
I have a list of data.frames. I would like to "...
- 70.1k
48
votes
7
answers
61k
views
What is the most efficient way to cast a list as a data frame?
Very often I want to convert a list wherein each index has identical element types to a data frame. For example, I may have a list:
> my.list
[[1]]
[[1]]$global_stdev_ppb
[1] 24267673
[[1]]$...
- 5,447
42
votes
5
answers
45k
views
Recombining a list of Data.frames into a single data frame [duplicate]
I am sorry if this question has been answered already. Also, this is my first time on stackoverflow.
I have a beginner R question concerning lists , data frames and merge() and/or rbind().
I ...
- 697
39
votes
2
answers
32k
views
"unstack" a pandas column containing lists into multiple rows [duplicate]
Say I have the following Pandas Dataframe:
df = pd.DataFrame({"a" : [1,2,3], "b" : [[1,2],[2,3,4],[5]]})
a b
0 1 [1, 2]
1 2 [2, 3, 4]
2 3 [5]
How would I "unstack" the ...
- 4,166
38
votes
1
answer
59k
views
AttributeError: 'numpy.ndarray' object has no attribute 'toList'
I'm trying to append certain columns of Pandas Data Frames from CSV files into a numpy array. I have no idea how to instantiate an empty numpy array, so I'm testing it first with a list.
def windows(...
user2398046
37
votes
4
answers
51k
views
Can I combine a list of similar dataframes into a single dataframe? [duplicate]
I have a dataframe:
foo <- list(df1 = data.frame(x=c('a', 'b', 'c'),y = c(1,2,3)),
df2 = data.frame(x=c('d', 'e', 'f'),y = c(4,5,6)))
Can I convert it to a single dataframe of the ...
- 31.3k
32
votes
6
answers
22k
views
Mean of elements in a list of data.frames
Suppose I had a list of data.frames (of equal rows and columns)
dat1 <- as.data.frame(matrix(rnorm(25), ncol=5))
dat2 <- as.data.frame(matrix(rnorm(25), ncol=5))
dat3 <- as.data.frame(matrix(...
- 466
30
votes
3
answers
29k
views
rbind dataframes in a list of lists
I have a list of lists that looks like this: x[[state]][[year]]. Each element of this is a data frame, and accessing them individually is not a problem.
However, I'd like to rbind data frames across ...
- 4,919
30
votes
3
answers
41k
views
Add Columns to an empty data frame in R
I have searched extensively but not found an answer to this question on Stack Overflow.
Lets say I have a data frame a.
I define:
a <- NULL
a <- as.data.frame(a)
If I wanted to add a column ...
- 1,873
29
votes
1
answer
27k
views
How to know if the data is a list or data.frame in R
How do I know if my data in R is a list or a data.frame?
If I use typeof(x) it says list, if I use class(x) it says data.frame?
- 327
27
votes
2
answers
68k
views
Nested List to Pandas Dataframe with headers
Basically I am trying to do the opposite of How to generate a list from a pandas DataFrame with the column name and column values?
To borrow that example, I want to go from the form:
data = [
['...
- 2,036
26
votes
5
answers
32k
views
Converting a list of data frames into individual data frames in R [duplicate]
I have been searching high and low for what I think is an easy solution.
I have a large data frame that I split by factors.
eqRegions <- split(eqDataAll, eqDataAll$SeismicRegion)
This now ...
- 1,790
26
votes
7
answers
67k
views
Combine two lists in a dataframe in R
I have two lists with different structure:
listA <- list(c("a","b","c"), c("d","e"))
listB <- list(0.05, 0.5)
listA
[[1]]
[1] "a" "b" "c"
[[2]]
[1] "d" "e"
listB
[[1]]
[1] 0.05
[[2]]
[1] 0....
- 627
26
votes
2
answers
18k
views
How to apply dplyr filter to list of data frames?
I have a list() of dataframes. I want to apply dplyr's filter() to all of them.
Example code of what I have tried so far...
require(dplyr)
list.DFs <- list(df1,df2)
lapply(
X = list.DFS,
FUN =...
- 3,553
26
votes
2
answers
57k
views
How do I create pandas DataFrame (with index or multiindex) from list of namedtuple instances?
Simple example:
from collections import namedtuple
import pandas
Price = namedtuple('Price', 'ticker date price')
a = Price('GE', '2010-01-01', 30.00)
b = Price('GE', '2010-01-02', 31.00)
l = [a, b]
...
- 4,808
25
votes
2
answers
40k
views
How to multiply each row in pandas dataframe by a different value
I am trying to multiply each row of a pandas dataframe by a different value and wondering what the best way to do this is.
For example if I have the following dataframe:
import numpy as np
import ...
- 13.4k
24
votes
4
answers
32k
views
Delete a column in a data frame within a list
I made a list out of my dataframe, based on the factor levels in column A. In the list I would like to remove that column. My head is saying lapply, but not anything else :P
$A
ID Test
A 1
A 1
$...
- 1,411
24
votes
2
answers
45k
views
Splitting a list in a Pandas cell into multiple columns [duplicate]
I have a really simple Pandas dataframe where each cell contains a list. I'd like to split each element of the list into it's own column. I can do that by exporting the values and then creating a new ...
- 9,003
24
votes
3
answers
4k
views
How do I turn a dataframe into a series of lists?
I have had to do this several times and I'm always frustrated. I have a dataframe:
df = pd.DataFrame([[1, 2, 3, 4], [5, 6, 7, 8]], ['a', 'b'], ['A', 'B', 'C', 'D'])
print df
A B C D
a 1 2 ...
- 1,605
23
votes
3
answers
40k
views
Unlist data frame column preserving information from other column
I have a data frame which consists of two column: a character vector col1 and a list column, col2.
myVector <- c("A","B","C","D")
myList <- list()
myList[[1]] <- c(1, 4, 6, 7)
myList[[2]] &...
- 6,625
22
votes
3
answers
32k
views
Put multiple data frames into list (smart way) [duplicate]
Is it possible to put a lot of data frames into a list in some easy way?
Meaning instead of having to write each name manually like the following way:
list_of_df <- list(data_frame1,data_frame2,...
- 2,257