Questions tagged [duplicates]
The "duplicates" tag concerns detecting and/or dealing with multiple instances of items in collections.
duplicates
16,725
questions
2448
votes
34
answers
3.7m
views
Finding duplicate values in a SQL table
It's easy to find duplicates with one field:
SELECT email, COUNT(email)
FROM users
GROUP BY email
HAVING COUNT(email) > 1
So if we have a table
ID NAME EMAIL
1 John [email protected]
2 Sam ...
- 35.3k
2395
votes
54
answers
3.1m
views
Remove duplicate values from JS array [duplicate]
I have a very simple JavaScript array that may or may not contain duplicates.
var names = ["Mike","Matt","Nancy","Adam","Jenny","Nancy",&...
- 24.2k
1476
votes
23
answers
1.3m
views
LINQ's Distinct() on a particular property
I am playing with LINQ to learn about it, but I can't figure out how to use Distinct when I do not have a simple list (a simple list of integers is pretty easy to do, this is not the question). What I ...
- 139k
1468
votes
58
answers
2.2m
views
Removing duplicates in lists
How can I check if a list has any duplicates and return a new list without duplicates?
- 20.5k
1375
votes
43
answers
1.4m
views
How can I remove duplicate rows?
I need to remove duplicate rows from a fairly large SQL Server table (i.e. 300,000+ rows).
The rows, of course, will not be perfect duplicates because of the existence of the RowID identity field.
...
- 69.5k
1052
votes
31
answers
730k
views
How do I remove duplicates from a list, while preserving order?
How do I remove duplicates from a list, while preserving order? Using a set to remove duplicates destroys the original order.
Is there a built-in or a Pythonic idiom?
- 25.7k
917
votes
78
answers
1.2m
views
How to remove all duplicates from an array of objects?
I have an object that contains an array of objects.
obj = {};
obj.arr = new Array();
obj.arr.push({place:"here",name:"stuff"});
obj.arr.push({place:"there",name:"...
- 10.6k
783
votes
25
answers
797k
views
Fastest way to duplicate an array in JavaScript - slice vs. 'for' loop
In order to duplicate an array in JavaScript: Which of the following is faster to use?
Slice method
var dup_array = original_array.slice();
For loop
for(var i = 0, len = original_array.length; i < ...
- 34k
743
votes
28
answers
936k
views
Find duplicate records in MySQL
I want to pull out duplicate records in a MySQL Database. This can be done with:
SELECT address, count(id) as cnt FROM list
GROUP BY address HAVING cnt > 1
Which results in:
100 MAIN ST 2
I ...
- 15k
708
votes
43
answers
1.4m
views
How do I find the duplicates in a list and create another list with them?
How do I find the duplicates in a list of integers and create another list of the duplicates?
- 19.4k
706
votes
7
answers
752k
views
Find duplicate lines in a file and count how many time each line was duplicated?
Suppose I have a file similar to the following:
123
123
234
234
123
345
I would like to find how many times '123' was duplicated, how many times '234' was duplicated, etc.
So ideally, the ...
- 7,123
693
votes
17
answers
905k
views
Removing duplicate rows in Notepad++
Is it possible to remove duplicated rows in Notepad++, leaving only a single occurrence of a line?
- 9,717
637
votes
32
answers
849k
views
Remove duplicates from a List<T> in C#
Anyone have a quick method for de-duplicating a generic List in C#?
- 39.7k
585
votes
29
answers
1.4m
views
Delete duplicate rows keeping the first row
How can I delete duplicate rows where no unique row id exists?
My table is
col1 col2 col3 col4 col5 col6 col7
john 1 1 1 1 1 1
john 1 1 1 1 1 1
sally 2 2 2 ...
- 10.9k
581
votes
40
answers
1.0m
views
How do I remove repeated elements from ArrayList?
I have an ArrayList<String>, and I want to remove repeated strings from it. How can I do this?
- 5,981
581
votes
5
answers
1.3m
views
How do I (or can I) SELECT DISTINCT on multiple columns?
I need to retrieve all rows from a table where 2 columns combined are all different. So I want all the sales that do not have any other sales that happened on the same day for the same price. The ...
- 33.8k
525
votes
13
answers
513k
views
C# LINQ find duplicates in List
Using LINQ, from a List<int>, how can I retrieve a list that contains entries repeated more than once and their values?
- 5,323
477
votes
2
answers
539k
views
Delete all Duplicate Rows except for One in MySQL? [duplicate]
How would I delete all duplicate data from a MySQL Table?
For example, with the following data:
SELECT * FROM names;
+----+--------+
| id | name |
+----+--------+
| 1 | google |
| 2 | yahoo |
|...
- 23.5k
461
votes
7
answers
477k
views
Remove pandas rows with duplicate indices
How to remove rows with duplicate index values?
In the weather DataFrame below, sometimes a scientist goes back and corrects observations -- not by editing the erroneous rows, but by appending a ...
- 66.8k
445
votes
28
answers
472k
views
Remove duplicate rows in MySQL
I have a table with the following fields:
id (Unique)
url (Unique)
title
company
site_id
Now, I need to remove rows having same title, company and site_id. One way to do it will be using the ...
- 4,915
383
votes
8
answers
276k
views
Remove duplicate elements from array in Ruby
I have a Ruby array which contains duplicate elements.
array = [1,2,2,1,4,4,5,6,7,8,5,6]
How can I remove all the duplicate elements from this array while retaining all unique elements without using ...
- 20.8k
374
votes
27
answers
592k
views
What's the most efficient way to erase duplicates and sort a vector?
I need to take a C++ vector with potentially a lot of elements, erase duplicates, and sort it.
I currently have the below code, but it doesn't work.
vec.erase(
std::unique(vec.begin(), vec.end(...
- 4,361
369
votes
18
answers
362k
views
How to remove duplicate values from a multi-dimensional array in PHP
How can I remove duplicate values from a multi-dimensional array in PHP?
Example array:
Array
(
[0] => Array
(
[0] => abc
[1] => def
)
[1] => Array
(
...
- 12.1k
362
votes
9
answers
467k
views
How to find duplicate records in PostgreSQL
I have a PostgreSQL database table called "user_links" which currently allows the following duplicate fields:
year, user_id, sid, cid
The unique constraint is currently the first field called "id", ...
- 6,737
325
votes
15
answers
462k
views
Remove duplicates by columns A, keeping the row with the highest value in column B
I have a dataframe with repeat values in column A. I want to drop duplicates, keeping the row with the highest value in column B.
So this:
A B
1 10
1 20
2 30
2 40
3 10
Should turn into this:
A B
1 ...
- 23.3k
315
votes
13
answers
858k
views
How do I find duplicate values in a table in Oracle?
What's the simplest SQL statement that will return the duplicate values for a given column and the count of their occurrences in an Oracle database table?
For example: I have a JOBS table with the ...
- 13.1k
307
votes
13
answers
619k
views
How do I get a list of all the duplicate items using pandas in python?
I have a list of items that likely has some export issues. I would like to get a list of the duplicate items so I can manually compare them. When I try to use pandas duplicated method, it only ...
- 5,133
259
votes
5
answers
164k
views
MySQL ON DUPLICATE KEY UPDATE for multiple rows insert in single query
I have a SQL query where I want to insert multiple rows in single query. so I used something like:
$sql = "INSERT INTO beautiful (name, age)
VALUES
('Helen', 24),
('Katrina', 21),
('Samia', ...
- 5,431
258
votes
15
answers
377k
views
How do I check if there are duplicates in a flat list?
For example, given the list ['one', 'two', 'one'], the algorithm should return True, whereas given ['one', 'two', 'three'] it should return False.
- 6,145
257
votes
8
answers
567k
views
Drop all duplicate rows across multiple columns in Python Pandas
The pandas drop_duplicates function is great for "uniquifying" a dataframe. I would like to drop all rows which are duplicates across a subset of columns. Is this possible?
A B C
0 ...
- 13.2k
247
votes
29
answers
423k
views
How do I remove duplicates from a C# array?
I have been working with a string[] array in C# that gets returned from a function call. I could possibly cast to a Generic collection, but I was wondering if there was a better way to do it, possibly ...
- 115k
247
votes
18
answers
581k
views
Finding duplicate rows in SQL Server
I have a SQL Server database of organizations, and there are many duplicate rows. I want to run a select statement to grab all of these and the amount of dupes, but also return the ids that are ...
- 2,619
245
votes
22
answers
449k
views
How to remove duplicate values from an array in PHP
How can I remove duplicate values from an array in PHP?
- 12.1k
225
votes
10
answers
608k
views
Remove duplicated rows
I have read a CSV file into an R data.frame. Some of the rows have the same element in one of the columns. I would like to remove rows that are duplicates in that column. For example:
...
- 2,401
213
votes
6
answers
213k
views
How to merge 2 List<T> and removing duplicate values from it in C#
I have two lists List that I need to combine in third list and remove duplicate values from that lists
A bit hard to explain, so let me show an example of what the code looks like and what I want as ...
- 14.7k
201
votes
24
answers
691k
views
Removing duplicate rows from table in Oracle
I'm testing something in Oracle and populated a table with some sample data, but in the process I accidentally loaded duplicate records, so now I can't create a primary key using some of the columns.
...
- 81.1k
198
votes
8
answers
417k
views
How to "select distinct" across multiple data frame columns in pandas?
I'm looking for a way to do the equivalent to the SQL
SELECT DISTINCT col1, col2 FROM dataframe_table
The pandas sql comparison doesn't have anything about distinct.
.unique() only works for a ...
- 8,151
194
votes
26
answers
278k
views
In MySQL, can I copy one row to insert into the same table?
insert into table select * from table where primarykey=1
I just want to copy one row to insert into the same table (i.e., I want to duplicate an existing row in the table) but I want to do this ...
- 1,941
183
votes
9
answers
202k
views
How to get duplicate items from a list using LINQ? [duplicate]
I'm having a List<string> like:
List<String> list = new List<String>{"6","1","2","4","6","5","1"};
I need to get the duplicate items in the list into a new list. Now I'm using a ...
- 14.5k
179
votes
17
answers
171k
views
Regular expression for duplicate words
I'm a regular expression newbie and I can't quite figure out how to write a single regular expression that would "match" any duplicate consecutive words such as:
Paris in the the spring.
...
- 6,773
175
votes
25
answers
251k
views
How to delete duplicates on a MySQL table?
I need to DELETE duplicated rows for specified sid on a MySQL table.
How can I do this with an SQL query?
DELETE (DUPLICATED TITLES) FROM table WHERE SID = "1"
Something like this, but I don't know ...
- 5,392
175
votes
11
answers
243k
views
How do I remove duplicate items from an array in Perl?
I have an array in Perl:
my @my_array = ("one","two","three","two","three");
How do I remove the duplicates from the array?
- 14.1k
170
votes
15
answers
113k
views
Removing duplicate rows in vi?
I have a text file that contains a long list of entries (one on each line). Some of these are duplicates, and I would like to know if it is possible (and if so, how) to remove any duplicates. I am ...
- 13.9k
166
votes
10
answers
69k
views
Finding ALL duplicate rows, including "elements with smaller subscripts"
R's duplicated returns a vector showing whether each element of a vector or data frame is a duplicate of an element with a smaller subscript. So if rows 3, 4, and 5 of a 5-row data frame are the same, ...
- 2,499
161
votes
13
answers
275k
views
In Javascript, how do I check if an array has duplicate values?
Possible Duplicate:
Easiest way to find duplicate values in a javascript array
How do I check if an array has duplicate values?
If some elements in the array are the same, then return true. ...
- 10k
161
votes
9
answers
256k
views
Does adding a duplicate value to a HashSet/HashMap replace the previous value
Please consider the below piece of code:
HashSet hs = new HashSet();
hs.add("hi"); -- (1)
hs.add("hi"); -- (2)
hs.size() will give 1 as HashSet doesn't allow duplicates so only one element will be ...
- 21k
156
votes
22
answers
66k
views
Xcode duplicate line
There is a Duplicate command in the Edit Menu (with a default shortcut of ⌘D), but it is (as Halley pointed out) meant for duplication in the Interface Builder part of Xcode.
So, how do you (easily) ...
- 3,588
143
votes
10
answers
266k
views
How do I find duplicates across multiple columns?
So I want to do something like this sql code below:
select s.id, s.name,s.city
from stuff s
group by s.name having count(where city and name are identical) > 1
To produce the following, (but ...
- 46.9k
143
votes
8
answers
108k
views
Linux command or script counting duplicated lines in a text file? [duplicate]
If I have a text file with the following conent
red apple
green apple
green apple
orange
orange
orange
Is there a Linux command or script that I can use to get the following result?
1 red apple
2 ...
- 2,204
138
votes
20
answers
316k
views
Map implementation with duplicate keys
I want to have a map with duplicate keys.
I know there are many map implementations (Eclipse shows me about 50), so I bet there must be one that allows this. I know it's easy to write your own map ...
- 63.3k