Questions tagged [pruning]
Use pruning for questions related to algorithms used on data structures to facilitate generalization and performance optimization.
pruning
191
questions
58
votes
3
answers
199k
views
TypeError: can’t convert CUDA tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first
I am using a modified predict.py for testing a pruned SqueezeNet Model
[phung@archlinux SqueezeNet-Pruning]$ python predict.py --image 3_100.jpg --model model_prunned --num_class 2
prediction in ...
- 979
21
votes
4
answers
10k
views
Pruning in Keras
I'm trying to design a neural network using Keras with priority on prediction performance, and I cannot get sufficiently high accuracy by further reducing the number of layers and nodes per layer. I ...
- 1,606
12
votes
3
answers
7k
views
Prune unnecessary leaves in sklearn DecisionTreeClassifier
I use sklearn.tree.DecisionTreeClassifier to build a decision tree. With the optimal parameter settings, I get a tree that has unnecessary leaves (see example picture below - I do not need ...
- 4,906
11
votes
5
answers
27k
views
TypeError: Tensor is unhashable. Instead, use tensor.ref() as the key. in Keras Surgeon
I'm using Kerassurgeon module for pruning.I encountered this error while i'm working with VGG-16 in google colab.It works fine for other models.Can someone help me fix this.
---> 17 model_new = ...
- 853
11
votes
2
answers
9k
views
How do I enable partition pruning in spark
I am reading parquet data and I see that it is listing all the directories on driver side
Listing s3://xxxx/defloc/warehouse/products_parquet_151/month=2016-01 on driver
Listing s3://xxxx/defloc/...
- 687
9
votes
2
answers
6k
views
How to prune weights less than a threshold in PyTorch?
How to prune weights of a CNN (convolution neural network) model which is less than a threshold value (let's consider prune all weights which are <= 1).
How we can achieve that for a weight file ...
- 1,668
9
votes
1
answer
12k
views
Why does the C4.5 algorithm use pruning in order to reduce the decision tree and how does pruning affect the predicion accuracy?
I have searched on google about this issue and I can't find something that explains this algorithm in a simple yet detailed way.
For instance, I know the id3 algorithm doesn't use pruning at all, so ...
- 3,852
9
votes
1
answer
4k
views
How to freeze/lock weights of one TensorFlow variable (e.g., one CNN kernel of one layer)
I have a TensorFlow CNN model that is performing well and we would like to implement this model in hardware; i.e., an FPGA. It's a relatively small network but it would be ideal if it were smaller. ...
- 101
8
votes
1
answer
828
views
How to prune history right in a CoreData+CloudKit app?
My app uses CoreData with iCloud as backend. Multiple devices can access the iCloud database which is thus .public.
The local CoreData store is synchronized with iCloud using an ...
- 14.6k
8
votes
1
answer
3k
views
Is rpart automatic pruning?
Is rpart automatic pruning?
The decision tree produced by rpart is much more levels than that produced by Oracle Data Mining which has the automatic pruning.
- 81
7
votes
1
answer
7k
views
What is XGBoost pruning step doing?
When I use XGBoost to fit a model, it usually shows a list of messages like "updater_prune.cc:74: tree pruning end, 1 roots, 6 extra nodes, 0 pruned nodes, max_depth=5". I wonder how XGBoost is ...
- 2,427
7
votes
3
answers
1k
views
Accelerating FFTW pruning to avoid massive zero padding
Suppose that I have a sequence x(n) which is K * N long and that only the first N elements are different from zero. I'm assuming that N << K, say, for example, N = 10 and K = 100000. I want to ...
- 21k
6
votes
1
answer
404
views
Is there a free tool capable of pruning unused code from a CLI assembly? [closed]
Is there a free tool capable of pruning unused code from a CLI assembly?
I know there are obfuscators that are capable of performing this optimization, but these all cost money. Is there a free (or ...
- 4,999
6
votes
1
answer
818
views
Minimum Mean Weight Cycle - Intuitive Explanation
In a directed graph, we are looking for the cycle that had the lowest average edge weights. For instance, a graph with nodes 1 and 2 with path from 1 to 2 of length 2 and from 2 to 1 of length 4 would ...
- 1,154
5
votes
4
answers
14k
views
How to quickly prune large tables?
I currently have a MySQL table of about 20 million rows, and I need to prune it. I'd like to remove every row whose updateTime (timestamp of insertion) was more than one month ago. I have not ...
- 3,004
5
votes
2
answers
1k
views
How to prune a Java program
Let's me start from what I want to do then raising some questions I have.
I want to develop a general Java program which is a superset of a number of programs (let's call them program variants). In ...
- 433
5
votes
1
answer
808
views
Why parameters in prunning increases in tensorflow's tfmot
I was prunning a model and came across a library TensorFlow model optimization so initially, we have
I trained this model on a default dataset and it gave me an accuracy of 96 percent which is good. ...
- 562
5
votes
2
answers
8k
views
Random Forest pruning
I have sklearn random forest regressor. It's very heavy, 1.6 GBytes, and works very long time when predicting values.
I want to prune it to make lighter. As I know pruning is not implemented for ...
- 467
5
votes
1
answer
2k
views
Freezing Individual Weights in Pytorch
The following question is not a duplicate of How to apply layer-wise learning rate in Pytorch? because this question aims at freezing a subset of a tensor from training rather than the entire layer.
...
5
votes
2
answers
2k
views
How can we prune the neural network in R?
I have 30 independent variables in my model. I wanted to prune the neural network based on importance of variables. I have tried using mlp function of RSNNS package, but I don't know what arguments ...
- 149
4
votes
1
answer
3k
views
Delete all git branches which have been merged - Fatal: branch name required?
I'm trying to delete all of my merged branches. I've always used
git branch --merged | egrep -v "(^\*|master|dev)" | xargs git branch -d but for some reason it won't work anymore, even though I've ...
- 4,744
4
votes
2
answers
10k
views
docker image prune is not working for version < 1.13
Have tried the following commands, but none of them delete the images.
sudo docker images prune --filter "dangling=true"
sudo docker images prune --all
sudo docker images prune -a
sudo docker images ...
- 381
4
votes
1
answer
1k
views
Can I make pruning to keras pretrained model with tensorflow keras model optimization tool kit?
I have keras pretrained model(model.h5). And I want to prune that model with tensorflow Magnitude-based weight pruning with Keras. One curious things is that my pretrained model is built with original ...
4
votes
2
answers
752
views
How can I accelerate inference speed in TensorFlow when I got sparse matrix from pruning?
I got a sparse weight matrix from Tensorflow-pruning to reduce SqueezeNet. After strip_pruning_vars, I checked the most of elements in weight matrix pruned to 0 successfully. However, the performance ...
- 41
4
votes
1
answer
913
views
Stop Cypher traversal when where condition on reduce() can no longer be satisfied
Suppose I have a neo4j database with a single node type and a single relationship type to keep things simple. All relationships have a "cost" property (as in classical graph problems), whose values ...
- 655
3
votes
2
answers
5k
views
Docker: Cannot delete intermediate images from a broken pull
There is a broken image which blocks further pulls. And I cannot remove or delete it
What I did:
- stopped all containers ("docker ps -a" shows empty list)
- cleaned everything with "docker system ...
- 33
3
votes
4
answers
691
views
Numerical hash for comparing lexical similarity
Is there some form of hashing algorithm that produces similar numerical values for similar words? I imagine there would be a number of false positives, but it seems like something that could be useful ...
- 1,835
3
votes
1
answer
2k
views
Pruning decision tree
How to prune decision tree build with ID3 when there are too few examples in the training set.
I cannot divide it into training, validation and test set, so that is out of the question.
Are there ...
- 5,609
3
votes
1
answer
3k
views
When does Postgresql do partition pruning with JOIN columns?
I have two tables in a Postgres 11 database:
client table
--------
client_id integer
client_name character_varying
file table
--------
file_id integer
client_id integer
file_name ...
- 59
3
votes
1
answer
800
views
Cost Complexity Pruning: Pruned error
Can anybody explain this two statements:
In Cost Complexity Pruning, the pruned tree error can never be less than the original tree on the training dataset.
In Cost Complexity Pruning, the pruned ...
- 35
3
votes
2
answers
3k
views
Pruning git branches
You can prune tracking branches in your repo that no longer exist on the remote repo using:
git remote prune [remoteName]
However this will only get rid of the tracking branches and not any local ...
- 28.6k
3
votes
1
answer
1k
views
How do I implement a pruning strategy with a DTW algorithm?
I've tried to implement a DTW algorithm for a speech recognition application, which I've done successfully, and I'm now trying to improve the performance of the DTW algorithm by pruning. I've tried ...
- 688
3
votes
2
answers
2k
views
Can PostgreSQL 12 do partition pruning at execution time with subquery returning a list?
I'm trying to take advantages of partitioning in one case:
I have table "events" which partitioned by list by field "dt_pk" which is foreign key to table "dates".
-- Schema
drop schema if exists test ...
3
votes
2
answers
2k
views
alpha beta pruning in python
In alpha, beta pruning algorithm,
I have a class in which a function def getAction(self,gamestate) id defined. I made 2 more function in def getAction
Like:
class BAC:
def you(self,gamestate):
...
- 1,855
3
votes
5
answers
2k
views
Are we able to prune a pre-trained model? Example: MobileNetV2
I'm trying to prune a pre-trained model: MobileNetV2 and I got this error. Tried searching online and couldn't understand. I'm running on Google Colab.
These are my imports.
import tensorflow as tf
...
- 49
3
votes
2
answers
647
views
Swamp/dead-end pruning in non-grid maps
Are there any existing algorithms for finding and avoiding problematic areas (swamps, dead-ends) in pathfinding when using non-grid maps? There are plenty available for grids that either avoid these ...
- 369
3
votes
0
answers
400
views
Partition pruning in Deltatable incremental merge
I want to merge a dataframe which contains incremental data to my base deltatable. I want to achieve partition pruning while doing it to avoid scanning a lot of files.
My data is partitioned based on ...
- 86
3
votes
0
answers
240
views
How to prune an existing tensorflow/keras model trained on imagenet
I am trying to prune InceptionNetV3 from keras trained on imagenet, right now I am using a tensorflow-datasets which has a subset of imagenet which I use for pruning. Currently my pruned models do not ...
3
votes
3
answers
1k
views
Pruning using Pytorch on a complicated model
So I am trying to use torch.nn.utils.prune.global_unstructured.
I did it on a simple model and that worked. model.cov2 or other layers and that works. I am trying to do it on a model that's (nested)? ...
- 31
3
votes
0
answers
380
views
How does discrete fourier transform pruning work?
I'm looking to utilise pruned DFTs.
I can't get my head around FFTW's example - they don't have any DFT pruning utilities in the library, so you have to code your own.
In general, how does it work? If ...
- 840
3
votes
0
answers
221
views
Threshold based weight pruning in keras/tensorflow?
Is there a way to prune a Keras model based on a certain threshold? The method prune_low_magnitude() from the tensorflow-model-optimization toolkit only allows pruning based on a defined final ...
- 364
2
votes
2
answers
66
views
Finding removed elements in a vector in matlab [duplicate]
I have a process that is iteratively and randomly pruning a huge vector of integers and I want to find what elements are removed between each iteration. This vector has a lot of repetitions and using ...
- 477
2
votes
2
answers
608
views
Pruning: When to Stop?
When does pruning stop being efficient in a depth-first search? I've been working on an efficient method to solve the N-Queens problem and I'm looking at pruning for the first time. I've implemented ...
- 8,843
2
votes
1
answer
2k
views
Why doesn't torch pruning actually remove filters or weights?
I work with one architecture and trying to sparse it via prune. I wrote functions for pruning, here is one of them:
def prune_model_l1_unstructured(model, layer_type, proportion):
for module in ...
- 159
2
votes
1
answer
2k
views
borg backup- how to keep weekly backups
I have a daily cronjob which backs up with borg backup like this:
borg create -v --stats \
$REPOSITORY::'{now:%Y-%m-%d_%H:%M}' \
/root \
/etc ...
- 354
2
votes
3
answers
827
views
MySQL Partition Pruning PARTITION BY LIST
I have created a MySQL table which is partitioned by the months of a date column like this:
CREATE TABLE `my_big_table` (
`some_id` varchar(60) COLLATE utf8_unicode_ci NOT NULL,
`some_value` ...
- 1,852
2
votes
1
answer
2k
views
Pytorch Global Pruning is not reducing the size of the model
I am trying to Prune my Deep Learning model via Global Pruning. The original UnPruned model is about 77.5 MB. However after pruning, when I am saving the model, the size of the model is the same as ...
2
votes
3
answers
1k
views
Keras-surgeon: For pruning and Model optimization
I am working on a Convolutional Neural Net built using Keras. And since the final model needs to be deployed on a processing unit with less processing capabilities, I started looking for ways to ...
- 151
2
votes
1
answer
110
views
Pruning before running convex hull algorithm
I have to form a Convex Hull from large amount of points and I came across this article. Whole process of pruning is described and well explained, except for one part.
I don't know what this part ...
- 21
2
votes
1
answer
2k
views
How does pytorch L1-norm pruning works?
Lets see the result that I got first. This is one of a convolution layer of my model, and im only showing 11 filter's weight of it (11 3x3 filter with channel=1)
Left side is original weight Right ...
- 23