All Questions
17
questions
13
votes
1
answer
656
views
Most scalable way for using generators with tf.data ? tf.data guide says `from_generator` has limited scalability
tf.data has a from_generator initializer, it doesn't seem like it's scalable. From the official guide
Caution: While this is a convienient approach it has limited
portability and scalibility. It ...
- 5,887
4
votes
2
answers
7k
views
Using flow_from_dataframe what is the correct "value" for y_col
I am reading in a csv file with pandas, and give the column names stored in colname
colnames=['file', 'label']
# Read data from file
data = pd.read_csv('./Hand_Annotations_2.csv',names=colnames, ...
- 2,091
3
votes
1
answer
634
views
Use generator in TensorFlow/Keras to fit when the model gets 2 inputs
I want to train a model that uses an extra output layer to compute the loss (ArcFace) so the model gets two inputs: the features and the true label: [X, y].
So far I did with the all data loaded at ...
- 781
1
vote
1
answer
651
views
Passing a Keras Generator to __call__ method of My Model
I have written my own keras Model, and I'm trying to pass a Keras Generator as input to model.fit.
The problem is that I don't know how to process the generator, when I'm in the call method of MyModel....
- 53
1
vote
1
answer
129
views
How to get the dictionary output from a generator, which outputs an array with a dictionary for custom keras image generator
I have a custom made generator to output multiple values to predict against. I'm trying to get the values to correspond to a given image, without success. Here is my output:
e(array([[[[0., 0., 0.],
...
- 1,028
1
vote
0
answers
205
views
Selecting according to labels in a TensorFlow generator
I have a very large dataset (VoxCeleb) and each datum has a label (multi-class, can assume that the label is a number between 1 to 5000) and an audio recording. Since it is too large to load entirely ...
- 781
1
vote
0
answers
151
views
Tensorflow Keras: Problems to handle variable length input, using generator?
We want to train our model on varying input dimensions. Every input in a given batch and across batches has different dimensions.
We cannot resize our input (since we’ll lose our microscopic features)....
- 675
1
vote
0
answers
369
views
Conv2DCustomBackpropInput: Size of out_backprop doesn't match computed:
The error is:
InvalidArgumentError: Conv2DCustomBackpropInput: Size of out_backprop doesn't match computed: actual = 16, computed = 5 spatial_dim: 1 input: 10 filter: 2 output: 16 stride: 2 dilation: ...
- 11
1
vote
0
answers
165
views
Keras: fit_generator() feeding text in batches to the model
I'm new to the fit_generator() function and since my data is too big to fit into RAM, I figured to solve it via a generator.
Now, I do have a folder which contains each sample as a separate file. ...
- 323
1
vote
0
answers
370
views
How to handle exception in custom Keras Sequence for fit_generator?
I implement my own Sequence (from tensorflow.python.keras.utils import Sequence). However, some bad file which result in exception at __getitem__. If I use model.fit_generator, then training process ...
- 5,347
0
votes
1
answer
2k
views
When using generator for representative dataset in quantization it "Failed to convert value into readable tensor"
I am quantizing a model. The model takes 224x224 input.
I preprocess the data with a build-in function preprocess_input() which subtracts some center pixels.
Now when using a simple image with this ...
- 2,091
0
votes
0
answers
68
views
How to improve training performance of model.fit() with generators?
Normally when I train models with tensorflow, I just feed my entire data set to the model training (Scenario 1). However, from time to time this data set is to large to fit into my GPU. For that ...
- 413
0
votes
1
answer
880
views
Error in Keras - 'int' object has no attribute 'shape'
I'm using VGGFace model. I am using tensorflow.keras with Tensorflow version 2.4.1. The following is the function for generator:
def gen(list_tuples, person_to_images_map, batch_size=16):
ppl = ...
0
votes
1
answer
99
views
Multi Input Modeling in TensorFlow, with Generator
import numpy as np
import tensorflow as tf
class ProbDistWRTChoices(tf.keras.layers.Layer):
def __init__(self, maxChoice):
super().__init__()
self.maxChoice = maxChoice
def ...
0
votes
0
answers
303
views
How to use sklearn.metrics.classification_report with generator?
I would like to use sklearn.metrics.classification_report but I cannot use it directly as my testing data is provided by a python generator. The python generator is given below.
Let me give you the ...
- 190
0
votes
1
answer
1k
views
Error occurred when finalizing GeneratorDataset iterator: Cancelled: Operation was cancelled - Tensorflow
I am really new with TensorFlow and model building and training. However, I was following a tutorial and everything went well until at one point I got the following error:
2020-04-29 17:24:35....
- 125
-1
votes
1
answer
211
views
Tensorflow 1.9.0: A question about training my model with generators
Situation
I am trying to build my own CNN with images I compiled.
I have two folders with training and validation data. This is my code so far:
import os
import sys
import shutil
import numpy as np
...
- 561