All Questions
Tagged with tf.keras tensorflow-datasets
64
questions
16
votes
3
answers
8k
views
tf.data.Dataset iterator returning Tensor("IteratorGetNext:1", shape=(None, 16), dtype=int32) but cannot get the values of the Tensors
I am trying to write a Custom Model in which I am writing a custom train_step function
I am creating a 'tf.data.Dataset` from a Custom Datagenerator like
tds = tf.data.Dataset.from_generator(tdg....
- 1,166
12
votes
1
answer
12k
views
How to use sample weights with tensorflow datasets?
I have been training a unet model for multiclass semantic segmentation in python using Tensorflow and Tensorflow Datasets.
I've noticed that one of my classes seems to be underrepresented in training. ...
- 137
11
votes
1
answer
4k
views
How to make custom loss with extra input in tensorflow 2.0
I'm having a lot of trouble getting a custom loss function with an extra argument to work in TF 2.0 using tf.keras and a dataset.
In the following case, the extra argument is the input data into the ...
- 6,879
8
votes
1
answer
16k
views
Tensorflow model.fit() using a Dataset generator
I am using the Dataset API to generate training data and sort it into batches for a NN.
Here is a minimum working example of my code:
import tensorflow as tf
import numpy as np
import random
def ...
- 2,065
5
votes
3
answers
5k
views
Keras ImageDataGenerator for multiple inputs and image based target output
I have a model which takes two Images as inputs and generates a single image as a Target output.
All of my training image-data is in the following sub-folders:
input1
input2
target
Can I use ...
- 3,324
5
votes
2
answers
2k
views
How does tf.keras.Model tell between features and label(s) in tf.data.Dataset and in TFRecords?
I'm trying to create tfrecords files from CSV data, then I want to use tf.data.TFRecordDataset() to create Dataset from them, and then feed the Dataset to tf.keras.Model. (In fact I'm using spark-...
- 3,178
4
votes
1
answer
985
views
Tensorflow model analysis, TFMA for keras model
I would like to use TFMA with keras model. The keras model was created with TF 2.0 alpha.
The model is a pretrained model with a classification layer:
...
- 679
4
votes
1
answer
2k
views
Validation loss is zero on first epoch only
Problem
I am trying to build a regression model in tensorflow using the dataset and keras API's. The target contains quite a lot of zero's and the non-zero values are roughly distributed normally ...
4
votes
1
answer
1k
views
tf.data or tf.keras.utils.Sequence. Improving efficiency of tf.data?
I am trying to develop an image colorizer using autoencoders. There are 13000 training images. Each epoch takes about 45 minutes if I use tf.data and about 25 minutes if I use tf.utils.keras.Sequence. ...
- 63
4
votes
0
answers
250
views
Tensorflow/Keras local training 6x faster than distributed training
I am training a Classifier (via RNN) in Tensorflow with tf.data api, tf.keras api and tf. estimator api. In local training i got a troughput of around 240 steps/sec, than i enter distributed mode with ...
- 69
3
votes
1
answer
3k
views
Why training accuracy and validation accuracy are different for the same dataset with tensorflow2.0?
I am training with tensorflow2.0 and tensorflow_datasets. But I am not understand: why does the training accuracy and loss and valdataion accuracy and loss are different?
This is my code:
import ...
- 4,857
3
votes
1
answer
3k
views
from_tensor_slices() with big numpy array while using tf.keras
I have some training data in a numpy array - it fits in the memory but it is bigger than 2GB. I'm using tf.keras and the dataset API. To give you a simplified, self-contained example:
import numpy as ...
- 582
3
votes
1
answer
1k
views
Is Tensorflow Dataset.from_generator deprecated in tensorflow 2.0 ? It throws tf.py_func deprecation error
When I create a tf dataset from generator and try to run tf2.0 code , It warns me with a deprication message.
Code:
import tensorflow as tf
from tensorflow.keras.layers import Dense, Flatten, ...
- 2,619
3
votes
1
answer
908
views
Is validation dataset initialized/created every epoch during the training process?
Setup:
U-Net network is trained to process small patches (e.g. 64x64 pixels).
The network is fed with a training dataset and validation dataset using Tensorflow Dataset API.
Small patches are ...
- 31
3
votes
2
answers
3k
views
How to extract all tf.data.Dataset object into features and labels and pass into ImageDataGenerator's flow() method?
I am acutally working on a mini-project based on cifar10 dataset. I have loaded the data from tfds.load(...) and practicing image augmentation techniques.
As I am using tf.data.Dataset object, which ...
- 313
3
votes
2
answers
2k
views
Error trying to feed a tf.keras model with a tf.data.Dataset instead of tensors
Why does the following tf2 tf.keras model 'work' when fitted with tensors but generates a ValueError when attempting to fit the same tensors in tf.data.Dataset.from_tensor_slices form?
EDIT: Put ...
- 41
3
votes
0
answers
897
views
Display currently used batch (input images, predictions) on tensorboard when using model.fit() with tf.Dataset in Tensorflow 2
I am training a deep network implementing the tf.keras.Model API using tf.keras.Model.fit() (Tensorflow 2.0) to segment images. The input data comes from a dataset using tf.data.TFRecordDataset. Thus, ...
- 63
2
votes
1
answer
1k
views
create tensorflow dataset from list_files
I am trying to create tensroflow dataset :
path_imgs = ('./images/train/*.jpg')
path_masks =('./masks/train/*.jpg'
images = tf.data.Dataset.list_files(path_imgs, shuffle=False)
masks = tf.data....
- 5,511
2
votes
2
answers
1k
views
How to attach or get filenames from MapDataset from image_dataset_from_directory() in Keras?
I am training convolutional autoencoder and I have this code for loading data (images):
train_ds = tf.keras.preprocessing.image_dataset_from_directory(
'path/to/images',
image_size=image_size
)...
- 714
2
votes
2
answers
1k
views
Keras training with shuffled tf.data: if training is interrupted, how to continue training at last data iteration/order of last saved checkpoint
I am training with keras model.fit, and the data comes from tf.records, loaded into a tf.data object, which uses .shuffle to shuffle the data. I am also using callbacks.ModelCheckpoint to save the ...
- 5,887
2
votes
1
answer
648
views
TFRecord reading pipeline slows down after PREFETCH samples
I have divided my training data into multiple tf-record files, and read them using this piece of code:
SHUFFLE_BUFFER = 64
PREFETCH = 256
dataset = tf.data.TFRecordDataset(filenames)
dataset = ...
- 4,391
2
votes
2
answers
931
views
How to build input function for a estimator created from keras model
I'm creating an estimator from keras model as below
estimator = tf.keras.estimator.model_to_estimator(keras_model=keras_model, model_dir=model_dir)
my model is like
...
- 114
2
votes
0
answers
258
views
How to use multiple image datasets as inputs and outputs for the Keras network?
At the moment I'm trying to join a dataset that is scattered through different folders into one, this dataset have no labels as this is an autoencoder-like application. The code at the moment is ...
2
votes
1
answer
822
views
How to use file_paths on a tensorflow TakeDataset?
I am trying to build a image classifier using a dataset of images, split between two directories, one for each class.
data_dir = 'experimental.data'
batch_size = 9
img_size = (160, 160)
train_ds = tf....
- 63
2
votes
0
answers
550
views
keras fit with tf.data.Dataset generator runs out of memory and deadlock
I have a CNN training scheme where I defined methods to import and preprocess data using the tf.data.Dataset API and eventually training with keras fit.
my model trains on large arrays of 5 dim ...
- 31
2
votes
0
answers
2k
views
tf.data object into a Keras model with multiple inputs
I am having trouble using tf.Data with Keras of multiple input.
I am reading data from a PostgreSQL table using Python generator which returns three arrays:
class PairGenerator(object):
return {...
- 942
1
vote
2
answers
4k
views
Is it possible to split a tensorflow dataset into train, validation AND test datasets when using image_dataset_from_directory?
I am using tf.keras.utils.image_dataset_from_directory to load a dataset of 4575 images. While this function allows to split the data into two subsets (with the validation_split parameter), I want to ...
1
vote
1
answer
3k
views
The established way to use TF Dataset API in Keras is to feed `model.fit` with `make_one_shot_iterator()`, But this iterator only good for one Epoch
Edit:
To clarify why this question is different from the suggested duplicates, this SO question follows up on those suggested duplicates, on what exactly is Keras doing with the techniques described ...
- 5,887
1
vote
1
answer
529
views
apply dataset.repeat of prefetched dataset
I'm trying to implement AdaRound quantization algorithm and I need to train my layers one by one.
I'm using a dataset with 1024 with batch-size of 32 and I need to iterate over the dataset roughly 312 ...
- 31
1
vote
1
answer
1k
views
How to use ImageDataGenerator.flow_from_directory to save each class/label in other subfolders?
Edit:
Setup:
Win 10 x64
Python 3.8.7
TF version 2.4.0
Keras version 2.4.3 #not needed I think but
I have dataset with images with structure like:
dataset/
class1/
img1class1.jpg
...
...
1
vote
1
answer
296
views
Numpy arrays used in training in TF1--Keras have much lower accuracy in TF2
I had a neural net in keras that performed well. Now with the deprecation that came with Tensorflow 2 I had to rewrite the model. Now it is giving me worse accuracy metrics.
My suspicion is that tf2 ...
- 487
1
vote
1
answer
116
views
Tensorflow: Cannot train neural network due to the ValueError: slice index 0 of dimension 0 out of bounds error
I'm precticing building neural networks, but I keep getting stuck on the training of the NN.
This time I wanted to do Intel Image Classification from (Kaggle).
I'm using tensorflow 2.12 in WSL.
...
- 15
1
vote
1
answer
389
views
Failed to convert a NumPy array to a Tensor, when trying to train the image model
I'm trying to train a model of image recognition.But the model.fit is always returning the error:
ValueError Traceback (most recent call last)
Cell In\[106\], line 1
\---...
- 11
1
vote
0
answers
129
views
"ValueError: Failed to find data adapter that can handle input" when trying to create Functional API model with Keras
This is the error I get:
ValueError: Failed to find data adapter that can handle input: (<class 'list'> containing values of types {"<class 'tensorflow.python.data.ops.dataset_ops....
1
vote
0
answers
125
views
Is it possible to do the whole training procedure in GPU with Tensorflow/Keras?
If the dataset is small enough to fit in the GPU memory, is it possible with Tensorflow to allocate it all initially and then do the training without having data transfers between CPU and GPU?
It ...
- 71
1
vote
0
answers
72
views
I am trying to convert some tfrecord files back to jpeg images but have some errors
I am trying to convert some tfrecord files back to jpeg images using the code sample section below:
for image_features in tqdm(parsed_image_dataset):
image_raw = image_features['image'].numpy() ...
- 11
1
vote
0
answers
777
views
Getting ValueError: as_list() is not defined on an unknown TensorShape. but the as_list() works fine on (next(iter(dataset)))
I'm trying to use tf.data.Dataset.list_files to load .tiff images and infer their labels from their names.
I use the following code but stumbled upon a strange issue, as described bellow:
import os
...
- 2,283
1
vote
0
answers
107
views
Index 51 is out of bounds for axis 1 with size 50
I am getting this code since I tried it for the very first time. I think I am not getting the error but it is continuously showing that index 51 is out of bounds for axis 1 with size 50.
batch_size = ...
1
vote
1
answer
3k
views
Iterating a tf.data.Dataset.from_generator for a keras image generator.flow_from_dir throw errors
This is the case of using Keras ImageDataGenerator with .flow_from_directory, wrapping it with tf.data.Dataset.from_generator(...). The dataset failed in any attempt to iterate through it.
Error ...
- 3,769
1
vote
0
answers
221
views
Image feature extraction with TF2 Keras API and TF Dataset
How should I use a tf Dataset in order run model.predict(data) and have access to the other features of the tf Dataset?
For example: my tf dataset has this format:
(tensor<(100,224,224,3)>, ...
- 1,184
1
vote
1
answer
71
views
Using generators on networks ending with tensorflow probability layer
I have a network which ends with probability layer something like this:
model = tfk.Sequential([
tfkl.InputLayer(10),
tfkl.Dense(tfpl.MultivariateNormalTriL.params_size(2)),
...
- 474
1
vote
1
answer
491
views
How to create windowed multivariate dataset from SequenceExample TFRecord
I am trying to set up a Tensorflow pipeline using tf.data.datasets in order to load some TFRecord into a Keras model. These data are multivariate timeseries.
I am currently using Tensorflow 2.0
...
- 11
0
votes
1
answer
2k
views
Should we apply repeat, batch shuffle to tf.data.Dataset when passing it to fit function?
I still don't after having read documentation about tf.keras.Model.fit and tf.data.Dataset, when passing tf.data.Dataset to fit function, should I call repeat and batch on the dataset object or should ...
- 1,057
0
votes
1
answer
3k
views
How to save tensorflow dataset as TFRecord file?
How to save or convert tensorflow dataset as TFRecord file?
I would like to convert my data from csv file to TFRecord to speed up training.
titanic = pd.read_csv("https://storage.googleapis.com/...
- 142
0
votes
1
answer
457
views
How to remove black canvas from image in TensorFlow
I'm currenly trying working with tensorflow dataset 'tf_flowers', and noticed that a lot of images consist mostly of black canvas, like this:
flower1
flower2
Is there any easy way to remove/or filter ...
- 63
0
votes
1
answer
2k
views
Training a single model jointly over multiple datasets in tensorflow
I want to train a single variational autoencoder model or even a standard autoencoder over many datasets jointly (e.g. mnist, cifar, svhn, etc. where all the images in the datasets are resized to be ...
- 3,237
0
votes
1
answer
4k
views
CIFAR10 dataset has no attribute load_data()
I am using tensorflow version 2.0.1. In my Jupyter notebook I use:
(train_x, train_y), (test_x, test_y) = tf.keras.datasets.cifar100.load_data()
however, it does not work and I get the following ...
0
votes
1
answer
262
views
Unable to give Keras neural network multiple inputs
I am trying to get a data pipeline of text based data into a neural network with two heads. Made use of the official documentation that tells you to zip it into a dictionary of values, but it did not ...
0
votes
0
answers
33
views
model.fit function using datasets stops training after 2 epochs
Trying to train a autoencoder model using tensorflow datasets. Trying to use model.fit function to train the model. The training is exiting after 2 epochs. Even after changing the epochs >2. It ...
0
votes
0
answers
34
views
Build a Tensorflow dataset from generator
I have 10000 large files.
From each file, we generate large number of training samples.
def generate_sample(tokens, id):
for i in range(len(tokens):
r = tokens[i]
x =.. #list
...
- 1,682