Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

modify dropoutnet in case of batch size mismatch #505

Merged
merged 39 commits into from
Dec 10, 2024
Merged

modify dropoutnet in case of batch size mismatch #505

Changes from all commits
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
d39af13
fix bug of `package` syntax of backbone
yangxudong Sep 16, 2023
8ee5af2
fix bug of `package` syntax of backbone
yangxudong Sep 16, 2023
8593b79
add document and test cases of BST & DIN backbone model
yangxudong Sep 17, 2023
c8e1257
modify input layer to support reuse
yangxudong Sep 18, 2023
bab2cea
add support for contrastive learning
yangxudong Sep 18, 2023
bd6dbe8
modify document
yangxudong Sep 18, 2023
670458e
add powerful support for contrastive learning
yangxudong Sep 18, 2023
a357cb9
add powerful support for contrastive learning
yangxudong Sep 19, 2023
61e238f
modify document
yangxudong Sep 19, 2023
4141461
modify document
yangxudong Sep 19, 2023
02d4dce
fix bug of bst when output all token embedding
yangxudong Sep 19, 2023
8672e47
add documents
yangxudong Sep 19, 2023
5bbead1
decrease num_steps of test cases
yangxudong Sep 19, 2023
5cec266
Merge branch 'master' of https://github.com/alibaba/EasyRec into bug_fix
yangxudong Sep 20, 2023
798d72f
fix bug of NaN output of tf.norm
yangxudong Sep 27, 2023
a7cc673
Merge branch 'master' into bug_fix
yangxudong Dec 13, 2023
5157f50
fix doc build problem
yangxudong Dec 13, 2023
2ee7015
fix doc build problem
yangxudong Dec 13, 2023
5cafa73
fix doc build problem
yangxudong Dec 13, 2023
ac9fb01
fix doc build problem
yangxudong Dec 14, 2023
7e8fa34
Merge branch 'master' of https://github.com/alibaba/EasyRec into bug_fix
yangxudong Dec 14, 2023
40ed612
Merge branch 'master' of https://github.com/alibaba/EasyRec into bug_fix
yangxudong Dec 21, 2023
27c95df
fix doc build problem
yangxudong Dec 22, 2023
db73211
fix doc build problem
yangxudong Dec 25, 2023
6ec43f7
fix bug of SENet when run with tf.distribute.MirroredStrategy
yangxudong Dec 26, 2023
6b3eba8
fix bug of SENet when run with tf.distribute.MirroredStrategy
yangxudong Dec 26, 2023
71d91eb
add test case
yangxudong Dec 27, 2023
39d3c05
add test case
yangxudong Dec 27, 2023
0d83b0b
add test case
yangxudong Dec 27, 2023
e524243
add test case
yangxudong Dec 27, 2023
0557c9b
add test case
yangxudong Dec 28, 2023
8572072
add LayerNormalization Layer
yangxudong Dec 28, 2023
9da20c1
modify
yangxudong Jan 11, 2024
bbecebf
Merge branch 'master' into bug_fix
yangxudong Oct 28, 2024
ee8d9aa
fix bug of jrc loss when sample weight is a scalar
yangxudong Oct 28, 2024
1dffb67
add documents
yangxudong Nov 8, 2024
65b0d43
Merge branch 'master' into bug_fix
yangxudong Nov 21, 2024
501139d
modify dropoutnet in case of batch size mismatch
yangxudong Dec 4, 2024
a139c02
modify dropoutnet in case of batch size mismatch
yangxudong Dec 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 14 additions & 21 deletions easy_rec/python/model/dropoutnet.py
View file
Open in desktop
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Draw some random uniform numbers, then check them with keep_rate. Maybe it's simpler in this way to get bernoulli numbers?

mask = tf.random.uniform(shape=(20,)) < keep_rate
mask = tf.cast(mask, tf.int32)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah! both do almost the same thing~

Copy link
Collaborator

@eric-gecheng eric-gecheng Dec 5, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tested on higher version of tensorflow(2.18), below statement is not working, giving some error messages, maybe due to tf and numpy compatibility issue:

import tensorflow as tf 
tf = tf.compat.v1
keep_rate = 0.2
dist = tf.distributions.Bernoulli(probs=keep_rate)
dist.sample(sample_shape=(5,))

error msg:
AttributeError: module 'numpy.core.multiarray' has no attribute 'integer'

Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,6 @@
from easy_rec.python.model.easy_rec_model import EasyRecModel
from easy_rec.python.protos.loss_pb2 import LossType
from easy_rec.python.utils.proto_util import copy_obj
from easy_rec.python.utils.shape_utils import get_shape_list

from easy_rec.python.protos.dropoutnet_pb2 import DropoutNet as DropoutNetConfig # NOQA
from easy_rec.python.loss.softmax_loss_with_negative_mining import softmax_loss_with_negative_mining # NOQA
Expand All @@ -22,6 +21,14 @@ def cosine_similarity(user_emb, item_emb):
tf.multiply(user_emb, item_emb), axis=1, name='cosine')
return user_item_sim

def bernoulli_dropout(x, rate, training=False):
if rate == 0.0 or not training:
return x
keep_rate = 1.0 - rate
dist = tf.distributions.Bernoulli(probs=keep_rate, dtype=x.dtype)
mask = dist.sample(sample_shape=tf.stack([tf.shape(x)[0], 1]))
return x * mask / keep_rate


class DropoutNet(EasyRecModel):

Expand Down Expand Up @@ -68,8 +75,6 @@ def __init__(self,
assert self.item_content_feature is not None or self.item_preference_feature is not None, 'no item feature'

def build_predict_graph(self):
batch_size = get_shape_list(self.item_content_feature)[0]

num_user_dnn_layer = len(self.user_tower_layers.hidden_units)
last_user_hidden = self.user_tower_layers.hidden_units.pop()
num_item_dnn_layer = len(self.item_tower_layers.hidden_units)
Expand All @@ -85,15 +90,9 @@ def build_predict_graph(self):
content_feature = user_content_dnn(self.user_content_feature)
user_features.append(content_feature)
if self.user_preference_feature is not None:
if self._is_training:
prob = tf.random.uniform([batch_size])
user_prefer_feature = tf.where(
tf.less(prob, self._model_config.user_dropout_rate),
tf.zeros_like(self.user_preference_feature),
self.user_preference_feature)
else:
user_prefer_feature = self.user_preference_feature

user_prefer_feature = bernoulli_dropout(self.user_preference_feature,
self._model_config.user_dropout_rate,
self._is_training)
user_prefer_dnn = dnn.DNN(self.user_preference_layers, self._l2_reg,
'user_preference', self._is_training)
prefer_feature = user_prefer_dnn(user_prefer_feature)
Expand All @@ -119,15 +118,9 @@ def build_predict_graph(self):
content_feature = item_content_dnn(self.item_content_feature)
item_features.append(content_feature)
if self.item_preference_feature is not None:
if self._is_training:
prob = tf.random.uniform([batch_size])
item_prefer_feature = tf.where(
tf.less(prob, self._model_config.item_dropout_rate),
tf.zeros_like(self.item_preference_feature),
self.item_preference_feature)
else:
item_prefer_feature = self.item_preference_feature

item_prefer_feature = bernoulli_dropout(self.item_preference_feature,
self._model_config.item_dropout_rate,
self._is_training)
item_prefer_dnn = dnn.DNN(self.item_preference_layers, self._l2_reg,
'item_preference', self._is_training)
prefer_feature = item_prefer_dnn(item_prefer_feature)
Expand Down