Add multitask models：SharedBottom, ESSM, MMOE, PLE

- add multitask learning models：SharedBottom, ESSM, MMOE, PLE - set `dnn_hidden_units` parameter to `(256, 128, 64)` - add new package `deepctr.models.multitask` and `deepctr.models.sequence` - fix truncated feature value error when padding string sequence in run_multivalue_movielens_hash.py
shenweichen · Sep 3, 2021 · 35288ae · 35288ae
2 parents 9f15559 + 1cbcb45
commit 35288ae
Show file tree

Hide file tree

Showing 88 changed files with 1,406 additions and 360 deletions.
diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md
@@ -20,7 +20,7 @@ Steps to reproduce the behavior:
 **Operating environment(运行环境):**
  - python version [e.g. 3.6, 3.7]
  - tensorflow version [e.g. 1.4.0, 1.15.0, 2.5.0]
- - deepctr version [e.g. 0.8.6,]
+ - deepctr version [e.g. 0.9.0,]
 
 **Additional context**
 Add any other context about the problem here.
diff --git a/.github/ISSUE_TEMPLATE/question.md b/.github/ISSUE_TEMPLATE/question.md
@@ -17,4 +17,4 @@ Add any other context about the problem here.
 **Operating environment(运行环境):**
  - python version [e.g. 3.6]
  - tensorflow version [e.g. 1.4.0, 1.15.0, 2.5.0]
- - deepctr version [e.g. 0.8.6,]
+ - deepctr version [e.g. 0.9.0,]
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -18,7 +18,7 @@ jobs:
     strategy:
       matrix:
         python-version: [3.6,3.7]
-        tf-version: [1.4.0,1.15.0,2.1.0,2.5.0]
+        tf-version: [1.4.0,1.15.0,2.2.0,2.5.0]
 
         exclude:
           - python-version: 3.7
@@ -28,10 +28,10 @@ jobs:
 
     steps:
 
-    - uses: actions/checkout@v1
+    - uses: actions/checkout@v2
 
     - name: Setup python environment
-      uses: actions/setup-python@v1
+      uses: actions/setup-python@v2.2.2
       with:
         python-version: ${{ matrix.python-version }}
 
@@ -49,7 +49,7 @@ jobs:
         pip install -q python-coveralls
         pytest --cov=deepctr --cov-report=xml
     - name: Upload coverage to Codecov  
-      uses: codecov/codecov-action@v1.0.2
+      uses: codecov/codecov-action@v2.0.3
       with:
         token: ${{secrets.CODECOV_TOKEN}}
         file: ./coverage.xml

diff --git a/README.md b/README.md
@@ -18,18 +18,23 @@
 <!-- [![Gitter](https://badges.gitter.im/DeepCTR/community.svg)](https://gitter.im/DeepCTR/community?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge) -->
 
 
-DeepCTR is a **Easy-to-use**,**Modular** and **Extendible** package of deep-learning based CTR models along with lots of core components layers which can be used to easily build custom models.You can use any complex model with `model.fit()`，and `model.predict()` .
-
-- Provide `tf.keras.Model` like interface for **quick experiment**. [example](https://deepctr-doc.readthedocs.io/en/latest/Quick-Start.html#getting-started-4-steps-to-deepctr)
-- Provide  `tensorflow estimator` interface for **large scale data** and **distributed training**. [example](https://deepctr-doc.readthedocs.io/en/latest/Quick-Start.html#getting-started-4-steps-to-deepctr-estimator-with-tfrecord)
+DeepCTR is a **Easy-to-use**,**Modular** and **Extendible** package of deep-learning based CTR models along with lots of
+core components layers which can be used to easily build custom models.You can use any complex model with `model.fit()`
+，and `model.predict()` .
+
+- Provide `tf.keras.Model` like interface for **quick experiment**
+  . [example](https://deepctr-doc.readthedocs.io/en/latest/Quick-Start.html#getting-started-4-steps-to-deepctr)
+- Provide  `tensorflow estimator` interface for **large scale data** and **distributed training**
+  . [example](https://deepctr-doc.readthedocs.io/en/latest/Quick-Start.html#getting-started-4-steps-to-deepctr-estimator-with-tfrecord)
 - It is compatible with both `tf 1.x`  and `tf 2.x`.
 
 Some related projects:
+
 - DeepMatch: https://github.com/shenweichen/DeepMatch
 - DeepCTR-Torch: https://github.com/shenweichen/DeepCTR-Torch
 
-
-Let's [**Get Started!**](https://deepctr-doc.readthedocs.io/en/latest/Quick-Start.html)([Chinese Introduction](https://zhuanlan.zhihu.com/p/53231955)) and [welcome to join us!](./CONTRIBUTING.md)
+Let's [**Get Started!**](https://deepctr-doc.readthedocs.io/en/latest/Quick-Start.html)([Chinese
+Introduction](https://zhuanlan.zhihu.com/p/53231955)) and [welcome to join us!](./CONTRIBUTING.md)
 
 ## Models List
 
@@ -45,8 +50,8 @@ Let's [**Get Started!**](https://deepctr-doc.readthedocs.io/en/latest/Quick-Star
 |   Attentional Factorization Machine    | [IJCAI 2017][Attentional Factorization Machines: Learning the Weight of Feature Interactions via Attention Networks](http://www.ijcai.org/proceedings/2017/435) |
 |      Neural Factorization Machine      | [SIGIR 2017][Neural Factorization Machines for Sparse Predictive Analytics](https://arxiv.org/pdf/1708.05027.pdf)                                               |
 |                xDeepFM                 | [KDD 2018][xDeepFM: Combining Explicit and Implicit Feature Interactions for Recommender Systems](https://arxiv.org/pdf/1803.05170.pdf)                         |
-|         Deep Interest Network          | [KDD 2018][Deep Interest Network for Click-Through Rate Prediction](https://arxiv.org/pdf/1706.06978.pdf)     
-|                AutoInt                 | [CIKM 2019][AutoInt: Automatic Feature Interaction Learning via Self-Attentive Neural Networks](https://arxiv.org/abs/1810.11921)                              ||
+|         Deep Interest Network          | [KDD 2018][Deep Interest Network for Click-Through Rate Prediction](https://arxiv.org/pdf/1706.06978.pdf)     |
+|                AutoInt                 | [CIKM 2019][AutoInt: Automatic Feature Interaction Learning via Self-Attentive Neural Networks](https://arxiv.org/abs/1810.11921)                              |
 |    Deep Interest Evolution Network     | [AAAI 2019][Deep Interest Evolution Network for Click-Through Rate Prediction](https://arxiv.org/pdf/1809.03672.pdf)                                            |
 |                FwFM                    | [WWW 2018][Field-weighted Factorization Machines for Click-Through Rate Prediction in Display Advertising](https://arxiv.org/pdf/1806.03514.pdf)                |
 |                  ONN                  | [arxiv 2019][Operation-aware Neural Networks for User Response Prediction](https://arxiv.org/pdf/1904.12579.pdf)                                                |
@@ -59,11 +64,15 @@ Let's [**Get Started!**](https://deepctr-doc.readthedocs.io/en/latest/Quick-Star
 |                DCN V2                    | [arxiv 2020][DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systems](https://arxiv.org/abs/2008.13535)   |
 |                DIFM                 | [IJCAI 2020][A Dual Input-aware Factorization Machine for CTR Prediction](https://www.ijcai.org/Proceedings/2020/0434.pdf)   |
 |   FEFM and DeepFEFM                    | [arxiv 2020][Field-Embedded Factorization Machines for Click-through rate prediction](https://arxiv.org/abs/2009.09931)                                         |
+|              SharedBottom               | [arxiv 2017][An Overview of Multi-Task Learning in Deep Neural Networks](https://arxiv.org/pdf/1706.05098.pdf)  |
+|   ESMM                    | [SIGIR 2018][Entire Space Multi-Task Model: An Effective Approach for Estimating Post-Click Conversion Rate](https://arxiv.org/abs/1804.07931)                       |
+|   MMOE                    | [KDD 2018][Modeling Task Relationships in Multi-task Learning with Multi-gate Mixture-of-Experts](https://dl.acm.org/doi/abs/10.1145/3219819.3220007)                   |
+|   PLE                    | [RecSys 2020][Progressive Layered Extraction (PLE): A Novel Multi-Task Learning (MTL) Model for Personalized Recommendations](https://dl.acm.org/doi/10.1145/3383313.3412236)                   |
 
 ## Citation
 
-- Weichen Shen. (2017). DeepCTR: Easy-to-use,Modular and Extendible package of deep-learning based CTR models. https://github.com/shenweichen/deepctr.
-
+- Weichen Shen. (2017). DeepCTR: Easy-to-use,Modular and Extendible package of deep-learning based CTR
+  models. https://github.com/shenweichen/deepctr.
 
 If you find this code useful in your research, please cite it using the following BibTeX:
 
@@ -81,11 +90,10 @@ If you find this code useful in your research, please cite it using the followin
 ## DisscussionGroup
 
 - [Discussions](https://github.com/shenweichen/DeepCTR/discussions)
-- 公众号：**浅梦学习笔记**  
-- wechat ID: **deepctrbot**
+- 公众号：**浅梦学习笔记**
+- wechat ID: **deepctrbot**
 
   ![wechat](./docs/pics/code.png)
-
 
 ## Main contributors([welcome to join us!](./CONTRIBUTING.md))
 
@@ -108,6 +116,11 @@ If you find this code useful in your research, please cite it using the followin
          <a href="https://github.com/pandeconscious">Harshit Pande</a>
         <p> Amazon   </p>
       </td>
+      <td>
+         <a href="https://github.com/morningsky"><img width="70" height="70" src="https://github.com/morningsky.png?s=40" alt="pic"></a><br>
+         <a href="https://github.com/morningsky">Lai Mincai</a>
+        <p> ShanghaiTech University </p>
+      </td>
       <td>
          <a href="https://github.com/codewithzichao"><img width="70" height="70" src="https://github.com/codewithzichao.png?s=40" alt="pic"></a><br>
          <a href="https://github.com/codewithzichao">Li Zichao</a>

diff --git a/deepctr/__init__.py b/deepctr/__init__.py
@@ -1,4 +1,4 @@
 from .utils import check_version
 
-__version__ = '0.8.7'
+__version__ = '0.9.0'
 check_version(__version__)
diff --git a/deepctr/estimator/feature_column.py b/deepctr/estimator/feature_column.py
@@ -47,6 +47,6 @@ def input_from_feature_columns(features, feature_columns, l2_reg_embedding=0.0):
 def is_embedding(feature_column):
     try:
         from tensorflow.python.feature_column.feature_column_v2 import EmbeddingColumn
-    except:
+    except ImportError:
         EmbeddingColumn = _EmbeddingColumn
     return isinstance(feature_column, (_EmbeddingColumn, EmbeddingColumn))
diff --git a/deepctr/estimator/models/autoint.py b/deepctr/estimator/models/autoint.py
@@ -20,7 +20,7 @@
 
 def AutoIntEstimator(linear_feature_columns, dnn_feature_columns, att_layer_num=3, att_embedding_size=8, att_head_num=2,
                      att_res=True,
-                     dnn_hidden_units=(256, 256), dnn_activation='relu', l2_reg_linear=1e-5,
+                     dnn_hidden_units=(256, 128, 64), dnn_activation='relu', l2_reg_linear=1e-5,
                      l2_reg_embedding=1e-5, l2_reg_dnn=0, dnn_use_bn=False, dnn_dropout=0, seed=1024,
                      task='binary', model_dir=None, config=None, linear_optimizer='Ftrl',
                      dnn_optimizer='Adagrad', training_chief_hooks=None):

diff --git a/deepctr/estimator/models/ccpm.py b/deepctr/estimator/models/ccpm.py
@@ -19,7 +19,7 @@
 
 
 def CCPMEstimator(linear_feature_columns, dnn_feature_columns, conv_kernel_width=(6, 5), conv_filters=(4, 4),
-                  dnn_hidden_units=(256,), l2_reg_linear=1e-5, l2_reg_embedding=1e-5, l2_reg_dnn=0, dnn_dropout=0,
+                  dnn_hidden_units=(128, 64), l2_reg_linear=1e-5, l2_reg_embedding=1e-5, l2_reg_dnn=0, dnn_dropout=0,
                   seed=1024, task='binary', model_dir=None, config=None, linear_optimizer='Ftrl',
                   dnn_optimizer='Adagrad', training_chief_hooks=None):
     """Instantiates the Convolutional Click Prediction Model architecture.

diff --git a/deepctr/estimator/models/dcn.py b/deepctr/estimator/models/dcn.py
@@ -15,7 +15,7 @@
 from ...layers.utils import combined_dnn_input
 
 
-def DCNEstimator(linear_feature_columns, dnn_feature_columns, cross_num=2, dnn_hidden_units=(128, 128,),
+def DCNEstimator(linear_feature_columns, dnn_feature_columns, cross_num=2, dnn_hidden_units=(256, 128, 64),
                  l2_reg_linear=1e-5,
                  l2_reg_embedding=1e-5,
                  l2_reg_cross=1e-5, l2_reg_dnn=0, seed=1024, dnn_dropout=0, dnn_use_bn=False,

diff --git a/deepctr/estimator/models/deepfefm.py b/deepctr/estimator/models/deepfefm.py
@@ -19,7 +19,7 @@
 
 
 def DeepFEFMEstimator(linear_feature_columns, dnn_feature_columns,
-                      dnn_hidden_units=(128, 128), l2_reg_linear=0.00001, l2_reg_embedding_feat=0.00001,
+                      dnn_hidden_units=(256, 128, 64), l2_reg_linear=0.00001, l2_reg_embedding_feat=0.00001,
                       l2_reg_embedding_field=0.00001, l2_reg_dnn=0, seed=1024, dnn_dropout=0.0,
                       dnn_activation='relu', dnn_use_bn=False, task='binary', model_dir=None,
                       config=None, linear_optimizer='Ftrl', dnn_optimizer='Adagrad', training_chief_hooks=None):

diff --git a/deepctr/estimator/models/deepfm.py b/deepctr/estimator/models/deepfm.py
@@ -17,7 +17,7 @@
 from ...layers.utils import concat_func, combined_dnn_input
 
 
-def DeepFMEstimator(linear_feature_columns, dnn_feature_columns, dnn_hidden_units=(128, 128),
+def DeepFMEstimator(linear_feature_columns, dnn_feature_columns, dnn_hidden_units=(256, 128, 64),
                     l2_reg_linear=0.00001, l2_reg_embedding=0.00001, l2_reg_dnn=0, seed=1024, dnn_dropout=0,
                     dnn_activation='relu', dnn_use_bn=False, task='binary', model_dir=None, config=None,
                     linear_optimizer='Ftrl',

diff --git a/deepctr/estimator/models/fibinet.py b/deepctr/estimator/models/fibinet.py
@@ -18,7 +18,7 @@
 
 
 def FiBiNETEstimator(linear_feature_columns, dnn_feature_columns, bilinear_type='interaction', reduction_ratio=3,
-                     dnn_hidden_units=(128, 128), l2_reg_linear=1e-5,
+                     dnn_hidden_units=(256, 128, 64), l2_reg_linear=1e-5,
                      l2_reg_embedding=1e-5, l2_reg_dnn=0, seed=1024, dnn_dropout=0, dnn_activation='relu',
                      task='binary', model_dir=None, config=None, linear_optimizer='Ftrl',
                      dnn_optimizer='Adagrad', training_chief_hooks=None):

diff --git a/deepctr/estimator/models/fnn.py b/deepctr/estimator/models/fnn.py
@@ -14,7 +14,7 @@
 from ...layers.utils import combined_dnn_input
 
 
-def FNNEstimator(linear_feature_columns, dnn_feature_columns, dnn_hidden_units=(128, 128),
+def FNNEstimator(linear_feature_columns, dnn_feature_columns, dnn_hidden_units=(256, 128, 64),
                  l2_reg_embedding=1e-5, l2_reg_linear=1e-5, l2_reg_dnn=0, seed=1024, dnn_dropout=0,
                  dnn_activation='relu', task='binary', model_dir=None, config=None, linear_optimizer='Ftrl',
                  dnn_optimizer='Adagrad', training_chief_hooks=None):

diff --git a/deepctr/estimator/models/fwfm.py b/deepctr/estimator/models/fwfm.py
@@ -19,7 +19,7 @@
 from ...layers.utils import concat_func, add_func, combined_dnn_input
 
 
-def FwFMEstimator(linear_feature_columns, dnn_feature_columns, dnn_hidden_units=(128, 128),
+def FwFMEstimator(linear_feature_columns, dnn_feature_columns, dnn_hidden_units=(256, 128, 64),
                   l2_reg_linear=0.00001, l2_reg_embedding=0.00001, l2_reg_field_strength=0.00001, l2_reg_dnn=0,
                   seed=1024, dnn_dropout=0, dnn_activation='relu', dnn_use_bn=False, task='binary', model_dir=None,
                   config=None, linear_optimizer='Ftrl',

diff --git a/deepctr/estimator/models/nfm.py b/deepctr/estimator/models/nfm.py
@@ -15,7 +15,7 @@
 from ...layers.utils import concat_func, combined_dnn_input
 
 
-def NFMEstimator(linear_feature_columns, dnn_feature_columns, dnn_hidden_units=(128, 128),
+def NFMEstimator(linear_feature_columns, dnn_feature_columns, dnn_hidden_units=(256, 128, 64),
                  l2_reg_embedding=1e-5, l2_reg_linear=1e-5, l2_reg_dnn=0, seed=1024, bi_dropout=0,
                  dnn_dropout=0, dnn_activation='relu', task='binary', model_dir=None, config=None,
                  linear_optimizer='Ftrl',

diff --git a/deepctr/estimator/models/pnn.py b/deepctr/estimator/models/pnn.py
@@ -16,7 +16,7 @@
 from ...layers.utils import concat_func, combined_dnn_input
 
 
-def PNNEstimator(dnn_feature_columns, dnn_hidden_units=(128, 128), l2_reg_embedding=1e-5, l2_reg_dnn=0,
+def PNNEstimator(dnn_feature_columns, dnn_hidden_units=(256, 128, 64), l2_reg_embedding=1e-5, l2_reg_dnn=0,
                  seed=1024, dnn_dropout=0, dnn_activation='relu', use_inner=True, use_outter=False, kernel_type='mat',
                  task='binary', model_dir=None, config=None,
                  linear_optimizer='Ftrl',

diff --git a/deepctr/estimator/models/wdl.py b/deepctr/estimator/models/wdl.py
@@ -15,7 +15,7 @@
 from ...layers import DNN, combined_dnn_input
 
 
-def WDLEstimator(linear_feature_columns, dnn_feature_columns, dnn_hidden_units=(128, 128), l2_reg_linear=1e-5,
+def WDLEstimator(linear_feature_columns, dnn_feature_columns, dnn_hidden_units=(256, 128, 64), l2_reg_linear=1e-5,
                  l2_reg_embedding=1e-5, l2_reg_dnn=0, seed=1024, dnn_dropout=0, dnn_activation='relu',
                  task='binary', model_dir=None, config=None, linear_optimizer='Ftrl',
                  dnn_optimizer='Adagrad', training_chief_hooks=None):

diff --git a/deepctr/estimator/models/xdeepfm.py b/deepctr/estimator/models/xdeepfm.py
@@ -15,7 +15,7 @@
 from ...layers.utils import concat_func, add_func, combined_dnn_input
 
 
-def xDeepFMEstimator(linear_feature_columns, dnn_feature_columns, dnn_hidden_units=(256, 256),
+def xDeepFMEstimator(linear_feature_columns, dnn_feature_columns, dnn_hidden_units=(256, 128, 64),
                      cin_layer_size=(128, 128,), cin_split_half=True, cin_activation='relu', l2_reg_linear=0.00001,
                      l2_reg_embedding=0.00001, l2_reg_dnn=0, l2_reg_cin=0, seed=1024, dnn_dropout=0,
                      dnn_activation='relu', dnn_use_bn=False, task='binary', model_dir=None, config=None,

diff --git a/deepctr/layers/sequence.py b/deepctr/layers/sequence.py
@@ -560,10 +560,10 @@ def call(self, inputs, mask=None, training=None, **kwargs):
         if self.blinding:
             try:
                 outputs = tf.matrix_set_diag(outputs, tf.ones_like(outputs)[
-                    :, :, 0] * (-2 ** 32 + 1))
+                                                      :, :, 0] * (-2 ** 32 + 1))
             except AttributeError:
                 outputs = tf.compat.v1.matrix_set_diag(outputs, tf.ones_like(outputs)[
-                    :, :, 0] * (-2 ** 32 + 1))
+                                                                :, :, 0] * (-2 ** 32 + 1))
 
         outputs -= reduce_max(outputs, axis=-1, keep_dims=True)
         outputs = softmax(outputs)
@@ -633,7 +633,7 @@ def build(self, input_shape):
         _, T, num_units = input_shape.as_list()  # inputs.get_shape().as_list()
         # First part of the PE function: sin and cos argument
         position_enc = np.array([
-            [pos / np.power(10000, 2. * (i//2) / num_units) for i in range(num_units)]
+            [pos / np.power(10000, 2. * (i // 2) / num_units) for i in range(num_units)]
             for pos in range(T)])
 
         # Second part, apply the cosine to even columns and sin to odds.
@@ -684,8 +684,12 @@ def build(self, input_shape):
             embed_size = input_shape[2].value
             seq_len_max = input_shape[1].value
         else:
-            embed_size = input_shape[0][2].value
-            seq_len_max = input_shape[0][1].value
+            try:
+                embed_size = input_shape[0][2].value
+                seq_len_max = input_shape[0][1].value
+            except AttributeError:
+                embed_size = input_shape[0][2]
+                seq_len_max = input_shape[0][1]
 
         self.sess_bias_embedding = self.add_weight('sess_bias_embedding', shape=(self.sess_max_count, 1, 1),
                                                    initializer=TruncatedNormal(
@@ -745,7 +749,7 @@ def build(self, input_shape):
         else:
             try:
                 self.gru_cell = tf.nn.rnn_cell.GRUCell(self.num_units)  # tf.keras.layers.GRUCell
-            except:
+            except AttributeError:
                 self.gru_cell = tf.compat.v1.nn.rnn_cell.GRUCell(self.num_units)
 
         # Be sure to call this somewhere!
@@ -841,7 +845,6 @@ def get_config(self, ):
         base_config = super(KMaxPooling, self).get_config()
         return dict(list(base_config.items()) + list(config.items()))
 
-
 # def positional_encoding(inputs,
 #                         pos_embedding_trainable=True,
 #                         zero_pad=False,

diff --git a/deepctr/layers/utils.py b/deepctr/layers/utils.py
@@ -8,9 +8,10 @@
 import tensorflow as tf
 from tensorflow.python.keras.layers import Flatten
 from tensorflow.python.ops.lookup_ops import TextFileInitializer
+
 try:
     from tensorflow.python.ops.lookup_ops import StaticHashTable
-except ImportError as e:
+except ImportError:
     from tensorflow.python.ops.lookup_ops import HashTable as StaticHashTable
 
 
@@ -93,7 +94,7 @@ def call(self, x, mask=None, **kwargs):
         try:
             hash_x = tf.string_to_hash_bucket_fast(x, num_buckets,
                                                    name=None)  # weak hash
-        except:
+        except AttributeError:
             hash_x = tf.strings.to_hash_bucket_fast(x, num_buckets,
                                                     name=None)  # weak hash
         if self.mask_zero:
@@ -106,7 +107,8 @@ def compute_output_shape(self, input_shape):
         return input_shape
 
     def get_config(self, ):
-        config = {'num_buckets': self.num_buckets, 'mask_zero': self.mask_zero, 'vocabulary_path': self.vocabulary_path, 'default_value': self.default_value}
+        config = {'num_buckets': self.num_buckets, 'mask_zero': self.mask_zero, 'vocabulary_path': self.vocabulary_path,
+                  'default_value': self.default_value}
         base_config = super(Hash, self).get_config()
         return dict(list(base_config.items()) + list(config.items()))