Code

Hf integration#1

Code

Draft

TimotheeMickus wants to merge 21 commits intomainHelsinki-NLP/mammoth:mainfrom hf-integrationHelsinki-NLP/mammoth:hf-integrationCopy head branch name to clipboard

+32,573-7

Commits

Commits on Aug 28, 2023

• create hf dir in tools

  

  Maksim Tikhonov

  committed
  ead6f10

  Copy full SHA for ead6f10

• example configuration file to trian a modular mNMT model

  jrvc

  committed
  285bca4

  Copy full SHA for 285bca4

• better cfg

  jrvc

  committed
  76e764d

  Copy full SHA for 76e764d

• add conversion script for hf

  

  Maksim Tikhonov

  committed
  32c23ff

  Copy full SHA for 32c23ff

Commits on Aug 29, 2023

• add marian config initialization

  

  Maksim Tikhonov

  committed
  4b99a5e

  Copy full SHA for 4b99a5e

• update conversion script using onmt arg parser

  

  Mickus Timothee

  committed
  152fb94

  Copy full SHA for 152fb94

• add sum for enc/dec layers

  

  Maksim Tikhonov

  committed
  ae7e070

  Copy full SHA for ae7e070

• add MarianTokenizer initialization

  

  Maksim Tikhonov

  committed
  cfddb2c

  Copy full SHA for cfddb2c

• fix typos

  

  Maksim Tikhonov

  committed
  3c502f2

  Copy full SHA for 3c502f2

• provide the default tokens properly

  

  Maksim Tikhonov

  committed
  77a47d7

  Copy full SHA for 77a47d7

• bugfix (double normalization)

  

  Mickus Timothee

  committed
  fb689af

  Copy full SHA for fb689af

• add direct spm initialization

  

  Maksim Tikhonov

  committed
  2635be7

  Copy full SHA for 2635be7

• add toy spm model/vocab

  

  Maksim Tikhonov

  committed
  8865b5b

  Copy full SHA for 8865b5b

• Merge branch 'hf-integration' of https://github.com/Helsinki-NLP/mammoth into hf-integration

  

  Maksim Tikhonov

  committed
  b8643c4

  Copy full SHA for b8643c4

• add the check for .tsv format

  

  Maksim Tikhonov

  committed
  5346a26

  Copy full SHA for 5346a26

• shallow util for HF integration

  

  Mickus Timothee

  committed
  56a31b8

  Copy full SHA for 56a31b8

Commits on Aug 30, 2023

• bugfix

  

  Mickus Timothee

  committed
  d2f9b23

  Copy full SHA for d2f9b23

• debug, again, soz

  

  Mickus Timothee

  committed
  3df22de

  Copy full SHA for 3df22de

• debug, again, soz

  

  Mickus Timothee

  committed
  f4ee810

  Copy full SHA for f4ee810

• curse you stig

  

  Mickus Timothee

  committed
  f28066f

  Copy full SHA for f28066f

• add convert_mammoth_checkpoint() fun

  

  Maksim Tikhonov

  committed
  b2a4ba2

  Copy full SHA for b2a4ba2