Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Modified #17

Open
wants to merge 46 commits into
base: main
Choose a base branch
from
Open

Modified #17

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
96475c3
update the requirement of numpy version
implicitfaith Aug 8, 2024
07b6446
update the version of env requirement
implicitfaith Aug 9, 2024
9231f66
Update requirements.txt
implicitfaith Aug 12, 2024
1cba1a6
Create devcontainer.json
implicitfaith Aug 13, 2024
edef3a8
to test on clusters
implicitfaith Aug 15, 2024
6964b2a
debug for modified
implicitfaith Aug 16, 2024
78043a7
debug for modified
implicitfaith Aug 16, 2024
6643407
debug for modified
implicitfaith Aug 16, 2024
48b2b07
modified
implicitfaith Aug 16, 2024
e52b004
add random data in the training data
implicitfaith Aug 16, 2024
f96307e
data cleaning in modified
implicitfaith Aug 16, 2024
4186d12
use cosine similarity w = w + gamma * cos; ignore points with small w…
implicitfaith Aug 16, 2024
b828ef8
Update base.py
implicitfaith Aug 16, 2024
cbb4124
dc, bm, nz settings
implicitfaith Aug 16, 2024
9f554ac
Merge branch 'modified' of https://github.com/implicitfaith/llm-basel…
implicitfaith Aug 16, 2024
635a1b3
align eval freq with dc
implicitfaith Aug 17, 2024
ca7a118
debug
implicitfaith Aug 17, 2024
d84787f
log more in wandb
implicitfaith Aug 17, 2024
ac850a2
grad is not weighted by w, average over all curated batches to comput…
implicitfaith Aug 18, 2024
5fed19a
update the scripts
implicitfaith Aug 18, 2024
03c3c22
add main.py training on only curated data in dc
implicitfaith Aug 18, 2024
9281339
change cos_sim to the real one, no like before it was the sum of part…
implicitfaith Aug 18, 2024
f5d8a77
change cos_sim to the real one, no like before it was the sum of part…
implicitfaith Aug 18, 2024
ec53dd2
gamma=1 for cos sim
implicitfaith Aug 18, 2024
1aa06ef
update scripts
implicitfaith Aug 18, 2024
e1d3d42
update scripts
implicitfaith Aug 18, 2024
2e47d38
add wandb saving checkpoint after training
implicitfaith Aug 18, 2024
9769333
wj = wj + ||g0|| cos_sim
implicitfaith Aug 18, 2024
46633c9
align the pertry to align the permutation of wutation, use pre-traine…
implicitfaith Aug 20, 2024
62abe2b
fix the bug of cr
implicitfaith Aug 20, 2024
b499329
try to make w permuted as dataset, add w_gt_sum for reset w
implicitfaith Aug 20, 2024
f0cb2c8
remove reset, change gamma to 0.001
implicitfaith Aug 21, 2024
43cda8f
no need to download pre-trained model
implicitfaith Aug 22, 2024
3ff6e37
from scratch by myself
implicitfaith Aug 26, 2024
1e7ec8c
record
implicitfaith Aug 26, 2024
ca226bf
record
implicitfaith Aug 26, 2024
f5b8453
record
implicitfaith Aug 26, 2024
1412fe0
finally it works, by writing val loss from the output logits to avoid…
implicitfaith Aug 26, 2024
1e3f99c
don't need to split the data in x and y
implicitfaith Aug 26, 2024
f05e3c5
record
implicitfaith Aug 27, 2024
592a7b0
record
implicitfaith Aug 27, 2024
bc39caa
no need to split the labels or targets as y
implicitfaith Aug 27, 2024
f87e5dd
record
implicitfaith Aug 27, 2024
6c9e6f3
record
implicitfaith Aug 28, 2024
d336922
record
implicitfaith Aug 28, 2024
aee3f01
record
implicitfaith Aug 28, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions .devcontainer/devcontainer.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
{
// Name this configuration
"name": "Codespace for Skills!",
"customizations": {
"vscode": {
"extensions": [
"GitHub.copilot"
]
}
}
}
1 change: 1 addition & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ src/data/datasets/
wandb/
exps/
scripts/
artifacts/

# Byte-compiled / optimized / DLL files
__pycache__/
Expand Down
Loading